Practical Observability with Prometheus, Grafana, and OpenTelemetry

Scenario

Your e-commerce platform runs on Kubernetes. Users report slow checkout. SRE team sees P99 latency spike from 200ms to 2s, but CPU and memory are normal. OpenTelemetry traces show missing spans for database queries.

Symptoms

• Grafana dashboard shows elevated P99 latency.
• Prometheus metric http_request_duration_seconds increased, but node_cpu_seconds_total flat.
• Trace waterfall incomplete—database spans absent.
• No error logs.

Diagnosis

1. Check OpenTelemetry Collector config: The collector config lacks a batch processor, causing span send delays.
2. Verify Prometheus scrape targets: Ensure the collector's metrics endpoint is scraped.
3. Inspect Collector logs: Run kubectl logs <otel-collector-pod>. See timeouts and retries.

Commands

Backup current config

kubectl get configmap otel-collector-config -o yaml > otel-collector-config.bak.yaml

Update and apply new config

Add batch processor:

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000

Apply:

kubectl apply -f otel-collector-config.yaml
kubectl rollout restart deployment otel-collector

Verify Prometheus target

kubectl port-forward svc/otel-collector 8888
curl http://localhost:8888/metrics | grep otel

Risk Controls

• Test config changes in staging first.
• Use blue/green deployment for collector updates.
• Backup current ConfigMap and Deployment.
• Monitor collector resource usage to avoid OOM.

Rollback

If issues arise, rollback immediately:

kubectl apply -f otel-collector-config.bak.yaml
kubectl rollout restart deployment otel-collector

Verification

1. Check Grafana: P99 latency should drop.2. OpenTelemetry traces now show database spans.3. Run PromQL: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])).

When to Submit an OpsGlobal Ticket

• Issue persists after all steps.2. Core metrics or traces are completely missing.3. Need help optimizing collector performance or configuration.4. Suspect unknown infrastructure bottleneck.