Unified Observability with Prometheus, Grafana, and OpenTelemetry: A Practical Guide

Scenario

A microservices-based e-commerce platform runs on Kubernetes. The team experiences intermittent high latency and occasional HTTP 500 errors. They already use Prometheus and Grafana for monitoring but lack distributed tracing, making root cause analysis difficult.

Symptoms

• P99 latency exceeding 2 seconds
• Spikes in 5xx error rates
• Customer complaints about timeouts

Diagnosis

1. Check Grafana dashboards for anomaly in http_requests_duration_seconds_count{status="5xx"}.
2. Query PromQL: rate(http_requests_duration_seconds_count{status="5xx"}[5m]) to observe error rate trends.
3. Deploy OpenTelemetry Collector and instrument the application to capture traces. Identify slow spans, e.g., a payment service span taking 800ms.
4. Correlate with logs to pinpoint the root cause—a timeout in the payment gateway.

Commands

Deploy OpenTelemetry Collector

helm install otel-collector open-telemetry/opentelemetry-collector -f values.yaml

Instrument Application

Add the OpenTelemetry Java agent:

java -javaagent:opentelemetry-javaagent.jar -jar app.jar

Configure Prometheus

Add a scrape target in Prometheus config:

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8888']

Import Grafana Dashboard

Use Grafana Explore to query traces or import a pre-built OpenTelemetry dashboard.

Risk Controls

• Back up Grafana dashboard JSON and Prometheus config files before changes.
• Deploy the Collector in a separate namespace to avoid service disruption.
• Use canary deployments for application instrumentation; enable OpenTelemetry on a small subset of instances first.

Rollback

• If Collector fails: helm rollback otel-collector 0
• For application code issues: revert via git revert and redeploy.
• For Prometheus config: restore the backup and reload Prometheus.

Verification

1. Verify new traces appear in Grafana Explore.
2. Check Prometheus target status: kubectl port-forward svc/prometheus-server 9090 then visit /targets.
3. Run a load test to confirm reduced latency and error rates.

When to Submit an OpsGlobal Ticket

• Persistent data loss or Collector crashes.
• Need custom instrumentation for non-standard frameworks (e.g., Go, Python).
• Large-scale cluster requiring optimized sampling and storage strategies.