Prometheus, Grafana, and OpenTelemetry: A Production Observability Field Guide

Scenario

An e-commerce platform experiences slow checkout and page timeouts during a flash sale. CPU usage spikes, but the root cause is unclear. The team already uses Prometheus and Grafana but lacks distributed tracing.

Symptoms

• Grafana dashboard shows pod CPU >80%
• Request latency P99 rises from 200ms to 2s
• Users report checkout API timeouts
• Some services return 502 errors

Diagnosis

1. Check Infrastructure

Query high-CPU pods with PromQL:

sum(rate(container_cpu_usage_seconds_total{namespace="production"}[5m])) by (pod)

Identifies checkout-service pods as hot spots.

2. Add OpenTelemetry Tracing

Install OpenTelemetry Collector exporting to Jaeger. Instrument services with OpenTelemetry SDK. Redeploy. In Jaeger UI, find that checkout-service traces show 90% of time spent in calls to payment-gateway.

3. Deep Dive

Check logs for payment-gateway:

kubectl logs -n production -l app=payment-gateway --tail=100

Reveals many connection refused errors — downstream bank API is down.

Commands

Enable Debug Logs Temporarily

kubectl set env deployment/checkout-service LOG_LEVEL=debug -n production

Hotfix: Circuit Breaker

Add config via ConfigMap:

# payment-gateway-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: payment-gateway-config
data:
  circuit-breaker: |
    timeout: 500ms
    maxFailures: 3

Apply:

kubectl create -f payment-gateway-config.yaml -n production --dry-run=client -o yaml | kubectl apply -f -

Verify Circuit Breaker

Check circuit_breaker_state metric in Prometheus. State should change to open after failures.

Risk Controls

• Assess disk space before enabling debug logs; disable after diagnosis.
• Test circuit breaker config on non-critical nodes first.
• All changes via CI/CD pipeline, avoid direct production edits.

Rollback

Rollback Log Level

kubectl set env deployment/checkout-service LOG_LEVEL=info -n production

Rollback Circuit Breaker

Delete ConfigMap and restart:

kubectl delete configmap -n production payment-gateway-config
kubectl rollout restart deployment/payment-gateway -n production

Verification

• Grafana latency panel: P99 <500ms
• payment-gateway error rate zero
• OpenTelemetry traces show circuit breaker blocking failed calls

When to Submit an OpsGlobal Ticket

• If bank API issue is upstream and needs third-party coordination from OpsGlobal
• If OpenTelemetry Collector setup is too complex or troubleshooting exceeds 2 hours
• When cascading failures require architectural changes (e.g., retry queues)

Conclusion: Combining Prometheus (metrics), Grafana (visualization), and OpenTelemetry (tracing) dramatically reduces MTTR. Always implement resilience patterns like circuit breakers in production.