Prometheus, Grafana, and OpenTelemetry in Action: A Practical Guide to Kubernetes Observability Troubleshooting

Scenario

An e-commerce platform's Kubernetes cluster experiences a sudden latency spike in the order-service microservice during peak hours, from an average of 200ms to 2s, causing order timeouts. The team has Prometheus + Grafana for monitoring and OpenTelemetry for distributed tracing.

Symptoms

• Grafana dashboard shows P99 HTTP request latency for order-service rising from 200ms to 2s
• Pod CPU usage not significantly increased, but memory consumption steadily climbs
• OpenTelemetry traces reveal the /checkout endpoint is the bottleneck, with payment-gateway calls taking 1.5s

Diagnosis

1. Open Grafana, examine the http_request_duration_seconds histogram for order-service to confirm P99 latency
2. Check Pod resource usage: kubectl top pod -n production -l app=order-service; memory is growing and near OOM limits
3. In OpenTelemetry UI (e.g., Jaeger), select a /checkout trace; payment-gateway shows 3 retries due to timeout
4. Inspect payment-gateway metrics: its Grafana panel shows 5% error rate and P95 latency of 8s

Commands

# Query latency histogram (PromQL)
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service="order-service", endpoint="/checkout"}[5m])) by (le))

# Check pod resources
kubectl top pod -n production -l app=order-service

# View last 100 logs
kubectl logs --tail=100 -n production deployment/order-service

echo "Payment gateway is slow; consider circuit breaking"

Risk Controls

• Ensure Grafana slow-query alerts are active and notified to SRE before changes
• Disable automatic circuit breaker on payment-gateway to avoid cascading failures
• Backup ConfigMap: kubectl get configmap -n production order-service-config -o yaml > /tmp/order-config-backup.yaml

Rollback

If latency does not improve after fix:

kubectl rollout undo deployment/order-service -n production
kubectl rollout status deployment/order-service -n production

If configuration changed, revert:

kubectl apply -f /tmp/order-config-backup.yaml

Verification

1. Run load test: hey -z 30s -c 10 http://order-service.production/checkout
2. Confirm Grafana shows P99 latency below 300ms
3. Check OpenTelemetry traces: payment-gateway call under 500ms
4. Verify no new error logs

When to Submit an OpsGlobal Ticket

• If your team lacks OpenTelemetry tracing and needs deployment assistance
• When cascading failures across multiple services cannot be isolated
• For advanced alerting configuration (e.g., SLO-based alerts)
• If cluster resource bottleneck analysis exceeds team's capability