Building Production-Grade Observability with Prometheus, Grafana, and OpenTelemetry

Scenario

You have a Kubernetes cluster running microservices instrumented with OpenTelemetry SDKs, using an OpenTelemetry Collector to export metrics to Prometheus, and Grafana for visualization. Suddenly, certain service metrics stop appearing in Grafana dashboards, and alerts based on those metrics fail to fire.

Symptoms

• "No data" errors in Grafana panels or missing data points.
• Prometheus targets showing status "down" or "unknown".
• OpenTelemetry Collector logs filled with errors (e.g., connection timeout, export failure).

Diagnosis

1. Check OpenTelemetry Collector health and logs: bash kubectl logs -n observability <collector-pod-name>
2. Verify Prometheus targets: Use the Prometheus UI or query its API: bash curl -s http://prometheus:9090/api/v1/targets | jq
3. Review the OpenTelemetry Collector configuration (usually a ConfigMap) for correct exporters, receivers, and pipelines.
4. Test network connectivity between components—for example, send a test metric from the collector pod to Prometheus: bash kubectl exec -it <collector-pod> -- curl -X POST -H "Content-Type: application/json" -d '{"resourceMetrics":[{"resource":{},"scopeMetrics":[{"scope":{},"metrics":[{"name":"test","unit":"1","sum":{"dataPoints":[{"asDouble":1.0}],"aggregationTemporality":1}}]}]}]}' http://prometheus:9090/api/v1/otlp/v1/metrics

Commands

• List all related pods: kubectl get pods -n observability
• Get collector config: kubectl get configmap -n observability otel-collector-config -o yaml
• Inspect Prometheus scrape configuration: kubectl get configmap -n monitoring prometheus-config -o yaml
• Tail collector logs: kubectl logs -f -n observability <collector-pod>

Risk Controls

• Test all changes in a staging environment before production.
• Use kubectl diff to preview modifications: kubectl diff -f new-config.yaml
• Avoid editing running ConfigMaps directly; update via version-controlled YAML files.

Rollback

• If the collector configuration causes issues, revert to the last known good configuration: bash kubectl apply -f previous-otel-config.yaml -n observability
• If Prometheus scrape config is broken, restore the previous Prometheus ConfigMap and restart the Prometheus pod.
• Maintain Git history of configuration files for quick rollback.

Verification

• Confirm Prometheus targets are "UP".
• Refresh Grafana dashboards to see metrics reappear.
• Trigger a test alert to verify alert rules work.
• Check collector logs for zero errors.

When to Submit an OpsGlobal Ticket

If the issue persists after following these steps, or you need expert assistance optimizing OpenTelemetry Collector performance, scaling Prometheus storage, designing complex dashboards, or if you encounter infrastructure-level problems (e.g., network, storage), submit a ticket immediately. We provide 24×7 SRE support to restore your observability stack swiftly.