Scenario
You have a microservices application on Kubernetes using OpenTelemetry for tracing, metrics, and logs. Prometheus collects metrics, and Grafana visualizes them. Suddenly, Grafana dashboards show gaps or no data, and alerts stop firing.
Symptoms
- Grafana panels display “No data” or missing time series
- Prometheus target page shows some targets as DOWN
- OpenTelemetry Collector logs show errors or warnings
- Traces in Jaeger or Tempo are incomplete
Diagnosis
1. Check Prometheus Targets
kubectl port-forward svc/prometheus-server 9090:80 -n monitoring
# Visit http://localhost:9090/targets
Look for DOWN targets. Common causes: scaled-down pods, RBAC or network policies blocking scrape.
2. Check OpenTelemetry Collector
kubectl logs -l app=otel-collector -n observability --tail=200 | grep -i error
Verify the collector is sending metrics to Prometheus (or via remote write). Errors: unreachable endpoint, auth failure, data format mismatch.
3. Network Policies
Ensure Prometheus can access /metrics on all target pods, and the collector can reach backends (Jaeger, Tempo).
Risk Controls
- Backup configs before changes:
kubectl get configmap -n monitoring prometheus-server -o yaml > prom-config-backup.yaml - Avoid changes during peak hours
- Test changes in a non-critical namespace first
Rollback Steps
If the issue worsens after changes, restore the backup:
kubectl apply -f prom-config-backup.yaml -n monitoring
kubectl rollout restart deployment prometheus-server -n monitoring
For the OpenTelemetry Collector, rollback its ConfigMap and restart pods.
Verification
- Confirm all Prometheus targets are UP:
http://localhost:9090/targets - Query recent metrics in Grafana (e.g.,
up,http_requests_total) - Check collector logs for no errors
When to Submit an OpsGlobal Ticket
Submit a ticket if: - Prometheus or collector pods are in CrashLoopBackOff - Critical metrics are missing for over 10 minutes - You need help scaling or tuning persistent storage
Our SRE team will quickly intervene to minimize business impact.
Use cases
Useful for teams handling Observability issues and needing a clear troubleshooting and delivery workflow.
Problem background
A practical guide to diagnosing missing metrics and traces in your observability stack built with Prometheus, Grafana, and OpenTelemetry on Kubernetes.
Troubleshooting steps
Confirm impact and recent changes, collect logs, configuration and metrics, then apply fixes from low to high risk.
Command examples
Replace sample resource names with real values and store passwords, tokens and keys in environment variables.
Risks
Before production changes, confirm backups, access boundaries, change windows and rollback paths.
Rollback plan
Keep original configuration and release versions; roll back config, images or database changes if metrics degrade.
Deliverables
Root-cause notes, key commands, remediation steps, verification results and follow-up recommendations.
Need help with a similar technical issue?
If your servers, Kubernetes, Docker, CI/CD, databases or monitoring systems have similar issues, submit logs and config files for remote diagnosis.