Scenario
A microservices application runs on a Kubernetes cluster. The team separately uses Prometheus (metrics), Elasticsearch (logs), and Jaeger (traces), but there's no correlation between metrics, logs, and traces, leading to high Mean Time to Resolution (MTTR).
Symptoms
- Frequent alerts with unclear root causes.
- Metrics show high latency but cannot be linked to specific services or requests.
- Incomplete trace data, missing cross-service context.
Diagnosis
The core issue is the lack of a unified observability pipeline. OpenTelemetry, a CNCF standard, can collect metrics, logs, and traces uniformly and export them to Prometheus (metrics) and Grafana Tempo (traces). Grafana serves as the single dashboard.
Commands
1. Deploy OpenTelemetry Collector
Install the OpenTelemetry Collector via Helm as a DaemonSet to collect data from each node.
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector \
--set mode=daemonset \
--set config.receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317 \
--set config.exporters.prometheus.endpoint=0.0.0.0:8889 \
--set config.exporters.otlp.endpoint=tempo.default.svc.cluster.local:4317
2. Configure Prometheus Scrape Targets
Add the OpenTelemetry Collector's metrics endpoint to the Prometheus configuration.
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['otel-collector.default.svc.cluster.local:8889']
3. Deploy Grafana Tempo for Traces
helm install tempo grafana/tempo --set storage.trace.backend=local
4. Configure Grafana Data Sources
Add Prometheus and Tempo data sources in Grafana, and set OpenTelemetry as the trace data source.
Risk Controls
- Limit collector resources (CPU/memory) to avoid impacting business containers.
- Use TLS encryption for transmission to prevent data leaks.
- Set sampling rates (e.g., 10%) to avoid trace data explosion.
Rollback
If issues arise, delete the Helm releases and restore original configurations:
helm uninstall otel-collector
helm uninstall tempo
# Restore Prometheus config
Remove newly added Grafana data sources.
Verification
- Check Prometheus targets at http://prometheus:9090/targets; status should be UP.
- In Grafana, add a "Tempo" dashboard, run queries, and view trace details.
- Use
kubectl logsto confirm the collector logs show no errors.
When to Submit an OpsGlobal Ticket
- Need to handle high-cardinality metrics or custom sampling strategies.
- Cluster exceeds 100 nodes, causing collector performance bottlenecks.
- Need custom OpenTelemetry processors or exporters.
- Advanced requirements for integrating alerts and dashboards. OpsGlobal provides 24/7 expert support to accelerate your observability journey.
Use cases
Useful for teams handling Observability issues and needing a clear troubleshooting and delivery workflow.
Problem background
This guide walks through integrating Prometheus, Grafana, and OpenTelemetry for a unified observability stack on Kubernetes, covering scenario, symptoms, diagnosis, deployment commands, risk controls, rollback, verification, and when to engage OpsGlobal.
Troubleshooting steps
Confirm impact and recent changes, collect logs, configuration and metrics, then apply fixes from low to high risk.
Command examples
Replace sample resource names with real values and store passwords, tokens and keys in environment variables.
Risks
Before production changes, confirm backups, access boundaries, change windows and rollback paths.
Rollback plan
Keep original configuration and release versions; roll back config, images or database changes if metrics degrade.
Deliverables
Root-cause notes, key commands, remediation steps, verification results and follow-up recommendations.
Need help with a similar technical issue?
If your servers, Kubernetes, Docker, CI/CD, databases or monitoring systems have similar issues, submit logs and config files for remote diagnosis.