Building a Unified Observability Pipeline: Prometheus, Grafana, and OpenTelemetry in Practice

Scenario

An e-commerce platform runs microservices on Kubernetes. Users report slow order placement and timeouts. The ops team has Prometheus and Grafana for basic metrics but cannot pinpoint the root cause.

Symptoms

• Prometheus alerts: Order service error rate rises to 5%, p99 latency >2s.
• Grafana dashboards show normal CPU/memory but abnormal network throughput.
• Logs contain only intermittent "upstream service timeout" errors.

Diagnosis

Introduce distributed tracing via OpenTelemetry. Auto-instrumentation and a trace backend (Jaeger/Tempo) reveal full call chains.

1. Deploy OpenTelemetry Collector

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector --values otel-collector-values.yaml

Example otel-collector-values.yaml:

mode: deployment
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
  processors:
    batch:
      timeout: 1s
  exporters:
    prometheus:
      endpoint: 0.0.0.0:8889
    otlp:
      endpoint: tempo.monitoring:4317
      tls:
        insecure: true

Note: Configure TLS and auth in production.

2. Configure Prometheus to Scrape OpenTelemetry Metrics

Add a job in Prometheus config:

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

3. Create Grafana Dashboards

Import pre-built OpenTelemetry dashboard (ID: 14032) or custom panels for trace latency, error rates, and dependencies.

Diagnosis Result

Traces revealed that the order service experienced DNS resolution delays when calling the payment gateway. Optimization of DNS caching resolved the issue.

Risk Controls

• Test OpenTelemetry configuration in non-production first.
• Set resource limits for Collector (CPU 500m, memory 512Mi).
• Enable sampling (e.g., 100 traces/sec) to avoid overload.

Rollback

If the Collector causes performance issues:

helm uninstall otel-collector

and remove the corresponding job from Prometheus.

Verification

1. Confirm metric drop in Grafana.
2. Check Collector logs: kubectl logs -l app=otel-collector.
3. Send test requests and view traces in Jaeger UI.

When to Submit an OpsGlobal Ticket

• Encounter OpenTelemetry SDK incompatibility with your language.
• Need custom business metrics or trace granularity.
• Large cluster requiring optimized sampling or storage solutions.