Practical Prometheus, Grafana, and OpenTelemetry: Building Observability for Kubernetes

Scenario

A microservices application experiences intermittent high latency after deployment, burning through error budget quickly. The team already has Prometheus and Grafana but lacks distributed tracing, making root cause analysis difficult.

Symptoms

• Users report slow page loads (>5 seconds)
• Grafana shows p99 latency spiking from 200ms to 2s
• Error rate rises from 0.5% to 3%, nearing error budget exhaustion

Diagnosis

1. Check Prometheus metrics: Determine if high latency is global or local. Run query: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)). Find that frontend service p99 is normal, but payments service p99 is 4s.
2. Deploy OpenTelemetry Collector: Auto-inject traces to locate slow call chains.

Commands and Configuration

Install OpenTelemetry Collector (Helm)

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install otel-collector open-telemetry/opentelemetry-collector \
  --set mode=deployment \
  --set config.receivers.otlp.protocols.grpc.endpoint=0.0.0.0:4317 \
  --set config.exporters.prometheus.endpoint=0.0.0.0:8889 \
  --set-file config.pipelines.traces=traces.yaml

Prometheus Configuration (add scrape target)

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8889']

Grafana Dashboard (import JSON)

Create a trace dashboard using tempo or other compatible data source.

Risk Controls

• Namespace isolation: Deploy collector in a separate namespace to avoid affecting production.
• Sampling rate control: Default 10% sampling to prevent storage explosion.
• Configuration backup: Backup Prometheus and Grafana configs.

Rollback

helm rollback otel-collector 0  # Rollback to previous version
kubectl delete -f prometheus-config.yaml  # Remove scrape config

Verification

• Query: rate(traces_sampled_total[5m]) to confirm trace data inflow.
• In Grafana, check latency dashboard; p99 should return to normal (<500ms).
• Run end-to-end tests; error rate drops to 0.5%.

When to Submit an OpsGlobal Ticket

• Cannot identify root cause; expert analysis of call chains required.
• No metric changes after configuration; collector or exporter may be misconfigured.
• Need optimization of sampling strategy or storage solution.