Practical Guide: Observability with Prometheus, Grafana, and OpenTelemetry

Scenario

An e-commerce platform running microservices on Kubernetes experiences intermittent latency spikes and increased error rates. The team uses Prometheus for metrics and Grafana for dashboards, but lacks distributed tracing to pinpoint the root cause.

Symptoms

• Users report slow page loads during peak hours
• Some API endpoints return 5xx errors
• Existing Prometheus alerts (e.g., high request latency) fire, but it's unclear whether the issue is due to database slowness, network jitter, or code logic

Diagnosis

Introduce OpenTelemetry (OTel) by auto-instrumenting sidecars or using the OTel Operator, export traces to Jaeger (or Grafana Tempo), and correlate with metrics in Grafana.

Steps

1. Deploy the OpenTelemetry Operator: bash kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
2. Create an OTel Collector instance (example YAML): yaml apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otel-collector spec: config: | receivers: otlp: protocols: grpc: http: exporters: jaeger: endpoint: jaeger-collector:14250 service: pipelines: traces: receivers: [otlp] exporters: [jaeger]
3. Annotate deployments for automatic injection: bash kubectl annotate deployment my-service instrumentation.opentelemetry.io/inject-java="true"
4. In Grafana, add Jaeger as a data source and use Explore to correlate traces with logs and metrics via TraceID.

Risk Controls

• Test OTel integration in staging first
• Set sampling strategy: start with 1% and gradually increase
• Configure resource limits and rate limiting for the OTel Collector
• Keep original Prometheus configuration as a fallback

Rollback

To roll back:

kubectl delete -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
kubectl delete opentelemetrycollector otel-collector
kubectl annotate deployment my-service instrumentation.opentelemetry.io/inject-java-

Additionally, clean up Jaeger storage and revert any Prometheus remote write changes.

Verification

• Confirm service graph and trace data in Jaeger UI
• Build a Grafana dashboard showing trace latency overlaying Prometheus metrics (e.g., request rate)
• Trigger a known failure (e.g., simulate slow SQL) and verify the trace clearly identifies the bottleneck

When to Submit an OpsGlobal Ticket

• Internal debugging exceeds 2 hours without identifying the root cause
• Cluster-wide performance degradation affecting multiple services
• Need expert tuning of sampling rates, storage capacity, or Collector high-availability configuration
• Team lacks OpenTelemetry experience and requires architecture review