Unified Observability with Prometheus, Grafana, and OpenTelemetry: A Practical Incident Response Playbook

Scenario

Your Kubernetes platform runs dozens of microservices, each producing metrics, logs, and traces. Prometheus scrapes metrics, Grafana visualizes them, and you use a mix of tools for tracing. When an incident hits, you often find yourself switching between dashboards, log queries, and trace views, trying to piece together what happened. This is a classic observability silo problem.

In this post, we’ll build a unified observability pipeline using OpenTelemetry (OTel) for instrumentation and telemetry collection, Prometheus for metric storage, and Grafana for dashboards and alerting. We’ll also use exemplars to jump directly from a metric to a trace, drastically reducing mean time to diagnosis (MTTD).

Symptoms

• p99 latency on the checkout service exceeds 500ms, breaching the SLO.
• Error rate for the payments API climbs to 2%, consuming error budget quickly.
• CPU and memory dashboards show no obvious anomaly, but user reports indicate slowdowns.
• Support engineers manually correlate timestamps across logs and traces, wasting 30 minutes per incident.

Diagnosis

Start by localizing the problem using PromQL. Query the latency histogram for the checkout service:

histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket{service="checkout"}[5m])) by (le, route)
)

This reveals which routes are slow. Suppose /api/checkout is the culprit. Next, use the same histogram to find an exemplar—a representative trace ID. In Grafana, enable exemplars in the query editor. Click the sparkle icon on a data point to open the associated trace in your tracing backend (e.g., Tempo).

If you don’t have exemplars yet, you can still dig into traces by correlating timestamps. But exemplars make it much faster.

Commands and Configuration

1. Deploy the OpenTelemetry Collector

The OTel Collector is a vendor-agnostic gateway. Create a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  namespace: observability
data:
  otel-collector-config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
        timeout: 5s
    exporters:
      prometheusremotewrite:
        endpoint: http://prometheus:9090/api/v1/write
        retry_on_failure:
          enabled: true
      otlp:
        endpoint: tempo:4317
        tls:
          insecure: true
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheusremotewrite]
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp]

Apply it:

kubectl create ns observability
kubectl apply -f otel-collector.yaml

Also deploy the collector as a DaemonSet to collect node-level metrics and receive OTLP from services. For simplicity, we’ll use a single collector Deployment.

2. Instrument Services with OpenTelemetry

Use the OpenTelemetry Operator to inject auto-instrumentation:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Then annotate your deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout
  namespace: prod
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"

Or use SDKs manually. For a Python service:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317"))
)

3. Configure Prometheus for Remote Write

Enable the remote write receiver when starting Prometheus:

prometheus --web.enable-remote-write-receiver

In your Prometheus config, ensure you have a scrape job for the collector:

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8888']

4. Set Up Grafana Data Sources and Exemplars

In Grafana, add Prometheus and Tempo as data sources. For Prometheus, in the data source settings, enable “Exemplars” and add a derived field with the trace ID lookup:

• TraceID field: traceID
• Data source: Tempo
• Query: {traceID="${__value.raw}"}

Now you can click exemplars in any Prometheus graph to open the full trace.

5. Build Dashboards and Alerts

Create a dashboard with panels for latency, error rate, and saturation (the USE method). For alerting, define a Prometheus rule:

groups:
  - name: checkout-slo
    rules:
      - alert: CheckoutP99High
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket{service="checkout"}[5m]))
            by (le, route)
          ) > 0.5
        for: 10m
        annotations:
          summary: "Checkout service p99 latency high"

Risk Controls

• Canary the collector: Start with one instance and monitor its resource usage.
• Use resource limits: Set memory and CPU limits on the collector to avoid noisy neighbors.
• Rate-limit remote writes: Use queue_size and retry_on_failure to avoid flooding Prometheus.
• Back up configs: Store Prometheus, Grafana, and OTel configs in Git before rolling out.
• Enable TLS for OTLP endpoints in production, using commercial certs or mTLS.

Rollback

If the OTel collector causes instability, you can quickly roll back:

kubectl delete deployment otel-collector -n observability

This stops all OTLP ingestion. Prometheus will still query its existing scrape targets. For auto-instrumentation, remove the annotations from the deployment:

kubectl annotate deployment checkout instrumentation.opentelemetry.io/inject-java-

For Prometheus config changes, use the previous config file:

cp prometheus.yml.bak prometheus.yml
kill -HUP $(pidof prometheus)

Verification

After deployment, verify each tier:

1. Collector: Check logs for errors: kubectl logs <otel-collector-pod>
2. Prometheus: Query a metric that only exists via OTLP remote write, e.g., otelcol_exporter_sent_metric_points.
3. Grafana: Open a dashboard and confirm new data appears.
4. Exemplars: In Grafana Explore, run a PromQL query, hover over a data point, and click “Exemplar” to open the trace.
5. Alerting: Use curl to generate high latency and verify the alert fires in 10 minutes.

When to Submit an OpsGlobal Ticket

You should consider hiring OpsGlobal if:

• Your team is spending more than 10 hours per week maintaining your observability stack.
• You need advanced instrumentation for legacy services that don't support OTel natively.
• You want 24/7 monitoring of your observability infrastructure itself.
• You're about to migrate to a service mesh and need help integrating telemetry.

OpsGlobal's remote SREs can design, deploy, and operate your Prometheus, Grafana, and OpenTelemetry stack, ensuring you meet your SLOs and get the visibility you need.