Building Unified Observability: A Practical Guide with OpenTelemetry, Prometheus, and Grafana in Kubernetes

Scenario

An e-commerce platform runs microservices on Kubernetes. Users report slow page loads. Operations team sees p99 latency spike from 200ms to 2s, but existing Prometheus and Grafana only collect CPU/memory metrics. They decide to adopt OpenTelemetry (OTel) for distributed tracing, metrics, and logs.

Symptoms

• Grafana dashboards show p99 latency >1.5s for multiple services.
• Application logs show no errors, but HTTP 503 responses increase.
• Prometheus alert "HighLatencyCritical" fires.

Diagnosis

Current setup lacks correlation between service calls. - Prometheus scrapes basic metrics via kube-state-metrics. - No tracing libraries in applications. - No centralized log aggregation.

Conclusion: Missing distributed tracing prevents bottleneck identification.

Solution: Integrate OpenTelemetry

1. Deploy OpenTelemetry Collector as a sidecar per service pod, collecting traces and exporting to Prometheus (via remote write) and Grafana Tempo.
2. Instrument application code with OTel SDK (auto or manual).
3. Configure Grafana with Prometheus and Tempo data sources; create dashboard for trace details.

Commands

Step 1: Deploy OTel Collector (Sidecar)

Create ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      prometheus:
        endpoint: 0.0.0.0:8889
      otlp:
        endpoint: tempo-sample:4317  # assuming Tempo service
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp]
        metrics:
          receivers: [otlp]
          exporters: [prometheus]

Inject sidecar into Deployment:

spec:
  template:
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://localhost:4317
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:latest
        args: ["--config=/conf/config.yaml"]
        volumeMounts:
        - name: otel-collector-config-vol
          mountPath: /conf
        ports:
        - containerPort: 4317
        - containerPort: 4318
        - containerPort: 8889
      volumes:
      - name: otel-collector-config-vol
        configMap:
          name: otel-collector-conf

Step 2: Configure Prometheus Remote Write

In Prometheus configuration:

remote_write:
  - url: http://prometheus-server:9090/api/v1/write

Note: Use dedicated remote write endpoint in production.

Step 3: Grafana Setup

Add Prometheus (http://prometheus-server:9090) and Tempo (http://tempo:3100) data sources. Create Dashboard: query spans, e.g., kind=SPAN_KIND_SERVER.

Risk Controls

• Performance impact: Sidecar consumes ~50MB memory each. For high-throughput apps, reduce sampling rate (e.g., 10%).
• Data security: Trace data may contain sensitive info. Use OTel processors to filter/redact.
• Version compatibility: Ensure OTel Collector version matches SDK.

Rollback Plan

If issues arise: 1. Rollback application Deployment: kubectl rollout undo deployment/myapp. 2. Delete OTel ConfigMap: kubectl delete configmap otel-collector-conf. 3. Remove new data sources in Grafana; restore old dashboards.

Verification

1. Check traces: In Grafana Explore, query traces – should see full call chains.
2. Metrics: Prometheus has custom metrics like service_latency_seconds.
3. Load test: Use hey to simulate traffic; p99 should normalize.

When to Submit an OpsGlobal Ticket

• Encounter configuration errors in OTel Collector (e.g., missing data).
• Need help optimizing sampling strategy or scaling Tempo storage.
• Grafana dashboard design consultation.

Our SRE experts respond within 15 minutes, offering end-to-end observability solutions.