Prometheus + Grafana + OpenTelemetry: A Production Observability Guide

Scenario

An e-commerce platform with microservices on Kubernetes experiences slow order responses (P99 up to 2s). Existing monitoring shows only CPU/memory, not service-level or code-level bottlenecks. The team adopts OpenTelemetry for unified instrumentation, Prometheus for metrics storage, and Grafana for dashboards.

Symptoms

• Order API P99 latency increases from 200ms to 2s
• No spike in error rates, but some requests time out
• No correlation between metrics and request traces

Diagnosis

Add OpenTelemetry Java auto-instrumentation to services. Deploy OpenTelemetry Collector as a sidecar or daemonset to receive OTLP data and export metrics to Prometheus and traces to Jaeger/Tempo.

Commands

1. Deploy OpenTelemetry Collector

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
    exporters:
      prometheus:
        endpoint: "0.0.0.0:8889"
      logging:
        loglevel: debug
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus, logging]
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  selector:
    app: otel-collector
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
  - name: otlp-http
    port: 4318
    targetPort: 4318
  - name: prometheus
    port: 8889
    targetPort: 8889
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.96.0
        args: ["--config=/conf/config.yaml"]
        ports:
        - containerPort: 4317
        - containerPort: 4318
        - containerPort: 8889
        volumeMounts:
        - name: otel-collector-config
          mountPath: /conf
      volumes:
      - name: otel-collector-config
        configMap:
          name: otel-collector-conf
EOF

2. Configure Prometheus Scrape

Update prometheus.yml:

scrape_configs:
  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
      - targets: ['otel-collector:8889']

3. Grafana Dashboard

Add Prometheus datasource, import dashboard (e.g., OTel Metrics Mixin).

Risk Controls

• Deploy Collector in staging first
• Set resource limits: CPU 200m, memory 512Mi
• Enable batch processor
• Monitor Collector logs via logging exporter

Rollback

1. Delete Collector deployment: kubectl delete deployment otel-collector
2. Revert prometheus scrape config
3. Remove instrumentation from application deployment

Verification

• Prometheus target page shows UP status
• Grafana shows metrics like http_server_duration_ms
• Traces visible in Jaeger UI with correct spans

When to Submit an OpsGlobal Ticket

• Custom instrumentation causes performance degradation or data loss
• Need help with high-cardinality metric tuning (e.g., user_id labels)
• Complex exemplar configuration for metrics-traces correlation
• Collector becomes a bottleneck or runs out of memory