Mastering Observability with Prometheus, Grafana, and OpenTelemetry

Scenario

An e-commerce platform's Kubernetes cluster experiences intermittent latency and errors in production. Users report slow page loads, but CPU and memory metrics appear normal. The SRE team needs to quickly root cause the issue.

Symptoms

• Application response time spikes from 200ms to 2s
• Error rate jumps from 0.1% to 5%
• Cluster resource usage is low (CPU 30%, memory 50%)
• No obvious anomalies in logs

Diagnosis

Existing monitoring covers only infrastructure metrics, missing distributed tracing and business context. We need OpenTelemetry for request-level tracing, Prometheus for metrics, and Grafana for unified visualization.

Deployment Commands

1. Deploy OpenTelemetry Collector

# otel-collector-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: observability
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.88.0
        ports:
        - containerPort: 4318
          name: otlp-http
        - containerPort: 4317
          name: otlp-grpc
        - containerPort: 8888
          name: metrics
        volumeMounts:
        - name: config
          mountPath: /etc/otelcol-contrib/config.yaml
          subPath: config.yaml
      volumes:
      - name: config
        configMap:
          name: otel-collector-config

2. ConfigMap Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
        timeout: 1s
        send_batch_size: 1024
    exporters:
      otlp:
        endpoint: "jaeger-collector.observability:4317"
        tls:
          insecure: true
      prometheus:
        endpoint: "0.0.0.0:8889"
        namespace: "app"
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus]

3. Configure Prometheus Scrape

# prometheus-scrape-config.yaml
scrape_configs:
- job_name: 'otel-collector'
  static_configs:
  - targets: ['otel-collector.observability:8889']
- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__

4. Deploy Jaeger (for trace storage)

kubectl create namespace observability
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm install jaeger jaegertracing/jaeger -n observability --set storage.backend=elasticsearch --set storage.elasticsearch.host=elasticsearch-master.observability

5. Deploy Grafana

helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana -n observability --set datasources."datasources\.yaml".apiVersion=1 --set datasources."datasources\.yaml".datasources[0].name=Prometheus --set datasources."datasources\.yaml".datasources[0].type=prometheus --set datasources."datasources\.yaml".datasources[0].url=http://prometheus-server.observability --set datasources."datasources\.yaml".datasources[0].access=proxy --set datasources."datasources\.yaml".datasources[0].isDefault=true

Risk Controls

• Sampling rate: Set a reasonable sampling rate (e.g., 10%) to avoid performance overhead from full tracing.
• High-cardinality labels: Avoid using unique values like user IDs in trace or metric labels.
• Configuration changes: Validate in staging environment before pushing to production.
• Resource limits: Set CPU/memory limits for the Collector to prevent OOM.

Rollback

1. Delete DaemonSet: kubectl delete -f otel-collector-daemonset.yaml
2. Remove Prometheus scrape config and restart Prometheus.
3. Delete Jaeger and Grafana helm charts: helm delete jaeger -n observability && helm delete grafana -n observability
4. Restore original monitoring configuration for applications.

Verification

1. Traces: Access Jaeger UI, search for requests, view span details.
2. Metrics: Query app_request_duration_seconds in Prometheus.
3. Dashboards: Import predefined dashboard in Grafana, verify latency and error rate decreased.
4. End-to-end: Send a test request manually, observe traces and metrics updating in real-time.

When to Submit an OpsGlobal Ticket

• No data is reported after deploying the Collector.
• Prometheus cannot scrape targets.
• Unusual spike in cluster resource consumption.
• Need expert assistance to optimize sampling strategy or design custom dashboards.