Implementing Observability with Prometheus, Grafana, and OpenTelemetry on Production Kubernetes

Scenario

An e-commerce platform runs 50+ microservices on Kubernetes. The team experiences high mean time to resolution (MTTR) due to fragmented monitoring. They decide to implement Prometheus, Grafana, and OpenTelemetry to unify observability.

Symptoms

• Switching between multiple tools during incident response
• Inability to correlate metrics with traces
• Incident resolution time exceeds 4 hours

Diagnosis

• No auto-instrumentation and standard collection
• Inconsistent metric naming and labeling across teams
• No unified trace backend

Commands

Below are example commands to deploy OpenTelemetry Collector and configure Prometheus on Kubernetes (assuming Helm is installed).

1. Install OpenTelemetry Collector

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector --values otel-values.yaml

Sample otel-values.yaml:

mode: deployment
config:
  receivers:
    otlp:
      protocols:
        grpc:
        http:
  processors:
    batch:
  exporters:
    prometheus:
      endpoint: "0.0.0.0:8889"
    logging:
    # Optional: export traces to Jaeger or Zipkin
    otlp:
      endpoint: "jaeger-collector:4317"
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus, logging]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp, logging]

2. Configure Prometheus to scrape OpenTelemetry Collector metrics

# prometheus-scrape-config.yaml
scrape_configs:
  - job_name: 'opentelemetry-collector'
    scrape_interval: 15s
    static_configs:
      - targets: ['otel-collector:8889']

3. Deploy Grafana and add Prometheus data source

helm install grafana grafana/grafana --set persistence.enabled=true

After logging into Grafana, add Prometheus data source (URL: http://prometheus-server) and import pre-built dashboards.

Risk Controls

• Deploy to a non-critical namespace first
• Use --dry-run with Helm to validate configuration
• Monitor Collector resource usage (CPU/memory)
• Set Pod resource limits
• Backup Prometheus and Grafana configurations

Rollback

# Uninstall OpenTelemetry Collector
helm uninstall otel-collector
# Restore Prometheus configuration
kubectl replace -f backup-prometheus-config.yaml
# If needed, restart Prometheus Pod
kubectl rollout restart -n monitoring deployment/prometheus-server

Verification

1. Check OpenTelemetry Collector Pods are running: kubectl get pods -l app.kubernetes.io/name=opentelemetry-collector
2. Query Prometheus for metrics: rate(otelcol_exporter_sent_metric_points_total[5m])
3. View dashboards in Grafana: ensure data appears
4. Send a test request and inspect traces (e.g., using Jaeger UI or kubectl logs of Collector)

When to Submit an OpsGlobal Ticket

• Custom OpenTelemetry Collector processors (e.g., filtering, sampling) needed but lacking in-house expertise
• Cross-cluster Prometheus federation requiring expert guidance
• Compatibility issues when injecting OpenTelemetry SDKs into legacy systems
• Performance tuning (e.g., adjusting batch size, memory limits)

If any of these complex scenarios arise, contact OpsGlobal for professional support.