Unifying Telemetry: A Practical Guide to Prometheus, Grafana, and OpenTelemetry for Kubernetes SREs

Scenario: Your organization runs a microservices-based platform on Kubernetes. Each service emits metrics, logs, and traces using different formats. Recently, customer-facing APIs have started experiencing intermittent latency spikes, but your current monitoring tools are siloed. Prometheus scrapes metrics, Grafana visualizes dashboards, but traces are only available in a separate APM tool, and logs are in another system. You need a unified view to correlate metrics, traces, and logs efficiently.

Symptoms: - Intermittent p99 latency spikes on the payments service and the product catalogue API. - Prometheus alerts show CPU throttling on some pods, but not consistently. - Developers spend hours cross-referencing dashboards, logs, and traces to identify the root cause. - Alert fatigue due to many redundant alerts from Kubernetes components. - No single source of truth for telemetry.

Diagnosis: The root cause of these symptoms is the lack of a standard telemetry generation and collection layer. We need to instrument services with OpenTelemetry, route all telemetry through the OpenTelemetry Collector, and use Prometheus and Grafana to store and visualize metrics. A unified collection layer will allow you to perform a root-cause analysis by correlating metrics, traces, and logs.

Commands: We'll break down the implementation into steps:

1. Deploy the OpenTelemetry Collector using Helm:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install otel-collector open-telemetry/opentelemetry-collector \
  --namespace observability --create-namespace \
  -f otel-collector-config.yaml

Sample otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: "otel"
  debug:
    verbosity: normal
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
    traces:
      receivers: [otlp]
      exporters: [debug]

1. Instrument a sample service with OpenTelemetry (using the Java agent or Python SDK). For a Java service:

java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.service.name=payments-service \
  -Dotel.exporter.otlp.endpoint=http://otel-collector.observability:4317 \
  -jar payments-service.jar

1. Configure Prometheus to scrape the OpenTelemetry Collector metrics endpoint. Add a scrape config in Prometheus:

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector.observability:8889']

1. Import a Grafana dashboard using the config file or via UI. You can use the Grafana provisioning approach:

apiVersion: 1
providers:
  - name: 'default'
    folder: 'SRE'
    type: file
    options:
      path: /var/lib/grafana/dashboards

Risk Controls: - Before modifying the production observability stack, take a snapshot of the existing Grafana dashboards and Prometheus rules using the API. - Deploy the OpenTelemetry Collector in a staging environment first and validate that telemetry is flowing correctly. - Use a separate namespace observability and limit resource usage (CPU/memory) with requests and limits. - Never expose the OpenTelemetry Collector endpoint to the public internet; use internal networking. - For critical services, use a canary deployment: route a small percentage of traffic to a service with OpenTelemetry instrumentation while leaving the rest unchanged.

Rollback: If the OpenTelemetry Collector causes performance degradation or forwarding issues, roll back quickly: 1. Remove the Prometheus scrape configuration for the collector and reload Prometheus. 2. Scale down the collector instance: kubectl scale deployment otel-collector -n observability --replicas=0 3. If you instrumented services with the Java agent, redeploy the previous version without the agent or with the agent disabled. 4. Restore previous Grafana dashboards using the snapshot taken earlier.

Verification: - Confirm metrics are being scraped by Prometheus: curl -G http://prometheus:9090/api/v1/targets | grep otel-collector - Verify traces are flowing: check the collector logs for exported spans or use the debug exporter. - Create a Grafana dashboard and add a Prometheus query like histogram_quantile(0.99, sum(rate(http_server_duration_seconds_bucket{job="payments-service"}[5m])) by (le, service_name)) to visualize p99 latency. - Correlate a latency spike with a trace: query the span data using Jaeger or Tempo (in a follow-up guide). - Use the top command on the collector pod to ensure it's not resource-constrained.

When to Submit an OpsGlobal Ticket: OpsGlobal provides 24/7 remote SRE support and can help you if: - Your team has limited experience with OpenTelemetry and needs expert guidance. - The observability stack is critical to production, and you need round-the-clock monitoring. - You're facing a complex incident where correlation of metrics, traces, and logs requires deep expertise. - You want to scale your observability setup to handle growing telemetry volume without hiring additional staff.