Mastering Observability with Prometheus, Grafana, and OpenTelemetry in Kubernetes

Scenario

You run a microservices application on Kubernetes and need end-to-end observability including metrics, traces, and logs. You have deployed Prometheus for metrics, Grafana for dashboards, and OpenTelemetry Collector for traces and logs. Recently, users report slow responses and 5xx errors on some endpoints. Prometheus alerts show high request latency and increased error rates.

Symptoms

• Users complain about slow page loads, occasional 5xx errors.
• Prometheus Alertmanager fires HighRequestLatency and HighErrorRate alerts.
• Grafana dashboards show p99 latency > 500ms and error rate > 5%.

Diagnosis

1. Open Grafana, check the prebuilt "Kubernetes Services" dashboard, filter to the affected namespace.
2. Identify high-latency services – often payment-service and order-service.
3. Navigate to the Traces page (e.g., Grafana Tempo datasource), search by affected endpoint, view distributed traces.
4. Notice payment-service makes a database call taking >1 second, with some failures.
5. Check logs for payment-service via kubectl logs or OpenTelemetry log export.

Commands

Deploy OpenTelemetry Collector

Install via Helm:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-collector open-telemetry/opentelemetry-collector \
  --set config.exporters.prometheus.endpoint="0.0.0.0:8889" \
  --set config.service.pipelines.metrics.exporters=[prometheus]

Configure Prometheus to scrape OpenTelemetry Collector metrics

Add to scrape_configs:

- job_name: 'opentelemetry-collector'
  static_configs:
    - targets: ['otel-collector:8889']

Restart application with OTel SDK (example for Java)

kubectl set env deployment/payment-service JAVA_TOOL_OPTIONS="-javaagent:/opt/opentelemetry-javaagent.jar"
kubectl rollout restart deployment/payment-service

Query Prometheus to verify metrics

kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090
curl http://localhost:9090/api/v1/query?query=rate(http_server_duration_ms_sum[5m])

Risk Controls

• Backup existing configs before changes: bash kubectl get configmap -n monitoring prometheus-kube-prometheus-prometheus -o yaml > prometheus-config-backup.yaml
• Canary test new OTel Collector versions by enabling only for a subset of services.
• Rate-limit metrics export to avoid overload: add a limit processor in OpenTelemetry Collector config.
• Validate changes in a staging environment first.

Rollback

If changes cause issues:

1. Restore Prometheus ConfigMap: bash kubectl apply -f prometheus-config-backup.yaml kubectl delete pod -n monitoring -l app.kubernetes.io/component=prometheus
2. Rollback OpenTelemetry Collector Helm release: bash helm rollback otel-collector 0
3. Rollback application deployment (e.g., remove Javaagent): bash kubectl set env deployment/payment-service JAVA_TOOL_OPTIONS- kubectl rollout restart deployment/payment-service

Verification

• Check Prometheus targets: http://localhost:9090/targets – OpenTelemetry Collector should be UP.
• View the "OpenTelemetry Collector" dashboard in Grafana to see metrics flowing.
• Perform a test request and check for the trace in Grafana Tempo.
• Confirm alerts recover: kubectl get alerts -n monitoring.

When to Submit an OpsGlobal Ticket

Submit a ticket if:

• You cannot integrate OpenTelemetry with existing Prometheus or Grafana.
• You need help configuring distributed tracing for custom services.
• Issues persist after following rollback steps.
• You require a production-ready configuration with security best practices.