Unified Observability with Prometheus, Grafana, and OpenTelemetry on Kubernetes

Scenario

An e-commerce platform experiences intermittent request latency on its Kubernetes cluster. Traditional monitoring (CPU, memory) fails to correlate application traces with infrastructure metrics. The SRE team decides to deploy OpenTelemetry to unify traces, metrics, and logs, and combine it with Prometheus and Grafana to build a comprehensive observability platform.

Symptoms

• Users report slow page loads, but Prometheus node utilization appears normal.
• Grafana dashboards show P99 latency spikes for some services, but lack root cause context.
• Difficult to determine whether the issue is a slow database query or network jitter.

Diagnosis

1. Verify OpenTelemetry Collector pods: kubectl get pods -n opentelemetry.
2. Check Prometheus target status: In Prometheus UI, ensure targets are all UP.
3. Test Grafana data source connection: Go to Grafana configuration and test the Prometheus data source.
4. Review OpenTelemetry Collector logs for export errors: kubectl logs -n <namespace> <collector-pod>.

Commands

# Install OpenTelemetry Operator
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install opentelemetry-operator open-telemetry/opentelemetry-operator --namespace opentelemetry --create-namespace

# Deploy OpenTelemetry Collector (custom config)
kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
  namespace: opentelemetry
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch:
    exporters:
      prometheus:
        endpoint: 0.0.0.0:8889
        namespace: otel
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus]
EOF

# Configure Prometheus to scrape OpenTelemetry metrics
# In Prometheus configuration add:
# - job_name: 'otel-collector'
#   static_configs:
#     - targets: ['otel-collector.opentelemetry:8889']

# Restart Prometheus or use Helm upgrade
helm upgrade --install prometheus prometheus-community/prometheus --namespace monitoring --set extraScrapeConfigs="- job_name: 'otel-collector'
  static_configs:
  - targets: ['otel-collector.opentelemetry:8889']"

Risk Controls

• Namespace isolation: Deploy OpenTelemetry, Prometheus, and Grafana in separate namespaces (e.g., opentelemetry, monitoring) to avoid resource conflicts.
• Resource limits: Set CPU/memory limits for OpenTelemetry Collector (e.g., 500m/512Mi) to prevent traffic spikes from overwhelming the cluster.
• Backup configuration: Before modifying Prometheus or Grafana, back up existing Prometheus volumes and Grafana databases.
• Canary deployment: Validate OpenTelemetry configuration in a non-production cluster before rolling to production.

Rollback

# Rollback OpenTelemetry Collector to previous version
kubectl delete -f otel-collector.yaml  # if using file deployment
helm rollback opentelemetry-operator 0  # using Helm rollback to previous revision

# Rollback Prometheus configuration
kubectl delete configmap prometheus-server -n monitoring  # may need to rebuild
# or helm rollback prometheus 1

Verification

1. Confirm OpenTelemetry Collector Pod is running: kubectl get pods -n opentelemetry -l app.kubernetes.io/name=opentelemetry-collector.
2. Check Prometheus targets: Visit Prometheus UI, search for job="otel-collector" and ensure status is UP.
3. In Grafana, import a sample dashboard (e.g., "OpenTelemetry Collector Metrics") and view time-series data.
4. Simulate a slow query by injecting latency into a test service; observe the OTel metrics update in real time.

When to Submit an OpsGlobal Ticket

• The diagnosis steps above do not resolve the issue, and expert assistance is needed to adjust OpenTelemetry Collector configuration.
• Prometheus scrape fails despite correct network policies, suggesting underlying cluster issues.
• Custom Grafana dashboards are required to correlate traces, metrics, and logs, but the team lacks experience.
• Performance regression after a canary deployment in production; immediate rollback and root cause analysis are needed.