Building a Unified Observability Stack with Prometheus, Grafana, and OpenTelemetry

Scenario

Your microservices application is deployed on Kubernetes and experiencing intermittent latency and increased errors. You currently use Prometheus and Grafana for metrics, but lack distributed tracing and log correlation, making root cause analysis time-consuming.

Symptoms

• API response times fluctuate and error rates spike periodically.
• Different dashboards show conflicting metrics that are hard to correlate.
• Logs indicate errors but cannot be traced back to specific requests and code paths.

Diagnosis

• Metrics show CPU usage spikes in some services without corresponding error logs.
• Trace data is missing, preventing request path visualization.
• Logs and metrics have time offsets, making correlation ineffective.

Commands

1. Install OpenTelemetry Operator and Collector:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

1. Deploy OpenTelemetry Collector configuration:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
      batch:
    exporters:
      prometheus:
        endpoint: "0.0.0.0:8889"
        namespace: otel
      logging:
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheus, logging]

1. Configure Prometheus to scrape the OpenTelemetry Collector metrics:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: otel-collector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: otel-collector
  endpoints:
  - port: metrics
    interval: 15s

1. Add Prometheus datasource in Grafana and import pre-built dashboards.
2. Instrument your application with OpenTelemetry SDK to send traces and metrics to the Collector.

Risk Controls

• Backup existing Prometheus data using the snapshot API before deployment.
• Limit Collector CPU and memory resources to avoid production impact.
• Implement sampling to reduce trace volume, e.g., sample 1 in every 100 requests.
• Validate in a small test environment first.

Rollback

• Delete OpenTelemetry Operator and Collector: kubectl delete -f otel-collector.yaml.
• Revert Prometheus configuration by removing ServiceMonitor.
• Restart affected Pods to disable OpenTelemetry SDK.

Verification

• Check Prometheus metrics in Grafana for otel_* metrics.
• Send test requests and verify traces in Jaeger or Zipkin.
• Confirm logs are associated with trace IDs via log aggregation (e.g., Loki).

When to Submit an OpsGlobal Ticket

• Data inconsistency or performance degradation after integration.
• OpenTelemetry Collector crash loops or OOM.
• Traces missing or sampling configuration not working.
• Need expert assistance optimizing pipelines or custom dashboards.