Mastering OpenTelemetry, Prometheus, and Grafana: A Practical Troubleshooting Guide for Kubernetes SREs

Scenario

Your Kubernetes cluster runs Prometheus and Grafana. The application team has enabled OpenTelemetry to send traces and metrics through the OpenTelemetry Collector to Prometheus's remote write endpoint. Suddenly, Grafana dashboards for some services start showing gaps, Prometheus's CPU and memory usage spike, and you face OOM risks. At the same time, the Collector logs show remote write errors.

Symptoms

• Grafana panels show "No data" or intermittent gaps.
• Prometheus pod memory grows continuously and may hit limits.
• OpenTelemetry Collector responds with HTTP 500 or 429.
• The up metric indicates targets are unreachable, or prometheus_remote_storage_queue_length keeps increasing.

Diagnosis

1. Check Prometheus Targets

First, verify that Prometheus is still scraping all targets. Visit the Prometheus UI Targets page or use the command line:

kubectl exec -n monitoring prometheus-0 -- wget -qO- http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {scrapeUrl, health, lastError}'

Look for targets that have become down. Common causes include label changes that break selectors or exporter port changes.

2. Check Remote Write Configuration

Verify that Prometheus's remote write configuration is correct. Inspect the configuration file:

kubectl exec -n monitoring prometheus-0 -- cat /etc/prometheus/prometheus.yml | grep -A 20 'remote_write'

Confirm that the url points to the OpenTelemetry Collector endpoint and that authentication credentials (if any) are not expired.

3. Check OpenTelemetry Collector Logs

Tail the Collector logs to find specific errors:

kubectl logs -n otel-collector -l app=otel-collector --tail=100

Common errors include:

• 416 Requested Range Not Satisfiable — time series went backwards.
• 429 Too Many Requests — write limits exceeded; you need rate limiting or scaling.
• 500 Internal Server Error — Collector internal issue, possibly exporter misconfiguration.

4. Check for Cardinality Explosion

High cardinality causes Prometheus memory pressure. Use PromQL to detect:

kubectl exec -n monitoring prometheus-0 -- wget -qO- 'http://localhost:9090/api/v1/query?query=topk(10,%20count%20by%20(__name__)(%7B__name__%3D~%22.%2B%22%7D))'

Or execute topk(10, count by (__name__)({__name__=~".+"})) in the Web UI.

Also check if any label values are exploding, such as url or request_id:

count(count by (url) (http_requests_total)) > 1000

5. Verify Whether the Collector Is Losing Metrics

Enable the debug exporter on the Collector to also send data to stdout and observe if metrics arrive:

exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    metrics:
      exporters: [debug, prometheus]

Commands and Actions

Backup Current Configurations

kubectl get configmap -n monitoring prometheus-config -o yaml > prometheus-config-backup.yaml
kubectl get configmap -n otel-collector otel-collector-config -o yaml > collector-config-backup.yaml

Apply Config Changes Safely

Always use --dry-run=client and --dry-run=server first:

kubectl apply -f prometheus-config-backup.yaml --dry-run=client -o yaml
kubectl apply -f prometheus-config-backup.yaml --dry-run=server

Adjust metrics_relabel_configs to Limit Cardinality

In the Prometheus config, add relabel rules to drop high-cardinality labels from metrics sent to remote write:

remote_write:
  - url: http://otel-collector.otel-collector:4318/api/v1/metrics
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'http_requests_total'
        action: keep
      - source_labels: [request_id]
        regex: '.*'
        action: drop

Reload Prometheus Configuration

kubectl exec -n monitoring prometheus-0 -- kill -HUP 1

Scale OpenTelemetry Collector

If the Collector is a bottleneck, increase replicas and resources:

kubectl scale deployment/otel-collector --replicas=3 -n otel-collector
kubectl set resources deployment/otel-collector -n otel-collector --requests='cpu=500m,memory=1Gi' --limits='cpu=1,memory=2Gi'

Risk Controls

• Do not modify production config without taking a backup.
• Misconfigured write_relabel_configs can silently drop all data; always test in a staging environment first.
• Running kill -HUP reloads config without restarting the process. If the config is invalid, Prometheus may fail to reload; use promtool check config before applying.
• Dropping/retaining labels at scale may prevent Prometheus from discovering targets; assess the impact first.
• Scaling the Collector may increase cost; adjust based on actual resource usage.

Rollback Plan

1. Restore configuration backups:

kubectl apply -f prometheus-config-backup.yaml
kubectl apply -f collector-config-backup.yaml

1. Reload Prometheus to apply the config.

2. If Collector changes caused the issue, roll back the Deployment to the previous image version:

kubectl rollout undo deployment/otel-collector -n otel-collector

1. Verify all Pods are healthy:

kubectl get pods -n monitoring -n otel-collector

Verification

Verify Prometheus Receives Remote Writes

Check remote write metrics:

prometheus_remote_storage_succeeded_samples_total - prometheus_remote_storage_failed_samples_total

Or query the up metric:

up

Verify OpenTelemetry Metrics Reach Prometheus

Assuming the application exposes http_requests_total, create a Grafana panel and query:

sum(rate(http_requests_total[5m])) by (service)

Verify Cardinality Has Decreased

Run the cardinality query again and confirm that the number of label values is within a reasonable range.

Check Memory Usage

Observe the Prometheus container memory to confirm it has stopped growing.

When to Submit an OpsGlobal Ticket

• If you cannot restore data after half an hour of the above steps.
• Prometheus is OOM-killed repeatedly and needs sharding or long-term storage solutions.
• The OpenTelemetry Collector is crash-looping and you cannot locate the configuration issue.
• You need expert help to design a large-scale observability architecture to prevent similar problems.

OpsGlobal provides 24×7 SRE support. Our experts can remotely access your cluster, quickly diagnose and resolve complex Prometheus, Grafana, and OpenTelemetry issues, ensuring your observability stack remains stable and reliable.