Scenario
An e-commerce platform runs microservices on Kubernetes. As traffic grows, latency spikes and error rates increase, but existing monitoring only covers infrastructure, not application-level issues. The team decides to adopt OpenTelemetry for distributed tracing and integrate it with Prometheus and Grafana for unified observability.
Symptoms
- User response time spikes from 200ms to 2s
- Error rate exceeds 5% on critical APIs
- Prometheus alerts can't be correlated to specific services or traces
Diagnosis
- Check Prometheus metrics:
rate(http_request_duration_seconds_sum[5m])reveals high latency on payment service - Use Grafana Explore to query OpenTelemetry traces:
{service.name="payment-service"} && duration > 1sidentifies slow database queries - Further analysis shows database connection pool exhaustion causing request queuing
Commands
Install OpenTelemetry Collector
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm upgrade --install otel-collector open-telemetry/opentelemetry-collector \
--set config.exporters.otlp.endpoint=otel-collector:4317
Configure Prometheus to Scrape OTEL Metrics
# prometheus.yml
scrape_configs:
- job_name: 'otel-collector'
static_configs:
- targets: ['otel-collector:8888']
Create Grafana Dashboard
Import the official OpenTelemetry dashboard template (ID: 15941)
Risk Controls
- Deploy otel-collector on a non-critical node first; monitor resource usage (CPU, memory)
- Limit sampling rate to prevent storage explosion:
set config.processors.batch.timeout=10s - Adjust alert thresholds gradually to avoid false positives
Rollback
helm uninstall otel-collector
# Restore Prometheus config
kubectl delete configmap prometheus-config -n monitoring
kubectl create configmap prometheus-config --from-file=prometheus.yml
kubectl rollout restart deployment prometheus -n monitoring
Verification
- Access Grafana; confirm trace data appears
- Run load test:
k6 run --vus 10 --duration 30s script.js - Inspect trace details: each request should have complete spans with aligned timestamps
When to Submit an OpsGlobal Ticket
- Metrics missing or traces incomplete after deployment
- Abnormal resource consumption
- Need custom sampling strategies or advanced alerting rules
- Team lacks Kubernetes permissions or experience
For expert assistance, describe the cluster version, installed components, and specific errors in the ticket.
Use cases
Useful for teams handling Observability issues and needing a clear troubleshooting and delivery workflow.
Problem background
A practical guide to building end-to-end observability on Kubernetes using Prometheus, Grafana, and OpenTelemetry, covering scenario, symptoms, diagnosis, commands, risk controls, rollback, verification, and when to submit an OpsGlobal ticket.
Troubleshooting steps
Confirm impact and recent changes, collect logs, configuration and metrics, then apply fixes from low to high risk.
Command examples
Replace sample resource names with real values and store passwords, tokens and keys in environment variables.
Risks
Before production changes, confirm backups, access boundaries, change windows and rollback paths.
Rollback plan
Keep original configuration and release versions; roll back config, images or database changes if metrics degrade.
Deliverables
Root-cause notes, key commands, remediation steps, verification results and follow-up recommendations.
Need help with a similar technical issue?
If your servers, Kubernetes, Docker, CI/CD, databases or monitoring systems have similar issues, submit logs and config files for remote diagnosis.