Overview
In production, failures are inevitable. SRE runbooks are crucial for maintaining service reliability. This post uses a common Kubernetes scenario—Pod crash loops—to demonstrate the complete troubleshooting workflow from symptom detection to automated response.
Scenario
A microservice application running in Kubernetes experiences Pods in CrashLoopBackOff state.
Symptoms
- Pod status:
CrashLoopBackOff - Logs:
OutOfMemoryErroror process killed by OOM Killer - Monitoring: Memory usage consistently above threshold
Diagnosis Steps
- Check Pod status
bash kubectl get pods -n <namespace> | grep -v Running - Examine Pod logs
bash kubectl logs <pod-name> -n <namespace> --previous - Describe Pod events
bash kubectl describe pod <pod-name> -n <namespace> - Check node resources
bash kubectl top nodes kubectl top pods -n <namespace> - Analyze memory usage
bash # Exec into container (if still running) kubectl exec -it <pod-name> -n <namespace> -- bash # View process memory ps aux --sort=-%mem | head -n 10
Risk Controls
- Perform actions during off-peak hours to minimize user impact.
- Adjust resource quotas first:
kubectl edit deployment <deployment>to increase memory limits. - Never delete stateful pods without prior evaluation.
- Use
kubectl cordon <node>to isolate faulty nodes.
Rollback Plan
- Confirm issue is due to a recent change.
- Rollback deployment:
bash kubectl rollout undo deployment/<deployment> -n <namespace> - If rollback fails, revert to a known good revision:
bash kubectl rollout history deployment/<deployment> -n <namespace> kubectl rollout undo deployment/<deployment> --to-revision=<revision>
Verification
- Confirm Pod status is Running and stable.
- Monitor memory usage curve return to baseline.
- Execute normal traffic tests.
When to Submit an OpsGlobal Ticket
- Issue persists after multiple rollback attempts.
- Problem involves low-level system configuration (kernel parameters, filesystem).
- Need performance tuning or architecture review.
- Rapid response required outside business hours.
Structured runbooks enable SRE teams to reduce MTTR. For complex issues, timely expert support is key.
Use cases
Useful for teams handling DevOps issues and needing a clear troubleshooting and delivery workflow.
Problem background
This article provides a deep dive into practical Linux SRE runbooks for production troubleshooting, covering scenario analysis, symptom identification, diagnosis commands, risk controls, rollback strategies, verification steps, and when to escalate to OpsGlobal experts. Focused on Kubernetes environments.
Troubleshooting steps
Confirm impact and recent changes, collect logs, configuration and metrics, then apply fixes from low to high risk.
Command examples
Replace sample resource names with real values and store passwords, tokens and keys in environment variables.
Risks
Before production changes, confirm backups, access boundaries, change windows and rollback paths.
Rollback plan
Keep original configuration and release versions; roll back config, images or database changes if metrics degrade.
Deliverables
Root-cause notes, key commands, remediation steps, verification results and follow-up recommendations.
Need help with a similar technical issue?
If your servers, Kubernetes, Docker, CI/CD, databases or monitoring systems have similar issues, submit logs and config files for remote diagnosis.