Scenario Your microservice cluster relies on Redis for caching, RabbitMQ for message queuing, and Kafka for event streaming. One day, service response times spike and order processing delays increase.
Symptoms
- Redis: redis-cli --latency shows >10ms; INFO reports a spike in total_connections_received; some keys miss their TTL expiry.
- RabbitMQ: Management UI shows deep queues; consumer acknowledgements time out; rabbitmqctl list_queues shows >1000 unacked messages.
- Kafka: Consumer lag grows; kafka-consumer-groups --describe shows increasing LAG; broker disk usage hits 85%.
Diagnosis
1. Redis – Check slow log: SLOWLOG GET 100. Many SET/GET operations >100µs indicate fragmentation or poor persistence settings. Run MEMORY DOCTOR. Verify maxmemory-policy is not something like noeviction which can cause eviction storms.
2. RabbitMQ – Check inter-node latency: rabbitmq-diagnostics check_port_connectivity. Inspect connections: rabbitmqctl list_connections. Too many short-lived connections? Adjust client pools. Verify mirrored queues: rabbitmqctl list_policies. For message paging, increase vm_memory_high_watermark or enable lazy queues.
3. Kafka – Check ISR: kafka-topics --describe --topic your-topic. Missing replicas? Check network/disk I/O. Dump log segments: kafka-run-class kafka.tools.DumpLogSegments --files. Monitor request handler avg idle ratio: kafka-run-class kafka.tools.JmxTool.
Commands
- Redis dynamic connection limit: CLIENT SETNAME monitor; CONFIG SET maxclients 5000 (caution: assess resources).
- RabbitMQ requeue unacked: rabbitmqctl eval 'rabbit_amqqueue:requeue(rabbit_misc:r(<<"queue-name">>, queue)).' or purge (high risk).
- Kafka increase partitions: kafka-topics --alter --topic your-topic --partitions 6 (ensure key distribution).
Risk Controls
- Always test parameter changes in non-production.
- Before changing Redis eviction policy, estimate max memory usage.
- For RabbitMQ, apply mirroring changes gradually to avoid split-brain.
- Kafka partition reassignment: use kafka-reassign-partitions.sh with --throttle to limit network impact.
Rollback
- Redis: Restore config: backup /etc/redis/redis.conf then CONFIG SET original values.
- RabbitMQ: Remove policy: rabbitmqctl clear_policy; revert queue properties by restarting node.
- Kafka: Move partitions back: generate old assignment with kafka-reassign-partitions --generate and execute.
Verification
- Redis: Run redis-benchmark -q -n 1000 compare baseline latency.
- RabbitMQ: Publish a test message and verify consumer ack within expected time.
- Kafka: Use kafka-producer-perf-test and kafka-consumer-perf-test to validate throughput.
When to Submit an OpsGlobal Ticket
- Issues persist after following above steps.
- Need cluster-wide tuning (e.g., Kafka num.io.threads or Redis cluster resizing).
- Data loss or persistence anomalies detected.
Use cases
Useful for teams handling NoSQL issues and needing a clear troubleshooting and delivery workflow.
Problem background
A deep-dive into common reliability issues of Redis, RabbitMQ, and Kafka in production Kubernetes environments, with step-by-step diagnosis, commands, risk controls, rollback, and verification procedures.
Troubleshooting steps
Confirm impact and recent changes, collect logs, configuration and metrics, then apply fixes from low to high risk.
Command examples
Replace sample resource names with real values and store passwords, tokens and keys in environment variables.
Risks
Before production changes, confirm backups, access boundaries, change windows and rollback paths.
Rollback plan
Keep original configuration and release versions; roll back config, images or database changes if metrics degrade.
Deliverables
Root-cause notes, key commands, remediation steps, verification results and follow-up recommendations.
Need help with a similar technical issue?
If your servers, Kubernetes, Docker, CI/CD, databases or monitoring systems have similar issues, submit logs and config files for remote diagnosis.