Deep Practical Guide to Redis, RabbitMQ, and Kafka Middleware Reliability

Scenario

Redis, RabbitMQ, and Kafka are critical components in modern microservices architectures, serving as caching, message queuing, and stream processing backbones. Reliability issues can cause service outages, data loss, or latency spikes.

Common Symptoms

• Redis: Response timeouts, frequent failovers, memory usage >80%, persistence failures
• RabbitMQ: Queue buildup, consumer disconnections, disk alarms, mirror queue sync issues
• Kafka: Shrinking ISR, frequent leader elections, increased produce/consume latency, broker crashes

Diagnosis Steps

Redis

1. Check slow logs: SLOWLOG GET 10
2. View memory usage: INFO memory
3. Check persistence: INFO persistence – confirm RDB/AOF success
4. Cluster status: CLUSTER INFO (if clustered)

RabbitMQ

1. List queues: rabbitmqctl list_queues name messages consumers
2. Check cluster health: rabbitmqctl cluster_status
3. Monitor disk/memory: rabbitmq-diagnostics check_port_connectivity
4. Inspect logs: /var/log/rabbitmq/

Kafka

1. Describe topic partitions: kafka-topics --describe --bootstrap-server localhost:9092 --topic my-topic
2. Check consumer group offsets: kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --describe
3. Broker logs: /var/log/kafka/server.log
4. JMX monitoring via Prometheus

Repair Commands

Redis

• OOM: CONFIG SET maxmemory 4gb and set eviction policy allkeys-lru
• Replication lag: SLAVEOF <master-ip> <master-port>
• Persistence failure: free disk space, adjust RDB save SAVE 900 1

RabbitMQ

• Queue backlog: add consumers or restart consumer app
• Node down: restart node, ensure cookie match
• Mirrored queue sync: rabbitmqctl sync_queue <queue-name>

Kafka

• Shrinking ISR: increase min.insync.replicas or fix corrupt replicas
• Unclean leader election: set unclean.leader.election.enable=false
• Latency: adjust fetch.max.bytes or add partitions

Risk Controls

Before making changes, always back up config and data. For Redis: BGSAVE then copy dump.rdb; RabbitMQ: rabbitmqadmin export foo.json; Kafka: backup config/server.properties and log directories.

Rollback Strategies

• Redis: restore dump.rdb and restart, or revert config
• RabbitMQ: restore config and restart node, or rebuild queues from backup
• Kafka: restore server.properties and restart broker, replay logs via kafka-replay-log-producer if needed

Verification

• Redis: PING, check INFO for connections and latency
• RabbitMQ: publish test message and consume, observe queue depth
• Kafka: produce and consume a message, check offset and latency

When to Submit an OpsGlobal Ticket

Contact OpsGlobal immediately if: - Middleware completely unavailable and cannot be recovered - Data loss exceeds acceptable threshold - Split-brain cannot be resolved - Need expert assistance for capacity planning or architecture optimization

Our SRE team is available 24/7 for remote support and automated remediation.