Ensuring Reliability of Redis, RabbitMQ, and Kafka in Production: A Practical Guide

Scenario

Middleware such as Redis, RabbitMQ, and Kafka are critical to modern distributed systems. Common failure scenarios include: - Redis: Out-of-memory (OOM) errors, replication lag, cluster failover issues. - RabbitMQ: Queue buildup, disk full, memory alarms. - Kafka: Broker failure, under-replicated partitions, ISR shrinkage.

Symptoms

• Latency spikes, increased error rates, service degradation.
• Redis client connection failures or timeouts.
• RabbitMQ publisher confirms failing, consumers stopping.
• Kafka producer errors, increased consumer lag.

Diagnosis

Use built-in tools: - Redis: redis-cli ping, INFO memory, SLOWLOG GET - RabbitMQ: rabbitmq-diagnostics check_alarms, rabbitmqctl list_queues - Kafka: kafka-broker-api-versions.sh, kafka-topics --describe

Check logs: - Redis: /var/log/redis/redis-server.log - RabbitMQ: /var/log/rabbitmq/ - Kafka: logs/server.log

Commands Examples

Redis memory usage:

redis-cli -h host -p 6379 INFO memory | grep used_memory_human

RabbitMQ queue backlog:

rabbitmqctl list_queues name messages_ready messages_unacknowledged

Kafka partition status:

kafka-topics --bootstrap-server localhost:9092 --describe --topic test

Risk Controls

• Set resource limits (memory, disk).
• Configure alerts (Prometheus + Alertmanager).
• Use replicas and redundancy.
• Implement graceful shutdowns and health checks.

Rollback

• Redis: Promote replica to master if node fails; use Sentinel for automatic failover.
• RabbitMQ: Reset memory high-watermark, delete blocked queues, or rebuild mirrored queues.
• Kafka: Restart failed Broker, reassign partitions, or manually adjust ISR.

Verification

• Redis: SET key value and GET to verify.
• RabbitMQ: Publish and consume a message.
• Kafka: Produce and consume a message, check consumer lag.

When to Submit an OpsGlobal Ticket

When issues exceed team capability or require urgent support: - Critical data loss not recoverable by yourself. - Cluster brain-split or data inconsistency. - Need architectural review or performance tuning.