Deep Practical Guide to Middleware Reliability: Redis, RabbitMQ, Kafka

Scenario

During a flash sale, an e-commerce order processing system slows down. Alerts show Redis memory spiking, RabbitMQ queue depth soaring, and Kafka consumer lag growing.

Symptoms

• Redis: Client connections timeout; INFO stats reveals evicted_keys skyrocketing, memory usage near limit.
• RabbitMQ: Messages pile up; rabbitmqctl list_queues shows high unacknowledged count; disk write I/O wait increases.
• Kafka: Consumer group lag (kafka-consumer-groups --describe) keeps rising; consumer logs report "Rebalance in progress".

Diagnosis

1. Redis: Run redis-cli info memory to check fragmentation ratio and eviction policy. Use redis-cli --bigkeys to find large keys. If many temporary keys lack TTL, memory leak is likely.
2. RabbitMQ: Run rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers memory to pinpoint blocked queues. Check disk space via rabbitmqctl eval 'rabbit_disk_monitor:get_disk_free_list().'. If persistent messages accumulate, consumers may be slow.
3. Kafka: Monitor incoming rate with kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec. Check partition assignment with kafka-consumer-groups --describe --members --verbose. Frequent rebalances can starve consumers.

Useful Commands

# Redis
redis-cli info stats | grep evicted_keys
redis-cli -h <host> -p <port> --bigkeys

# RabbitMQ
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers memory --no-table-headers
rabbitmqctl set_policy ha-all ".*" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

# Kafka
kafka-consumer-groups --bootstrap-server localhost:9092 --group order-group --describe
kafka-topics --bootstrap-server localhost:9092 --describe --topic orders

Risk Controls

• Redis: Set maxmemory-policy allkeys-lru (non-blocking). Avoid large keys (>1MB). Use Sentinel or Cluster for HA.
• RabbitMQ: Configure mirrored queues for HA. Limit queue length (x-max-length or x-max-length-bytes). Set consumer prefetch count to 1 to prevent overload.
• Kafka: Tune session.timeout.ms and heartbeat interval for rebalance. Use sticky partitioner to reduce rebalances. Monitor consumer lag with alerts.

Rollback

• Redis: If AOF rewrite causes lag, disable it temporarily (CONFIG SET auto-aof-rewrite-percentage 0), rewrite manually later. If memory exhausted, scale up or use swap (not recommended for production).
• RabbitMQ: Pause producers (e.g., via circuit breaker) to let consumers drain. If disk full, migrate queues to healthy nodes (rabbitmqctl set_cluster_name ha).
• Kafka: Add more consumer instances or adjust fetch.min.bytes. If replicas out of sync, set min.insync.replicas to 1 temporarily (high risk, cautious).

Verification

• Redis: redis-cli ping returns PONG; redis-cli info commandstats shows normal latency.
• RabbitMQ: rabbitmqctl status shows running; rabbitmqctl list_queues queue depth normalizes.
• Kafka: kafka-consumer-groups --describe LAG near zero; no errors in consumer logs.

When to Submit an OpsGlobal Ticket

Escalate when internal team cannot recover despite following steps: - Persistent hardware faults (disk failure, network partition). - Need advanced data recovery (corrupted RDB repair). - Large-scale cluster performance tuning (e.g., too many partitions causing controller bottleneck). - Lack of specific middleware expertise (e.g., RabbitMQ cluster split-brain resolution).

OpsGlobal SRE team provides 24/7 remote support including real-time troubleshooting, architecture optimization, and disaster recovery planning.