A Practical Guide to Middleware Reliability: Redis, RabbitMQ, and Kafka

Scenario

During a flash sale, order processing latency spikes. Redis cluster timeouts, RabbitMQ queues backlog millions, Kafka consumer lag skyrockets. Core services degrade.

Symptoms

• Redis: redis-cli --latency shows >100ms; INFO memory near limit. Key evictions start.
• RabbitMQ: rabbitmqctl list_queues shows queue depth exceeding 1M; nodes swapping.
• Kafka: kafka-consumer-groups --bootstrap-server ... --group ... --describe shows increasing lag; some consumers stop.

Diagnosis

Redis

# Slow queries
redis-cli SLOWLOG GET 50
# Large keys
redis-cli --bigkeys
# Persistence status
redis-cli INFO persistence

RabbitMQ

# Queue details
rabbitmqctl list_queues name messages consumers memory
# Memory usage
rabbitmq-diagnostics status | grep memory

Kafka

# Consumer lag
kafka-consumer-groups --bootstrap-server localhost:9092 --group mygroup --describe
# Partition distribution
kafka-topics --describe --topic mytopic --bootstrap-server localhost:9092

Risk Controls

• Redis: Enable slow log; avoid KEYS; set maxmemory-policy allkeys-lru; use Redis Cluster.
• RabbitMQ: Set x-message-ttl on queues; configure max-length; use lazy queues.
• Kafka: Increase partitions; tune replica.lag.time.max.ms; ensure min.insync.replicas≥2.

Rollback

• Redis: Revert maxmemory-policy to previous value and scale memory.
• RabbitMQ: Remove or adjust TTL; delete and recreate queue if needed.
• Kafka: Cannot reduce partitions; adjust consumer group settings or pause rebalance.

Verification

• Re-run diagnostic commands to confirm reduced lag/backlog.
• Monitor dashboards: Redis hit rate, RabbitMQ publish/consume rates, Kafka consumer offset progress.
• Run chaos experiments (network latency, node failures) to validate resilience.

When to Submit an OpsGlobal Ticket

• Root cause unclear, suspected middleware bug.
• Need cross-team coordination (application changes).
• Architectural advice required (e.g., switching from RabbitMQ to Kafka).