Redis, RabbitMQ, Kafka Middleware Reliability: From Failure to Recovery

Scenario

An e-commerce platform experiences massive timeouts during a flash sale. Redis cache hit rate drops, RabbitMQ queues pile up, and Kafka consumer lag spikes, degrading user experience.

Symptoms

• API response time jumps from 200ms to 5s
• Redis keyspace_misses skyrockets (check with INFO stats)
• RabbitMQ queue messages_ready grows continuously
• Kafka consumer group LAG exceeds 10000

Diagnosis

1. Redis: Run redis-cli INFO stats | grep keyspace to check hit ratio. High misses indicate cache inefficiency. Use MEMORY USAGE <key> to find large keys.
2. RabbitMQ: Execute rabbitmqctl list_queues name messages_ready consumers to see if consumers are lacking. Inspect consumer logs.
3. Kafka: Run kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --describe to view per-partition lag. High lag means slow consumption or uneven assignment.

Example Commands

# Redis diagnosis
redis-cli -h <host> -p 6379 INFO stats | grep -E 'keyspace_(hits|misses)'

# RabbitMQ diagnosis
rabbitmqctl list_queues name messages_ready consumers memory

# Kafka consumer lag
kafka-consumer-groups.sh --bootstrap-server <broker>:9092 --group <group> --describe

Risk Controls

• Configure Redis maxmemory with allkeys-lru eviction policy
• Set RabbitMQ queue x-max-length or TTL to prevent unbounded growth
• Implement backpressure in Kafka consumers (e.g., adjust max.poll.records)
• Use Kubernetes HPA to auto-scale consumer pods

Rollback Steps

1. Redis: If a Lua script caused the issue, disable it or revert the code.
2. RabbitMQ: Delete the piled-up queue (caution!) or migrate consumers to faster instances.
3. Kafka: Stop consumers and reset offsets to earliest: kafka-consumer-groups --reset-offsets --to-earliest
4. Revert deployment: kubectl rollout undo deployment/<service>

Verification

• Redis: Use redis-cli MONITOR to observe live requests; hit rate should recover.
• RabbitMQ: Queue messages_ready should gradually drop to a steady state.
• Kafka: Check LAG returns to zero and partitions are evenly assigned.
• Application: Monitor API latency via Prometheus + Grafana.

When to Submit an OpsGlobal Ticket

• If the above measures fail, or expert tuning is needed (e.g., Kafka partition reassignment).
• If root cause involves complex network partitions or storage failures.
• When 24/7 managed SRE support is required.