Mastering Middleware Reliability: Redis, RabbitMQ, Kafka in Production

Scenario

Your e-commerce platform experiences checkout failures. Monitoring shows Redis latency spikes to 500ms, RabbitMQ queues grow to 100k messages, and Kafka consumer lag exceeds 10k per partition. The cause? A combination of misconfigured memory limits, unoptimized consumer code, and a network partition.

Symptoms

• High p99 latency in Redis queries
• RabbitMQ queues warning of memory alarm
• Kafka consumer lag alerts
• User-facing errors in payment processing

Diagnosis

Redis

Run redis-cli info stats | tail -20 to check instantaneous_ops_per_sec and rejected_connections. Use redis-cli memory stats to see fragmentation ratio.

RabbitMQ

Use rabbitmqctl list_queues name messages messages_ready messages_unacknowledged consumers memory to identify bloated queues. Check rabbitmq-diagnostics status for alarms.

Kafka

Run kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --describe to see lag. Check kafka-log-dirs for disk usage.

Commands

# Redis: Check latency
redis-cli --intrinsic-latency 100
# RabbitMQ: Clear queue (use with caution)
rabbitmqctl purge_queue my_queue
# Kafka: Reset consumer offset (careful)
kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --topic my-topic --reset-offsets --to-earliest --execute

Note: Purge and reset can cause data loss; use only when message loss is acceptable.

Risk Controls

• Enable Redis lazy freeing: CONFIG SET lazyfree-lazy-eviction yes
• Limit RabbitMQ queue length with x-max-length
• Increase Kafka partitions or consumer parallelism
• Implement circuit breakers in application to prevent cascading failures

Rollback

• Redis: Revert config changes with CONFIG SET lazyfree-lazy-eviction no
• RabbitMQ: Reapply original queue limits
• Kafka: Restore previous consumer offset or restart with correct config
• General: Rollback application code changes

Verification

• Check Redis latency drops below 10ms
• RabbitMQ queue count decreases
• Kafka consumer lag reduces to zero
• Monitor application response times

When to Submit an OpsGlobal Ticket

If you lack bandwidth to implement controls, or if issues persist after basic troubleshooting, submit a ticket. OpsGlobal SREs can diagnose deep issues like bufferbloat, kernel tuning, and complex network partitions.