Keep Your Middleware Mesh Alive: A Field Guide to Redis, RabbitMQ, and Kafka Reliability

Scenario

A typical e-commerce platform relies on Redis for caching, RabbitMQ for task queues, and Kafka for event streaming. One day, application latency spikes, order processing slows, and page loads time out. The infrastructure team notices Redis memory usage is near its limit, RabbitMQ queue depth is growing continuously, and Kafka consumers are hundreds of thousands of messages behind. Users start complaining, and business is impacted.

Symptoms

• Redis: INFO memory shows used_memory approaching maxmemory, INFO stats shows rising evicted_keys and expired_keys, and slow log captures many KEYS or SMEMBERS commands.
• RabbitMQ: The management UI or rabbitmqctl list_queues shows a sharp increase in messages in queues, consumer count drops, and rabbitmq-diagnostics reports memory and disk alarms.
• Kafka: kafka-consumer-groups shows consumer group lag continuously growing, broker CPU and disk I/O usage rises, and partition replicas experience ISR shrinkage.

Diagnosis

1. Identify hotspots: Use APM and monitoring tools (e.g., Prometheus/Grafana) to inspect request volume, error rates, and latency. Check for burst traffic or code defects.
2. Redis diagnosis: Run redis-cli --latency and redis-cli --stat, and retrieve slow commands with SLOWLOG GET. Analyze for large keys, hotkeys, and whether the eviction policy is appropriate.
3. RabbitMQ diagnosis: Inspect connection count, channel count, and unacknowledged messages. Use rabbitmqctl list_consumers to confirm consumers are online, and rabbitmqctl list_queues name messages consumers to see queue status. Common causes include insufficient consumer processing power or infinite loops.
4. Kafka diagnosis: Use kafka-consumer-groups --describe --group <group> to check per-partition lag. Examine consumer config like max.poll.records and session.timeout.ms, along with broker disk and network metrics.

Commands

Redis

# Check memory and keyspace statistics
redis-cli INFO memory
redis-cli INFO keyspace

# View slow queries
redis-cli SLOWLOG GET 10

# Check large keys (using third-party tools like redis-rdb-tools)
redis-cli --bigkeys

# If using cluster, view cluster status
redis-cli CLUSTER INFO

RabbitMQ

# View queue status
rabbitmqctl list_queues name messages consumers

# View connections and channels
rabbitmqctl list_connections
rabbitmqctl list_channels

# Check memory and disk alarms
rabbitmq-diagnostics check_running
rabbitmq-diagnostics memory_breakdown

# Purge a queue (dangerous, use with caution)
rabbitmqctl purge_queue <queue_name>

Kafka

# View consumer group lag
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group my-consumer-group

# View topic details
kafka-topics --bootstrap-server localhost:9092 --describe --topic orders

# View broker logs for exceptions
journalctl -u kafka -n 200

Risk Controls

• Redis: Set a sensible maxmemory-policy (e.g., allkeys-lru or volatile-lru), avoid KEYS commands, and use SCAN instead. Ensure spare memory when persistence (RDB/AOF) is enabled.
• RabbitMQ: Configure queue length limits (x-max-length) and message TTL, and use dead-letter queues for unconsumable messages. Enable lazy queues for high-throughput queues to reduce memory pressure.
• Kafka: Tune replica.lag.time.max.ms and min.insync.replicas to avoid ISR shrinkage. Optimize consumer-side fetch.max.bytes and max.poll.records to prevent processing timeouts.
• General: Set up monitoring alerts for each middleware, such as Redis memory usage, RabbitMQ queue depth, and Kafka consumer lag. Conduct load tests and back up configurations before changes.

Rollback

If a configuration change worsens the situation, roll back immediately.

• Redis: Use CONFIG GET and CONFIG SET for temporary changes, but update the configuration file manually for persistence. If AOF is enabled, rolling back config does not require restart. If you accidentally use FLUSHALL, stop Redis immediately and restore the last backup (RDB).
• RabbitMQ: If a policy causes issues, remove it with rabbitmqctl clear_policy <name>. If a queue is purged, data cannot be recovered; you must rebuild from producers or backups.
• Kafka: If broker config was modified, restart Kafka instances (do rolling restarts). For consumer groups, reset offsets with --to-earliest or --to-latest, but be aware this may duplicate or lose messages.
• Safety note: Ensure backups exist and notify team members before any destructive command.

Verification

After applying fixes, verify that the middleware has recovered.

• Redis: INFO stats shows evicted_keys stabilizing, and latency returns to normal. Confirm latency <1ms with redis-cli --latency.
• RabbitMQ: Queue depth decreases, consumer count recovers, and rabbitmq-diagnostics status shows all nodes running.
• Kafka: Consumer group lag drops to zero or near zero, and ISR is stable. Use kafka-producer-perf-test and kafka-consumer-perf-test to verify throughput.
• Business level: Core API response times drop, and order processing stops backing up.

When to Submit an OpsGlobal Ticket

If your team faces any of the following, it is time to call OpsGlobal:

• The same middleware issue recurs frequently without a clear root cause.
• Deep tuning or architectural changes are required (e.g., sharding, cluster migration).
• Production data loss or inconsistency occurs and needs rapid recovery.
• Your team lacks 24x7 support while issues happen during critical hours.

OpsGlobal's SRE experts can quickly pinpoint bottlenecks, provide production-grade hardening solutions, and respond according to SLA. Don't wait until failure spreads; submit a ticket early to ensure business continuity.