Middleware Reliability in Production: A Practical Guide to Redis, RabbitMQ, and Kafka Troubleshooting

Scenario

During a flash sale, an e-commerce platform's order service relies on Redis cache, RabbitMQ message queue, and Kafka event bus. Suddenly, order latency spikes, some requests time out, and monitoring alarms fire.

Symptoms

• Redis: redis-cli --latency shows >100ms (normal <1ms); INFO command displays used_memory_peak nearing maxmemory.
• RabbitMQ: Management UI shows queue accumulation; rabbitmqctl list_queues messages_ready reveals millions of unconsumed messages; node memory alarm.
• Kafka: kafka-consumer-groups --describe shows consumers lagging by tens of thousands; kafka-run-class.sh kafka.tools.JmxTool reports elevated request processing time.

Diagnosis

Redis

# Check slow queries
redis-cli SLOWLOG GET 10
# Check memory usage
redis-cli INFO memory | grep used_memory_human
# Find large keys
redis-cli --bigkeys
# Check connection count
redis-cli CLIENT LIST | wc -l

RabbitMQ

# Queue details
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers
# Node memory
rabbitmqctl status | grep memory
# Connections
rabbitmqctl list_connections state user
# Unacknowledged messages per channel
rabbitmqctl list_channels connection messages_unacknowledged

Kafka

# Consumer group lag
kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --describe
# Partition leader distribution
kafka-topics --describe --topic my-topic --bootstrap-server localhost:9092
# Log segment size
kafka-log-dirs --bootstrap-server localhost:9092 --describe --topic-list my-topic
# Request processing time (requires JMX enabled)

Risk Controls

Before any changes: - Redis: If memory is near limit, first set maxmemory and eviction policy allkeys-lru. Never run FLUSHALL in production without explicit approval and backup. - RabbitMQ: Verify consumer health before scaling. Deleting queues drops data; avoid unless necessary. - Kafka: Backup server.properties before config changes. Do not use kafka-delete-records on production topics without snapshot.

Rollback

Redis

If config changed: CONFIG SET maxmemory 0 to revert to default. If FLUSHALL was executed, restore from RDB/AOF—note potential data loss.

RabbitMQ

If a policy (e.g., TTL) was added: rabbitmqctl clear_policy <queue>. If parameter changed: use rabbitmqctl set_parameter to revert.

Kafka

If log.retention.hours was modified: change back to original value and rolling-restart brokers.

Verification

• Redis: Run redis-cli --latency to confirm latency drops; observe INFO memory usage declining.
• RabbitMQ: Check queue depth decreases; rabbitmqctl list_queues messages_ready shows fewer messages.
• Kafka: Consumer lag reaches 0; kafka-consumer-groups --describe displays LAG as 0.

When to Submit an OpsGlobal Ticket

• Changes requiring cross-team coordination (e.g., storage expansion).
• Root cause unclear after basic diagnosis.
• Issues persist after rollback, or need cluster rebuild.
• Security patches or architectural refactoring needed.

Conclusion

With systematic symptom recognition, safe diagnosis commands, and controlled rollback, most middleware issues can be resolved within 10 minutes. OpsGlobal provides 24/7 expert support to keep your critical path always available.