Middleware Reliability: Taming Redis, RabbitMQ, and Kafka in Production

Scenario

A financial services client experiences severe latency spikes and data inconsistencies during peak hours. Their Kubernetes platform runs Redis as a cache and session store, RabbitMQ for transactional messaging, and Kafka for event streaming. The mixed workload begins to affect each other, causing cascading failure.

Symptoms

• Redis: Cache hit ratio drops, eviction count (evicted_keys) rises, latency exceeds 100ms. Clients see OOM command not allowed when used memory > maxmemory.
• RabbitMQ: Queue backlog grows into millions of messages. Consumers get disconnected with channel error and precondition failed.
• Kafka: Consumer group lag continuously increases, partition replication factor falls below configured value, and ISR (in-sync replicas) shrinks.

Diagnosis

1. Collect Metrics

Use Prometheus and Grafana to observe these key metrics:

• Redis: redis_memory_used, redis_evicted_keys, redis_commands_processed.
• RabbitMQ: rabbitmq_queue_messages, rabbitmq_connection_open, rabbitmq_channel_open.
• Kafka: kafka_consumergroup_lag, kafka_partition_underreplicated, kafka_server_replica_fetcher_manager_leader_count.

2. Inspect Logs

# Get Redis pod logs
kubectl logs -l app=redis -n middleware

# Get RabbitMQ and Kafka logs
kubectl logs -l app=rabbitmq -n middleware
kubectl logs -l app=kafka -n middleware

Look for anomalies like Redis WARNING about memory fragmentation, RabbitMQ connection_closed_abruptly, Kafka OffsetOutOfRangeException or ReplicaFetcherThread errors.

3. Live Commands

Redis

kubectl exec -it <redis-pod> -- redis-cli INFO memory
kubectl exec -it <redis-pod> -- redis-cli INFO stats | grep evicted
kubectl exec -it <redis-pod> -- redis-cli --bigkeys

Check if memory is near maxmemory and if there are large keys.

RabbitMQ

kubectl exec -it <rabbitmq-pod> -- rabbitmqctl list_queues name messages consumers
kubectl exec -it <rabbitmq-pod> -- rabbitmqctl list_connections name state

Identify which queues are backlogged and whether consumers are still connected.

Kafka

kubectl exec -it <kafka-pod> -- kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group <group>
kubectl exec -it <kafka-pod> -- kafka-topics --bootstrap-server localhost:9092 --describe --topic <topic>

Check consumer lag and partition replication status.

Risk Controls

• Before any operation, backup Redis via SAVE or BGSAVE; export RabbitMQ metadata; ensure Kafka replication factor > 1 and ISR healthy.
• Avoid FLUSHALL or deleting queues/topics directly in production.
• Always have a rollback plan and test connectivity before changing configuration.
• Use kubectl rollout restart instead of deleting pods manually to reduce disruption.

Remediation Commands (Examples)

Redis: Adjust Memory Policy

kubectl exec -it <redis-pod> -- redis-cli CONFIG SET maxmemory-policy allkeys-lru
kubectl exec -it <redis-pod> -- redis-cli CONFIG REWRITE

RabbitMQ: Temporarily Scale Consumers

kubectl scale deployment <consumer-deployment> --replicas=5 -n middleware

Kafka: Reset Consumer Group Offsets (Use with caution)

kubectl exec -it <kafka-pod> -- kafka-consumer-groups --bootstrap-server localhost:9092 --group <group> --topic <topic> --reset-offsets --to-earliest --execute

Rollback

• If Redis policy change degrades performance, revert with CONFIG SET maxmemory-policy volatile-lru.
• If scaling consumers fails to help, reduce replicas back.
• If Kafka offset reset causes disorder, roll back using --to-latest or the original offsets.

Verification

• Confirm Redis evicted_keys and used_memory return to normal.
• Check RabbitMQ queue lengths drop and consumer connections stabilize.
• Verify Kafka consumer lag approaches zero and ISR is healthy.
• Test latency with redis-benchmark or actual business requests.

When to Submit an OpsGlobal Ticket

If you are still unable to find the root cause after diagnosis, or if you need 24/7 monitoring and rapid response, submit an OpsGlobal ticket. Our SRE experts provide: - 24/7 full-stack monitoring and alerting - Proactive capacity planning and performance tuning - Disaster recovery and failover drills - Advanced troubleshooting and root cause analysis

We intervene within minutes to help your team reduce MTTR and ensure business continuity.