Redis, RabbitMQ, and Kafka Reliability in Kubernetes: A Field Guide

Scenario

Your application relies on Redis for caching, RabbitMQ for task queues, and Kafka for event streaming. Suddenly, the platform team reports a cascade of alerts: Redis memory usage is spiking, RabbitMQ queues are backing up, and Kafka consumer lag is growing. Users complain about slow checkouts and missing notifications. You need to act fast without making things worse.

Symptoms

• Redis: Latency spikes, OOM kills, eviction storms, READONLY errors during failovers.
• RabbitMQ: Queue depth grows, consumer connection drops, high CPU on broker nodes, channel.error in logs.
• Kafka: Consumer groups lag, ISR shrinks, NotLeaderForPartition exceptions, broker under-replicated partitions.

Diagnosis

Start with Kubernetes-level checks: resource usage, pod health, and network. Then dig into each middleware.

1. Kubernetes level: - kubectl get pods -n <namespace> - look for restarts or CrashLoopBackOff. - kubectl top pods -n <namespace> - check CPU/memory. - kubectl describe pod <pod> - inspect events, probes, limits.

2. Redis diagnosis: - redis-cli info memory - check used_memory, maxmemory, evicted_keys. - redis-cli info stats - look for rejected connections and errors. - redis-cli latency doctor - identify latency sources. - redis-cli --bigkeys - find large keys causing partition.

3. RabbitMQ diagnosis: - rabbitmqctl list_queues name messages consumers - observe queue depth. - rabbitmqctl list_channels - inspect channel states. - rabbitmq-diagnostics -q ping - check broker heartbeats. - rabbitmq-diagnostics runtime - check memory, file descriptors, and process details.

4. Kafka diagnosis: - kafka-consumer-groups.sh --bootstrap-server <broker> --describe --group <group> - check lag. - kafka-topics.sh --describe --topic <topic> - look at partition leaders and ISR. - kafka-broker-api-versions.sh --bootstrap-server <broker> - confirm connectivity. - From within the Kafka pod, inspect logs for errors like "NotLeaderForPartition".

Commands

Redis

# Check memory and keyspace hit ratio
redis-cli info memory | grep -E "used_memory_human|maxmemory_human|mem_fragmentation_ratio"

# Monitor evictions and OOM kills
redis-cli info stats | grep -E "evicted_keys|rejected_connections"

# Slow query log
redis-cli slowlog get 10

# RDB save status
redis-cli info persistence | grep rdb_last_save

RabbitMQ

# List queues with message count and consumers
rabbitmqctl list_queues name messages consumers

# Check connection and channel counts
rabbitmqctl list_connections state recv_oct discards

# Get memory and alarm status
rabbitmqctl status | grep -A 10 "Memory"

# Enable/disable the management API? Not needed.

Kafka

# List topics with partitions and replication
kafka-topics.sh --describe --bootstrap-server localhost:9092

# Check consumer group lag
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group

# Alter message retention if needed (emergency only)
kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name my-topic --add-config retention.ms=3600000

Risk Controls

• Set resource requests and limits for CPU/memory on all middleware pods. Overprovisioning can lead to noisy neighbors; underprovisioning causes OOM kills.
• Configure Kubernetes liveness and readiness probes correctly. For Redis, use redis-cli ping; for RabbitMQ, use the health check endpoint or rabbitmq-diagnostics ping; for Kafka, use kafka-broker-api-versions.sh or the built-in JMX exporter.
• Use PodDisruptionBudgets to prevent all brokers from being evicted during node maintenance.
• Enable persistence: Redis AOF (with appendfsync everysec), RabbitMQ queues with durable flags, Kafka with log.dirs on persistent volumes.
• Leverage Redis Cluster, RabbitMQ quorum queues, and Kafka rack awareness for high availability.
• Set alarms and dashboards for key metrics: memory usage, connection count, message lag, CPU utilization.

Rollback

• Redis: If a recent CONFIG SET caused instability, revert with redis-cli CONFIG REWRITE after restoring the previous config from backup. For Kubernetes, roll back the ConfigMap and do a rolling restart.
• RabbitMQ: If plugin changes or policy updates caused issues, disable the plugin or revert the policy. Use rabbitmqctl set_policy with the old values. For version issues, switch back to the previous image version and perform a rolling update.
• Kafka: If a topic configuration change (like retention or partition count) backfired, use kafka-configs.sh to revert the topic override. If the issue is from a broker version upgrade, rollback the image version while keeping the same storage.

Always test rollback in a staging environment first. Never roll back stateful clusters without suspending client traffic.

Verification

• Redis: Check redis-cli info stats for evicted_keys decreasing and hit ratio stabilizing. Run redis-cli ping from clients.
• RabbitMQ: Confirm queue depths are trending downward and consumer connections are stable. Use rabbitmqctl list_queues again.
• Kafka: Verify consumer lag drops to normal ranges and kafka-topics.sh --describe shows all ISR in-sync.

Also monitor application-level error rates and latency percentiles to ensure the user experience is restored.

When to Submit an OpsGlobal Ticket

Submit a ticket if: - The root cause is unclear or the same failure repeats after multiple interventions. - You need to recover a corrupted Redis RDB/AOF file or perform a manual Kafka partition reassignment. - The middleware version is beyond end-of-life and requires an upgrade strategy. - You need help designing a comprehensive multi-region deployment or tuning low-level kernel parameters.