Middleware Reliability in Practice: Redis, RabbitMQ, and Kafka

Scenario

In a modern microservices architecture, Redis, RabbitMQ, and Kafka form the core middleware backbone. One day, the business team reports intermittent failures: user requests slow down, some order processing is delayed, and occasionally messages are lost. Initial checks reveal that Redis cache hit ratio has dropped, RabbitMQ queues are backing up significantly, and Kafka consumer groups show noticeable lag.

Symptoms

• Redis: INFO stats shows evicted_keys continuously increasing, keyspace_hits vs keyspace_misses ratio is off, and SLOWLOG shows many slow queries.
• RabbitMQ: Queue messages and messages_unacknowledged are rising, consumer connection count drops, and memory/disk alarms are triggered.
• Kafka: Consumer group LAG keeps growing, partition ISR (In-Sync Replicas) shrinks, or you see OutOfRangeException errors.

Diagnosis Commands

Redis Diagnostics

# Check key statistics
redis-cli INFO stats
redis-cli INFO keyspace

# View recent slow queries (last 10)
redis-cli SLOWLOG GET 10

# Check persistence state to avoid blocking during RDB/AOF rewrite
redis-cli INFO persistence

Note: These commands are read-only and do not affect production. However, SLOWLOG RESET clears logs; avoid it if you need to preserve them.

RabbitMQ Diagnostics

# List queue details (message count, unacknowledged, consumers)
rabbitmqctl list_queues name messages messages_unacknowledged consumers

# Check connection states and channels
rabbitmqctl list_connections name state

# Check for memory or disk alarms
rabbitmqctl list_nodes name mem_used disk_free_alarm

These commands do not modify any state but require admin privileges. Run them during a maintenance window.

Kafka Diagnostics

# View all consumer group consumption progress and lag
kafka-consumer-groups.sh --bootstrap-server <broker>:9092 --describe --all-groups

# Inspect topic partition replicas and ISR status
kafka-topics.sh --bootstrap-server <broker>:9092 --describe --topic <topic>

# Check broker logs and disk usage (via JMX or logs)
# Example: Get number of under-replicated partitions
kafka-run-class.sh kafka.tools.JmxTool --object-name kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

kafka-run-class.sh launches a JVM, which may temporarily increase memory usage; assess before running. All commands are read-only.

Risk Controls

Before implementing fixes, risk must be contained:

1. Backup Configurations: Backup redis.conf, rabbitmq.conf, Kafka server.properties, and consumer group settings.
2. Enable Maintenance Mode: For Kafka, pause production writers if possible; for RabbitMQ, temporarily stop some consumers (but carefully to avoid piling up more messages).
3. Rate Limiting and Circuit Breakers: Add rate limiting at the entrance and open circuit breakers to prevent cascading failures.
4. Set Change Windows: All operations must be done during off-peak hours, and relevant teams must be notified.
5. Enhanced Monitoring: Enable detailed logging and metrics collection to detect anomalies immediately.

Rollback Plan

Redis Rollback

• If configuration changes (e.g., maxmemory-policy) caused issues, restore the backup and restart Redis.
• Safe rollback steps: bash # Copy original config cp /etc/redis/redis.conf.bak /etc/redis/redis.conf # Restart Redis (caution: this interrupts service) sudo systemctl restart redis
• Never use FLUSHALL or FLUSHDB as a rollback; they will lose data.

RabbitMQ Rollback

• If policy or parameter changes caused issues, delete or restore policies: bash rabbitmqctl clear_policy <vhost> <policy_name> # Or re-apply backup policy definitions
• Disabling automatic synchronization for high-availability queues might cause mirrored queue inconsistencies; ensure you have a clear plan.

Kafka Rollback

• If consumer group offsets were modified, reset them back: bash kafka-consumer-groups.sh --bootstrap-server <broker>:9092 --group <group> --topic <topic> --reset-offsets --to-earliest --execute

  This will re-consume historical messages, potentially causing duplicate processing; proceed with caution.

• If the issue was caused by a version upgrade, revert to the previous binary and restore the original configuration files.

Verification Steps

After fixes, verify:

• Redis: Check if evicted_keys has fallen, hit ratio recovered, SLOWLOG shortened, and INFO ping latency returns to normal.
• RabbitMQ: Queue messages and unacknowledged continue to decrease, consumer connections stabilize, memory/disk alarms clear.
• Kafka: Consumer group LAG gradually reaches zero, ISR is fully restored, and all partition replicas are in sync.

Also perform business-level verification: simulate payment, order, and other critical flows to confirm no message loss or duplication. Monitor SLO metrics (e.g., p99 latency) to ensure they return to baseline.

When to Submit an OpsGlobal Ticket

Submit a ticket immediately if:

1. You cannot identify the root cause after 30+ minutes of troubleshooting.
2. There is data loss or corruption requiring professional data repair services.
3. Underlying hardware or kernel issues are suspected, such as abnormal disk I/O or network jitter.
4. Complex failover across availability zones or clusters is needed.
5. Your team lacks deep operational experience with these middleware components and needs expert guidance.

OpsGlobal's SRE team can provide rapid diagnosis, performance tuning, and disaster recovery support without invading business code, helping you minimize downtime.