Middleware Reliability: A Practical Guide to Diagnosing and Recovering Redis, RabbitMQ, and Kafka

Scenario

A microservices platform relies on Redis for caching, RabbitMQ for async messaging, and Kafka for event streaming. Suddenly, the platform experiences latency spikes, message processing failures, and data inconsistencies.

Redis Reliability

Symptoms

• Cache hit rate drops below 80%
• Command timeouts (e.g., GET latency > 10ms)
• Memory usage exceeds 80% (high used_memory_rss in INFO memory)

Diagnosis

• Check slow queries: SLOWLOG GET 10
• Check memory fragmentation: INFO memory mem_fragmentation_ratio (>1.5 indicates fragmentation)
• Check persistence issues: INFO persistence shows long rdb_last_bgsave_time_sec

Commands & Risk Controls

# View slow queries
redis-cli -h localhost -p 6379 SLOWLOG GET 10
# Analyze memory fragmentation
redis-cli MEMORY PURGE
# Note: MEMORY PURGE can block; run during low traffic.

Rollback

• If caused by config changes, revert config and restart: redis-cli CONFIG SET save "" to revert persistence.
• Use REPLICAOF NO ONE to stop replication, then resync.

Verification

• Re-check slow queries: SLOWLOG RESET then benchmark.
• Verify memory: redis-cli INFO memory | grep used_memory should be stable.

When to Submit an OpsGlobal Ticket

• When slow queries or memory issues persist despite internal optimization.
• When cluster resharding or upgrade is needed.

RabbitMQ Reliability

Symptoms

• Message backlog: queue messages grow continuously (rabbitmqctl list_queues shows increasing message count).
• Connection drops: rabbitmqctl list_connections shows many unexpected disconnections.
• High memory alarm: rabbitmqctl status shows vm_memory_high_watermark_paging_ratio > 0.5.

Diagnosis

• Check queue details: rabbitmqctl list_queues name messages consumers memory
• Check logs: /var/log/rabbitmq/rabbit@host.log for "alarm"
• Check network: netstat -an | grep 5672

Commands & Risk Controls

# View all queues
rabbitmqctl list_queues --silent name messages consumers memory
# Force purge a queue (destructive)
rabbitmqctl purge_queue <queue_name>
# Note: Purge loses all messages; use only if certain it's safe.

Rollback

• If memory threshold was changed, revert: rabbitmqctl set_vm_memory_high_watermark 0.4
• If queue mirroring was added, remove: rabbitmqctl set_policy ha-all "" ".*" '{"ha-mode":"exactly","ha-params":1}'

Verification

• Confirm message consumption rate recovers: use rabbitmqctl list_queues and see message count decreasing.
• Test publish and consume with sample code.

When to Submit an OpsGlobal Ticket

• When message backlog impacts business and internal tuning fails.
• When migration or upgrade of RabbitMQ cluster is required.

Kafka Reliability

Symptoms

• Consumer lag increases: kafka-consumer-groups --bootstrap-server localhost:9092 --group <group> --describe shows LAG growing.
• Replicas out of sync: kafka-topics --describe --topic <topic> shows ISR missing or smaller than replication factor.
• High disk usage: Broker logs report "Disk usage exceeds threshold".

Diagnosis

• View consumer group lag: kafka-consumer-groups --bootstrap-server localhost:9092 --all-groups --describe
• Check replica sync: kafka-topics --describe --topic <topic>
• Check disk: df -h and Kafka logs for disk warnings.

Commands & Risk Controls

# View specific consumer group
kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --describe
# Change retention to purge old data (risky)
kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name <topic> --alter --add-config retention.ms=604800000
# Note: Shorter retention loses data; confirm necessity.

Rollback

• If retention was changed, revert: retention.ms=original_value
• If partition reassignment was done, stop and revert to original.

Verification

• Check lag decreases: re-run --describe.
• Check ISR restored: kafka-topics --describe shows all replicas in-sync.

When to Submit an OpsGlobal Ticket

• When consumer lag persists despite consumer optimization.
• When cluster expansion or disk I/O bottleneck resolution is needed.