Scenario

You manage an e-commerce platform relying on Redis for caching, RabbitMQ for order messages, and Kafka for user behavior logs. One day, orders are delayed, pages load slowly, and logs pile up, affecting user experience.

Symptoms

• Redis: Cache hit rate drops sharply, response time spikes from 1ms to 100ms, INFO stats shows evicted_keys surging.
• RabbitMQ: Queue depth exceeds threshold, rabbitmqctl list_queues shows thousands of messages_ready, consumer connections drop.
• Kafka: Consumer lag increases, kafka-consumer-groups --describe shows LAG growing, high disk I/O.

Diagnosis

Redis

1. Check memory: redis-cli INFO memory for used_memory and maxmemory.
2. View slow queries: SLOWLOG GET 100 to analyze costly commands.
3. Verify connections: CLIENT LIST for exceeding maxclients.

RabbitMQ

1. Queues and connections: rabbitmqctl list_queues name messages_ready messages_unacknowledged.
2. Node health: rabbitmqctl status for running and alarms.
3. Memory pressure: rabbitmqctl report for memory usage.

Kafka

1. Consumer lag: kafka-consumer-groups --bootstrap-server localhost:9092 --group <group> --describe.
2. Partition replica status: kafka-topics --describe --topic <topic> --bootstrap-server localhost:9092.
3. Disk usage: df -h monitoring log.dirs partitions.

Commands

Redis

# Set maxmemory policy (recommend allkeys-lru for production)
redis-cli CONFIG SET maxmemory-policy allkeys-lru
# Limit connections
redis-cli CONFIG SET maxclients 5000
# Enable keyspace events
redis-cli CONFIG SET notify-keyspace-events KEA

RabbitMQ

# Increase queue max length to avoid memory overflow
rabbitmqctl set_policy DLQ ".*" '{"max-length":1000000}' --apply-to queues
# Enable lazy queues to reduce memory
rabbitmqctl set_policy LazyQueue ".*" '{"queue-mode":"lazy"}' --apply-to queues
# List consumers
rabbitmqctl list_consumers

Kafka

# Adjust log retention time to free disk
kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name <topic> --alter --add-config retention.ms=604800000
# Increase partitions for higher parallelism
kafka-topics --alter --topic <topic> --partitions 12 --bootstrap-server localhost:9092

Risk Controls

• Backup configurations before changes: cp /etc/redis/redis.conf /etc/redis/redis.conf.bak.
• RabbitMQ policy changes affect entire cluster; test in non-production first.
• Kafka partitions cannot be decreased; ensure downstream consumers support rebalancing before increasing.

Rollback

Redis

redis-cli CONFIG SET maxmemory-policy volatile-lru  # revert to default

RabbitMQ

rabbitmqctl clear_policy DLQ  # remove policy

Kafka

kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name <topic> --alter --delete-config retention.ms

Verification

• Redis: Run redis-benchmark -q -n 1000 to compare latency; check INFO stats for reduced evicted_keys.
• RabbitMQ: Confirm queue depth drops and consumers reconnect: rabbitmqctl list_queues name messages_ready.
• Kafka: Monitor consumer LAG reaching zero; disk usage normalizing.

When to Submit a Ticket

• Problem persists after applying changes.
• OOM or disk full causing service unavailability.
• Need data migration or cluster reconfiguration (e.g., Kafka controller reelection).
• Suspect hardware or network issues.

OpsGlobal team provides 24/7 expert support, capable of remote diagnosis and complex recovery operations.