Optimizing Nginx as an API Gateway: A Performance Operations Guide

Scenario

Your Nginx-based API gateway is experiencing high latency and increased error rates during peak traffic. Upstream services are healthy, but the gateway becomes a bottleneck.

Symptoms

• Client timeouts increase (502/504 errors)
• Response latency spikes from 50ms to 2s
• Nginx error logs show 'upstream timed out' or 'connection refused'
• System metrics show abnormal CPU or memory usage

Diagnosis

1. Check Nginx worker processes: ps aux | grep nginx. Ensure worker count equals CPU cores.
2. Examine error logs: tail -f /var/log/nginx/error.log. Look for timeouts or connection errors.
3. Analyze access logs: tail -100 /var/log/nginx/access.log | awk '{print $NF}' | sort | uniq -c | sort -rn. Check status code distribution.
4. Validate config syntax: nginx -t.
5. Monitor system resources: top, free -m, iostat -x 1.

Commands (Core Actions)

# Optimize upstream keepalive
http {
    upstream backend {
        server backend1.example.com:8080;
        keepalive 32;          # Persistent connections
    }
}

# Tune worker processes and connections
worker_processes auto;
events {
    worker_connections 1024;
    multi_accept on;
}

# Increase timeouts
proxy_connect_timeout 5s;
proxy_send_timeout 10s;
proxy_read_timeout 10s;

Risk Controls

• Test configuration changes outside production
• Validate syntax with nginx -t
• Apply changes incrementally (e.g., modify one upstream server first)
• Monitor error rates and prepare rollback

Rollback

# Restore original config and reload
cp /etc/nginx/nginx.conf.backup /etc/nginx/nginx.conf
nginx -s reload

Verification

• Check latency: curl -w "%{time_total}" -o /dev/null -s https://api.example.com/health
• Confirm error rate drop: tail -100 /var/log/nginx/access.log | grep -c ' 50[0-9] '
• Resource usage returns to normal

When to Submit an OpsGlobal Ticket

• Performance not restored after following above steps
• Need kernel parameter tuning (e.g., somaxconn, tcp_tw_reuse)
• DDoS mitigation or WAF configuration required
• Architecture changes needed (e.g., adding cache layer)