Nginx API Gateway Performance Tuning: From Diagnosis to Recovery

Scenario

An e-commerce API gateway (Nginx) experienced P99 latency spikes from 50ms to 3s during a flash sale, with some requests returning 502. Upstream was a Spring Boot app in Kubernetes.

Symptoms

• P99 latency >2s
• Increased upstream connection errors (upstream timed out)
• Nginx error log: "upstream prematurely closed connection"
• High Nginx worker CPU even though traffic below peak

Diagnosis

1. Check nginx status: curl http://localhost/nginx_status – look at Active connections and Waiting.
2. Stub_status: poor accepts/handled ratio indicates low connection reuse.
3. Load test with wrk: wrk -t4 -c200 -d30s http://gateway/api measure latency distribution.
4. Analyze access log: awk '{print $NF}' access.log | sort | uniq -c | sort -rn to see response time buckets.

Tuning Commands

1. Enable Upstream Keepalive

Edit upstream block:

upstream backend {
    server app-svc:8080;
    keepalive 32;
}

Pass keep-alive header in location:

proxy_set_header Connection "";
proxy_http_version 1.1;

2. Buffer Tuning

proxy_buffering on;
proxy_buffer_size 8k;
proxy_buffers 8 32k;
proxy_busy_buffers_size 64k;
client_body_buffer_size 128K;

3. Timeout Adjustments

proxy_connect_timeout 10s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;

4. Worker Connections

event {
    worker_connections 4096;
}

Risk Controls

• Backup config: cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak
• Validate syntax: nginx -t
• Graceful reload: nginx -s reload (non-disruptive)
• Test in staging first

Rollback

cp /etc/nginx/nginx.conf.bak /etc/nginx/nginx.conf
nginx -s reload

Verification

1. Re-run load test: P99 latency should drop below 200ms.
2. Check nginx_status: Active connections stable, Waiting queue low.
3. Error log: no more timeout entries.
4. Monitor alerts cleared.

When to Submit an OpsGlobal Ticket

• Tuning ineffective, errors persist.
• Need deep analysis of Nginx epoll event loop.
• Multi-cloud gateway architecture complexity.
• Custom Nginx module development required.