Nginx API Gateway Performance Tuning: From Symptoms to Solutions

Scenario

Imagine Nginx serving as the API gateway in a Kubernetes cluster. During peak hours, users report slow load times and occasional 502 Bad Gateway errors. The operations team is tasked with resolution.

Symptoms

• P99 latency spikes from 20ms to 500ms
• Nginx error logs show 'upstream timed out' and 'no live upstreams'
• Worker process CPU usage hits 90%
• Upstream health checks pass, but Nginx connection pool exhausts

Diagnosis

1. Check Nginx connection status: curl -s http://localhost/nginx_status | grep -E 'Active|Waiting'
2. View upstream statistics: curl -s http://localhost/upstream_status (requires ngx_http_upstream_module)
3. Examine kernel parameters: sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog
4. Review proxy_pass and buffers in config: nginx -T | grep -E 'proxy_buffer|proxy_busy|worker_connections'
5. Load test with h2load: h2load -c 100 -n 10000 -H 'Host: example.com' http://localhost/

Commands & Actions

Tune worker processes and connections

worker_processes auto;
worker_rlimit_nofile 65535;
events {
    worker_connections 10240;
    multi_accept on;
    use epoll;
}

Reload: nginx -s reload

Optimize proxy buffers

proxy_buffer_size 4k;
proxy_buffers 8 16k;
proxy_busy_buffers_size 32k;
proxy_temp_file_write_size 64k;

Increase upstream timeouts

proxy_connect_timeout 10s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;

Adjust kernel parameters

sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535
sysctl -w net.ipv4.tcp_tw_reuse=1

Risk Controls

• Test config changes in a non-production environment first
• Validate syntax with nginx -t
• Perform graceful reload (nginx -s reload) to avoid traffic interruption
• For kernel changes, backup original values and persist via /etc/sysctl.conf

Rollback

1. Restore Nginx config: cp /etc/nginx/nginx.conf.bak /etc/nginx/nginx.conf && nginx -s reload
2. Revert kernel parameters: sysctl -p /etc/sysctl.conf.bak
3. Monitor error rate recovery

Verification

• Check latency metrics: P99 should drop below 50ms
• Error log: no 502/504 entries
• Load test: h2load should show 100% success
• Observe for 24 hours to confirm stability

When to Submit an OpsGlobal Ticket

• The above steps don't resolve the issue
• Deep packet analysis (Wireshark) is needed
• Upstream services have performance bottlenecks requiring joint effort
• Long-term optimization or architecture changes are required

OpsGlobal SRE experts can provide remote diagnosis and tuning to keep your API gateway performing optimally.