Tuning Nginx and API Gateway Middleware: A Practical Performance Playbook

Scenario

A typical microservices deployment uses Nginx as a reverse proxy and TLS terminator, fronting an integrated API gateway middleware (e.g., Kong, based on OpenResty/Nginx) that handles authentication, rate limiting, and request routing. Within one month of a traffic spike, the operations team observes that average response time has grown from 120 ms to 800 ms, and p99 latency exceeds 2 seconds. Some users start receiving 504 Gateway Timeout errors. The gateway's CPU utilization periodically hits 95%, and upstream application logs show increased connection resets. The system is not down, but user experience is degrading rapidly.

Symptoms

• High latency (p99 > 2s) and frequent timeouts
• CPU saturation of Nginx/gateway nodes
• Increased 5xx errors, particularly 502/504
• Upstream connections being dropped
• Error logs filled with "worker_connections are not enough" and "upstream timed out"

Diagnosis

Start with the natural sequence: check system metrics, Nginx status, logs, and then drill down with load testing.

1. Verify system resources: Use top, vmstat, and mpstat to see CPU, memory, and I/O. High sy time indicates kernel/system calls, often from excessive context switching.
2. Check Nginx worker connections: The ngx_http_stub_status_module exposes basic counters. Enable it in a location block and curl it: location /nginx_status { stub_status; allow 127.0.0.1; deny all; } Then curl http://127.0.0.1/nginx_status. Look at "Active connections" and "Waiting" vs "Writing".
3. Inspect Nginx error log: tail -n 100 /var/log/nginx/error.log. Common messages: - "upstream timed out (110: Connection timed out)" — check proxy_read_timeout. - "no live upstreams" — check upstream server health checks. - "worker_connections are not enough" — increase worker_connections or reduce keepalive connections.
4. Check API gateway middleware logs (e.g., Kong's error.log). For Kong, you can also query /status endpoint for cluster status.
5. Analyze Nginx access log format with response times. If you use the $request_time variable, you can see slow requests directly. Example: tail -f /var/log/nginx/access.log | awk '{print $NF}' if time is the last field.
6. Perform a controlled load test using wrk or ab against a test endpoint to isolate whether the bottleneck is in Nginx or the gateway middleware: wrk -t4 -c100 -d30s http://your-gateway/health Compare with a direct hit to an upstream application to see the difference.

Commands and Immediate Actions

Here are practical commands to gather more data and begin remediation, with safety notes.

• Validate configuration before any reload: nginx -t -c /etc/nginx/nginx.conf (must be run as root or with appropriate permissions).
• Reload Nginx gracefully to apply changes: nginx -s reload (works if you have permission; in systemd, systemctl reload nginx).
• Increase worker_connections temporarily in the events block if you see "worker_connections are not enough". This requires a reload, and it's safe as long as file descriptors are available (ulimit -n).
• Tune keepalive to upstream: In the upstream block of Nginx, ensure keepalive 32; and in the location block, set proxy_http_version 1.1; and proxy_set_header Connection ""; to reuse upstream connections. This can drastically reduce memory and CPU.
• For upstream timeouts, adjust proxy_connect_timeout, proxy_send_timeout, proxy_read_timeout to reasonable values (e.g., 5s, 10s, 60s). Never set them too high as it ties up worker connections.
• For the API gateway middleware (assuming Kong), use its admin API to check performance plugins. If rate limiting is enabled, query counters to ensure limits are not causing throttling.
• Use strace -p <nginx_pid> -f -e trace=network for a few seconds to see if system calls are blocking (only if you are comfortable; it can cause overhead).
• Check open file descriptors: lsof -p $(pidof nginx) | wc -l to see if worker processes hit the limit.

Risk controls: Any change should be applied to one node first (if you have a load balancer in front). Maintain a backup of the current config: cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak. Use a canary by slowly shifting traffic using weight in the upstream block. If you modify plugins in the API gateway, do it through the admin API and test on a staging environment.

Rollback

If an adjustment leads to worse behavior or errors, revert immediately.

• For Nginx config changes, replace the config with the .bak file and run nginx -t then nginx -s reload. This is a graceful rollback that drops no active connections.
• For API gateway plugin changes, you can disable the plugin via the admin API or revert to a previous declarative config (if using declarative mode).
• Always have a rollback plan for certificate changes (e.g., TLS sessions). Reloading with a bad config will be caught by nginx -t.

Verification

After applying changes, verify that the problem is solved.

• Monitor key metrics: nginx_status values (Active connections should be stable), response times in access log ($request_time), and error rates.
• Run a load test again with the same parameters as before. Compare p99 and error rates.
• Use dashboard tools like Grafana and Prometheus if available to visualize changes.
• Check curl -v https://your-api/endpoint to ensure TLS handshake is smooth and no extra latency.

When to Submit an OpsGlobal Ticket

If you still see high CPU usage after tuning worker_connections, keepalive, and timeouts, or if the gateway middleware itself becomes the bottleneck (e.g., Kong's database connection pool, Lua JIT, or custom plugins), it's time to bring in the experts. OpsGlobal can do a deep dive with tracing tools like eBPF/perf, analyze Nginx and gateway internals, and implement advanced performance strategies like dynamic upstream management, config optimization, and kernel tuning. If the issue is affecting production availability, submit a ticket immediately to minimize downtime.