Nginx as an API Gateway: A Performance-Oriented Operations Playbook

Scenario

You run Nginx as an API gateway in a Kubernetes cluster, routing external traffic to multiple microservices. One day monitoring alerts fire: API response time jumps from an average of 80ms to 800ms, and some requests begin failing with 502 and 504 errors. Business teams report a severe degradation in user experience, but you don't know where the bottleneck lies.

Symptoms

• Increased API latency, especially for cross-service calls.
• Upstream timeouts, with upstream timed out or no live upstreams appearing in Nginx error logs.
• Nginx worker process CPU usage near 100%.
• Connections hitting the worker_connections limit, resulting in accept failed errors.
• Clients receiving 503 or 504 status codes.

Diagnosis

1. Check Nginx Status and Logs

First, exec into the Nginx pod and inspect error and access logs:

kubectl exec -it <nginx-pod> -n <namespace> -- /bin/bash

tail -f /var/log/nginx/error.log
tail -f /var/log/nginx/access.log

Common signals in the error log:

• worker_connections are not enough: insufficient connection capacity.
• upstream timed out (110: Connection timed out) while connecting to upstream: the upstream service is slow or unreachable.
• recv() failed (104: Connection reset by peer): the upstream service actively closed the connection, often due to timeout or a crash.

2. Use the Built-in Status Module

If the ngx_http_stub_status_module is enabled, you can inspect active connections:

curl http://localhost:80/nginx_status

Example output:

Active connections: 291
server accepts handled requests
 16630948 16630948 31070465
Reading: 6 Writing: 179 Waiting: 106

• High Reading and Writing indicate a busy request flow; Waiting are idle keep-alive connections.
• If accepts and handled diverge significantly, connections are being dropped.

3. Analyze Access Log Latency

Enable $request_time and $upstream_response_time fields in the log format to see downstream vs upstream timings:

tail -n 100 /var/log/nginx/access.log | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20

If $request_time is much larger than $upstream_response_time, the bottleneck is Nginx itself (e.g., extensions, rate limiting, SSL handshake). If they are close, the upstream service is the culprit.

4. Check Upstream Health

If you use an upstream block with health checks, verify that upstream servers are in the available list:

kubectl get endpoints -n <namespace>
kubectl get pods -n <namespace> -o wide

You can also test connectivity from the pod:

telnet <upstream-ip> <port>

Commands and Immediate Actions

1. Adjust Worker Processes and Connections

Edit the Nginx configuration to increase worker_processes and worker_connections:

events {
    worker_connections 4096;
}

error_log /var/log/nginx/error.log warn;
worker_rlimit_nofile 65535;

Then reload:

nginx -t && nginx -s reload

2. Optimize Upstream Timeouts

Set appropriate proxy timeouts in location or server blocks:

location /api/ {
    proxy_read_timeout 15s;
    proxy_connect_timeout 5s;
    proxy_send_timeout 15s;
}

Note: Too-short timeouts can abort requests before the upstream finishes; too-long timeouts can hold connections and leak resources.

3. Enable Nginx Caching

Cache infrequently changing responses:

proxy_cache_path /tmp/nginx_cache levels=1:2 keys_zone=my_cache:10m max_size=1g inactivity=60m;

location /static/ {
    proxy_cache my_cache;
    proxy_cache_valid 200 60m;
    proxy_cache_valid 404 1m;
}

4. Rate Limiting for Protection

Use limit_req to throttle requests and prevent sudden traffic surges from overwhelming backends:

limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;

location /api/ {
    limit_req zone=api_limit burst=20 nodelay;
}

5. Tune Kernel Parameters

If connection spikes are extreme, you may need to adjust system-level parameters, e.g., in the container start command:

sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.ip_local_port_range="1024 65535"

Risk Controls

• Always back up the original configuration before making changes.
• Use nginx -t to validate configuration syntax.
• For production, adopt canary releases: reload on one instance first, observe, then scale out.
• When changing kernel or global settings, confirm there are no side effects for other applications.
• Avoid setting worker_connections extremely high during peak hours to prevent memory exhaustion.

Rollback

If a change worsens the outage or introduces new errors, roll back immediately:

# Restore from backup
cp /etc/nginx/nginx.conf.bak /etc/nginx/nginx.conf
nginx -t && nginx -s reload

If the configuration is mounted via a ConfigMap, use kubectl rollout to undo:

kubectl rollout undo deployment/nginx-ingress -n <namespace>

Verification

• Check error.log for new timeouts or connection errors.
• Run load tests (e.g., ab, wrk, k6) to compare P95/P99 latency before and after the changes.
• Monitor Nginx connections, CPU, memory, and upstream response times.
• Observe whether the Waiting connections in nginx_status return to normal.

When to Submit an OpsGlobal Ticket

Contact OpsGlobal's remote SRE team immediately if:

• Performance does not improve after tuning and the root cause remains unknown.
• You need 24/7 performance monitoring and proactive alerting.
• The cluster is large and requires systematic Nginx and Kubernetes network stack optimization.
• Your organization lacks internal SRE expertise and the issue is affecting production availability.

OpsGlobal provides professional Nginx and API gateway operations support, including performance optimization, fault diagnosis, security hardening, and automation, helping you reduce MTTR and improve platform stability.