Nginx as API Gateway: A Deep Dive into Performance Troubleshooting

The Scenario

At OpsGlobal, we often see clients running Nginx as an API gateway in front of their microservices. The typical setup involves Nginx terminating TLS, routing requests to upstream services, and handling rate limiting or caching. Recently, one of our clients reported that their API latency had spiked from 50 ms to over 2 seconds, and they were seeing intermittent 504 Gateway Timeouts during peak business hours.

Symptoms

• Increased response latency (p95 latency tripled)
• 504 errors from Nginx when upstreams fail to respond within default timeout
• Nginx worker connections hitting worker_connections limit
• Upstream service CPU usage is high but not saturated
• No obvious code changes or deployment events

Diagnosis

We started by checking the basics:

1. Check Nginx error logs tail -f /var/log/nginx/error.log Look for upstream timed out or no live upstreams.

2. Check access logs for slow requests We enabled $request_time and $upstream_response_time in the log format. Commands to configure and tail logs.

3. Validate Nginx configuration: nginx -t and nginx -T to dump full configuration.

4. Test connectivity and timing to upstream: Use curl -w to measure DNS, connect, TLS, TTFB, and total time.

5. Check connection pool stats: ss -s to view socket statistics, also netstat -an | grep ESTABLISHED | wc -l.

6. Load test with wrk or ab to reproduce under controlled load.

We discovered that the Nginx config had proxy_connect_timeout and proxy_read_timeout set to the defaults (60s), but the real issue was that upstream keepalive connections were not configured properly. The upstream block had no keepalive directive, causing a new TCP connection and TLS handshake for every request. Under high concurrency, this added massive overhead.

Commands Used

Here are the key commands we used during diagnosis and fix:

Check error log

tail -f /var/log/nginx/error.log

Check access log with timing

Access log format definition:

log_format timed '$remote_addr - $remote_user [$time_local] '
                 '"$request" $status $body_bytes_sent '
                 '"$http_referer" "$http_user_agent" '
                 '$request_time $upstream_response_time';

Then tail it:

tail -f /var/log/nginx/access.log | awk '$10 > 2 {print}'

(assuming field 10 is request_time)

Validate config

nginx -t
nginx -T

Curl timing

curl -w "DNS: %{time_namelookup}s\nConnect: %{time_connect}s\nTLS: %{time_appconnect}s\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n" https://api.example.com/health

Socket statistics

ss -s
ss -tn state established '( dport = :443 or sport = :443 )' | wc -l

Load test with wrk

wrk -t8 -c200 -d30s https://api.example.com/health

Fix Implementation with Risk Controls

We made the following changes:

1. Enable upstream keepalive:

upstream backend {
    server api1:8080 weight=3;
    server api2:8080 weight=2;
    keepalive 32;
}

1. Adjust Nginx HTTP settings:

http {
    upstream backend {
        server api1:8080;
        keepalive 32;
    }

    server {
        listen 443 ssl;

        location /api/ {
            proxy_pass http://backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_connect_timeout 5s;
            proxy_read_timeout 10s;
            proxy_send_timeout 10s;
            proxy_buffering off; # only if needed
        }
    }
}

• proxy_http_version 1.1 is required for keepalive to upstream.
• Empty Connection header.
• Reduce timeouts to prevent prolonged hangs.

1. Tune worker processes/connections: - Set worker_processes auto; to match CPU cores. - Increase worker_connections to 4096 or higher. - Adjust keepalive_timeout for client connections.

Before applying, we: - Backed up original config: cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak - Tested in a staging environment that mirrored production. - Validated with nginx -t before reload.

We used nginx -s reload for a graceful reload, which forks new workers and drains old connections.

Rollback Strategy

In case the changes caused issues:

• Restore the backup: cp /etc/nginx/nginx.conf.bak /etc/nginx/nginx.conf
• Test: nginx -t
• Reload: nginx -s reload

If the issue is critical, we can quickly switch to old config using a symlink if multiple versions are kept.

Verification and Results

After reload, we monitored error logs, access logs, and external metrics (e.g., Datadog).

• tail -f /var/log/nginx/error.log showed no more timeout errors.
• wrk benchmark showed a 40% improvement in throughput and latency.
• The ss -s socket statistics showed fewer new TCP connections because keepalives were reused.
• Our client's p95 latency dropped back to ~80 ms.

When to Submit an OpsGlobal Ticket

This level of troubleshooting is often within the reach of an in-house DevOps team. However, if you see any of the following, it’s time to bring in OpsGlobal:

• You’re not familiar with Nginx internals or not comfortable editing production configs.
• The issue persists after basic tuning and might require kernel parameters (e.g., tcp_tw_reuse, somaxconn) or system-level tuning.
• You need to scale the gateway horizontally and require expertise in load balancer design or service mesh integration.
• The symptoms point to an upstream that’s misbehaving, and you need help tracing or instrumenting your microservices.

OpsGlobal offers 24/7 support. We can take over incident response, perform in-depth performance audits, and implement robust production changes with proper rollback plans.