Anatomy of a High-Performance API Gateway

An API gateway is not just "a reverse proxy with extra features" — it's the layer that owns a specific, ordered set of cross-cutting concerns so individual services don't each reimplement them: TLS termination (so services behind it can speak plain HTTP internally), request routing (mapping a path or host to the right upstream service), authentication/authorization enforcement at the edge, rate limiting per client, and load-balanced retries across healthy upstream instances. Getting the order right matters — routing has to happen before rate limiting can apply the right per-route limit, and rate limiting has to happen before the request reaches an upstream that would otherwise absorb load it never should have received.

The checkout request makes this concrete: TLS terminates at the gateway, so the certificate and cipher configuration live in exactly one place instead of every service; the path /api/checkout routes to the order service's upstream pool; a per-API-key token-bucket check runs before the request is forwarded; and if the order service's first instance is slow or down, the gateway retries against a healthy instance rather than the client seeing a raw connection failure.

NGINX's core strength is architectural simplicity: an event-loop worker model that handles enormous concurrent connection counts with very low per-request overhead, configured through a static file that's reloaded (not hot-swapped mid-request) when routes or upstreams change. That simplicity is also its constraint — adding or removing an upstream instance dynamically (as a Kubernetes deployment scales pods up and down) requires either a config reload or an external mechanism (like nginx-ingress watching Kubernetes and regenerating config) rather than a built-in service-discovery integration.

Envoy was built for exactly that dynamic case: it separates the data plane (the proxy handling traffic) from a control plane that pushes configuration updates over the xDS API in real time, so upstream endpoints, routing rules, and even rate-limit policy can change without a reload or restart. That buys live reconfiguration in a fast-scaling microservices environment, at the cost of running (or adopting) a control plane and a meaningfully larger operational surface than a static NGINX config file.

Trace the checkout request when the order service's primary instance starts timing out under load: the gateway's retry policy resends the request to another instance. That's correct behavior for one request — the problem appears when every gateway worker independently makes the same decision for every request hitting the slow instance simultaneously, multiplying load onto the remaining healthy instances at exactly the moment they're least able to absorb it. The trigger is a single degraded upstream instance; the symptom is the entire order-service pool becoming overloaded within seconds; the root cause is an unbounded retry policy with no shared budget across the gateway's concurrent requests.

The fix is a retry budget: cap retries as a percentage of total request volume (Envoy's retry budget mechanism, or an equivalent token-bucket-style limiter shared across workers) so retries can't exceed, say, 20% of primary traffic regardless of how many individual requests are failing. Combined with a circuit breaker that stops sending new requests to an instance failing above a threshold, this converts a retry storm into a bounded, contained degradation instead of a cascading outage.

The decision rule: choose NGINX when the backend topology is relatively stable, the team wants operational simplicity, and per-request overhead at very high connection counts is the dominant concern. Choose Envoy when the topology changes frequently (autoscaling, service mesh, canary deployments) and you need live configuration updates plus rich per-request observability (detailed L7 metrics, distributed tracing integration) that static config makes harder to get. Choose a serverless front door (a gateway built on Cloud Functions or equivalent) when request volume is bursty and low enough that cold-start tail latency is acceptable, and you'd rather not operate gateway infrastructure at all — the trade is cost-per-request and a latency profile you don't fully control.

Whichever gateway you choose, the retry budget and circuit breaker from the previous section apply regardless of vendor — they are the mechanism, not a feature specific to one product. Verify your chosen gateway actually supports a shared retry budget (not just a per-request retry count) before depending on it in production.