Why I built it
Rick's portal gun opens a door to whichever dimension he points it at, and when that dimension is on fire, he points it somewhere else. An API gateway does the same job thousands of times a second, with a fraction of a millisecond to decide. Everything a gateway promises is a number: little added latency, one quota across several instances, traffic off a dead upstream before many requests fail, and a bad release caught without rolling back good ones. I built one around those four numbers and measured each against something real.
Decisions
Build on the standard library, measure what it costs
The proxy is Go's httputil.ReverseProxy with my own Transport, which picks a dimension per attempt, so a retry moves to another dimension without ReverseProxy knowing. Hop-by-hop headers, streaming and upgrades hide subtle bugs, and the standard library has had years of them fixed. Against nginx, HAProxy, Envoy and Caddy on one vCPU each, my code costs a few percent over the bare ReverseProxy, and nginx and HAProxy hold about twice the rate, roughly what C event loops without garbage collection should do. Past their maximum the Go proxies queued in memory until they swapped, so the gateway refuses work past 256 requests in flight. I guessed 20-40 KB per request in flight. It was about 145.
- At 1,000 requests a second on one vCPU it adds 155 µs at p50, 16 µs over Go's bare ReverseProxy. For comparison, nginx adds 107 µs, HAProxy 108, Caddy 201 and Envoy 207.measured · README.md
- Its own code costs 1.03-1.09x the bare ReverseProxy's CPU per request at every rate up to 8,000 a second, while nginx and HAProxy held about twice the rate on one core.measured · README.md
- Without a cap on requests in flight, a cold start at 8,000 requests a second took down both Go proxies. With a cap of 128 or 256 nothing collapsed in 30 runs, and 256 is the default.measured · README.md
- A request in flight costs about 145 KB of heap and goroutine stack, not the 20-40 KB I guessed, so a cap of 1,024 would take half of a 256 MB container.measured · README.md
One quota, four designs, one referee
Several gateways have to agree on one bucket without asking anyone on every request. I built a static split, an exact limiter with one Redis call per request, a local budget synced with Redis on a timer, and leases, all behind one interface, and judged them by replaying every decision through one perfect token bucket. The local budget synced every 100 ms is the default: nothing measurable on the request and within 1.3% everywhere. The exact limiter taught me the most. Its 2 ms timeout never applied, and its first breaker let too many through. With the debt charged back it is the one I'd use for a limit that's a contract.
- Gateways share one rate limit through local budgets synced with a Redis bucket every 100 ms, which stayed within 1.3% of a perfect token bucket in every traffic pattern and added nothing measurable.measured · README.md
- Splitting a limit evenly between gateways turned away a third of what a client was allowed once traffic shifted 90/10, and 60% over four gateways.measured · README.md
- Rebuilding the exact limiter's Redis breaker, with what a gateway admitted on its own charged back later, took it from 11.6% too many requests to 0.2% too few at an even split.measured · README.md
- The first full limiter run found two bugs in my own code, a 2 ms Redis timeout that never applied and a bucket cap that cut the limit to two thirds at a 500 ms sync.measured · README.md
Three kinds of health, because each misses something
Active probes find a dead dimension that gets no traffic, passive ejection reacts to real failures within milliseconds, and a panic threshold stops one shared failure from ejecting the whole pool. Power of two choices does part of the job before any of them: a hung dimension collects requests in flight and stops being picked. With all three and a retry budget, six kinds of injected faults cost no requests. On a crash, probes on their own were worse than slow, because a refused connection fails in microseconds and looks like the cheapest dimension. The live demo's pool comes from FiveFamilies, my Raft registry, which no request ever waits on.
- With probes, passive ejection and retries together, no request was lost in any of six injected faults at 2,000 requests a second, and p99 during the fault stayed at 5.8-6.0 ms.measured · README.md
- When all three dimensions failed 20% of requests at once, the panic threshold kept losses at 4% instead of 59% by spreading over the whole pool rather than ejecting it.measured · README.md
- A dimension that stopped answering lost 11-18 requests at every timeout tried, because power of two choices stops picking it as requests pile up in flight, before any check fires.measured · README.md
- A retry goes to a different dimension, and retries are capped at 20% of requests over 10 seconds, so a dead pool can't triple the load.design parameter · README.md
- The live demo uses FiveFamilies, my Raft service registry, to discover its dimensions. They register themselves, and only the controller follows the registry, never the gateways.design · docker-compose.registry.yml
- In laptop runs a new dimension got its first request a median of 15 ms after registering, and with two retries on the route none of 443,047 requests was lost in 43 runs, some with the registry's leader or every node down.measured · bench/results/e5/20261021-225536-laptop/summary.json
A canary that can look every second
The canary decides with Wald's sequential test on who the errors came from. If the new version is no worse, an error lands on it with the same probability as its share of traffic, so a downstream blip that hits both versions cancels out, and looking every second doesn't inflate false rollbacks. A simulator calling the controller's own package checked it. The price is small regressions on thin traffic, where a test that waits for the whole step does better. On the live demo anyone can start a rollout and break the new version. I first kept rollouts for the admin, and that left the canary invisible to anyone looking.
- The canary uses a sequential test on who the errors came from. In 36 simulated cells with a good canary it rolled back 0.1-0.7%, against 22-52% for a z-test checked every second.measured · README.md
- Visitors can kill or slow a dimension on the live demo, one action at a time for at most 30 seconds, and start a canary rollout, three an hour per address. The controller never touches the Docker socket.design · README.md
How I know it works
Every number comes from a harness in the repo: an open-loop load generator checked against vegeta and wrk2, medians of three runs or more, every proxy on its own CPU set with the same limits. It ran on my laptop, so I only compare numbers from the same session. Limiter accuracy comes from replaying each gateway's decisions through one perfect bucket, failover from real TCP faults through Toxiproxy, and the canary from 72 simulated cells. Race tests, integration tests against Redis and Postgres and a Docker e2e suite run in CI on every push.