Why I built it
Press a button, a helper appears, does one job and disappears. If the job drags on, it calls more helpers, who call more helpers, until the whole thing falls over. It isn't only a cartoon problem: an autoscaler that adds workers when a queue falls behind, or a client that retries a slow call, can do the same, because the extra load makes everything later. I wanted to build the engineering version of that on one small server and find out, with numbers, when calling for help actually helps.
Decisions
Simulate before building
Before writing the service I simulated it, calibrated on benchmarks from the server it runs on, in a container capped like the real cage. One image is not linear in megapixels, most of a fresh worker's memory is shared libraries, and past 1.5 cores more workers only make everything slower. My laptop's numbers were useless for concurrency, because single and multi-process runs landed on different core types.
- Resizing one image costs about 152 · MP^0.40 ms on the server, 81 ms below 1 MP and 520 ms at 40-70 MP, so work is predicted from image dimensions rather than a flat average.measured · README.md
- The simulator replays an hour of arrivals across five scenarios, seventeen policies, three machine sizes and 20 seeds, with the same arrivals for every policy. The 5,100 runs take about six minutes.measured · README.md
The pool runs, splitting is the exhibit
A forecast beat a timer for deciding when to split, and anything without a global budget fell over even at half load. But a plain pool that hands out ten images at a time, earliest deadline first, beat the best splitting policy, and the reason is preemption, not the forecast. So the button runs the pool, and the splitting policies, the box included, run as one-minute exhibits that visitors can start.
- A plain earliest-deadline pool met 5 to 24 points more deadlines than splitting late work across helpers.measured · README.md
- Without a global worker budget, splitting falls over even at half load. The box spends 36-40% of its CPU on spawns and killed work and meets 17% of deadlines, where never splitting meets 44%.measured · README.md
- The pool wins because it is preemptive. Adding a yield to the splitting policy brings it level on the cage, but it still starts 1.3 to 4.6 processes per task against 0.1 to 0.25 for the pool.measured · README.md
Check the simulator against the cage
A simulator is only worth something if it predicts the real box, so I ran five policies in the real cage and replayed every run with the same tasks. With a budget it holds. Getting the storm right took three fixes to the simulator, and the comparison turned up three bugs in the real system. The server itself is not the same machine from one day to the next, so replays now take a speed probe from the day of the runs.
- With a worker budget the simulator matches the real cage run for run. Tasks turned away agree in 14 of 15 runs, and spawns are within one in every budgeted run.measured · README.md
- Comparing the simulator with the real cage found three bugs in the real system, a timeout that killed heavy images as hung, admission counting contention twice, and helpers that skipped their warm-up.measured · README.md
- The server's own speed moves by about a quarter between days. A single worker ran 27% slower on 5 October than on 2 October, so replays scale costs by a same-day speed probe.measured · README.md
A promise you can check, and crashes that stay small
A batch is accepted only if the deadlines still fit, and a refusal comes back with a Retry-After. Workers know nothing but IPC, and the supervisor kills one that goes over its memory, misses heartbeats or runs far past its prediction, and blames the image in flight, so one corrupt photo costs three restarts instead of the batch.
- A batch is admitted only if the earliest-deadline schedule of everything already admitted still fits. A flat per-preset average had turned away 45% of tasks in a calm hour.measured · README.md
- An image that crashes three workers goes to a dead letter queue, and workers raise their own oom_score_adj, so under memory pressure the kernel kills a worker and never the supervisor.design · README.md
- Only the supervisor changes scheduling state. It checkpoints cursors every two seconds and writes outputs through a rename, and a test stops it halfway through a task and checks no image is lost.design · README.md
How I know it works
5,100 seeded simulator runs with the same arrivals for every policy, calibrated on benchmarks from the server. Five policies run in the real cage and replayed in the simulator, with the raw runs in the repo. End-to-end tests run against real Postgres, Redis and worker processes, including one that stops the supervisor halfway through a task and one that fails on the old helper warm-up. Both suites run in CI on every push.