Why I built it
Money pushed through a car wash has to add up on paper when somebody looks. I wanted the honest version: a ledger for a chain of car washes where correctness is the whole product. A payment that is retried, cut off by a crash or reported twice by the bank must still move each cent exactly once, the books must always balance, and anyone editing history behind the ledger's back must be caught. Most of my working life has been payment pipelines, Stripe and Kafka, and none of it is public, so this takes the hardest part of that work and proves it with numbers.
Decisions
Let the database refuse bad books
Every movement is a journal of entries that sum to zero per currency, in integer cents, and a mistake is fixed with a reversing journal, never an edit. I didn't want that to depend on the code being right. A deferred constraint trigger checks every journal at commit, the app's role can't update or delete entries, and every entry copies its account's currency through a trigger. The invariant checker that judges all of it is separate code with its own SQL, so a bug in posting can't hide behind the same bug in the check.
- A deferred constraint trigger rejects an unbalanced journal at commit, and the app's role has no UPDATE or DELETE on entries, so the books balance even if the code is wrong.design · README.md
- The invariant checker is separate code with its own SQL, running as a read-only role, so a bug in posting can't hide behind the same bug in the check. It has eight safety and five liveness checks.design · README.md
Measure the hot account before picking a design
Every payment touches what the provider owes the platform and the platform's fee, so a plain balance row there is a lock every payment waits on. I measured four ways around it with an open-loop generator checked against pgbench. SERIALIZABLE with retries, the textbook answer, was the worst. Derived balances carried the most, but one long transaction makes every read sum a longer tail. I chose 16 split rows, which carry nearly as much with cheap reads. Posting is one round trip of locks and one CTE on plain pg, because Prisma's query API took twice the round trips.
- At 32 connections the hot account sustained 149 journals a second on one locked row, 132 with SERIALIZABLE and retries, 915 derived from entries, and 803 split into 16 rows, which I chose.measured · README.md
- On a host that pages, about 0.1% of commits stalled for around 100 ms, which behind a locked hot row becomes everyone's p99, so the benchmarks only start on a quiet machine.measured · README.md
- Posting a journal is one round trip of locks plus one CTE on plain pg. Prisma's query API took twice the round trips and 1.7 times the latency for the same journal.measured · README.md
Exactly-once effects, at-least-once delivery
The API takes payments with Stripe's idempotency rules, and a key is held by a lease that every write checks. Without it, the first chaos runs charged one key twice. A payment is a state machine of conditional updates, and the bank's word is checked, not trusted: a webhook only makes the API ask the provider what happened. Events go to an outbox in the same transaction, a relay publishes them to Kafka at least once, and the consumers record every event id they apply, so a repeat does nothing.
- Idempotency keys follow Stripe's rules and are held by a lease that every write checks. The first chaos runs showed that without it a slow request could create a second payment for its key.design · README.md
- In a chaos run with three Postgres crashes, the relay published 3,368 events as 3,744 Kafka messages, and both consumers still applied each event once.measured · README.md
- An event reaches Kafka in about 20 ms at the median and 36-46 ms at p99 from 100 to 2,000 events a second, most of it the relay's 50 ms poll.measured · README.md
- With Kafka killed for 30 s at 500 events a second, about 25,000 events piled up in the outbox and drained in under 2 s once the relay reconnected.measured · README.md
A witness the database can't fool
Every journal becomes a leaf in an RFC 9162 Merkle tree, the construction Certificate Transparency uses, and a separate witness signs the root every 10 seconds, only for a tree that provably extends the last one it signed. A rewrite that makes the database agree with itself passes every balance check, and only the signed roots catch it. The browser can check a journal on its own. What none of it catches is a rewrite in the seconds before the next checkpoint, and I measured that too.
- Of 900 tampers that rewrote the ledger's history so the database agreed with itself, the witness's signed roots caught all 900 and the database alone caught none.measured · README.md
- Every tamper that touched a journal the witness had signed was caught and the right journal named, 2,704 of 2,704. The same rewrite above the last checkpoint was caught 0 of 100 times.measured · README.md
- At a million journals an inclusion proof is 15-20 hashes, at most 640 bytes, a checkpoint takes about 12 ms, and verifying every journal from scratch takes 31 s.measured · README.md
- The browser checks a journal on its own, recomputing its leaf from the rows, following the server's proof to the root and verifying the witness's Ed25519 signature with WebCrypto.design · README.md
How I know it works
One container runs Postgres, Kafka, Toxiproxy, the simulated provider, two API processes, the worker and the witness. Clients drive payment journeys with real retries as a chaos harness kills processes at random and at 15 named points in the code, crashes Postgres and Kafka, breaks the network, has the provider time out and misstate amounts, and forges webhooks. After each run an independent checker reads everything. Each guard has a mutant to show a violation would be seen. The runs found real bugs: a dead pooled connection crashing Node, the unfenced lease, a deadlock with an exhausted pool, consumers holding partitions for 45 s after a crash. CI runs a three-run chaos smoke on every push.