Why I built it
Distributed consensus is the one thing none of my other projects touch, so I built Raft from scratch in Go. Raft is a university assignment and Go already has two good implementations, so having Raft wasn't the reason to build it. What I wanted to build was the rare part: evidence that it keeps its promises when the network splits, processes crash or freeze and disks lose what wasn't synced, and a number for what each guarantee costs. PortalGun's live demo discovers its servers through the registry on top, and in the dashboard the five nodes are the five families and the leader is the Don.
Decisions
A Raft core that can't touch the world
The core is a deterministic state machine, the shape etcd/raft has. Messages, ticks and "this write is durable" go in, and a batch of work comes out: what to persist, what to send now, what to send only once the write is on disk, and what to apply. Randomness comes from a seed and peers live in a sorted slice, because one loop over a Go map is enough to break replay. So every failure the simulator finds replays exactly, under a debugger, as often as I want. The price is more code than the paper's pseudocode and a contract (persist this, then send that) that tests of the core alone can't see, so the driver and the WAL's logic run inside the simulator unchanged too.
- The Raft core is a plain struct with no goroutines, clock, sockets or files and gets its randomness from a seed, so a run replays exactly. An import test guards that, and CI runs the same seeds twice and checks the trace hashes match.design · README.md
Plant bugs to prove the nets catch them
Two nets check it. The simulator runs the whole cluster on one seed with a simulated network, disk and clock, turns on a random mix of faults, checks Raft's invariants after every step and checks the client history with Porcupine at the end. A clean run proves nothing if the net has holes, so I planted ten bugs behind switches the production image can't turn on, and the simulator catches each one, one of them only with PreVote off. It caught bugs of mine too, like a leader elected before its own vote was on its disk. The second net is five real processes in Docker that a nemesis kills, freezes and partitions with iptables, and there only the client history and the final state count.
- The simulator catches each of the ten bugs I planted on purpose (one only with PreVote off), seven of them within 42 seeds, and CI replays a seed that catches each one on every pull request.measured · README.md
- In seed 2888 a node won an election before its own vote was on its disk, was killed, and won the same term again with a shorter log. The durability invariant and Log Matching caught it.measured · README.md
- Client histories of 287,516 operations stayed linearizable through 187 faults across six three-minute runs, from SIGKILL and freezes to partitions and membership changes on five nodes in Docker, every node ending at one state.measured · README.md
- On 2026-10-17 I ran 100,000 simulator seeds with random faults on the code as it was then, half of them modelling the server loop's batching, and none failed.measured · README.md
Heartbeats don't wait for the disk
The leader sends its appends before its own write finishes and counts itself toward the quorum only once its fsync is done. One goroutine owns the WAL and syncs everything that queued up during the last flush, so under load many writes share one. A short election timeout should cost elections nobody wanted, but not through the disk: an injected one-second stall on the leader caused nothing at 150 ms, while with the loop waiting on every fsync the same stall at 300 ms forced an election every time. On this laptop pauses are the limit, though those runs came before a fix. My client slowed failover at first by connecting to followers only after the leader died, so now it connects up front.
- When the leader is killed under 200 writes a second, the others elect a new leader and clients wait a median of 194 ms at a 150 ms election timeout and 1,206 ms at 1 s, so the timeout is nearly all of the failover.measured · README.md
- Injected leader disk stalls of up to 1 s caused no election at a 150 ms timeout, 0 of 15, because heartbeats never wait for the disk. With the loop waiting for every fsync, a 1 s stall forced one every time at 300 ms.measured · README.md
- One fsync per write caps five nodes at 110 writes a second. Group commit packs about 50 entries into one fsync near the knee and carries 5,000.measured · README.md
- My first failover sweep was slower because of the client. With cold connections its own part of the gap was 35-150 ms against 24-29 ms, so it now connects to every node when it's created.measured · README.md
Measured against etcd, including where it loses
ReadIndex is the default, because a lease read skips the heartbeat round but is only safe while clocks drift less than its margin, which clients can't see. My server loop asked the core for work after every event, so it never had more than one entry to batch, and fixing that halved the messages per write. Then the write knee wouldn't move, and a trace showed the event loop waiting for the node's one Go processor. Against etcd 3.6.15 with the same setup, a write is faster at low load, but under load etcd carries at least twice as many and doesn't collapse past its knee. Both ran with one processor per node, and the proper runs with two, for both, are still to do.
- At 2,000 operations a second, 95% of them reads, a lease read takes 0.3 ms at the median and ReadIndex 0.7 ms, against 13.4 ms through the log. ReadIndex is the default because a lease is only safe while clocks drift less than its margin.measured · README.md
- Batching the server loop's whole queue before asking the core for work halved the messages per write, 8.4 against 15.5 at 4,000 a second, and lifted ReadIndex's knee from 8,000 to 14,000 that night, 12,000 in three rounds a day later.measured · README.md
- Near the write knee the leader was only 39-54% busy. A trace showed the event loop waiting for the node's one Go processor, so every node now runs with at least two.measured · README.md
- Against etcd 3.6.15 on the same laptop, a write is faster at low load, 11-13 ms at the median against 23-36 ms, but under load etcd takes at least 16,000 writes a second on five nodes to my 6,000.measured · README.md
How I know it works
What I know about where it works and where it wasn't correct comes from the simulator, the lab and what they found in my code: a leader elected before its own vote was durable, a lease that couldn't see a pause, leases that expired a tick early and watches that waited forever. Every refactor of code the simulator runs that wasn't meant to change behaviour has to give identical totals for 1,000 seeds in five configurations before it's merged. An audit halfway through fixed nine more bugs, one of them live in production: cleaning up old snapshots could delete the one the WAL still pointed to. The numbers come from an open-loop generator that measures from each request's scheduled time.
What I would do next
The biggest limit is that everything runs on one machine, so real distance between nodes and clocks that really drift are only simulated, and joint consensus and TLS between nodes aren't done. Next I'd fix how the client leaves a leader that stopped answering, which should bring a registration during a failover down to about the election. Then the measurements with two Go processors on a quiet evening, etcd included, recovery numbers, and later Jepsen as an outside check.