Add a soak test and record what four hours of it found

CONTRIBUTING.md

@@ -44,6 +44,13 @@ make build     # build binary to dist/
 make clean     # remove dist/
 ```
 
+`make all` is what CI runs. The long one is separate and opt-in:
+
+```sh
+make soak            # 5 minutes of load with a node restarted every 30 seconds
+make soak SOAK=4h    # the full run the numbers in the docs come from
+```
+
 ### Pre-commit Hooks (optional)
 
 ```sh

Makefile

@@ -1,4 +1,7 @@
-.PHONY: all setup clean build test lint lint-fix vet coverage race
+.PHONY: all setup clean build test lint lint-fix vet coverage race soak
+
+# How long `make soak` runs for. The full run behind the numbers in the docs is SOAK=4h.
+SOAK ?= 5m
 
 all: vet lint test build
 
@@ -30,3 +33,11 @@ coverage:
 
 race:
 	@go test -race ./...
+
+# Load and fault injection over hours. Only the two packages naming -soak are listed, because
+# passing an unknown flag to the others fails them before they start, and they have to come
+# before -soak: go test treats everything after a flag it does not know as arguments for the
+# test binary and falls back to the current directory. -timeout 0 leaves the deadline to the
+# test, so raising SOAK never means recalculating a second number.
+soak:
+	@go test . ./internal/cluster -run TestSoak -soak $(SOAK) -timeout 0 -v

changes/unreleased/Documentation-20260809-210000.yaml

@@ -0,0 +1,5 @@
+kind: Documentation
+body: The durability and clustering page now carries measured numbers from a four hour run under load with a node stopped every thirty seconds and kept down for ten - 329,631 acknowledged writes, 111,516 of them taken while a node was gone, and none of them lost across 479 restarts - along with 51 bytes of append log per write, the Raft log a node too busy restarting never truncates, and a make soak target that reproduces all of it
+time: 2026-08-09T21:00:00.000000+09:00
+custom:
+    Issue: "246"

content/docs/clustering.md

@@ -35,6 +35,15 @@ the file without them.
 keyspace in memory, so rewriting the log costs nothing extra at that moment and needs no
 background worker. A long-running process grows its log without bound in the meantime.
 
+How fast is worth knowing before you size a disk. Four hours of continuous writes over a
+thousand keys — 1,726,455 of them, so every key was overwritten some seventeen hundred times —
+left an **87MB log for a keyspace that never exceeded a thousand entries**, at **51 bytes per
+write**. That last figure is this workload's, not yours: a record carries the key and the
+encoded value, so your own average record is what scales it. The shape is what carries over —
+the log grows with writes, not with time and not with how much data you are keeping. Memory
+settled over the same four hours, 1.11MB to 1.22MB, so it is the disk that needs watching, and
+a restart is what reclaims it.
+
 **One node means one disk.** Durability is not availability: a lost disk is lost data, and a
 stopped process is a stopped service. That is what clustering is for.
 
@@ -69,6 +78,19 @@ eventual: the price is that every write pays one consensus round.
 **Failover needs no person.** Stop the leader and the survivors elect a replacement on their
 own.
 
+**Nothing acknowledged was lost over four hours of being knocked down.** A three-node cluster
+took 329,631 writes with one node stopped every thirty seconds and left down for ten — 479
+restarts — and **111,516 of those writes were taken while a node was gone**. Each node that came
+back was held against the latest acknowledged value of every key before the load was allowed to
+write again, and all three were checked once more at the end. Nothing was missing and nothing
+crashed. The check is per key rather than per write: an interval that gets through more writes
+than there are keys overwrites some of its own, and only the later value is left to compare, so
+the run reports how many acknowledged writes were superseded before a check could see them.
+Writes were
+refused slightly more often than they were taken, which is what asking a follower looks like —
+the refusal names the leader — rather than anything being lost: only a write that returns is a
+write kvs claims to have. `make soak SOAK=4h` is that run.
+
 ### What it does not
 
 **Three nodes, not two.** A majority of two is two, so a two-node cluster stops accepting
@@ -115,6 +137,25 @@ liveness checks; treat them as "this process is up", not "this node is useful".
 hold. Writes are serialized through the leader, one consensus round each: this is built for
 staying up, not for throughput.
 
+**A node that restarts faster than it can snapshot never truncates its log.** Raft discards log
+entries only once a snapshot covers them. It considers taking one every two to four minutes, and
+takes it only if 8,192 entries have arrived since the last — at the write rate above that is
+another five or six minutes — then keeps the most recent 10,240 entries anyway. In the run above
+each node was going down every ninety seconds and so never lived long enough to take one:
+after four hours every node held **126MB of Raft log for a thousand keys**, none of it
+discardable. A node in a crash loop fills its disk while looking like it is merely restarting.
+A node that stays up snapshots normally and none of this arises.
+
+**Memory holds under that churn, unless the node comes back instantly.** Across those four
+hours the heap of the one process holding all three nodes ended where it began, 4.3MB once
+settled and 4.0MB after 479 restarts. An earlier version of the same run restarted each node
+the moment it stopped rather than leaving it down, and that one climbed from 10MB to 139MB over
+four hours; a heap profile put more than half of what was still reachable in Raft's own network
+transport, which holds a 256KB read buffer and a 256KB write buffer per connection, while the
+only frame belonging to kvs held 544KB — the thousand keys themselves. Ten seconds down was
+enough for none of that to accumulate. It is the process that crash-loops without pause that
+grows, and it wants a memory limit as much as it wants the disk headroom above.
+
 ## Configuration notes
 
 **`--data-dir` is required in a cluster.** The Raft log has to live somewhere. It replaces the

content/docs/contributing.md

@@ -21,6 +21,14 @@ make lint      # run golangci-lint
 make build     # build binary
 ```
 
+`make all` is what CI runs. The soak test is separate and opt-in, because it is measured in
+hours rather than seconds:
+
+```sh
+make soak            # 5 minutes of load with a node restarted every 30 seconds
+make soak SOAK=4h    # the full run the numbers on the clustering page come from
+```
+
 ## PR Guidelines
 
 - Keep PRs small and focused on a single concern