Repository navigation
Add a soak test and record what four hours of it found - #246
Conversation
github-actions
Bot
added
documentation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 85ebb48540
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
85ebb48 to
82aabb8
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82aabb8bd8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e5611139c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
8e56111 to
44815b2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 44815b25e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
44815b2 to
1136b24
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1136b24c4c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
`make soak` puts load on a three-node cluster with one node stopped and restarted every thirty seconds, and on a single node writing to its append log. Both skip unless -soak names a duration, so `make test` and CI cost what they did before. Four hours of it: 350,260 acknowledged writes across 457 node restarts, every one present on every node at the end holding the value it was acknowledged with, and no crashes. The single-node log grew 50 bytes per write to 76MB for a keyspace that never exceeded a thousand entries, and its memory settled. Two things did not come out clean, and the clustering page now says so. Raft discards log entries only once a snapshot covers them and takes one every two to four minutes, so a node going down every ninety seconds never takes one: each held 133MB of undiscardable log after four hours. And the heap went from 10MB to 139MB, more than half of it Raft's own per-connection transport buffers, the only kvs frame holding the thousand keys themselves. The cluster soak reports the heap rather than asserting on it. Convergence is what was measured not to happen under that churn, so a threshold would either be picked to pass or fail without telling anyone something they can act on. What it does assert is the one that matters: nothing acknowledged went missing.
The load and the fault injection shared one goroutine, so the node was stopped and restarted between two writes and never missed any. That measures a process being started twice, not a cluster taking writes a node short and handing them over when it comes back, which is what the run is for. The node now stays down for ten seconds while the writes carry on, and how many were taken in that state is counted rather than inferred. Every acknowledged value is also checked on the returning node before the run writes again. Checking only at the end let a later round overwrite a value that had gone missing, which took the loss out of the evidence along with the data.
The numbers on the page came from a run where the node never missed a write, so they described a weaker thing than they claimed. Four hours with the node kept down for ten seconds at a time: 329,631 acknowledged writes, 111,516 of them taken while a node was gone, none of them missing across 479 restarts. The heap is the part that changed its mind. It ended where it started, 4.3MB to 4.0MB, where the run that restarted a node the instant it stopped went from 10MB to 139MB. So the growth belongs to the process that comes back with no pause, and the page now says that rather than saying memory grows under churn. The Raft log a restarting node never truncates is unchanged in kind, 126MB a node. Bytes per write is qualified as this workload's, since a record carries the key and the encoded value and someone sizing a disk needs their own record, not this one's.
1136b24 to
6189ed1
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6189ed12be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| r.down = r.nodes[i] | ||
| r.down.stop(t) |
There was a problem hiding this comment.
When validating recovery from a process crash or power loss, this fault path only calls testNode.stop, which invokes Node.Close (internal/cluster/cluster_test.go:198-204); that method performs an orderly Raft shutdown and closes the transport and log store (internal/cluster/cluster.go:283-294). Consequently, the soak never exercises reopening after an interrupted write or unclean Raft termination, so it cannot support the new documentation's crash-loop/“knocked down” durability claims. Run each node in a killable process (or otherwise inject an abrupt termination), or narrow the claims to graceful restarts.
Useful? React with 👍 / 👎.
Three ways the long run reported better than it knew. Any error from a write counted as a refusal and sent the load to the next node. A follower refusing a write is the ordinary case and names the leader, but an apply that timed out, a store that failed, or an FSM that would not take a frame is not that, and retrying until some node says yes let a broken run finish looking clean while inflating the refusal count the docs quote. Only kvs.ErrNotLeader is retried now; anything else ends the run where it happened. The check at every restart compares the last acknowledged value of each key, and the load walks a thousand keys in order, so an interval that got through more than a thousand writes overwrote values nothing had compared yet. That is not a loss, but it is not evidence either, and the run said nothing about how much of it there was. It counts them now: a ninety-five second run on this machine put 1,383 of 4,383 acknowledged writes past a check, which is the number the claim has to be qualified by rather than a number to hide. And the single-node half asserted the heap against a baseline it might never have taken. Reaching the size cap before the tenth of the run when the baseline is due left it at zero, and a limit of zero fails, so the one outcome that cap exists to produce was a guaranteed failure. It says there is nothing to compare instead.
Two claims on the clustering page that went further than the run behind them. The restart check was described as making a lost write impossible to cover up. It compares the latest acknowledged value of every key, so an interval that gets through more writes than there are keys leaves only the later value to compare - per key, not per write. The page says which it is, and the run reports how many writes went past a check. And a healthy node was said to snapshot every two to four minutes. That is how often hashicorp/raft asks the question; it takes one only once 8,192 entries have arrived since the last, which at the rate measured here is another five or six minutes, and it keeps the most recent 10,240 entries even then. An operator waiting for the log to shrink was given the wrong minute.
1 participant
Summary
Adds
make soak: hours of load against a three-node cluster with one node stopped and restartedevery thirty seconds, and against a single node writing to its append log. Then runs it for four
hours and puts the numbers on the durability and clustering page.
Both tests skip unless
-soaknames a duration, somake testand CI cost exactly what they didbefore.
Stacked on #245 — base is
datadir-format-version, notmain. It retargets tomainonce #244and #245 land.
Type of Change
Marked Documentation rather than New feature: nothing in the shipped binary changes. What is new
is a test and the measurements it produced.
Changes
internal/cluster/soak_test.go— three-node cluster, fixed thousand-key set overwritten underload, one node stopped and restarted every thirty seconds in rotation, never two at once. Only a
write that returned
nilcounts as acknowledged; at the end every acknowledged write is comparedagainst every node. Reuses the existing
startCluster/stop/restart/eventuallyharness rather than adding a second one.
soak_test.go— the single-node half, measuring append-log growth. The cluster path cannotmeasure it: Raft's own snapshotting bounds that store, so the unbounded log is single-node only.
Stops at a 2GiB cap so a long run cannot fill the disk it is measuring.
Makefile—make soak,SOAK=4hfor the full run. Packages are listed before-soakonpurpose:
go testtreats everything after a flag it does not know as arguments for the testbinary and silently falls back to the current directory, which quietly ran only half of this.
content/docs/clustering.md— the measured numbers, in the same place and tone as the failovertimings already there.
CONTRIBUTING.md,content/docs/contributing.md— the command.What four hours measured
Passed:
end holding the value it was acknowledged with. No loss, no crashes.
Memory settled, 1.06MB to 1.11MB.
Did not come out clean, and the clustering page now says so:
only once a snapshot covers them and takes one every two to four minutes; a node going down every
ninety seconds never lives that long. Each held 133MB of undiscardable log after four hours. A
crash-looping node fills its disk while looking like it is merely restarting.
still reachable in Raft's own
NetworkTransport— a 256KB read buffer and a 256KB write bufferper connection. The only frame belonging to kvs held 544KB, which is the thousand keys
themselves. Whether it levels off past that is not established.
Why the cluster soak reports the heap instead of asserting on it
Convergence is the thing that was measured not to happen under this churn. An assertion would
either be a number picked so the test passes, or a failure whose cause is upstream and which nobody
can act on. So the measurement stays and the docs carry what it means. The assertion that remains
is the one this test exists for: nothing acknowledged went missing.
Testing
make allpassesNew tests added
make soak SOAK=45s— both packages green, failover exercised.make soak SOAK=4h— the run above.Diagnostic runs at 20m, 45m and 1h with a heap profile, to locate the growth before writing a
word about it in the docs.
go test ./...unchanged: both soaks skip without-soak.Related Issues
None.