Repository navigation
Differential fuzzer barely exercises the review path; make action selection state-aware and report coverage #156
Description
Follow-up from the #146 review. The fuzzer is merged and useful as it stands; this is what it needs before it can be read as covering the review profile.
Measured
40 seeds x 40 steps on the merged generator, counting envelopes sent and envelopes that returned a result:
| method | sent | succeeded |
|---|---|---|
decide.approve |
47 | 3 |
decide.override |
41 | 1 |
decide.reject |
35 | 1 |
abstain.declare |
35 | 5 |
control.resume |
75 | 5 |
handoff.accept |
37 | 14 |
task.complete |
87 | 49 |
The generator picks a random task and a random human for each decision, so nearly every decide.* lands on a task that is not in review_requested and the comparison is of error responses. The override path, which is the class of bug behind #124 (#103, JSON Patch index tokens), succeeds about once per 40 seeds and always with the same one-op replace /text patch.
Needed
- State-aware selection. Decide on tasks that are in
review_requested, resume tasks that are paused, accept handoffs that are open, answer whispers that are unanswered. Keep some random misses so error paths stay covered, but weight towards legal transitions. - Varied patches. Array indices,
add/remove/move/copy/test, nested paths, and the1e1/1_0style index tokens from Parse JSON Patch array indices with one strict shared rule #103, applied to artefacts with arrays in them. - A
--statsflag that prints sent and succeeded per method, and print it in the CI step, so coverage is visible in the log rather than assumed. - Fold back the held-out actions as their fixes land:
control.snapshotafter Define the canonical control.snapshot wire artefact and captured-state projection #148 is settled, and the explicit-empty-list cases after fix: reject explicitly empty handoff acceptance #151 and Honour an explicit empty include / what_to_restore in control ops #152.
The review profile is the one CHAP is judged on. Until the decisions succeed at a useful rate the fuzzer is checking the edges of that profile and not its centre.
Activity
brightbeamarsalan commentedon Sep 21, 2026
Done, on fix/156-state-aware-fuzzer. All four items, and it paid for itself twice on the first run.
Selection now reads the workspace. Decisions go to tasks in review_requested and are sent by a reviewer the review named. Resumes go to paused tasks, handoffs are resolved by the participant they were offered to, whispers are answered by someone they were addressed to, votes come from participants who have not voted. A share of the steps is a deliberate miss, so the refusals stay compared. When nothing is live it creates a task rather than acting on a settled one.
Overrides draw from the whole operation set against documents with arrays and nesting: add at the end and at an index, remove, move, copy, test, multi-op patches, and the 1e1, 1_0 and 01 index tokens from #103.
Measured over 40 seeds at 40 steps, accepted calls per method, against your table:
decide.approve 3 to 42, decide.override 1 to 55, decide.reject 1 to 37, abstain.declare 5 to 38, control.resume 5 to 30, task.complete 49 to 80, handoff.accept 14 to 18. review.request and task.complete are now at 100 per cent.
The empty-selection cases held out for #151 and #152 are folded back, so the fuzzer checks that both references refuse them identically.
--stats prints sent, accepted and rate per method, and the CI step runs with it.
Two divergences fell out of the wider action set, both in the patch engine, both on fix/patch-engine-parity:
The refusal messages differed word for word. Python quoted with repr and named Python types, TypeScript quoted with JSON.stringify and named JavaScript types, so the same refusal read two ways. Both now speak JSON, and a shared vector file holds every message so a drift in one fails the suite in both. 10 of 25 seeds.
The worse one. RFC 6901 gives - the position after the last array element, which no element occupies, so RFC 6902 admits it for add alone. TypeScript compared the token numerically, which NaN makes false both ways, so the range check passed: remove spliced at zero and deleted the first element, and replace wrote a - property onto the array, which serialises away, so the call reported success and changed nothing. A decide.override carrying a remove at /tags/- produced a different corrected artefact in each reference, and on the TypeScript side the wrong element was gone. 6 of 25 seeds.
With both fixed: 50 of 50 seeds at 40 steps, and 100 of 100 at 60 steps.
Order: #163, then #155, then the patch parity fix, then this.
Fuzzer: choose actions from the state, vary the patches, report cover…
Restore the state-aware fuzzer generator that #168 left behind (#174)
brightbeamarsalan commentedon Sep 21, 2026
All four items are on main and CI is green, so closing this.
It landed in two steps and that is worth recording. #168 took the CI flag and none of the generator, so main called fuzz.py --stats against a fuzzer without --stats and the conformance job failed on exit code 2. #174 restored the generator, merged with the recording invariant from #166 that had landed in between. My error, and the lesson is the plain one: run the command CI runs, not a near neighbour of it.
Where it ends up, accepted calls per method over 40 seeds against your table: decide.approve 3 to 42, decide.override 1 to 55, decide.reject 1 to 37, abstain.declare 5 to 38, control.resume 5 to 30, task.complete 49 to 80. --stats prints sent, accepted and rate, and CI runs with it.
The two divergences it found on the first run are fixed in #167: the patch engine's refusal messages, and remove and replace at the position after the last array element, which TypeScript accepted and which made the same decide.override produce a different corrected artefact in each reference.