Follow-up from #1996 (merged at ab63da7a1). Known limitation, documented there rather than silently shipped.
Summary
The finalized-SWM GC uses one traversal for two jobs with opposite requirements.
- Measurement must cover every context graph for the backlog total to be a whole-node figure.
- Deletion must resume exactly where its budget ran out and ignore rotation shape entirely.
Because deletion rides the measurement rotation, the node-wide deletion budget always lands at the front of the traversal, and tasks behind the front are never examined. #1996 fixed this within a meta graph (keyset cursor, 650174547); it remains open across meta graphs and context graphs.
Independently raised as a 🔴 by the review bot from a cold read of the pushed head, with a suggested direction (per-meta-graph pagination) matching the analysis below.
Why it matters in practice
A task is only re-selected forever if it can never be cleared. On a receiver whose SWM synced cleanly the task clears and leaves the set, so a healthy node behaves correctly. The failure is progressive: unclearable tasks accumulate at the front of the traversal, and deletion throughput degrades as they do. It does not recover without a process restart, because the cursor is in-memory.
This interacts with #2021 and #2022: a peer can deliberately seed unclearable tasks, and on Blazegraph the retirement path that would remove them is inert.
Evidence
Fixture: 2 context graphs × 3 meta graphs × 5 tasks = 30 tasks, maxCandidatesPerSweep: 2, 40 slices, all tasks permanently non-deleting.
| build |
distinct tasks ever examined |
| as shipped |
8 / 30 |
| + meta cursor advancing on budget exhaustion |
15 / 30 |
| + rotation restarting where discovery stopped |
15 / 30 (inert) |
| separated deletion traversal (flat cursor) |
30 / 30, first 30 all-distinct |
30/30 is reachable. Hitting 15/30 twice is the signal that this is not patchable level-by-level: each level fixed exposes the next, because the defect is the shared traversal rather than a missing cursor.
This is the fourth instance of one bug class in this subsystem — no cursor, restart at the beginning, successors never reached — after the context-graph rotation, the meta-graph walk, and task selection, all three fixed in #1996.
Why the permanently-non-deleting case is reachable
finalization-handler.ts sets layerVerification = vmVerification when VM verification succeeds, so the cleanup marker is armed from VM state and the live SWM is never inspected at arming time. A divergent non-empty SWM then fails clearIfStillExact's SWM check, returns preserved, and the task is never retired — while still being charged to the budget on every sweep.
Two traps that re-created the bug class while fixing it
Both looked obviously correct when written, and each re-introduced the exact starvation being fixed:
- Resetting the cursor after a full circuit. "Everything is drained, start from the top" is the head-of-line behaviour — on a node whose leading tasks are permanently unactionable, every sweep re-walks them.
- Wrapping the inner loop modulo one context graph's meta graphs. Cycles that context graph forever whenever it always has examinable tasks; the graphs behind it are never reached. Only the outermost loop may wrap.
The spec: three coupled contracts any fix must satisfy together
- Wall-clock split. Deletion-first starves the backlog metric under load (re-opens the "counters blank exactly when deferral matters" problem by another door). Measurement-first starves deletion, which is the bug being fixed. A fixed, non-configurable half-slice split works; a tunable ratio is a knob nobody can set correctly without knowing both costs.
- Rotation cursor persistence. Returning early on a deletion halt skips
yieldRotation, so the measurement cursor is never persisted.
- Pressure classification. Moving candidate discovery into the deletion pass means a scheduler rejection raised there no longer reaches the retryable-deferral classification added in
067e06f1e. Fixing (2) breaks (3) — the halt reason must be carried into the sweep result, not returned early and not dropped.
Fixing 1 and 2 together took the suite from 2 red to 4 red. Two of the newly-red were pressure-classification pins added in #1996 itself, and one was rotates a head that repeatedly cannot be finished so the tail is still served returning ['cg-slow'] — i.e. the measurement starvation guard fixed earlier in #1996 regressed. Any future design is constrained by that test.
Rebuild notes
The exploratory commit was not merged, so:
- Replace the per-meta-graph task cursor with a flat field:
{ contextGraphId, swmMetaGraph, afterTaskSubject? } | null.
- Add a
runDeletionPass walking from that cursor: outer loop over context graphs wrapping modulo length, inner loop from the resume index to the end of that CG only, no wrap. Park the cursor at {cg, metaGraph} before each cleanupMetaGraph call so a mid-page yield resumes there; set afterTaskSubject from lastTaskSubject only when the page was full.
- Do not reset the cursor after a full circuit (trap 1).
- Call it from
runSweep after context-graph enumeration and before resumeRotation, sharing a per-sweep memoised metaGraphsFor(cgId) so each CG's meta list is fetched once for both passes.
- Leave
rotationPending and metaGraphResumeCursor untouched — they remain correct for measurement.
Test strategy
Fairness and coverage are complementary; a flat cursor can pass either while failing the other — which is exactly what produced the 15/30 ceiling twice.
- Coverage — every task in every meta graph in every context graph is eventually examined (the 30/30 assertion above).
- Fairness — more preserved tasks than
maxCandidatesPerSweep followed by one cleanable task; assert the cleanable task is cleared, not merely examined. A traversal can reach it and still not clear it. Needs real head/VM/SWM state rather than a synthetic store.
Current state
Within-meta-graph starvation is fixed: 12/12 distinct over 3 sweeps where the pre-fix build reached 4/12. No test in the suite asserts cross-level fairness, so nothing green contradicts this limitation — the rotation suite seeds backlog-only markers that discovery can never select, so deletion is never exercised there.
Follow-up from #1996 (merged at
ab63da7a1). Known limitation, documented there rather than silently shipped.Summary
The finalized-SWM GC uses one traversal for two jobs with opposite requirements.
Because deletion rides the measurement rotation, the node-wide deletion budget always lands at the front of the traversal, and tasks behind the front are never examined. #1996 fixed this within a meta graph (keyset cursor,
650174547); it remains open across meta graphs and context graphs.Independently raised as a 🔴 by the review bot from a cold read of the pushed head, with a suggested direction (per-meta-graph pagination) matching the analysis below.
Why it matters in practice
A task is only re-selected forever if it can never be cleared. On a receiver whose SWM synced cleanly the task clears and leaves the set, so a healthy node behaves correctly. The failure is progressive: unclearable tasks accumulate at the front of the traversal, and deletion throughput degrades as they do. It does not recover without a process restart, because the cursor is in-memory.
This interacts with #2021 and #2022: a peer can deliberately seed unclearable tasks, and on Blazegraph the retirement path that would remove them is inert.
Evidence
Fixture: 2 context graphs × 3 meta graphs × 5 tasks = 30 tasks,
maxCandidatesPerSweep: 2, 40 slices, all tasks permanently non-deleting.30/30 is reachable. Hitting 15/30 twice is the signal that this is not patchable level-by-level: each level fixed exposes the next, because the defect is the shared traversal rather than a missing cursor.
This is the fourth instance of one bug class in this subsystem — no cursor, restart at the beginning, successors never reached — after the context-graph rotation, the meta-graph walk, and task selection, all three fixed in #1996.
Why the permanently-non-deleting case is reachable
finalization-handler.tssetslayerVerification = vmVerificationwhen VM verification succeeds, so the cleanup marker is armed from VM state and the live SWM is never inspected at arming time. A divergent non-empty SWM then failsclearIfStillExact's SWM check, returnspreserved, and the task is never retired — while still being charged to the budget on every sweep.Two traps that re-created the bug class while fixing it
Both looked obviously correct when written, and each re-introduced the exact starvation being fixed:
The spec: three coupled contracts any fix must satisfy together
yieldRotation, so the measurement cursor is never persisted.067e06f1e. Fixing (2) breaks (3) — the halt reason must be carried into the sweep result, not returned early and not dropped.Fixing 1 and 2 together took the suite from 2 red to 4 red. Two of the newly-red were pressure-classification pins added in #1996 itself, and one was
rotates a head that repeatedly cannot be finished so the tail is still servedreturning['cg-slow']— i.e. the measurement starvation guard fixed earlier in #1996 regressed. Any future design is constrained by that test.Rebuild notes
The exploratory commit was not merged, so:
{ contextGraphId, swmMetaGraph, afterTaskSubject? } | null.runDeletionPasswalking from that cursor: outer loop over context graphs wrapping modulo length, inner loop from the resume index to the end of that CG only, no wrap. Park the cursor at{cg, metaGraph}before eachcleanupMetaGraphcall so a mid-page yield resumes there; setafterTaskSubjectfromlastTaskSubjectonly when the page was full.runSweepafter context-graph enumeration and beforeresumeRotation, sharing a per-sweep memoisedmetaGraphsFor(cgId)so each CG's meta list is fetched once for both passes.rotationPendingandmetaGraphResumeCursoruntouched — they remain correct for measurement.Test strategy
Fairness and coverage are complementary; a flat cursor can pass either while failing the other — which is exactly what produced the 15/30 ceiling twice.
maxCandidatesPerSweepfollowed by one cleanable task; assert the cleanable task is cleared, not merely examined. A traversal can reach it and still not clear it. Needs real head/VM/SWM state rather than a synthetic store.Current state
Within-meta-graph starvation is fixed: 12/12 distinct over 3 sweeps where the pre-fix build reached 4/12. No test in the suite asserts cross-level fairness, so nothing green contradicts this limitation — the rotation suite seeds backlog-only markers that discovery can never select, so deletion is never exercised there.