Skip to content

Finalized-SWM GC: deletion traversal starves across meta graphs and context graphs #2020

Description

@Jurij89

Follow-up from #1996 (merged at ab63da7a1). Known limitation, documented there rather than silently shipped.

Summary

The finalized-SWM GC uses one traversal for two jobs with opposite requirements.

  • Measurement must cover every context graph for the backlog total to be a whole-node figure.
  • Deletion must resume exactly where its budget ran out and ignore rotation shape entirely.

Because deletion rides the measurement rotation, the node-wide deletion budget always lands at the front of the traversal, and tasks behind the front are never examined. #1996 fixed this within a meta graph (keyset cursor, 650174547); it remains open across meta graphs and context graphs.

Independently raised as a 🔴 by the review bot from a cold read of the pushed head, with a suggested direction (per-meta-graph pagination) matching the analysis below.

Why it matters in practice

A task is only re-selected forever if it can never be cleared. On a receiver whose SWM synced cleanly the task clears and leaves the set, so a healthy node behaves correctly. The failure is progressive: unclearable tasks accumulate at the front of the traversal, and deletion throughput degrades as they do. It does not recover without a process restart, because the cursor is in-memory.

This interacts with #2021 and #2022: a peer can deliberately seed unclearable tasks, and on Blazegraph the retirement path that would remove them is inert.

Evidence

Fixture: 2 context graphs × 3 meta graphs × 5 tasks = 30 tasks, maxCandidatesPerSweep: 2, 40 slices, all tasks permanently non-deleting.

build distinct tasks ever examined
as shipped 8 / 30
+ meta cursor advancing on budget exhaustion 15 / 30
+ rotation restarting where discovery stopped 15 / 30 (inert)
separated deletion traversal (flat cursor) 30 / 30, first 30 all-distinct

30/30 is reachable. Hitting 15/30 twice is the signal that this is not patchable level-by-level: each level fixed exposes the next, because the defect is the shared traversal rather than a missing cursor.

This is the fourth instance of one bug class in this subsystem — no cursor, restart at the beginning, successors never reached — after the context-graph rotation, the meta-graph walk, and task selection, all three fixed in #1996.

Why the permanently-non-deleting case is reachable

finalization-handler.ts sets layerVerification = vmVerification when VM verification succeeds, so the cleanup marker is armed from VM state and the live SWM is never inspected at arming time. A divergent non-empty SWM then fails clearIfStillExact's SWM check, returns preserved, and the task is never retired — while still being charged to the budget on every sweep.

Two traps that re-created the bug class while fixing it

Both looked obviously correct when written, and each re-introduced the exact starvation being fixed:

  1. Resetting the cursor after a full circuit. "Everything is drained, start from the top" is the head-of-line behaviour — on a node whose leading tasks are permanently unactionable, every sweep re-walks them.
  2. Wrapping the inner loop modulo one context graph's meta graphs. Cycles that context graph forever whenever it always has examinable tasks; the graphs behind it are never reached. Only the outermost loop may wrap.

The spec: three coupled contracts any fix must satisfy together

  1. Wall-clock split. Deletion-first starves the backlog metric under load (re-opens the "counters blank exactly when deferral matters" problem by another door). Measurement-first starves deletion, which is the bug being fixed. A fixed, non-configurable half-slice split works; a tunable ratio is a knob nobody can set correctly without knowing both costs.
  2. Rotation cursor persistence. Returning early on a deletion halt skips yieldRotation, so the measurement cursor is never persisted.
  3. Pressure classification. Moving candidate discovery into the deletion pass means a scheduler rejection raised there no longer reaches the retryable-deferral classification added in 067e06f1e. Fixing (2) breaks (3) — the halt reason must be carried into the sweep result, not returned early and not dropped.

Fixing 1 and 2 together took the suite from 2 red to 4 red. Two of the newly-red were pressure-classification pins added in #1996 itself, and one was rotates a head that repeatedly cannot be finished so the tail is still served returning ['cg-slow'] — i.e. the measurement starvation guard fixed earlier in #1996 regressed. Any future design is constrained by that test.

Rebuild notes

The exploratory commit was not merged, so:

  • Replace the per-meta-graph task cursor with a flat field: { contextGraphId, swmMetaGraph, afterTaskSubject? } | null.
  • Add a runDeletionPass walking from that cursor: outer loop over context graphs wrapping modulo length, inner loop from the resume index to the end of that CG only, no wrap. Park the cursor at {cg, metaGraph} before each cleanupMetaGraph call so a mid-page yield resumes there; set afterTaskSubject from lastTaskSubject only when the page was full.
  • Do not reset the cursor after a full circuit (trap 1).
  • Call it from runSweep after context-graph enumeration and before resumeRotation, sharing a per-sweep memoised metaGraphsFor(cgId) so each CG's meta list is fetched once for both passes.
  • Leave rotationPending and metaGraphResumeCursor untouched — they remain correct for measurement.

Test strategy

Fairness and coverage are complementary; a flat cursor can pass either while failing the other — which is exactly what produced the 15/30 ceiling twice.

  • Coverage — every task in every meta graph in every context graph is eventually examined (the 30/30 assertion above).
  • Fairness — more preserved tasks than maxCandidatesPerSweep followed by one cleanable task; assert the cleanable task is cleared, not merely examined. A traversal can reach it and still not clear it. Needs real head/VM/SWM state rather than a synthetic store.

Current state

Within-meta-graph starvation is fixed: 12/12 distinct over 3 sweeps where the pre-fix build reached 4/12. No test in the suite asserts cross-level fairness, so nothing green contradicts this limitation — the rotation suite seeds backlog-only markers that discovery can never select, so deletion is never exercised there.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions