Skip to content

feat(storage): add the live managed materialization boundary (#2052 Stack B2) - #2110

Open
Jurij89 wants to merge 27 commits into
feat/2052-system-record-corefrom
feat/2052-system-record-materializer
Open

feat(storage): add the live managed materialization boundary (#2052 Stack B2)#2110
Jurij89 wants to merge 27 commits into
feat/2052-system-record-corefrom
feat/2052-system-record-materializer

Conversation

@Jurij89

@Jurij89 Jurij89 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Stack B2 of #2052: the live managed materialization boundary. Default-unused — no producer, provider, requester, protocol handler, advertisement or activation is registered here, and nothing in this PR performs store, epoch, queue, permit or scheduler work unless a daemon supervisor hands over a live ownership lease.

Stacked on #2103 (feat/2052-system-record-core), which is still open. Base will move to testnet-canary once #2103 merges.

This PR does not close #2052. Per the plan's ownership table, closure needs Stacks C–F plus the loaded r26/r27 regression.

Scope note — read this first

B2 as specified in the Revision 17 plan is very large. This PR delivers the ownership, capability and lifecycle boundary plus its live evidence, and deliberately stops short of the full-state CAS transaction. Concretely:

  • SparqlHttpStore.executeSystemRecordApply() currently returns a typed deferred and dispatches nothing. The lane can be discovered, opened, sealed, disabled and shut down; it cannot yet apply a record.
  • That is fail-closed by construction rather than a stub that silently no-ops: no unproven write is ever dispatched.
  • The composition is live: with the supervisor handoff wired, the real daemon store advertises the lane, and a session opened against a proven-ownership generation restarts the owned child under a real scheduler barrier. Only the apply itself defers.

The verified-replacement command construction, bounded inspectAppliedState, and the exact prior/next subject-union transaction are the next PR in the stack. Splitting here keeps both halves reviewable; the alternative was one PR nobody could review carefully.

How it fits together

Capability travels by identity, not by config. Nothing below is reachable from a persisted file.

flowchart LR
    subgraph daemon["CLI daemon"]
        SUP["startOxigraphServer\n(lifecycle mutex + state machine)"]
        CHILD["owned oxigraph child\nPID + listen socket"]
        SUP -->|"spawn / SIGTERM,\nprove listener owner"| CHILD
    end

    subgraph storage["storage package"]
        CTRL["ownership controller\n(module-private WeakMap)"]
        LEASE(["lease handle\ndata-less, unique symbol"])
        ADAPTER["SparqlHttpStore"]
        LANE["system-record lane\nsession state machine"]
        SCHED["StorePriorityScheduler\ncontrol barrier"]
        POOL["OwnedManagedHttpClient\n(node:http.Agent)"]
    end

    SUP --- CTRL
    CTRL -->|"mints"| LEASE
    LEASE -->|"symbol-keyed option"| ADAPTER
    SUP -->|"supervisorHandoff"| ADAPTER
    ADAPTER -->|"advertises IFF lease live\nAND handoff present\nAND snapshot ready"| LANE
    LANE -->|"every lifecycle transition"| SCHED
    LANE -->|"apply (deferred in this PR)"| POOL
    POOL --> CHILD
    ADAPTER --> CHILD
Loading

The lease carries no data at all — its meaning lives in a module-private WeakMap keyed by identity — and travels under a unique symbol, which object spread copies (so it survives resolveAdapterOptions' rewrite) but JSON cannot produce or round-trip. Mutation authority is a separate controller the supervisor never hands out.

The generation handoff, as it now runs

sequenceDiagram
    participant Q as Ordinary store traffic
    participant S as StorePriorityScheduler
    participant L as System-record lane
    participant A as SparqlHttpStore
    participant O as Oxigraph supervisor
    participant X as Owned child

    Q->>S: run() — admitted, in flight
    L->>S: runControlBarrier(store, "system-record.enable", gen)
    S->>S: seal store; HOLD new tagged run() off-queue
    Q-->>S: in-flight work completes
    S->>S: quiesced (untagged 0, tagged-for-store 0)
    S->>L: transition starts in the reserved controller slot
    L->>A: destroyClient() — retire the owned pool
    L->>O: stopAndProveOwnedChildDead()
    O-->>X: SIGTERM, then prove the PORT released
    L->>A: awaitRetiredWork() — drain the retired pool
    L->>O: startAndProveCleanGeneration()
    O->>X: spawn; bind generation ONLY once the child is the proven listener
    L->>A: rotateMaterializationEpoch()
    L->>S: transition returns; seal commits
    S-->>Q: held work resumes against the replacement generation
Loading

The order is load-bearing and asserted step by step, not merely as a set: a replacement started before the old child is proven dead would let a request issued over the retired pool reach the new listener.

What lands

Reserved graph policy (internal-graph-policy.ts). B2's persistent graphs deliberately reuse the urn:dkg:internal:atomic-graph-replace: prefix, because all nine existing filter sites already hide it — predecessors get the downgrade property with no change on their side. That makes "internal" mean two things with opposite lifetimes, so reserved names match exactly, staging names match a canonical UUID shape, and generic mutation refuses everything else in the namespace. The module imports nothing from the system-record barrel because index.ts is on every process's startup path; a test imports the barrel and asserts the literals agree.

Unforgeable ownership lease (managed-oxigraph-ownership-v1-internal.ts). Every signal available today is forgeable from persisted config: managedByDkg is operator-writable and resolveAdapterOptions actively rewrites it to false, while atomicUpdates is synthesized as true by that same function. Hence the data-less, symbol-transported handle described above.

Lane session state machine (system-record-materializer-v1.ts). Owns lifecycle and policy but no bytes; the transaction is performed by an injected executor, which is what makes the plan's pure reference-model fault matrix possible without a live child. Every lifecycle transition runs inside a scheduler control barrier, supplied as a required dependency.

Always-on mutation denial. No always-on guard existed — the changelog's is gated on enabled, which is default-off. The guard now lives on the adapter, the only always-on choke point, and is checked before the atomicUpdates capability refusal since that flag is synthesized from config.

Owned connection pool (managed-http-client.ts). SparqlHttpStore dispatches through global fetch, whose sockets belong to Node's process-wide dispatcher, so "destroy the retired generation's client" was unimplementable. node:http.Agent gives the managed lane a pool it owns; undici is not a dependency here and does not resolve.

Explicit per-decorator capability discovery, in all four places — the two storage decorators, the changelog, and the agent's hand-rolled forwarder. Structural .innerStore walking would reach past a wrapper whose cache the lane must invalidate, and past the changelog's denial.

Scheduler admission (store-priority-scheduler.ts): generation permits, ordering domains, and one coalesced reserved control barrier — all strictly opt-in, with the existing run() signature unchanged and every pre-existing test passing unmodified. The hard constraint was the old behaviour: this scheduler is process-global and sits on every store operation, and its admission decision is deliberately entry-independent (canStart reads four integers and never inspects a queued entry). The entire extension therefore hangs off one integer comparison — while taggedQueuedCount is zero, nextRunnable() takes the original unconditional queue.shift(). That fast path has a witness rather than a comment: admissionEvaluations increments before the untagged early return, so it rises if selection ever walks entries the guard should have kept it away from, and a test asserts it stays at zero.

Managed child ownership (daemon/oxigraph-server.ts, oxigraph-managed.ts): one lifecycle mutex and explicit state machine replacing three unlocked closure booleans; a generation bound only after the spawned child is the proven listener owner, never at spawn. Three pre-existing defects fixed on the way — killSync() poisoned stop() so it never escalated to SIGKILL; the auto-revive timer handle was never stored and so could not be cancelled; and stop() resolved on the child's exit event without proving the port was released. A bind still served after the child exited now burns the lease terminal (port-release-unproven) rather than letting a replacement bind over a listener we do not own. Kills still go only through the tracked ChildProcess, so a foreign listener is never signalled.

Live conformance gate (devnet/issue-2052-managed-ownership/, .github/workflows/system-record-managed-ownership.yml, pnpm test:live:system-record-managed-ownership). Drives one real generation handoff end to end plus the capability, socket-ownership and predecessor matrices — see below.

Two defects this work caught

  1. ChangelogStore silently swallowed reserved writes. Its insert()/delete() strip reserved-graph quads rather than refusing them, so with the changelog enabled a reserved write returned a resolved promise for a write that never happened — a lost update with no signal. Stripping stays correct for the changelog plane and ephemeral staging graphs; persistent state now denies.

  2. The live gate failed on a real socket leak in this PR's own code. OwnedManagedHttpClient reported one live socket after destroyAndSettle(), because Agent.destroy() tears sockets down asynchronously. Returning there would have let a replacement child bind while a retired keep-alive socket was still open — precisely the stale-generation window the design exists to close. It now polls to zero under a bounded deadline and treats failure as terminal.

Merge-readiness review — all six findings addressed

Reviewed at b1c1bdf6d; fixes at d8b2a2d02. Every finding was validated against the code before being fixed, and every fix is mutation-proven rather than merely green.

# Finding Fix Kill proof
1 BLOCKER — the live handoff bypassed the scheduler control barrier barrier is a required dep on the lane; enable/disable/shutdown each open a named section; the adapter supplies one backed by runControlBarrier adapter barrier → pass-through reproduces the reviewer's exact symptom; the live gate fails 3 of 20 checks
2 BLOCKER — a generation-raced success returned indeterminate without sealing the final outcome is derived first and the seal keyed on it; attribution widened from a generation-string compare to the whole ownership snapshot regression test: readiness lost with the generation string unchanged must still be indeterminate
3 HIGH — concurrent shutdowns each ran a teardown runShutdown is no longer async, so this.transition is assigned synchronously before any await removing the synchronous assignment reddens all three shutdown tests
4 HIGH — unknown names in the internal namespace were writable but hidden generic mutation refuses the whole namespace except canonical UUID staging graphs near-miss and future-name negative tests
5 MEDIUM — the live gate did not exercise what it claimed the gate now drives the real supervisor, child and barrier; the workflow header enumerates what is and is not proven see below
6 LOW — source encoding damage repaired to clean UTF-8, verified byte-identical to base on pre-existing lines git diff against base

Finding 1 in detail

runControlBarrier and sealStoreGeneration shipped in this stack exported, documented and covered by 25 unit tests, with zero production callers. The lane stopped the owned child, asserted its port free and bound a replacement while ordinary requests were still in flight on the retired generation.

The materializer stays store-agnostic — it takes an exclusive-section runner, not a scheduler — and the adapter supplies one using this as the opaque store identity, re-reading the lease snapshot's generation per transition rather than capturing the one observed at construction (the lane outlives any single child).

Shutdown under a barrier that cannot be acquired does not run the teardown, and that is the safe side: the lane still reaches terminal so nothing writes through it again, while the child is left alive under the supervisor that still owns it and stops it at process exit. Stopping the child outside a section is the hazard; "could not quiesce" is not a reason to do it anyway. The error propagates, because a shutdown that could not quiesce the store is not a clean one.

Unit tests bracket each transition and assert every child-touching step ran inside the section — but they inject a pass-through stand-in, so they cannot show that asking for a section does anything. All 41 of them stay green under the pass-through mutant. system-record-control-barrier-integration-v1.test.ts is the one that fails: real adapter, real process-global scheduler, real handoff composition, genuinely held-open ordinary request.

The live gate now discriminates

The gate previously minted ownership by hand and injected no-op supervisor methods, so it never restarted a child, never opened a lane and never ran a barrier — which is exactly why finding 1 passed it. Of the two directions the review offered (narrow the claims, or drive the real thing), this takes the second: narrowing alone would have left the gate structurally unable to catch a regression of the defect it is named after.

It now runs one real handoff first — startOxigraphServer owns and spawns the child, its lease and supervisorHandoff travel to the adapter under the symbol key exactly as the daemon composes them — against a 60,000-quad ordinary insert as the probe workload (one scheduler admission, one HTTP request over the store's own path). Ten measured checks: the write was still running when the lane opened; zero store requests inflight at the instant the child was stopped; the write survived; the generation advanced; a second OS process was spawned, the first PID is dead and the last alive; the replacement serves the same 60,000 quads; the store resumed against it; the lane reports enabled.

Proven to fail, not merely to pass. Built with the adapter's barrier replaced by a pass-through and re-run end to end:

childStoppedOnlyAfterOrdinaryWorkDrained: 1 store request(s) inflight
inflightOrdinaryWriteSurvivedTheHandoff: fetch failed
replacementServesTheSameData: 0/0 quads
-> FAILED (3 of 20 checks)

With the barrier restored: PASS: 20 checks, 3 predecessor entries. That negative control was not constructed — the first run of the new section hit a stale dist/ predating the fix and reproduced the reviewer's finding unprompted.

One measurement was reworked rather than tuned. The first shape compared wall-clock "stop happened after the write settled", which is subtly wrong: the scheduler releases admission when the store work resolves, and the caller's promise settles a few microtasks later, so a correct handoff measured as 0.09 ms too early. The pre-fix build read −49 ms, so any tolerance would have been chosen to sit between two numbers rather than to mean anything. Reading the scheduler's own inflight count at the instant of the stop has no such boundary — it is the quantity the barrier waits on.

Correction to an earlier revision of this description

A previous revision claimed a barrier asymmetry — a generation-scoped drain against a generation-blind hold — and warned that a future fix must be an allow-list. Re-read at this head, that description was wrong: isBarrierReady() waits on this store's whole taggedInflight, and the hold is seals > 0. Both are generation-blind, deliberately, because the transition is a child-process restart and the child it stops serves every generation. runningPermits feeds admissionGenerationsInflight and the timeout blocker report; the seal's generation is a diagnostic label, not an enforcement key. Narrowing either to the sealed generation would be strictly weaker and — generations being decimal counters — trivially bypassable. The scheduler's own docstring said "waits for the sealed generation's execution permits to drain", which is where the claim came from; it is corrected in this PR.

The related re-entry note also needs restating now that the barrier has production callers: a transition still must not re-enter run(), and the three that exist do not — they make supervisor calls, drain the adapter's owned pool, and invalidate a cache synchronously. The bound remains in place for the caller who gets it wrong.

Independent review — four earlier tracks

Every finding below was raised by an independent reviewer working read-only in its own worktree pinned to a specific commit, and each is fixed in this branch.

Protocol/safety. Two BLOCKERs, both demonstrated by an executed exploit rather than argued: shutdown superseded nothing while a transition was in flight, so a stalled enable resumed afterwards and started a replacement child after shutdown had proved the old one dead; and an open() that joined a shutdown never re-checked terminal state, so a shut-down lane revived and dispatched a record write to the executor. Both fixed and mutation-proven. The same track attempted eight forgery attacks on the ownership lease — structuredClone, spread, Object.assign, JSON.parse reviver, same-prototype, Proxy, and two more — and all were rejected; dual-package and worker-thread cases fail closed.

Execution/QA. The gate reported "PASS: 18 checks" while 13 were assertions over constants the generator hardcoded to zero. Deleted; it then reported 10, all measured, and now reports 20. leakedOwnedSockets was 0-or-throw by construction and is now a before/after pair. The manifest carried a fabricated commit SHA published as CI evidence — fixed, with a git cat-file -e check proven to catch it. Mutation-testing that fix surfaced a further hole: a crashed run left the previous PASS artifact verifiable, so run.ts now clears artifacts first.

Resource/operations. destroyAndSettle awaited before destroying, so its timeoutMs did not bound the phase it named (measured 3831 ms; no return at all within 8 s against a dribbling server) — now 5 ms. The negative capability memo latched permanently in 3 of 4 layers, so one probe inside a 1 s restart would have disabled the lane for the process lifetime. awaitRetiredWork was a structural no-op. A control barrier did not stop untagged traffic (20 ops started mid-transition → 0), and held calls were unbounded, rejecting 190/200 callers after they had waited out the seal → 0.

Supervisor handoff (036efc6b3). All remediated:

  • BLOCKERchild_process.spawn throws synchronously for EACCES/EFTYPE/E2BIG/EINVAL. The .once('error') handler catches only async failures, so the throw escaped startCleanGenerationLocked and bypassed the only code that clears handoffPhase and re-arms backoff. Measured: 30 backoff periods later, still one spawn, port unserved, self-heal refusing — and terminal: false, so the lease reported "momentarily not ready" indefinitely rather than failing closed.
  • HIGH — the port-release proof accepted a probe timeout as evidence of release. endpointAnswers() collapsed ECONNREFUSED (real evidence) and TimeoutError (no evidence) into one false.
  • HIGH — nothing bounded the retired phase, and runShutdown() reached the first half without the second, parking the supervisor there indefinitely.
  • MAJOR — a failed retire left the supervisor open and childless while ordinary store traffic still targeted the listener it could not account for; and the handoffPhase === 'retired' clause in scheduleRevive was inert (deleting it changed 0 of 57 test statuses).

The inert guard produced a standard now applied to this branch: every new guard ships with a test that fails when only that guard is removed. It was removed rather than papered over, and a second proposed guard was declined for the same reason. Later in the branch a shutdownWork coalescing field was added, its mutant survived twice, and it was deleted on the same standard.

One proposed fix was rejected as worse than the bug. The review (and I) initially proposed making a teardown-retire terminal for the supervisor. That would have been a serious regression: the managed child is the daemon's entire triple store, so disabling the system-record lane would have permanently killed node storage until restart. The phase is bounded instead — fail-closed for the lane, alive for every other consumer.

Measured, not asserted

Default-path cost of the scheduler admission work, re-measured independently after the fix:

variant ns/op (median, 200k untagged ops)
base 3530.4
this PR (pre-fix) 3462.8
this PR (post-fix) 3462.9

+0.003% versus pre-fix, against ±10–20% within-variant spread — three orders of magnitude below the noise floor. admissionEvaluations = 0 on a pure untagged workload held across every round.

One number in an earlier revision of this description was wrong and is corrected here: a claimed heap-delta improvement of 19.2 → 13.1 MB was a GC-timing artifact (heapUsed sampled without forcing GC), not allocation volume. Re-measured as GC count the reduction is real and stable (307 → 260 → 223) but pause time is identical across all three, so the practical benefit is nil.

Known deferral

After a retire whose port-release proof fails, the supervisor terminates and cannot spawn again — but ordinary store traffic never consults the ownership lease, so the daemon keeps issuing SPARQL to an endpoint served by a process it could not account for. A FATAL log names the condition and tells the operator to restart.

Closing this properly needs an adapter-side kill switch, deliberately not in this PR: refusing all store traffic is a large change to the daemon's hot path and deserves its own review rather than riding in on a default-unused change.

A failed retire has three causes, and they want different answers:

  • (a) something genuinely serves the bind — a leaked descendant of our own child, so the same RocksDB, and writes still land in the right store;
  • (b) a foreign process took the port — the dangerous case;
  • (c) the probes were merely inconclusive while nothing was on the port at all.

(c) is a direct consequence of the HIGH fix above. Once "no evidence" stops counting as "evidence of absence", an inconclusive probe is no longer treated as release. A closed loopback port normally RSTs immediately, so this needs the connect or its abort to lose a >1.5 s race under event-loop starvation — and AbortSignal.timeout fires from that same starved loop, so starvation often self-cancels. Plausible rather than likely. But #2052 is a store-pressure issue, so the one condition that produces (c) is the condition this feature exists for, and in (c) the FATAL-log-and-restart guidance is telling an operator to restart a node whose port was fine.

The coupling underneath it

A failed retire leaves the daemon storeless regardless of any kill switch, and closing the supervisor is not what causes it: bindReadyGeneration() throws on a terminal lease, so once the lease is terminal no generation can ever bind again. Without the close the supervisor is equally childless, merely less honest about it.

That is the same coupling rejected above, surfacing elsewhere: the lease is a lane capability, the child is the daemon's store, and today they share a fate. Burning the lease on an unproven release is correct and should stay irreversible; killing the daemon's storage as a side effect of it is not.

Attempting a revive after a failed retire would in fact be safe — if something really is on the port, the respawned child dies on EADDRINUSE and the ownership proof refuses to adopt the listener, which is an existing, tested guard. So the fix is decoupling lease-terminality from supervisor-liveness, not weakening any guard. The kill switch alone leaves the node storeless; the decoupling alone leaves bad traffic flowing. They want fixing together, and that is a larger design change than this PR.

Test plan

pnpm --filter @origintrail-official/dkg-storage exec vitest run \
  test/internal-graph-policy.test.ts \
  test/managed-oxigraph-ownership-v1.test.ts \
  test/reserved-internal-graph-mutation-guard.test.ts \
  test/system-record-materializer-lifecycle-v1.test.ts \
  test/system-record-control-barrier-integration-v1.test.ts \
  test/system-record-capability-discovery-v1.test.ts \
  test/store-priority-scheduler.test.ts \
  test/store-scheduler-system-record-admission.test.ts

pnpm --filter @origintrail-official/dkg-storage build
pnpm --filter @origintrail-official/dkg-core test
pnpm test:live:system-record-managed-ownership

Guards are mutation-proven rather than merely green. Beyond the barrier proofs above: forcing the lease identity check to true kills exactly the two identity-forgery tests; relaxing the reserved-name check to a prefix match kills the near-miss test. Every mutant was verified applied by grep and then reverted.

⚠️ Full CI is NOT running on this PR yet — do not read the green ticks as full validation

.github/workflows/ci.yml triggers on pull_request: branches: [main, testnet-canary, release/rc.12, rc17-vm-wip]. This PR is stacked on feat/2052-system-record-core, which is not in that list, so the root pnpm test lanes do not run here. The checks currently green are only the path-filtered workflows: protocol evidence, SPARQL scalability lint, the managed-ownership live gate, and the RFC-64 Windows gate.

That is expected for a stacked PR and not a defect, but it is a merge gate, not a footnote:

  • Do not merge this while it is based on feat/2052-system-record-core.
  • Once feat(core): add bounded system-record V1 contracts #2103 merges, rebase only the B2 commits onto updated testnet-canary, inspect git range-diff, change this PR's base to testnet-canary, and let the full CI lanes run.
  • Root pnpm test must be green there before merge, per the Revision 17 plan's Full Validation Gate.

Known environment caveats

  • packages/cli/test/oxigraph-*.test.ts is effectively POSIX-only. On Windows it fails for environmental reasons (spawn EFTYPE on the extension-less Node stand-in; No prebuilt Oxigraph 0.5.8 binary for freebsd-x64, which those tests force deliberately; one systemd/uid-only case). Verified on Linux CI only.
  • The plan's own CLI gate command does not work as written. pnpm --filter @origintrail-official/dkg exec vitest run test/oxigraph-managed.test.ts test/oxigraph-server.test.ts reports "No test files found" with two files but runs fine with one; npx vitest from packages/cli runs both. Flagging rather than silently substituting.
  • packages/storage full-suite failures are load-dependent on Windows, always confined to the embedded oxigraph-worker path, which V1 leaves legacy. Established by measurement: the same files fail on main with none of this stack present (8 of 11 red in oxigraph-worker-resilience.test.ts), and they fail worse in isolation than in the full run — 5 s test timeouts against ~70 s of import cost. Linux CI is the gate.
  • The CLI ownership work was likewise A/B-verified: control 16 failed / 24 passed, with changes 16 failed / 36 passed — identical failure set, 12 new passing tests, zero regressions.
  • packages/core/test/system-record-golden-v1 fails on a Windows CRLF checkout (core.autocrlf=true, no .gitattributes) because the generator byte-compares against LF. Not a code defect; Linux CI is green.

Related

🤖 Generated with Claude Code

Jurij89 and others added 5 commits August 6, 2026 08:53
…h ownership lease

Stack B2 of #2052 needs two primitives before any materialization code can be
written safely. Both are default-unused: nothing constructs or consumes them yet.

internal-graph-policy.ts distinguishes the first PERSISTENT graphs under the
`urn:dkg:internal:atomic-graph-replace:` prefix from the ephemeral per-operation
staging graphs that prefix used to hold exclusively. Reusing the prefix is
deliberate: every supported predecessor already filters it out of listGraphs(),
hasGraph(), the graph-set index, the changelog and the sync responder, so an
older binary neither enumerates nor serves the new reserved state without any
change on its side. The reserved names are matched EXACTLY and the ephemeral
names by canonical UUID shape, so a staging sweep can never classify durable
state as garbage and an unrecognised internal name is hidden but neither
sweepable nor writable. The module stays dependency-free because index.ts is on
every process's startup path and B2 must add zero default-off cost; a test
imports the frozen system-record-v1 barrel and asserts the literals agree.

managed-oxigraph-ownership-v1-internal.ts gates capability on a live lease that
only the supervisor can mint. Every signal available today is forgeable from
persisted config: `managedByDkg` is operator-writable and resolveAdapterOptions
actively rewrites it to false, while `atomicUpdates` is synthesized as true by
that same function. The lease handle instead carries no data at all -- its
meaning lives in a module-private WeakMap keyed by identity -- and travels under
a unique symbol, which object spread copies (surviving the factory rewrite) but
JSON cannot produce or round-trip. Mutation authority is a separate controller
the supervisor never hands out, so holding a lease lets a store ask whether it
is live, never assert that it is.

Both guards are mutation-proven: forcing the lease identity check true kills the
two identity-forgery tests, and relaxing the reserved-name check to a prefix
match kills the near-miss test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…state

The reserved graphs added in the previous commit are hidden from every
enumeration surface, but nothing stopped a caller from writing to them by
hardcoded IRI. Reserved state is writable only through the materializer's
structured, generation-bound command path, which derives its own scope; a
generic write would bypass the full-state CAS, the capacity accounting and the
materialization epoch.

The guard lives on SparqlHttpStore rather than on a decorator because every
decorator is optional -- the changelog defaults off and the graph-set index and
blob store are conditional -- while all of them delegate downward, so the
adapter is the only always-on choke point for the managed endpoint that holds
reserved state. It is checked before the atomicUpdates capability refusal,
since resolveAdapterOptions synthesizes that flag from plain config.

ChangelogStore needed a separate fix that the new test caught: its insert() and
delete() STRIP reserved-graph quads rather than refusing them, so an enabled
changelog turned a reserved write into a silently resolved no-op -- a lost
update with no signal. Stripping stays correct for the changelog plane and for
ephemeral staging graphs, which are decorator-internal, but persistent state now
denies.

update() is deliberately left unguarded: its argument is opaque SPARQL, and
scanning it for reserved IRIs is exactly the evadable check that
assertNoReservedRef already documents as insufficient. Opaque updates rotate the
materialization epoch instead, and Stack C migrates the raw callers.

The pass-through cases stub fetch rather than dialing an unbound port. An
earlier revision dialed for real and measurably destabilized the worker-thread
suites; the stub is both faster and a stronger claim, asserting the request was
actually issued. Remaining full-suite failures are confined to the embedded
oxigraph-worker adapter, pass 3/3 in isolation, and vary run to run (2-9
observed) with and without these changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane controller owns lifecycle and policy but deliberately owns no bytes:
the store transaction is performed by an injected transaction executor that the
mandatory sparql-http adapter will supply. That split is what lets the logical
fault matrix run as a pure reference model -- a test executor can fail at any
boundary without a live child -- while keeping one implementation of the state
machine. Nothing here is inert: the session is fully functional against any
executor, including the pure one the tests use.

The enable path asserts a physical fact rather than a logical one. It destroys
the client, proves the owned child and its port dead, drains retired work, and
only then starts a clean generation and rotates the epoch; the tests assert the
ORDER, not merely the set, because a replacement started before the old child
is proven dead reopens exactly the stale-generation window this design exists to
close. Any handoff failure is terminal `unavailable` rather than a retry: if we
cannot prove the old writer is gone, retrying does not make it so, and allowing
legacy work to bypass while that uncertainty stands is the forbidden fallback.

Every applyVerified re-reads the lease instead of trusting the state captured at
open, because a child exit revokes it with no notification. Two cases the tests
pin: an indeterminate dispatch seals the lane into `reconciling` so no further
work is admitted against a generation whose last write may or may not have
committed; and a success whose child generation changed UNDER the dispatch is
downgraded to indeterminate, since that response cannot be attributed to the
child we addressed. A `stale` outcome is deliberately NOT downgraded -- it means
the CAS did not match and carries no uncertainty about bytes written.

The process-global single-registration invariant is enforced before any
capability is exposed: the store scheduler has no store identity, so two managed
controllers could not be told apart during a control barrier.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ator capability

Capability discovery is explicit at every layer rather than structural. A
resolver that walked .inner/.innerStore the way asGraphWriteGenSource does would
reach PAST a wrapper whose cache the lane must invalidate, and past the
changelog's denial, handing out a controller that silently desynchronises an
index. So each decorator forwards the getter itself, in all FOUR places: the two
storage decorators, the changelog, and the agent's hand-rolled forwarder, which
rebuilds a fresh object literal and would otherwise drop it silently (the same
failure already documented there for replaceGraph).

Advertising is gated on three independent facts, all fail-closed: a live lease,
a supervisor handoff, and non-terminal ownership. The handoff requirement is the
correction of a real design error caught while wiring this: the adapter owns the
connection pool but cannot assert PROCESS facts (child exited, port released,
replacement is the proven listener) because only the supervisor holds the
ChildProcess. Without it the adapter would have advertised a lane that could
never open. A lease alone is therefore valid but deliberately silent.

managed-http-client.ts gives the lane a pool it actually OWNS. SparqlHttpStore
dispatches through global fetch, whose sockets belong to Node's process-wide
dispatcher and are shared with every other consumer, so "destroy the retired
generation's client" was unimplementable: you cannot destroy a pool you do not
own and must not destroy one everyone else is using. node:http rather than
undici because undici is not a dependency here and does not resolve, while
http.Agent is built in, destroys synchronously, and exposes live socket counts
for the leak assertions the live gate needs. The managed endpoint is always
loopback, so no TLS/HTTP2 capability is traded away.

The graph-set index advances only on applied and dirties on indeterminate,
mirroring its existing replace* discipline, because indeterminate means the
endpoint may have committed and a stale membership set becomes a silent wrong
answer. Every other outcome provably wrote nothing, so dirtying on those would
force the full-store scan this class exists to avoid. The agent facade also
invalidates on root-collision, which durably quarantines and rewrites state.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…B2

Launches the checksum-pinned Oxigraph v0.5.8 server, seeds reserved
system-record V1 state directly into it, and drives the real production store
stack against that live endpoint. Unit tests and the embedded adapter cannot
satisfy this gate by construction: the properties proven are about a separate OS
process, its listen socket, and what a supported predecessor binary observes.

The gate resolves the binary through the production OXIGRAPH_ASSETS table rather
than restating a version or checksum, so a bump cannot silently desynchronise
the manifest from the binary actually launched. Enforcement is exception-based,
matching the RFC-64 gate0/1/2 harnesses: verify.ts throws, so a bad verdict is a
non-zero exit rather than a file nobody reads. It pins HEAD on both sides of the
run, because a verdict attributed to the wrong commit certifies code that was
never executed. The evaluation table is data, so a missing check is visible as a
missing row rather than as absent code, and two checks exist purely to stop a
vacuous pass: every manifest entry must have executed, and the seeded fixture
quad count must match exactly.

The predecessor manifest is committed with immutable SHAs and expected outcomes.
Changing supported predecessors therefore requires reviewing a diff of that
file; a moving branch head or a new-binary-only filter is explicitly
insufficient.

This immediately earned its keep by failing on a real defect in my own code.
OwnedManagedHttpClient reported one live socket after destroyAndSettle(),
because Agent.destroy() tears sockets down ASYNCHRONOUSLY -- they leave the
agent's maps on their close event, a turn of the loop later at the earliest.
Returning there would have let a replacement child bind while a retired
keep-alive socket was still open, which is precisely the stale-generation window
the design exists to close. destroyAndSettle now polls to zero under a bounded
deadline and treats failure as terminal rather than retrying: if a socket will
not close, the old writer cannot be proven gone and the caller must fail closed.
So the gate has a genuine fail-before/pass-after record, not just a green run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… barrier

Two independent admission dimensions inside the existing bounded scheduler,
plus one coalesced reserved control-barrier entry. All strictly opt-in: the
existing four-argument run() signature is unchanged and every pre-existing test
passes unmodified.

The hard constraint here was not the new behaviour but the OLD behaviour. This
scheduler is process-global and sits on every store operation in the daemon, and
its admission decision is deliberately entry-INDEPENDENT: canStart() reads four
integers and never inspects a queued entry, which is what makes selection O(1)
regardless of queue depth. Any per-entry predicate on the default path would
convert that into a scan.

So the entire extension hangs off one integer comparison. While
taggedQueuedCount is zero every queued entry is provably untagged, and
nextRunnable() takes the original unconditional queue.shift() with no per-entry
scan and no allocation. The tagged path is only reached once something actually
carries admission metadata.

The fast path has a witness rather than a comment: admissionEvaluations is
incremented BEFORE the untagged early return in isEntryAdmissible, so it rises
if selection ever walks entries the guard should have kept it away from -- not
merely when a tagged entry is examined. A default-path test asserts it stays at
zero, so a future change that quietly starts scanning fails a test instead of
silently costing latency on every store operation.

Blocked entries are skipped in place rather than dequeued, preserving FIFO
within (priority, domain) while letting unrelated domains past. A store-wide
transition freezes its store even while merely QUEUED, because letting ordinary
work past it there is exactly how it would be starved. Untagged legacy entries
carry no store identity and so are never gated and never waited on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…ration

The supervisor previously kept three ad-hoc closure booleans (`stopping`,
`ready`, `restarts`) with no lock, so correctness rested entirely on those flags
being set synchronously. This replaces them with one lifecycle mutex and an
explicit state machine covering startup, backoff, automatic revive, controlled
recovery, stop and terminal close, and layers the process-local ownership lease
on top.

A generation is bound ONLY after the spawned child is the proven listener owner,
never at spawn. That distinction is the whole point: a spawned-but-unproven
child must not be able to satisfy a capability check, and the existing
findListenOwnerPid proof already fails closed when it cannot establish
ownership. Liveness is invalidated on child exit, revive, stop and shutdown, so
a store holding the lease observes revocation with no notification.

Three pre-existing defects fixed along the way:

- `killSync()` poisoned `stop()`. It set the same `stopping` flag, so a later
  `stop()` returned immediately and never escalated to SIGKILL. Terminating and
  closed are now distinct states.
- The auto-revive timer handle was never stored, so it could not be cancelled;
  it is now tracked and cleared, and the callback re-checks state so nothing can
  spawn after close.
- `stop()` resolved on the child's exit event without proving the PORT was
  released. It now re-probes under a bounded deadline, and a bind still served
  after the child exited burns the lease as terminal
  (`port-release-unproven`) rather than letting a replacement bind over a
  listener we do not own. Kills continue to go only through the tracked
  ChildProcess object, so a foreign listener is never signalled.

Recovery coalesces by expected generation, and a STALE expected generation
restarts nothing -- it returns the already-newer healthy generation instead of
queueing behind or triggering a redundant restart.

oxigraph-managed.ts now attaches the lease AND the supervisor handoff to the
rewritten adapter options. Both are symbol-keyed, so object spread carries them
through the storage factory rewrite while JSON.stringify drops them; a test
asserts the persisted projection is byte-identical to before and that a JSON
round trip yields no lease.

Verified by A/B on Windows: control (these changes stashed) 16 failed / 24
passed / 1 skipped; with them 16 failed / 36 passed / 1 skipped. Identical
failure set -- 12 new passing tests and zero regressions. The 16 are the
pre-existing Windows-environment set (spawn EFTYPE on the extension-less Node
stand-in, the deliberately-forced freebsd-x64 PATH fallback, and cgroup/systemd
Linux-only cases); the suite is proven on Linux CI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

The harness resolves @origintrail-official/dkg-storage through its published dist/ types, so the closure must exist before any typecheck. Omitting the build passed locally -- where a dist is usually left over from a previous run -- and failed only in CI, which starts from a clean checkout. Mirrors the RFC-64 gate0 workflow, which builds its closure for the same reason.

Reproduced locally by removing packages/storage/dist and confirming the exact CI error, then confirming the build step clears it. Note the repro needed tsconfig.tsbuildinfo removed too: with it present tsc reports 'Done' while emitting nothing, so dist/index.d.ts never reappears and the failure looks unfixable. CI is unaffected by that, having no incremental state.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…eduler work invalidated

The comment justified the process-global single-registration invariant by claiming the store scheduler has no store identity, so two managed controllers could not be told apart during a control barrier. That stopped being true in the same PR: the scheduler now carries an opaque storeId and scopes barriers per (storeId, purpose), and its own tests drive two distinct stores with concurrently pending barriers.

The invariant is still right, so this is a comment fix rather than a behaviour change -- but a reviewer reading the stack top-to-bottom would have hit a stated justification that the code two files over contradicts.

Replaced with the reason that actually holds: there is exactly one daemon-managed child by construction, so a second controller is necessarily wrong about what it owns, and its recovery could stop a child the first is mid-write against. Also records explicitly that enforcement lives here rather than in the scheduler on purpose -- 'who owns the managed store' needs one source of truth, and duplicating it would create a second one to drift.

Caught by the scheduler implementation review rather than by a test, since no test can assert that a comment is still true.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…end to end

Previously oxigraph-managed.ts attached only the lease, and the adapter requires
BOTH a live lease and a supervisor handoff before it advertises the lane. The
composition was therefore fail-closed but inert: correct, yet nothing could ever
open the lane in the real daemon. This wires the second half.

The handoff exists because the two halves of a clean generation swap have
different owners. The adapter owns the connection pool and can destroy its own
sockets; only the supervisor holds the ChildProcess and can assert PROCESS facts
-- that a child exited, that its port was released, that a replacement is the
proven listener. Neither can honestly assert the other's half.

stopAndProveOwnedChildDead is deliberately NOT stop(). It leaves the supervisor
open for a replacement, detaches the child before signalling so the exit it
causes cannot be misread as a crash and schedule a restart behind it, and then
THROWS if release cannot be proven -- with the lease already terminal at
port-release-unproven. stop() keeps its old non-throwing behaviour because it
runs from finally blocks and process teardown where a rejection would mask the
original error, and because nothing can bind a replacement after close anyway.
The handoff has the opposite constraint: its caller is about to bind one.

One guard beyond the brief, and it is the important one. The lifecycle mutex is
released between the two halves, so an armed backoff or a consumer
recoverGeneration() could bind a generation the lane never asked for, over the
port the lane believes it has freed -- which would defeat the entire handoff. A
handoffPhase flag disarms revive and refuses recovery while retired. If the
start half then fails it reaps the unproven child, clears the phase and re-arms
ordinary backoff, so a lane that abandons the handoff cannot leave the store
permanently childless.

Verified independently rather than on report: 16 failed / 41 passed / 1 skipped
against 16 / 36 / 1 before, so five new tests pass and the failure set is
unchanged (the 16 are the A/B-confirmed Windows environmental set). Four more
serial mutants each killed exactly their predicted test, including passing only
two arguments here -- which reproduces the inert state this commit fixes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…ive time

Found by probing the owned client directly rather than by a test, and it is the
dangerous kind of bug: it failed silently by SUCCEEDING.

`req.setTimeout()` bounds socket-ACTIVE time only. It does not count time spent
waiting for a free socket, and this pool is deliberately maxSockets:1, so a
second concurrent call queues behind the first. Measured: a call with a 500 ms
timeout resolved SUCCESSFULLY after 3822 ms. Nothing errored, nothing logged --
the caller simply got a success far outside the deadline it was promised, which
would blow both the 1,000 ms apply bound and the three-second slice while the
node looked healthy. The system-record lane treats its apply timeout as a SAFETY
bound, so queue wait has to count against it.

The fix needed two steps, and the intermediate state is worth recording because
it looked correct and was not. Arming a wall-clock timer that destroys the
request produced the right ERROR but the wrong LATENCY: still 3825 ms, because
destroying a request that has not yet been assigned a socket does not emit
`error` until one arrives. The deadline must bound when the CALLER is released,
not merely when teardown starts, so it now settles first and tears down after.
Measured again: 503 ms.

The socket-level `req.setTimeout` is kept alongside rather than replaced -- it
detects a connection that goes quiet mid-exchange, which the wall-clock deadline
would only catch at expiry.

Adds `managed-http-client-v1.test.ts` covering the queue-wait bound, the
already-aborted branch settling rather than hanging (it returns before
`req.end()`), the post-destroy socket count being proven zero, and a destroyed
client refusing to dispatch. The queue-wait test is mutation-proven: removing
the settle-first call turns exactly that test red at 1555 ms, and nothing else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

Five findings from the independent resource/operations track, four of them real
defects in code added by this PR.

**destroyAndSettle awaited before destroying (was the review's BLOCKER).**
`Promise.allSettled(inflight)` ran BEFORE `agent.destroy()`, so the method's
duration was `max(callerTimeoutMs across inflight) + poll` and the `timeoutMs`
parameter did not bound the phase it names. Reviewer measured 3831 ms for a
`destroyAndSettle(5000)` with one 4 s request, and no return at all within 8 s
against a server dribbling a byte every 100 ms, because the socket-inactivity
timer kept resetting. This is step 1 of the enable handoff, so a slow or chatty
endpoint could stall the transition, leave the seal uncommitted and hang daemon
teardown -- on a healthy-looking endpoint rather than a dead one. Destroying
first forces inflight requests to reject: the same scenario now returns in 5 ms
with zero sockets. The settle also shares the deadline, so the terminal-failure
branch below it is reachable rather than sitting behind an unbounded await.

**The negative capability memo latched permanently in three of four layers.**
The blob store, graph-set index and agent facade each cached `null` on first
absence and never re-probed, while the producer returns undefined during ANY
window in which the managed child is not the proven-ready listener -- i.e. every
ordinary revive. One probe landing inside a one-second Oxigraph restart would
have disabled the lane for the whole process lifetime, silently, recoverable
only by a daemon restart. Now only a PRESENT controller is memoized; absence is
re-probed. ChangelogStore already forwarded live and is unchanged.

**awaitRetiredWork was structurally a no-op.** `destroyClient` nulled
`managedClient` and `awaitRetiredWork` then read that same null, so step 3 of a
sequence whose own docstring calls it "load-bearing and asserted step by step"
could never drain anything. Harmless in enable only because step 1 already
awaited settlement -- but actively wrong in disable, which calls
`awaitRetiredWork` with no preceding `destroyClient` and so destroyed the LIVE
client and left the field pointing at a dead pool bound to the current
generation. A retired client is now held in its own field.

**maxSockets restated a frozen limit it cites.** Now imports
SYSTEM_RECORD_MAX_MATERIALIZER_WRITE_CONCURRENCY rather than hardcoding 1, so a
future bump cannot leave the pool pinned with a comment claiming they agree.

**Per-write allocation on the always-on guard.** `quads.map((q) => q.graph)`
built a throwaway array the length of every write batch -- measured +14.3 ms and
~8 MB transient at 1,000,000 quads. This is the only thing in B2 that runs on
every write on every node today, so it now iterates quads directly, matching
what ChangelogStore already does.

Also removes `excludeInternalGraphsV1`, which had zero production callers: every
enumeration surface still filters through `isAtomicGraphReplaceStagingGraph`, so
shipping a second unused filter implied a migration that has not happened. Its
test is replaced by one asserting the predicate production ACTUALLY calls, with
a positive control so a predicate returning true for everything would fail.
`isInternalGraphUriV1` and `isEphemeralInternalStagingGraphUriV1` drop off the
public barrel for the same reason and stay module-scoped for the tests that pin
the reserved/ephemeral partition.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…ceMs

The comment claimed the probes were 'spread across stopGraceMs so stop() stays a bounded operation'. The bound is real but it is not stopGraceMs: five attempts at up to readyIntervalMs+1000 each, separated by floor(stopGraceMs/attempts), is ~11.5s, on top of stopGraceMs for the SIGTERM/SIGKILL wait and the bounded listener lookup. Worst case ~22-25s.

This matters because stopGraceMs is exactly what an operator would reach for when tuning a shutdown timeout, and a stated bound is what a future reader budgets against.

The cost is defensible and the comment now says why: it accrues ONLY while something is still serving our bind after our own child exited, which is the leaked-descendant case where guessing 'released' is worse than waiting. A cheaper probe would buy speed with more false 'released' verdicts, which is the wrong trade for a proof. Also records that there is no durability exposure -- probing starts after the child exits, so RocksDB is already closed and a hard kill at an outer deadline costs a log line, not the store.

Comment only; no behaviour change. Raised by the independent resource/operations review, which also confirmed the timing arithmetic and the absence of durability risk.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

Two BLOCKERs from the independent protocol/safety review, both demonstrated by
an executed exploit rather than argued, and both squarely mine.

**Shutdown did not supersede anything.** `runShutdown()` inspected
`this.transition` only for `kind === 'shutdown'`, so an in-flight open was
neither awaited nor cancelled -- just silently clobbered by overwriting the
field, and left running concurrently. Observed: a stalled enable resumed AFTER
shutdown completed, set the lane back to `enabled` with activation generation 2,
and started a replacement child after shutdown had already proved the old one
dead and its port released. The same root cause made a shutdown issued during a
disable end at `disabled`, leaving the lane non-terminal and freely re-openable.
Shutdown now JOINS the in-flight transition. Superseding something means
outliving it, not racing it.

**A shut-down lane revived and dispatched a write.** `open()` checked terminal
state BEFORE `await this.transition.work` and never re-checked after it, so an
open that joined a shutdown proceeded to run the full handoff on a lane the
process had already terminated -- and step 5 of the exploit was the one that
mattered: `applyVerified` on that session dispatched to the executor and
returned `applied`. Every post-await path now re-reads state.

TypeScript actively resisted the fix, which is worth recording: it narrows
`this.current` and does NOT invalidate the narrowing across an `await`, so it
reported the re-checks as impossible comparisons on types with "no overlap".
That assumption -- nobody mutates the field while we are suspended -- is exactly
the bug. Reads now go through `readState()` so the union stays wide and a future
reader is not tempted to delete the guards as dead code.

Also from the same review:

- **MAJOR: `runEnable` had no post-condition.** It set `enabled` because no step
  threw, never verifying that `startAndProveCleanGeneration()` actually bound a
  ready generation; a handoff resolving without binding one produced an
  "enabled" lane over a child that was not the proven listener. It now asserts a
  ready, non-terminal snapshot and fails to `unavailable` otherwise.
- **MAJOR: the advertising gate checked `terminal` but not `ready`**, despite
  its own docstring promising exactly the `ready` check. Added.
- **MAJOR: the process-global registration was never released on shutdown**, so
  a replacement controller was unconstructable for the process lifetime. Worse,
  the adapter called the factory inside an unguarded capability probe, so the
  refusal propagated out of `getSystemRecordLaneControllerV1?.()` through three
  decorators -- merely ASKING whether the capability existed could throw. The
  registration is released on shutdown and the probe now fails closed to
  `undefined`, because absence is the correct answer to "can I have the lane?".
- **MAJOR: unscoped `deleteByPattern` bypassed the reserved-graph guard.** With
  no `pattern.graph` the guard body (`if (graph)`) is a no-op and the emitted
  update binds `?g_ctx` across every named graph, reserved ones included. The
  unbound form now excludes the operation-internal prefix in its WHERE; no
  legitimate caller targets those graphs, so no real caller's semantics change.

The suite drove this machine only SEQUENTIALLY, which is the sole reason both
blockers shipped. Four concurrency regression tests added. Both are
mutation-proven: restoring the clobbering shutdown reddens the stalled-re-open
test, and removing the post-await terminal re-check reddens the
dispatch-after-shutdown test -- each exactly one test, nothing else.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…d held calls

Two HIGH findings from the independent resource/operations review, both
reproduced to the reviewer's exact numbers before being fixed.

**HIGH 2 — a barrier did not stop untagged traffic, which is 100% of today's
store traffic.** The transition IS the Oxigraph stop-and-restart, so for the
whole restart window the scheduler kept dispatching legacy queries and writes at
full concurrency into a child being SIGTERM'd and then into no listener at all.
Measured at maxConcurrent 4: 20 new untagged ops STARTED during the transition;
now 0, with all 20 held in queue and completing after it.

Untagged entries are gated at selection while a barrier is pending, and barrier
readiness waits for untagged inflight derived as `total - tagged`, so the
untagged path pays nothing to be waitable.

Held work is strictly better than dispatched work here: entries stay queued,
keep their existing wait timer, and fail with a typed retryable
STORE_SCHEDULER_BUSY rather than transport-level ECONNREFUSED. That is
backpressure instead of an error burst, and it is something the caller can
actually retry.

Barrier readiness is deliberately no longer generation-scoped: one child process
serves every generation, so waiting only on the sealed one was wrong for exactly
the case the barrier exists to serve. Tagged work for OTHER stores is still not
waited on, preserving the two-store no-head-of-line-blocking property.

**HIGH 3 — held calls were unbounded, and release turned them into a queue_full
burst.** 200 tagged calls under one seal at queueLimits 8 produced 200 held and
then 190 rejections AT COMMIT -- callers rejected only after waiting out the
entire seal, which is strictly worse than rejecting at admission. Accumulation
had no cap: 100,000 held runs measured at 143.6 MB.

A held call has been ADMITTED, so it now counts against its lane's queue limit
exactly like a queued one. That single accounting change fixes both halves:
`queued + held <= limit` is invariant across hold, enqueue and release, so every
held call still fits when the seal commits. Now: 192 rejected at admission, 8
held, 8 ok, and ZERO rejected after waiting -- a provable property rather than
an empirical one. The ceiling is the queue limit (~385 KB worst case at the
measured ~1505 B per held call) rather than caller volume.

No timer was added. A timer would break the contract the seal exists to provide
and turn a control transition into a source of spurious failures; the barrier
commits its seal in a `finally`, so holding forever requires a control-path bug
whose blast radius is now bounded by the lane limit.

The review also diagnosed WHY HIGH 3 survived testing: the hold test used
exactly one call at queueLimits 1, the single arity at which a release cannot
overrun, so it could only confirm what it expected. The replacement holds 200 at
a limit of 8 and asserts the full outcome distribution split by WHEN rejection
happened. Two other tests used the same degenerate arity and were raised.

Seven mutants applied serially, presence grepped, all killed. One is worth
recording: the first attempt at M17 SURVIVED, but the mutant was invalid -- it
added a pump call instead of moving one, so the correct later pump still fired.
Rewritten as a genuine move it kills three tests. A surviving mutant that turns
out to be a bad mutant is not evidence of coverage.

Fast-path invariance is intact: the guard gained one integer term
(`taggedQueuedCount > 0 || barriers.length > 0`) and the invariance test still
asserts zero admission evaluations on a pure untagged workload, with a tagged
positive control. Independent re-benchmarking of three new per-operation costs
on the untagged path is outstanding and tracked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

Four findings from the independent execution/QA review. The gate was reporting
"PASS: 18 checks" while thirteen of those checks were assertions about literals.

**Thirteen verdict rows were constants the generator hardcoded.** Foreign-process
signals, old-generation dispatches, stale-facade dispatches, barrier slot-ms,
healthy deadline recoveries, indeterminate and recovery latencies -- all emitted
as `0` by `run.ts`, all asserted `=== 0` by the verifier. No code path could
have made any of them non-zero: the gate never restarts a child, never opens a
lane session and never runs a barrier. They are deleted rather than left in
place. A verdict that overclaims is worse than a smaller one, because the
smaller one does not stop anyone looking. Those rows return when something
actually drives them, which is the CAS stack.

The sharpest instance: `indeterminateReturnWithinThreeSeconds` was the one check
nominally bounding "the write returns in time", and it was asserted against a
hardcoded `0`. A request that returned after 3822 ms under a 500 ms deadline --
a real bug found in this PR by probing, not by testing -- would have passed it
by construction.

**`leakedOwnedSockets` was 0-or-throw.** `destroyAndSettle` loops until the
count reaches zero and throws otherwise, so reading it afterwards can only ever
be 0 and the check could never report a failure. It is now a PAIR: the socket
count is captured BEFORE destroy and must be non-zero, which makes the probe
demonstrably live, and the post-destroy zero means something.

**The manifest carried a fabricated commit SHA.** `843f5213d` zero-padded to 40
characters is not a commit and resolves to nothing, and an adjacent field
claimed "the gate resolves and records the exact 40-character SHA at run time
and fails if it cannot" -- which it did neither of. That value was copied
verbatim into the artifact CI uploads as evidence, so the published verdict
attributed a pass to a commit that does not exist. Replaced with the real SHA
843f521, the false claim removed, and a real
check added: every pinned commit must resolve via `git cat-file -e`. Verified to
discriminate -- re-injecting the fabricated SHA now fails the gate with
`everyManifestCommitResolves: a pinned commit does not exist`.

**A crashed run left the previous PASS artifact verifiable.** Found while
mutation-testing the above: a malformed manifest crashed the generator, the
stale artifact stayed on disk, and the standalone verifier re-verified it and
exited 0 -- certifying a run that never completed. `run.ts` now deletes both
artifacts before doing anything, and tolerates a UTF-8 BOM rather than crashing
on one.

The predecessor section is renamed to say what it does: these entries run
against the CURRENT binary, and no predecessor is checked out or built. The
manifest's role today is to pin WHICH commits must keep the property and to
require that each resolves. `manifestIsNonEmptyAndFullyIterated` is documented
as structural, its real content being the non-empty term.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…instead of hanging

Quiescing untagged work (previous commit) closed an ECONNREFUSED burst but
replaced it with a worse failure mode: a caller that issues store work from
INSIDE a barrier transition now waits for work that cannot proceed until the
transition it is inside returns. Circular wait, and silent. The independent
resource review hit it for real -- its own attack harness deadlocked with Node
exiting 13 on an unsettled top-level await. A deadlock is harder to diagnose and
harder to alert on than the burst it replaced.

Exact re-entry detection was considered and REFUSED on measured grounds rather
than deferred. `AsyncLocalStorage` latches async_hooks on for the rest of the
process after a single `run()`; `disable()` exists but enable/disable churn is
itself a deopt event and is version-dependent. That is a permanent process-wide
tax on every async operation in the daemon, paid to diagnose a caller bug
faster, on a scheduler whose measured default-path cost is +0.003%. Wrong trade.

The ambiguity is genuine, not incidental: during a running transition, a held
`run()` is EITHER the transition re-entering (deadlock) OR an unrelated caller
legitimately waiting. Those are indistinguishable at the call site without
ambient context. So the bound compensates for what it cannot prove -- the error
names re-entry as the usual cause and carries a `blockedBy` breakdown (untagged
inflight, tagged inflight, generations, held runs), putting the diagnosis on the
operator's screen even though the scheduler cannot demonstrate it.

`StoreControlBarrierTimeoutError` (`STORE_CONTROL_BARRIER_TIMEOUT`), default
60s, env-overridable, per-call override. Deliberately double the child's 30s
readyTimeoutMs: this is meant to catch a circular wait, not a slow disk.

Recovery differs by phase, and the asymmetry is deliberate:
- WAIT phase: the transition never started, so nothing is disrupted. The barrier
  withdraws completely -- seal released, held runs released, selection woken.
  Fully recoverable, and this is the phase the real deadlock lands in.
- TRANSITION phase: the transition may be part-way through stopping a child.
  Releasing the seal would admit work into a store that no longer exists, which
  is HIGH 2 recreated at the worst possible moment. The caller is told and the
  seal is left in place; if the transition ever settles its own `finally` still
  cleans up, so this reports without permanently freezing the lane.

Three tests, and the third is what stops the bound becoming a nuisance: the real
deadlock shape asserting wait-phase withdrawal AND that the store resumes; the
re-entrant-transition shape asserting the seal deliberately outlives the
rejection; and an ordinary slow transition completing untouched with
`barrierTimeouts: 0`, so a bound that fired on every real restart would be
caught. Five mutants, serial, all killed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…its value

The comment explained the CHOICE of 60s without stating the CONSTRAINT a tuner needs. It now says the relationship to the child's readyTimeoutMs is the constraint rather than the number, names the failure mode of setting it too low -- anything at or below readyTimeoutMs turns every genuinely slow restart into a spurious timeout -- and says why that is worse than no bound at all: it trains operators to ignore the one signal that means a transition actually deadlocked. Also names both tuning surfaces and the raise-together rule.

Comment-only, and verified as such rather than assumed: the diff touches no coverage directives or ts-directives, 47/47 tests still pass and the build exits 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…fire

The bound's timer is unref'd deliberately -- a safety timer that kept a daemon alive through its own shutdown would be worse than the deadlock it guards. The consequence is not obvious and was found by an independent re-run rather than by reasoning: the bound can only fire while something else holds the event loop open.

A daemon always qualifies, since a listening server holds a ref'd handle. A process that has quiesced down to nothing but the deadlock exits silently instead -- the pre-bound behaviour, in the one situation nobody is watching.

The practical trap is for tests: a real-timer test of this bound must hold the loop open itself, or the process exits before the timer fires and the run is recorded as a failure with a bare non-zero exit and no output. That is exactly how the reviewer's first re-run presented, and it looks identical to the deadlock the bound was added to fix. The tests here use fake timers and are unaffected.

Comment-only; no coverage or ts-directives touched. 47/47 pass, build exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…ase properly

Five findings from the independent review of the supervisor handoff.

**BLOCKER — a synchronous spawn throw stranded the handoff phase permanently.**
`child_process.spawn` throws SYNCHRONOUSLY for EACCES/EFTYPE/E2BIG/EINVAL; the
`.once('error')` handler catches only async failures, so the throw escaped
`startCleanGenerationLocked` and bypassed the only code that clears
`handoffPhase` and re-arms backoff. Measured: 30 backoff periods later, still
one spawn, port unserved, self-heal refusing, and `terminal: false` -- so the
lease reported "momentarily not ready" indefinitely instead of failing closed.
The daemon's triple store was dead for the process lifetime, reachable by a
binary being replaced or re-permissioned under a running daemon. Everything from
`state = 'recovering'` is now inside a try/catch that runs the existing failure
tail and rethrows, covering `spawnChild()`'s own assertion and
`bindProvenGeneration()` throwing on a terminal lease (the same class, milder).

**HIGH — the release proof accepted a probe TIMEOUT as evidence of release.**
`endpointAnswers()` collapsed ECONNREFUSED (positive evidence nothing is bound)
and TimeoutError (no evidence at all) into one `false`. That collapse is
fail-CLOSED in `probeReady()` and correct there; the release proof reused the
same primitive with the OPPOSITE polarity, making it fail-OPEN, and its loop
accepted the FIRST non-answer. Since #2052 is a store-PRESSURE issue, a 1.5s ASK
timeout against a loaded listener is the expected case: the retire resolved in
137ms with ownership non-terminal, which is exactly the state where a
replacement may bind over a live listener.

Fixed at the root rather than patched: `probeBind()` now returns three states --
`serving | refused | inconclusive` -- with refusal detection walking `cause` and
`AggregateError.errors`, since `fetch` wraps the OS error. ECONNRESET is
deliberately NOT proof: something was there. The loop is inverted to retry until
it sees a refusal and return false if it never does. `endpointAnswers()` is
unchanged for readiness, which genuinely wants the collapse.

**HIGH — nothing bounded the retired phase.** Fixed by bounding it rather than
by making a teardown terminal, which is where the reviewer's proposal (mine) was
wrong: the managed child is the DAEMON'S ENTIRE TRIPLE STORE, not the lane's
private one, so a terminal teardown would mean disabling the lane permanently
kills node storage until restart -- strictly worse than the parked state it
fixes. On expiry the phase clears and ordinary supervision resumes; a lane that
returns late is refused by the proven-dead-predecessor check. Fail-closed for
the lane, alive for everyone else.

**MAJOR — a failed retire left the supervisor open and childless.** It now
terminates and closes, so no path can spawn against the unaccounted-for
listener. The residual is real and outside this module: ordinary store traffic
never consults the lease, so the daemon keeps POSTing to that endpoint. A FATAL
log names the condition; closing it properly needs an adapter-side kill switch
and is tracked separately.

**MAJOR — an inert guard removed rather than papered over.** The
`handoffPhase === 'retired'` clause in `scheduleRevive` changed 0 of 57 test
statuses when deleted. Attempts to give it a discriminating test confirmed why:
every `scheduleRevive` caller is unreachable while retired. Replaced with a
comment explaining the invariant. For the same reason a proposed guard in
`reviveLocked`/`spawnChild` was NOT added -- it would have been inert by the
identical argument, and the property now holds by construction: both recovery
tails clear the phase before any spawn.

Four mutants, one test each, serial and grep-verified. One process note worth
keeping: a first attempt turned `try {` into `if (true) {`, leaving a dangling
`catch`; vitest reported "no tests" rather than failing, which is not a green
run but skims as one. A mutant only counts if the suite actually collected
tests. A second mutant was also re-run in isolation because two could each
explain the same red, and its failure message -- the retire wrongly resolving --
confirms the kill is for the right reason.

Verified: 16 failed / 44 passed / 1 skipped, up from 41 passed, with the same 16
Windows-environmental failures and zero regressions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

@Jurij89

Jurij89 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Merge-readiness review

Reviewed commit: b1c1bdf6dfda2a615f0be3ee13758dcbba86d583

Verdict: not ready to merge. The ownership, capability-discovery, and reserved-state foundations are directionally sound, and the focused suite is green, but the production composition currently advertises an openable live lane without routing the child handoff through the store control barrier. I also reproduced three independent state-machine/policy defects. The first four findings should be fixed before merge.

1. BLOCKER: the live handoff bypasses the scheduler control barrier

runEnable() destroys the client and stops/restarts the owned Oxigraph child directly. buildChildHandoff() delegates directly to the supervisor. The scheduler provides runControlBarrier(), but there are no production callers of it (or sealStoreGeneration) outside the scheduler itself.

I reproduced this with the real SparqlHttpStore: hold an ordinary request unresolved, open the advertised lane, and observe the supervisor's stop step execute while that request is still in flight (stopWhileOrdinaryStoreRequestInflight: true). This violates ADR-0002's admission-seal/drain requirement and can turn valid store traffic into transport failures or allow stale-generation work across the restart boundary.

Current behavior:

sequenceDiagram
    participant Q as Ordinary request
    participant S as Store scheduler
    participant L as System-record lane
    participant O as Oxigraph supervisor
    participant X as Oxigraph child
    Q->>S: start request
    S->>X: request remains in flight
    L->>O: stop child (no barrier)
    O-xX: terminate
    X--xQ: transport failure / ambiguous result
Loading

Required behavior:

sequenceDiagram
    participant Q as Store traffic
    participant S as Store scheduler
    participant L as System-record lane
    participant O as Oxigraph supervisor
    L->>S: runControlBarrier(storeId, generation)
    S-->>L: admission sealed; tagged and untagged work drained
    L->>O: stop and prove release
    L->>O: start and prove replacement
    L->>S: commit generation/epoch transition
    S-->>Q: resume against replacement generation
Loading

Fix direction: route every enable/disable/shutdown lifecycle transition that can invalidate the client/child through externalStorePriorityScheduler.runControlBarrier() using the adapter's stable store identity and generation. Add an integration test with a real adapter/supervisor and an in-flight ordinary request. If that wiring is intentionally deferred to the CAS follow-up, keep this capability unadvertised/fail-closed until the barrier-safe composition lands.

2. BLOCKER: a generation-raced success returns indeterminate without sealing admission

In applyVerified(), the lane moves to reconciling only when the executor itself returns indeterminate. If the executor returns applied/already-applied and the post-read detects a generation change, the method synthesizes indeterminate and returns it while leaving the lane enabled.

Executable reproduction on this head:

{"first":{"outcome":"indeterminate","recoveryGeneration":"2"},"stateAfterFirst":"enabled","second":{"outcome":"indeterminate","recoveryGeneration":"3"},"calls":2,"stateAfterSecond":"enabled"}

The second dispatch should never have been admitted after an ambiguous first write.

Fix direction: derive the final outcome first, then transition to reconciling for every final indeterminate result, including locally synthesized outcomes. Treat loss of ready/valid ownership as part of the same post-dispatch attribution check, not only a changed generation string. Add a regression test asserting that a second apply is deferred and the executor call count remains one.

3. HIGH: concurrent shutdown callers can execute teardown twice

runShutdown() checks for an existing shutdown only before awaiting an in-flight open/disable. Multiple shutdown callers can therefore wait on the same transition, resume together, each create a teardown, and overwrite this.transition.

With a stalled/released open followed by two concurrent shutdowns, I observed:

{"destroy":4,"stop":4,"start":2,"state":"shutdown"}

Only three destroy/stop calls are expected in that scenario (the preceding lifecycle work plus one coalesced shutdown). Double stop/release proof is not a harmless idempotency assumption at a process-ownership boundary.

Fix direction: establish shutdown intent synchronously before the first await, or serialize lifecycle intents through a single transition loop/mutex. After every awaited transition, re-read and coalesce/preempt the current transition again. Add concurrent-shutdown and disable-vs-shutdown tests behind a deliberately stalled open.

4. HIGH: unknown names in the internal namespace are writable but permanently hidden

The policy states that an unrecognized internal name is neither sweepable nor writable (internal-graph-policy.ts). However, assertNotReservedInternalGraphV1() rejects only the two known persistent names. Generic adapter mutations therefore accept a future/near-miss name under urn:dkg:internal:atomic-graph-replace:, while prefix-wide enumeration filtering hides it.

I confirmed a graph named urn:dkg:internal:atomic-graph-replace:system-record-v1:future-reserved was admitted and dispatched once. This creates invisible durable state and leaves future namespace takeover/confusion risk.

Fix direction: generic graph mutations should reject every graph in the internal prefix except canonical UUID-shaped ephemeral staging graphs where that internal operation explicitly requires them. Add negative insert/delete/drop/replace tests for unknown and near-miss internal names while retaining the intended UUID staging lifecycle.

5. MEDIUM: the “live managed ownership” gate does not exercise the behavior it claims

The workflow says it proves properties of the separate process and listen socket (workflow lines 3-7), but the gate manually mints ownership and injects no-op supervisor handoff methods (run.ts lines 234-240). Its own model notes that it never restarts a child, opens a lane, or runs a barrier (model.ts lines 29-44). This is why finding 1 passes the advertised live CI gate.

Fix direction: either narrow the workflow/name/PR claims to endpoint and capability-policy evidence, or preferably drive startManagedOxigraph/the real supervisor, open the lane with active ordinary work, and assert drain, restart, generation, port-release, and socket facts.

6. LOW: source encoding damage should be cleaned up

packages/storage/src/adapters/sparql-http.ts contains a UTF-8 BOM and multiple mojibake comments (—, §, …); the devnet runner also contains a replacement character. Runtime strings do not appear affected, but this should be restored to clean UTF-8 to avoid permanent diff/readability noise.

Verification performed

  • git diff --check: pass.
  • Storage dependency-closure build: pass.
  • Focused storage suite: 7 files / 129 tests passed (system-record-materializer-lifecycle-v1, scheduler admission, managed ownership/client, reserved mutation guard, capability discovery, internal graph policy).
  • Managed-ownership devnet typecheck: pass.
  • Four executable behavioral probes reproduced findings 1-4 on the reviewed head.
  • Current GitHub checks are green, but only the path-selected gates ran. Full CI still needs to run after the stacked base is merged/retargeted.
  • A CLI dependency-closure build was attempted, but two concurrent Hardhat compiler processes stalled in the pre-existing EVM-module build and were stopped; this is not counted as a PR failure.

Merge gates

  1. Resolve findings 1-4 and add the missing concurrency/live-boundary regressions.
  2. Make the live evidence gate exercise the real handoff/barrier, or accurately narrow its stated proof.
  3. Merge PR feat(core): add bounded system-record V1 contracts #2103, then rebase/retarget this PR to testnet-canary and run the full CI matrix.
  4. Refresh the PR description: its barrier asymmetry/re-entry notes are stale relative to this head, and the required architecture diagrams are missing.

The structure is promising, but advertising a live lane before the scheduler barrier and terminal-state semantics are correct would introduce exactly the restart/backpressure ambiguity this stack is intended to remove.

Jurij89 and others added 3 commits August 6, 2026 14:42
…lose the namespace guard

Four findings from the PR review on b1c1bdf, each reproduced before fixing.

**BLOCKER 2 — a generation-raced success returned indeterminate WITHOUT sealing.**
The lane sealed only when the EXECUTOR returned indeterminate. When a success
was synthesized into indeterminate because the generation changed under the
dispatch, it was returned to the caller while the lane stayed `enabled`, so the
next apply was admitted: two dispatches, both ambiguous, state `enabled`
throughout. The final outcome is now derived first and the seal keyed on IT,
however it was reached.

Attribution also widened from a generation-string comparison to the whole
ownership snapshot. A lease that went not-ready or terminal under the dispatch
is equally unable to attribute the response, and comparing only
`childGeneration` missed both. Its own regression test pins that: readiness is
lost while the generation string is unchanged, and the outcome must still be
indeterminate.

**HIGH 3 — concurrent shutdowns each ran a teardown.** Both callers passed the
"already shutting down?" check while the in-flight transition was still the
OPEN, both awaited it, and both ran a teardown -- 4 destroy / 4 stop where 3 of
each were expected. Double stop-and-prove-release is not a harmless idempotency
assumption at a process-ownership boundary: each one signals a child and asserts
a port fact.

The fix is one line's PLACEMENT. `runShutdown` is no longer `async`, so the
`this.transition` assignment happens synchronously before any await and a second
caller joins at the top.

Worth recording how that was established, because the first attempt was wrong: a
separate `shutdownWork` coalescing field was added and its mutant SURVIVED --
twice, including against the reviewer's exact stalled-open shape. It never
changed an outcome, because the synchronous assignment already coalesces. It was
removed rather than kept, per the standard adopted earlier in this branch: an
inert guard reads as protection and is worse than none. Removing the synchronous
assignment instead reddens all three shutdown tests, which is what identifies it
as the load-bearing change.

**HIGH 4 — unknown names in the internal namespace were writable but hidden.**
The guard rejected only the two known-reserved names, so a near-miss or future
name such as `...:system-record-v1:future-reserved` was accepted by generic
mutation while the prefix-wide enumeration filter hid it -- invisible durable
state, writable by anyone who can spell the prefix, and a standing
namespace-confusion risk against whatever a later stack reserves for real. It
also directly contradicted this module's own docstring, which claimed such names
were "not writable".

Generic mutation now refuses the WHOLE internal namespace except canonical
UUID-shaped ephemeral staging graphs, which stay writable because the
atomic-replace builders must be able to drop their own.

**LOW 6 — encoding damage, all of it self-inflicted.** Several whole-file
PowerShell rewrites in this branch re-encoded UTF-8 as cp1252, corrupting
comments in files I touched INCLUDING pre-existing lines. Verified against the
base, which had zero mojibake, and the restored pre-existing lines are now
byte-identical to it. Repaired with an escape-only script, because the first
repair attempt was itself corrupted in transit -- the same mechanism that caused
the damage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… barrier

BLOCKER 1 from the review on b1c1bdf, and the finding lands hardest because
the missing piece was already written: `runControlBarrier` / `sealStoreGeneration`
shipped in this stack exported, documented and covered by 25 unit tests, with
ZERO production callers. The live handoff stopped the owned child, asserted its
port free and bound a replacement while ordinary requests were still in flight
on the retired generation -- exactly the window the barrier exists to close.

`SystemRecordLaneControllerDepsV1.barrier` is REQUIRED, not optional. An optional
barrier is one that gets forgotten, which is the defect this commit fixes; making
it mandatory means a future construction site cannot omit it silently.

Enable, disable and shutdown each open their own named section. The materializer
stays store-agnostic -- it takes an exclusive-section runner, not a scheduler --
and the adapter supplies one backed by the real barrier, using `this` as the
opaque store identity and re-reading the lease snapshot's generation per
transition rather than capturing the one observed at construction (the lane
outlives any single child).

Shutdown under a barrier that cannot be acquired: the teardown does NOT run, and
that is the safe side. The lane still reaches terminal, so nothing writes through
it again, while the child is left alive under the daemon supervisor that still
owns it and stops it at process exit. Stopping the child outside a section is the
hazard; "could not quiesce" is not a reason to do it anyway. The error propagates,
because a shutdown that could not quiesce the store is not a clean one.

**Evidence.** The unit tests bracket each transition, so they pin that a section
is opened AND that every child-touching step runs inside it. They cannot show
that asking for a section does anything, because they inject a pass-through
stand-in -- so
`system-record-control-barrier-integration-v1.test.ts` drives a real adapter, the
real process-global scheduler and the real handoff composition against a
genuinely held-open ordinary request.

Each half was mutated separately:

- adapter barrier -> pass-through: the integration test fails with the reviewer's
  exact symptom, `['stop','start']` observed while the request is outstanding.
  All 41 lane unit tests stay GREEN under that mutant, which is the direct
  demonstration that they could not have caught this.
- enable/disable/shutdown wrapping removed one at a time: each reddens its own
  bracketing test and nothing else.

The integration file also carries its own discrimination: a quiesced-store
control proving the wait is caused by the in-flight request rather than by the
open being async, and a request issued DURING the section that must complete
afterwards -- a transition that sealed the store and never resumed it would be a
silent permanent outage, which "no stop happened" alone would score as a pass.

Storage suite: 90/90 across the B2 files. The full package shows 7-16 failures in
`oxigraph-worker-{resilience,respawn}` and one worker-backed factory case; those
are 5 s timeouts under ~70 s import cost and reproduce on `main` with none of
this stack present (8/11 red there), so they are box contention, not this change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MEDIUM 5 from the review: the gate's workflow header claimed properties of a
separate process and its listen socket, but the gate minted ownership by hand and
injected no-op supervisor methods -- it never restarted a child, never opened a
lane and never ran a barrier. The review's own words for it: "this is why finding
1 passes the advertised live CI gate."

Of the two offered directions -- narrow the claims, or drive the real thing --
this takes the second. Narrowing alone would have left the gate structurally
unable to catch a regression of the very defect it is named after.

The gate now runs ONE real generation handoff first: `startOxigraphServer` from
the CLI daemon owns and spawns the child, its `ownership.lease` and
`supervisorHandoff` travel to the adapter under the symbol key exactly as the
daemon composes them, and the lane transition goes through the process-global
scheduler's control barrier. The only interposition is an observer that records
a reading and delegates straight through.

The probe workload is a 60,000-quad ordinary insert: one scheduler admission, one
HTTP request over the store's own path, and interrupting it is precisely the harm
the barrier prevents.

Ten measured checks, each a fact about processes rather than about a literal:

  - the write was STILL RUNNING when the lane was asked to open (without this
    the ordering check is vacuous -- a finished write satisfies it regardless)
  - zero store requests inflight at the instant the child was stopped
  - the in-flight write survived the restart
  - the generation advanced; a SECOND OS process was spawned; the first PID is
    dead and the last is alive
  - the replacement serves the same 60,000 quads (a handoff that silently
    produced an empty store would otherwise satisfy everything above)
  - the store resumed against the replacement; the lane reports enabled

On the inflight reading: it replaced a wall-clock "stop happened after the write
settled" comparison, which was subtly wrong. The scheduler releases admission
when the store work resolves, and the caller's promise settles a few microtasks
later through `workLifecycle` and `insert`'s tail, so a CORRECT handoff measured
as 0.09 ms too early. The pre-fix build read -49 ms, so any tolerance would have
been chosen to sit between two numbers rather than to mean anything. The inflight
count has no such boundary -- it is the quantity the barrier waits on.

**The gate is proven to discriminate, not merely to pass.** Built with the
adapter's barrier replaced by a pass-through and re-run end to end:

    childStoppedOnlyAfterOrdinaryWorkDrained: 1 store request(s) inflight
    inflightOrdinaryWriteSurvivedTheHandoff: fetch failed
    replacementServesTheSameData: 0/0 quads
    -> FAILED (3 of 20 checks)

With the barrier restored: PASS, 20 checks, 3 predecessor entries. That negative
control was not constructed -- the first run of the new section hit a stale
`dist/` predating the fix and reproduced the reviewer's finding unprompted.

The workflow header is also rewritten to enumerate what is proven and, explicitly,
what is not: predecessor entries run against the CURRENT binary rather than each
pinned commit, and no verified apply is dispatched until the CAS lands. The unit
step gains the barrier integration and scheduler admission suites.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

…ally key on

The `runControlBarrier` docstring said it "waits for the sealed generation's
execution permits to drain". It does not: `isBarrierReady()` waits on this
store's whole `taggedInflight`, and the hold is `seals > 0`. Both are
generation-blind, deliberately -- the transition is a child-process restart and
the child it stops serves every generation.

This matters beyond the wording. The PR description carried a "known asymmetry
the CAS PR must resolve" section derived from this sentence, warning that a
future fix must be an allow-list rather than a not-equal predicate. There is no
asymmetry to fix; the sentence was the whole source of it. Left in place it
would have invited someone to NARROW the drain to the sealed generation to
restore a symmetry that already exists, which is strictly weaker and -- since
generations are decimal counters -- trivially bypassable.

`sealStoreGeneration` now states what its `generation` argument is: a diagnostic
label carried on the seal and reported through `admissionGenerationsInflight`,
not an enforcement key.

Comment-only, and checked as such: the scheduler suites are unchanged at 47/47.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Jurij89

Jurij89 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Review addressed — all six findings

Reviewed head b1c1bdf6d → now 8447314c1. Every finding was reproduced against the code before being fixed, and every fix carries a kill proof. All six confirmed; none disputed.

# Commit Kill proof
1 BLOCKER — handoff bypassed the barrier 1784b8c8d pass-through mutant reproduces your exact symptom; live gate fails 3 of 20
2 BLOCKER — raced success returned indeterminate unsealed 29dc6b1bd regression test: readiness lost, generation string unchanged, must still be indeterminate
3 HIGH — concurrent shutdowns ran teardown twice 29dc6b1bd removing the synchronous assignment reddens all three shutdown tests
4 HIGH — unknown internal names writable but hidden 29dc6b1bd near-miss and future-name negative tests
5 MEDIUM — live gate did not exercise its claims d8b2a2d02 gate FAILS 3 of 20 checks against a build without the barrier
6 LOW — encoding damage 29dc6b1bd repaired; pre-existing lines byte-identical to base

1 — the barrier

Confirmed exactly as reported: runControlBarrier/sealStoreGeneration shipped exported, documented and covered by 25 unit tests with zero production callers.

SystemRecordLaneControllerDepsV1.barrier is now required, not optional. An optional barrier is one that gets forgotten, which is this defect; making it mandatory means a future construction site cannot omit it silently. Enable, disable and shutdown each open their own named section. The materializer stays store-agnostic — it takes an exclusive-section runner, not a scheduler — and the adapter supplies one using this as the opaque store identity, re-reading the lease snapshot's generation per transition rather than capturing the construction-time one (the lane outlives any single child).

One decision worth flagging, since it is not what your fix direction implies: shutdown under an unacquirable barrier does not run the teardown. The lane still reaches terminal so nothing writes through it again, and the child is left alive under the supervisor that still owns it and stops it at process exit. Stopping the child outside a section is the hazard; "could not quiesce" is not a reason to do it anyway. The error propagates, because a shutdown that could not quiesce the store is not a clean one.

The integration test you asked for is packages/storage/test/system-record-control-barrier-integration-v1.test.ts. Note what it demonstrates about the unit tests: all 41 of them stay green under the pass-through mutant, because they inject a stand-in barrier. Only the integration test fails, with ['stop','start'] observed while the request is outstanding.

5 — the gate now discriminates

You offered two directions; this takes the second. Narrowing the claims alone would have left the gate structurally unable to catch a regression of the defect it is named after.

The gate drives startOxigraphServer, its real child, and the real barrier, against a 60,000-quad ordinary insert. Ten measured checks, including: zero store requests inflight at the instant the child was stopped; a second OS process spawned with the first PID dead and the last alive; the replacement serving the same 60,000 quads. Built with the barrier replaced by a pass-through:

childStoppedOnlyAfterOrdinaryWorkDrained: 1 store request(s) inflight
inflightOrdinaryWriteSurvivedTheHandoff: fetch failed
replacementServesTheSameData: 0/0 quads
-> FAILED (3 of 20 checks)

That negative control was not constructed. The first run of the new section hit a stale dist/ predating the fix and reproduced your finding unprompted.

The workflow header now enumerates what is proven and what is not — predecessor entries run against the current binary rather than each pinned commit, and no verified apply is dispatched until the CAS lands.

3 — how the fix was reached, because the first attempt was wrong

I first added a separate shutdownWork coalescing field. Its mutant survived twice, including against your exact stalled-open shape: it never changed an outcome, because making runShutdown non-async already coalesces. It was deleted rather than kept — an inert guard reads as protection and is worse than none. Removing the synchronous assignment instead reddens all three shutdown tests, which is what identifies it as the load-bearing change.

A stale claim in my own PR description, now corrected

The description carried a "known asymmetry the CAS PR must resolve" section: a generation-scoped drain against a generation-blind hold. Re-read at this head, that was wrong. isBarrierReady() waits on this store's whole taggedInflight and the hold is seals > 0 — both generation-blind, deliberately, because the transition is a child-process restart and the child it stops serves every generation.

It came from the scheduler's own docstring, which claimed the barrier "waits for the sealed generation's execution permits to drain". Corrected in 8447314c1, because left in place it invited someone to narrow the drain to restore a symmetry that already exists — strictly weaker, and (generations being decimal counters) trivially bypassable.

Merge gates

gate status
1. Findings 1–4 + regressions done — 90/90 across the B2 storage files, plus the new integration suite
2. Live gate exercises the real handoff donePASS: 20 checks, 3 predecessor entries, proven to fail without the barrier
3. Merge #2103, rebase, retarget, full CI blocked#2103 is still open. Unchanged: do not merge this while it is based on feat/2052-system-record-core
4. Refresh the PR description done — stale barrier/re-entry notes corrected, architecture and handoff diagrams added

On the storage full-suite failures in your verification notes: the oxigraph-worker files fail on main with none of this stack present (8 of 11 red in oxigraph-worker-resilience.test.ts), and fail worse in isolation than in the full run — 5 s test timeouts against ~70 s of import cost. Box contention, not this change. Linux CI is the gate.

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

The gate's typecheck went red in CI with a wall of TS2307 and the implicit-any /
unknown-catch errors that cascade from the degraded types, while the exact same
command passed locally.

Cause, and it is the harness's own import graph rather than anything about the
packages: `run.ts` now imports `daemon/oxigraph-server` by relative SOURCE path,
because the CLI package exports no such subpath. That module imports
`./routes/status.js`, which reaches `dkg-agent`, `dkg-node-ui`, `dkg-publisher`,
`dkg-chain`, `dkg-mcp` and both adapters. tsc follows all of it, so every one of
those needs real `dist/` types -- and the build step only built storage's
closure. Locally they were all present from earlier builds, which is precisely
why this passed here and only here.

Reproduced before fixing rather than reasoned about: renaming `packages/agent/dist`
out of the way reproduces the same TS2307 class locally, and restoring it clears
them. The wide build was then run end to end and the typecheck re-run clean.

The filter now matches what `test:live:system-record-managed-ownership:generate`
already builds. That alignment is the point, not a convenience: a typecheck over
a narrower closure than the run itself uses can only ever produce a false result
in one direction or the other. No added cost either -- the generate step was
already running this same wide build seconds later.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

@Jurij89

Jurij89 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

CI follow-up — the gate went red on its own harness closure, now green

My previous comment reported the live gate passing. It passed locally and then failed in CI on the first run at 8447314c1, so the record needs correcting: it was red for about fifteen minutes.

Cause, and it is worth stating because it is the same class as the finding above. run.ts now imports daemon/oxigraph-server by relative source path (the CLI exports no such subpath), and that module imports ./routes/status.js, which reaches dkg-agent, dkg-node-ui, dkg-publisher, dkg-chain, dkg-mcp and both adapters. tsc follows all of it, so every one of those needs real dist/ types — and the workflow's build step only built storage's closure. On my box they were all present from earlier builds, which is exactly why it passed here and only here.

Reproduced before fixing rather than reasoned about: renaming packages/agent/dist out of the way reproduces the same TS2307 class locally, restoring it clears them. The build filter now matches what the generate script already runs, which is the point rather than a convenience — a typecheck over a narrower closure than the run uses can only produce a false result in one direction or the other. No added cost; the generate step was running the same wide build seconds later anyway.

Fixed in e8788a840. Current state at that head:

check result
Managed Oxigraph ownership (live) passPASS: 20 checks, 3 predecessor entries, real handoff included
Storage unit conformance (inside that job) pass — 7 files, incl. system-record-control-barrier-integration-v1
Protocol evidence pass
SPARQL scalability lint pass
SQLite lifecycle (Windows) pending

Merge gate 3 is unchanged and still blocking: #2103 is open, so the full CI matrix has not run. Do not merge while this is based on feat/2052-system-record-core.

@Jurij89

Jurij89 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Merge-readiness review — round 2

Reviewed commit: e8788a840bbea5ab53aa231874e5394e14a4dc0f

Verdict: not ready to merge; one lifecycle blocker remains. Findings 1, 2, 4, 5, and 6 from the previous review are materially addressed: production lifecycle calls now use the real scheduler barrier, synthesized indeterminate outcomes seal admission, the internal namespace guard is prefix-wide, the live gate drives a real child handoff, and the encoding damage is repaired. The shutdown fix narrows finding 3, but an older transition can still erase the shutdown intent and allow a later transition to revive the terminal session.

1. BLOCKER: a superseded open() can erase shutdown intent and let disable revive the terminal session

The new synchronous assignment in runShutdown() correctly coalesces callers that arrive while this.transition still points at shutdown. However, an older open() still clears this.transition unconditionally in its finally, even after shutdown has replaced that pointer. close('disable') has the same unconditional cleanup pattern (lines 371-377).

I reproduced this against the real StorePriorityScheduler, not a pass-through barrier:

  1. Open, disable, then begin a re-open stalled in startAndProveCleanGeneration.
  2. Request shutdown; it synchronously installs the shutdown transition and waits for the re-open.
  3. Release the re-open. Its older finally clears the shutdown transition while the shutdown handoff is stalled in stopAndProveOwnedChildDead.
  4. Request disable. Because the shutdown pointer is gone and state is still enabled, disable is admitted and queues behind the real shutdown barrier.
  5. Release shutdown. Shutdown sets state to shutdown and releases the process-global registration; then the queued disable runs and overwrites state to disabled.

Executed result:

expected session.state: shutdown
received session.state: disabled

This breaks the terminal-state and single-controller invariants: the old controller is nonterminal again after its registration has been released, so it can coexist with a replacement controller and may be reopened in a later stack.

The race is:

sequenceDiagram
    participant O as Older open caller
    participant S as Shutdown caller
    participant D as Disable caller
    participant B as Scheduler barrier
    O->>B: enable transition (stalled)
    S->>S: install shutdown intent
    S->>O: await older open
    O-->>S: enable completes
    O->>S: finally clears transition (erases shutdown)
    S->>B: shutdown barrier / teardown
    D->>B: disable admitted behind shutdown
    B-->>S: shutdown completes; state=shutdown
    B-->>D: disable runs; state=disabled
Loading

Fix direction: do not let a transition clear a successor it does not own (if (this.transition?.work === work) ...), but do not stop there. Both open() and close('disable') resume after awaits using stale intent and must loop/re-read transition precedence before installing new work. A dedicated synchronous terminal/shutdown latch or a single serialized lifecycle-intent loop is safer than relying on a replaceable promise pointer. Release registeredController only once shutdown remains authoritative after every superseded continuation has settled. Add the exact real-scheduler regression above, plus trailing open and already-waiting disable variants; assert terminal state, one teardown, no post-shutdown barrier, and no second live controller.

2. MEDIUM: the live gate has no failure-safe process cleanup

measureLiveHandoff() stops the session/store/supervisor and removes the location only on the success tail. Any exception after startOxigraphServer() and before that tail can leave a live Oxigraph child and temporary store behind. That is particularly likely when the gate is doing its job and detects a broken handoff.

Fix direction: wrap the acquired handle/store/session/location in try/finally, preserve the original test failure, and independently attempt session.close, store.close, handle.stop, and rm. A deliberately failing handoff should exit promptly with no surviving child/listener.

3. LOW: the live workload uses a timing assumption instead of observing admission

The gate sleeps a fixed 250 ms before checking whether the 60,000-quad write is still active (run.ts lines 260-268). On a sufficiently fast runner, correct code can fail because the write completed before the sleep; on a slow runner it adds a fixed delay and a large payload merely to create overlap.

Fix direction: poll the scheduler snapshot until the write is observed in flight, then request the lane open immediately. Fail with a bounded diagnostic if the probe cannot be observed; optionally retry with a larger workload. This preserves the non-vacuous proof without tying it to machine speed.

Validation

  • Incremental diff from b1c1bdf6d inspected; git diff --check passes.
  • Storage dependency-closure build passes.
  • Focused suite: 8 files / 142 tests passed, including the new real control-barrier integration test.
  • A temporary real-scheduler regression test reproduced finding 1 and was removed after the run; the review worktree is clean.
  • Current GitHub checks are green, including the revised live managed-ownership gate. The full matrix still has to run after PR feat(core): add bounded system-record V1 contracts #2103 merges and this PR is retargeted to testnet-canary.
  • The local gate typecheck was not counted because its new CLI source import requires the full CLI build closure; the current CI job builds that closure and passes.

Once finding 1 is fixed and the lifecycle test matrix covers stale-transition cleanup, the architecture is substantially closer to merge-ready. Findings 2 and 3 are gate robustness improvements; they do not change the runtime lane semantics, but fixing cleanup in this PR is strongly recommended because this PR introduces the live process gate.

Jurij89 and others added 2 commits August 6, 2026 17:59
Round-2 BLOCKER on e8788a8, confirmed and then found to be wider than
reported. This does not patch the pointer a third time; it moves what the
pointer was carrying onto the state field.

**The finding, reproduced against the REAL scheduler.** An older `open()` cleared
`this.transition` unconditionally in its `finally`, erasing the shutdown entry
that `runShutdown` had installed while joining it. A later `disable` then saw no
pointer and a non-terminal state, was admitted, queued behind the shutdown
barrier, and wrote `disabled` over the committed terminal state. Downstream, and
worse than the review reported: `open()` then RESOLVED, spawned a fresh child
after shutdown had proved the old one dead and asserted its port released, and
`applyVerified` dispatched and returned `applied`. A second `close('shutdown')`
ran a SECOND full teardown -- 4 destroy / 4 stop for one session. The mirror
case through `close('disable')`'s own `finally` is equally real.

**Root cause, which is one layer under that.** Seven of the nine writes to
`this.current` happen AFTER an await and none re-validated that it was still the
authoritative transition. Terminality was asserted by READERS
(`assertNotTerminal`, `applyVerified`'s ladder) and enforced by no writer, so it
could only ever be stale. And shutdown had no representation before it committed:
`runEnable`/`runDisable` publish `enabling`/`disabling` synchronously, but
`runShutdown` wrote `current` only in its `finally`, so for the whole teardown
the lane still read `enabled`.

**The change.** Shutdown commits `current = 'shutdown'` SYNCHRONOUSLY at intent,
before the method can suspend. Every other write to `current` goes through
`commitState`, which refuses to move off a terminal state. That is a change of
CARRIER, not a third patch -- and it is why `applyVerified` needed no admission
change: a dispatch during the teardown now sees a terminal lane and returns
`capability-lost` instead of being admitted.

Four consequences that follow, each verified:

- `release(entry)` clears the pointer only if it is still ours. The pointer is
  demoted to a coalescing hint: it decides who joins whom, never whether a
  transition runs or what state results, so a stale one degrades to a redundant
  join rather than a wrong transition.
- `runShutdown` joins BEFORE checking state. With the latch, checking first would
  hand every later caller a resolved promise while the child was still being
  stopped, and they would never learn the teardown failed.
- `runEnable` publishes state and activation as ONE commit -- a terminal lane
  advertising a fresh activation generation is the resurrection the latch exists
  to prevent.
- Two disables queued behind one open ran two `system-record.disable` sections
  and rotated the epoch twice. One new line fixes it; no design in the review or
  in my own analysis caught this one.

**Three guards DELETED rather than kept**, because under the latch each became a
branch that can no longer change an outcome: `close`'s `kind === 'shutdown'`
join, the `current = 'shutdown'` write in the teardown's `finally`, and the
`if (readState() === 'shutdown') return` inside the teardown -- that last one is
now ALWAYS true and would have skipped every teardown.

**Evidence.**

- 41/41 pre-existing tests pass with ZERO assertion changes.
- 12 new interleaving tests. 7 of them FAIL against the shipped model; the other
  5 are regression pins for paths the shipped model happens to get right (or for
  risk this change itself introduces, like the join/state order). I am not
  claiming 12 discriminators.
- Solo-mutant sweep, one guard removed at a time: 11 of 11 killed.
- The 11th was earned rather than assumed. `assertNotTerminal()`'s post-join
  mutant SURVIVED the first sweep with all 52 tests green -- while, without it, a
  superseded `open()` ran a full destroy/stop/start handoff on an already
  terminal lane. The guard was load-bearing and the suite could not see the path.
  The test that now kills it is the last one added.

The harness needed upgrading to express any of this: `StallingHandoff` can pause
exactly one step, which is why the suite could express "shutdown behind a stalled
open" but not "a caller arrives while the TEARDOWN is stalled" -- and the second
shape is where both review rounds found their blockers. `GatedHandoff` pauses any
named step with several gates live at once, and `track()` makes "has not resolved
yet" an assertion rather than a timing hope.

Declared behaviour deltas: `state` reads `shutdown` from the instant
`close('shutdown')` is called (strictly stronger; both consumers only forward
it); a superseded `open()` now rejects `/terminal/` instead of resolving with a
terminal session; `close('disable')` during a teardown returns instead of
joining; two concurrent disables behind one open run one section.

Knowingly stated rather than hidden: `commitState('enabling')` and
`commitState('disabling')` are reached with no await between them and their
caller's terminal check, so their refusal is currently unreachable. They go
through the chokepoint anyway, because the property being relied on is
"`commitState` is the only writer of `current`", and a raw write at those two
sites reintroduces exactly the await-placement audit that has now been wrong
twice.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ean up on failure

Round-2 findings 2 and 3, plus four "checks that cannot fail" found while
verifying them.

**MEDIUM 2 — no failure-safe cleanup.** The four teardown steps were bare
statements on the SUCCESS tail, so any throw in between leaked a live Oxigraph
child holding an ephemeral port, a temp RocksDB directory, an un-closed store
and — once the capability probe had run — the process-global lane registration.
Six throw sites sit in that window; the dominant one is `lane.open()`, which
rejects on any failed handoff step, so the leak was most likely exactly when the
gate was doing its job. On `ubuntu-latest` a non-detached child is re-parented
rather than reaped and would outlive the job. Resources now register a release
step as they are acquired and a single `finally` unwinds them newest-first, each
caught independently, never masking the original error.

**LOW 3 — a fixed sleep raced against the payload.** Replaced with polling
`getPressureSnapshot()` until the write is observed in flight, bounded at 30 s
and failing the verdict rather than proceeding vacuously. Measured: the 60k-quad
write commits in ~2.1-2.3 s here, so the old 250 ms sample held with ~8x margin
-- but that margin was an accident of the payload size, and at 10k quads the
sample already reads `false`. The witness was also on the wrong side of the same
microtask boundary this file documents elsewhere: `ordinarySettled` flips when
the CALLER's promise settles, a few microtasks after the scheduler releases
admission, so in the marginal regime both halves of the pair could be green and
vacuous at once. It now samples the same counter the other half does.

**The gate now drives the round-2 interleaving, and this is the part that
matters.** My first attempt added terminal-lifecycle checks around a SEQUENTIAL
`close('shutdown')`. I built the pre-fix materializer into `dist/` and re-ran:
all 27 checks still passed. A sequential shutdown on a quiesced lane is correct
even in the broken model, so those checks were regression pins, not
discriminators -- the same defect the review found in the gate one level up.

So the gate now reproduces the defect instead: disable, start a re-open and HOLD
it inside the supervisor's start step, request shutdown while it is held, release
the re-open so its `finally` runs, then issue a disable into that window. Against
the pre-fix build:

    laneIsTerminalAfterShutdown: disabled
    secondShutdownRanNoSecondTeardown: 2 child stop(s) across both calls
    -> FAILED (2 of 27 checks)

`disabled` is the reviewer's exact reported symptom, now produced by the real
supervisor, the real child and the real scheduler rather than by hand. With the
fix: PASS, 27 checks.

Seven new measured checks: terminal after shutdown, shutdown completed cleanly,
exactly one teardown, still one after a second `close('shutdown')`, dispatch
refused, re-open refused, and ZERO children spawned after shutdown -- that last
one is a process fact from the real `spawn`, because a state string can be wrong
and a spawned PID cannot.

**Four checks that could not fail, found while verifying the above.**

1. Probing `full` REGISTERS the process-global controller, after which every
   later probe that reaches the factory gets a duplicate-registration refusal the
   adapter reports as `undefined`. With `full` probed third, the enabled-changelog
   and terminal-ownership checks were satisfied by that backstop rather than by
   the guards they name: deleting either guard would not have turned them red.
   Every negative case is now probed while the registry is still empty, `full`
   goes last, and the remaining coupling is loud -- a negative case that wrongly
   advertises takes the registration and reddens `capabilityPresentWhenFullyProven`.
2. The deletion detector was one-shot. Entry 1's `dropGraph` either failed (as it
   must) or emptied the graph; entries 2 and 3 then read before=0/after=0 and
   PASSED. Reserved state is re-seeded per entry, and the seed count is re-counted
   per entry rather than hoisted.
3. Enumeration was measured only on the graph-set-index composition, where two
   independent filters hide the internal prefix, so deleting either left the check
   green. It now also enumerates on the index-free store, making the always-on
   adapter layer independently falsifiable.
4. `verify.ts` cast the parsed artifact straight to the result type and stamped
   the CURRENT HEAD onto it. Running verify alone, or a CI cache restoring
   `artifacts/`, would re-certify old measurements against a newer commit. The
   generator now records `sourceCommit` and the verifier refuses a mismatch or an
   unknown schemaVersion.

Also: the pinned-server temp store was `mkdir`'d without a preceding `rm` and
never removed -- 12 had accumulated locally, and a PID collision would seed on
top of leftover reserved quads and fail every predecessor row with a misleading
`seed incomplete`. Now removed before and after.

Predecessor rows carry `executedAgainst: 'current-binary'`. No predecessor is
checked out, built or executed; the caveat previously lived only in prose that
the uploaded evidence does not carry, while the artifact itself said
`<commit>: pass`. The dead `advertisedSystemRecordLane` literal is deleted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)

@Jurij89

Jurij89 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Round 2 addressed — all three findings, plus the class under finding 1

e8788a84091d38870e. All three confirmed; one part of the round-1 fix direction is refuted with evidence, below.

# Finding Commit
1 BLOCKER superseded transition erases shutdown intent d5240115f
2 MEDIUM live gate has no failure-safe cleanup 91d38870e
3 LOW fixed sleep instead of observing admission 91d38870e

1 — confirmed, and worse than reported

Reproduced against the real StorePriorityScheduler, with adapter-exact wiring. Your expected shutdown, received disabled reproduces exactly. Downstream of it, measured:

  • controller.open() then RESOLVES, spawning a fresh child after shutdown had proved the old one dead and asserted its port released, and applyVerified dispatches and returns applied — the round-1 hazard reachable again by a different route;
  • a second close('shutdown') runs a second full teardown, 4 destroy / 4 stop for one session;
  • the mirror through close('disable')'s own finally is equally real.

Negative control, which is what rules out a harness artifact: issuing the disable one step earlier — while the shutdown pointer is still live — ends correctly at shutdown. The erased pointer is the sole cause.

I did not patch the pointer a third time. Rounds 1 and 2 are the same root: seven of the nine writes to this.current happen after an await and none re-validated that it was still the authoritative transition, so terminality was asserted by readers and enforced by no writer. And shutdown had no representation before it committed — runEnable/runDisable publish enabling/disabling synchronously, but runShutdown wrote current only in its finally, so for the whole teardown the lane still read enabled.

Shutdown now commits current = 'shutdown' synchronously at intent, and every other write goes through commitState, which refuses to move off a terminal state. That is a change of carrier, not a third patch — and it is why applyVerified needed no admission change: a dispatch during the teardown now sees a terminal lane and returns capability-lost.

Three guards were deleted because the latch makes each one a branch that can no longer change an outcome: close's kind === 'shutdown' join, the current = 'shutdown' write in the teardown's finally, and if (readState() === 'shutdown') return inside the teardown — that last one is now always true and would have skipped every teardown.

One part of your fix direction does not ship, and I want to be explicit rather than quietly omit it. "Release registeredController only once shutdown remains authoritative after every superseded continuation has settled" is inert and harmful. Inert: runShutdown already joins the in-flight transition before the teardown starts, so any continuation that existed has fully settled, and with the install-guard none can install afterwards. Harmful: the finally is currently four await-free statements, so the terminal commit and the release are atomic; deferring the release opens a window in which the lane is terminal but the slot is still held, and a probe landing there hits the factory throw that sparql-http catches and memoizes as null for the store's lifetime — reintroducing a weaker form of the failure the release exists to prevent.

Relatedly, the write-order premise is inverted: the finally writes current = 'shutdown' before releasing the registration, atomically. The real ordering defect is the reverse direction — runDisable/runEnable writing current as their last statements after the terminal commit — which is what the latch and commitState fix.

Evidence. 41/41 pre-existing tests pass with zero assertion changes. 12 new interleaving tests; 7 fail against the shipped model, the other 5 are regression pins (I am not claiming 12 discriminators). Solo-mutant sweep: 11 of 11 guards killed. The 11th was earned — assertNotTerminal()'s post-join mutant SURVIVED the first sweep with all 52 green, while without it a superseded open() ran a full destroy/stop/start handoff on an already-terminal lane. The guard was load-bearing and the suite could not see the path; the test that kills it is the last one added.

The harness needed upgrading first: StallingHandoff can pause exactly one step, which is why the suite could express "shutdown behind a stalled open" but not "a caller arrives while the TEARDOWN is stalled" — and that second shape is where both rounds found their blockers.

2 and 3 — fixed, and the gate now catches finding 1

Cleanup is a resource stack unwound in a single finally, newest-first, each step caught independently and never masking the original error. I enumerated six throw sites in the unguarded window; the dominant one is lane.open(), so the leak was most likely exactly when the gate was detecting a broken handoff.

The sleep is replaced by polling getPressureSnapshot() until the write is observed in flight. Measured: the 60k-quad write commits in ~2.1–2.3 s, so 250 ms held with ~8x margin — but that was an accident of payload size, and at 10k quads the sample already reads false. The witness was also on the wrong side of the microtask boundary this file documents elsewhere, so both halves of the pair could have been green and vacuous at once.

The part worth your attention: my first attempt wrapped terminal-lifecycle checks around a sequential close('shutdown'). I built the pre-fix materializer into dist/ and re-ran — all 27 checks still passed. A sequential shutdown on a quiesced lane is correct even in the broken model, so those checks were regression pins, not discriminators: the same defect you found in the gate, one level up.

The gate now drives the interleaving instead — hold a re-open inside the supervisor's start step, request shutdown, release, then issue a disable into that window. Against the pre-fix build:

laneIsTerminalAfterShutdown: disabled
secondShutdownRanNoSecondTeardown: 2 child stop(s) across both calls
-> FAILED (2 of 27 checks)

disabled is your exact symptom, produced by the real supervisor, real child and real scheduler. With the fix: PASS: 27 checks.

Four checks that could not fail, found while verifying the above

  1. Capability matrix ordering. Probing full registers the process-global controller; every later probe reaching the factory then gets a duplicate-registration refusal the adapter reports as undefined. So capabilityDeniedByEnabledChangelog and capabilityAbsentOnTerminalOwnership were satisfied by that backstop, not by the guards they name — deleting either guard would not have reddened them. Negatives are now probed with an empty registry and full goes last.
  2. The deletion detector was one-shot. Entry 1's dropGraph either failed (as it must) or emptied the graph; entries 2 and 3 then read before=0/after=0 and passed. Reserved state is re-seeded per entry.
  3. Enumeration was doubly shadowed — measured only on the graph-set-index composition, where two independent filters hide the prefix. It now also enumerates on the index-free store.
  4. verify.ts would certify any artifact it found — no schemaVersion check, and sourceCommit stamped from current HEAD rather than recorded by the generator. Verify alone, or a CI cache restoring artifacts/, re-certified old bytes. Both are now bound.

Plus: the pinned-server temp store was never removed (12 had accumulated locally; a PID collision would seed on top of leftover quads and fail every predecessor row with a misleading seed incomplete), and predecessor rows now carry executedAgainst: 'current-binary' — the caveat previously lived only in prose the uploaded evidence does not carry.

Merge gates

gate status
1. Findings + concurrency regressions done — 124/124 across the B2 storage files; 11/11 solo mutants killed
2. Live gate exercises the real behaviour done — 27 checks, proven to fail on the pre-fix model
3. Merge #2103, rebase, retarget, full CI blocked#2103 still open
4. PR description refreshed last round; I will fold the terminality change in once CI on this head reports

Still standing: do not merge while this is based on feat/2052-system-record-core.

@Jurij89

Jurij89 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

CI note at 91d38870e — the gate is green; two checks are being cancelled by the platform

Managed Oxigraph ownership (live): passPASS: 27 checks, 3 predecessor entries, and its storage unit step is 149/149 across 7 files. That is the check that actually exercises this change, including the new terminal-lifecycle interleaving.

Protocol evidence: pass.

The other two are not content failures, and I want the record straight rather than "CI is red":

  • SPARQL scalability lint — its own scan step passed on every attempt (sparql-scale-lint: 0 new blocking, 0 acknowledged, 2 grandfathered, self-test 37/37), and the job was then cancelled during cleanup. The last attempt was cancelled before recording a single step.
  • SQLite lifecycle (Windows) — first attempt died in Set up job with Failed to resolve action download info. Error: Service Unavailable, before any repository code ran; the next was cancelled mid-step.
  • The live gate itself failed the same way once (Service Unavailable in Set up job) and passed on re-run with no code change.

That is a GitHub Actions degradation window, not this branch. I have re-run each several times; the conclusions are cancelled, not failure, and no step of ours reported an error. Worth a fresh re-run before anyone reads the ticks — I have stopped burning job minutes on it.

Merge gate 3 is unchanged and still blocking: #2103 is open, so the full matrix has not run. Do not merge while this is based on feat/2052-system-record-core.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants