feat(storage): add the live managed materialization boundary (#2052 Stack B2) - #2110
feat(storage): add the live managed materialization boundary (#2052 Stack B2)#2110Jurij89 wants to merge 27 commits into
Conversation
…h ownership lease Stack B2 of #2052 needs two primitives before any materialization code can be written safely. Both are default-unused: nothing constructs or consumes them yet. internal-graph-policy.ts distinguishes the first PERSISTENT graphs under the `urn:dkg:internal:atomic-graph-replace:` prefix from the ephemeral per-operation staging graphs that prefix used to hold exclusively. Reusing the prefix is deliberate: every supported predecessor already filters it out of listGraphs(), hasGraph(), the graph-set index, the changelog and the sync responder, so an older binary neither enumerates nor serves the new reserved state without any change on its side. The reserved names are matched EXACTLY and the ephemeral names by canonical UUID shape, so a staging sweep can never classify durable state as garbage and an unrecognised internal name is hidden but neither sweepable nor writable. The module stays dependency-free because index.ts is on every process's startup path and B2 must add zero default-off cost; a test imports the frozen system-record-v1 barrel and asserts the literals agree. managed-oxigraph-ownership-v1-internal.ts gates capability on a live lease that only the supervisor can mint. Every signal available today is forgeable from persisted config: `managedByDkg` is operator-writable and resolveAdapterOptions actively rewrites it to false, while `atomicUpdates` is synthesized as true by that same function. The lease handle instead carries no data at all -- its meaning lives in a module-private WeakMap keyed by identity -- and travels under a unique symbol, which object spread copies (surviving the factory rewrite) but JSON cannot produce or round-trip. Mutation authority is a separate controller the supervisor never hands out, so holding a lease lets a store ask whether it is live, never assert that it is. Both guards are mutation-proven: forcing the lease identity check true kills the two identity-forgery tests, and relaxing the reserved-name check to a prefix match kills the near-miss test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…state The reserved graphs added in the previous commit are hidden from every enumeration surface, but nothing stopped a caller from writing to them by hardcoded IRI. Reserved state is writable only through the materializer's structured, generation-bound command path, which derives its own scope; a generic write would bypass the full-state CAS, the capacity accounting and the materialization epoch. The guard lives on SparqlHttpStore rather than on a decorator because every decorator is optional -- the changelog defaults off and the graph-set index and blob store are conditional -- while all of them delegate downward, so the adapter is the only always-on choke point for the managed endpoint that holds reserved state. It is checked before the atomicUpdates capability refusal, since resolveAdapterOptions synthesizes that flag from plain config. ChangelogStore needed a separate fix that the new test caught: its insert() and delete() STRIP reserved-graph quads rather than refusing them, so an enabled changelog turned a reserved write into a silently resolved no-op -- a lost update with no signal. Stripping stays correct for the changelog plane and for ephemeral staging graphs, which are decorator-internal, but persistent state now denies. update() is deliberately left unguarded: its argument is opaque SPARQL, and scanning it for reserved IRIs is exactly the evadable check that assertNoReservedRef already documents as insufficient. Opaque updates rotate the materialization epoch instead, and Stack C migrates the raw callers. The pass-through cases stub fetch rather than dialing an unbound port. An earlier revision dialed for real and measurably destabilized the worker-thread suites; the stub is both faster and a stronger claim, asserting the request was actually issued. Remaining full-suite failures are confined to the embedded oxigraph-worker adapter, pass 3/3 in isolation, and vary run to run (2-9 observed) with and without these changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane controller owns lifecycle and policy but deliberately owns no bytes: the store transaction is performed by an injected transaction executor that the mandatory sparql-http adapter will supply. That split is what lets the logical fault matrix run as a pure reference model -- a test executor can fail at any boundary without a live child -- while keeping one implementation of the state machine. Nothing here is inert: the session is fully functional against any executor, including the pure one the tests use. The enable path asserts a physical fact rather than a logical one. It destroys the client, proves the owned child and its port dead, drains retired work, and only then starts a clean generation and rotates the epoch; the tests assert the ORDER, not merely the set, because a replacement started before the old child is proven dead reopens exactly the stale-generation window this design exists to close. Any handoff failure is terminal `unavailable` rather than a retry: if we cannot prove the old writer is gone, retrying does not make it so, and allowing legacy work to bypass while that uncertainty stands is the forbidden fallback. Every applyVerified re-reads the lease instead of trusting the state captured at open, because a child exit revokes it with no notification. Two cases the tests pin: an indeterminate dispatch seals the lane into `reconciling` so no further work is admitted against a generation whose last write may or may not have committed; and a success whose child generation changed UNDER the dispatch is downgraded to indeterminate, since that response cannot be attributed to the child we addressed. A `stale` outcome is deliberately NOT downgraded -- it means the CAS did not match and carries no uncertainty about bytes written. The process-global single-registration invariant is enforced before any capability is exposed: the store scheduler has no store identity, so two managed controllers could not be told apart during a control barrier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ator capability Capability discovery is explicit at every layer rather than structural. A resolver that walked .inner/.innerStore the way asGraphWriteGenSource does would reach PAST a wrapper whose cache the lane must invalidate, and past the changelog's denial, handing out a controller that silently desynchronises an index. So each decorator forwards the getter itself, in all FOUR places: the two storage decorators, the changelog, and the agent's hand-rolled forwarder, which rebuilds a fresh object literal and would otherwise drop it silently (the same failure already documented there for replaceGraph). Advertising is gated on three independent facts, all fail-closed: a live lease, a supervisor handoff, and non-terminal ownership. The handoff requirement is the correction of a real design error caught while wiring this: the adapter owns the connection pool but cannot assert PROCESS facts (child exited, port released, replacement is the proven listener) because only the supervisor holds the ChildProcess. Without it the adapter would have advertised a lane that could never open. A lease alone is therefore valid but deliberately silent. managed-http-client.ts gives the lane a pool it actually OWNS. SparqlHttpStore dispatches through global fetch, whose sockets belong to Node's process-wide dispatcher and are shared with every other consumer, so "destroy the retired generation's client" was unimplementable: you cannot destroy a pool you do not own and must not destroy one everyone else is using. node:http rather than undici because undici is not a dependency here and does not resolve, while http.Agent is built in, destroys synchronously, and exposes live socket counts for the leak assertions the live gate needs. The managed endpoint is always loopback, so no TLS/HTTP2 capability is traded away. The graph-set index advances only on applied and dirties on indeterminate, mirroring its existing replace* discipline, because indeterminate means the endpoint may have committed and a stale membership set becomes a silent wrong answer. Every other outcome provably wrote nothing, so dirtying on those would force the full-store scan this class exists to avoid. The agent facade also invalidates on root-collision, which durably quarantines and rewrites state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…B2 Launches the checksum-pinned Oxigraph v0.5.8 server, seeds reserved system-record V1 state directly into it, and drives the real production store stack against that live endpoint. Unit tests and the embedded adapter cannot satisfy this gate by construction: the properties proven are about a separate OS process, its listen socket, and what a supported predecessor binary observes. The gate resolves the binary through the production OXIGRAPH_ASSETS table rather than restating a version or checksum, so a bump cannot silently desynchronise the manifest from the binary actually launched. Enforcement is exception-based, matching the RFC-64 gate0/1/2 harnesses: verify.ts throws, so a bad verdict is a non-zero exit rather than a file nobody reads. It pins HEAD on both sides of the run, because a verdict attributed to the wrong commit certifies code that was never executed. The evaluation table is data, so a missing check is visible as a missing row rather than as absent code, and two checks exist purely to stop a vacuous pass: every manifest entry must have executed, and the seeded fixture quad count must match exactly. The predecessor manifest is committed with immutable SHAs and expected outcomes. Changing supported predecessors therefore requires reviewing a diff of that file; a moving branch head or a new-binary-only filter is explicitly insufficient. This immediately earned its keep by failing on a real defect in my own code. OwnedManagedHttpClient reported one live socket after destroyAndSettle(), because Agent.destroy() tears sockets down ASYNCHRONOUSLY -- they leave the agent's maps on their close event, a turn of the loop later at the earliest. Returning there would have let a replacement child bind while a retired keep-alive socket was still open, which is precisely the stale-generation window the design exists to close. destroyAndSettle now polls to zero under a bounded deadline and treats failure as terminal rather than retrying: if a socket will not close, the old writer cannot be proven gone and the caller must fail closed. So the gate has a genuine fail-before/pass-after record, not just a green run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… barrier Two independent admission dimensions inside the existing bounded scheduler, plus one coalesced reserved control-barrier entry. All strictly opt-in: the existing four-argument run() signature is unchanged and every pre-existing test passes unmodified. The hard constraint here was not the new behaviour but the OLD behaviour. This scheduler is process-global and sits on every store operation in the daemon, and its admission decision is deliberately entry-INDEPENDENT: canStart() reads four integers and never inspects a queued entry, which is what makes selection O(1) regardless of queue depth. Any per-entry predicate on the default path would convert that into a scan. So the entire extension hangs off one integer comparison. While taggedQueuedCount is zero every queued entry is provably untagged, and nextRunnable() takes the original unconditional queue.shift() with no per-entry scan and no allocation. The tagged path is only reached once something actually carries admission metadata. The fast path has a witness rather than a comment: admissionEvaluations is incremented BEFORE the untagged early return in isEntryAdmissible, so it rises if selection ever walks entries the guard should have kept it away from -- not merely when a tagged entry is examined. A default-path test asserts it stays at zero, so a future change that quietly starts scanning fails a test instead of silently costing latency on every store operation. Blocked entries are skipped in place rather than dequeued, preserving FIFO within (priority, domain) while letting unrelated domains past. A store-wide transition freezes its store even while merely QUEUED, because letting ordinary work past it there is exactly how it would be starved. Untagged legacy entries carry no store identity and so are never gated and never waited on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…ration The supervisor previously kept three ad-hoc closure booleans (`stopping`, `ready`, `restarts`) with no lock, so correctness rested entirely on those flags being set synchronously. This replaces them with one lifecycle mutex and an explicit state machine covering startup, backoff, automatic revive, controlled recovery, stop and terminal close, and layers the process-local ownership lease on top. A generation is bound ONLY after the spawned child is the proven listener owner, never at spawn. That distinction is the whole point: a spawned-but-unproven child must not be able to satisfy a capability check, and the existing findListenOwnerPid proof already fails closed when it cannot establish ownership. Liveness is invalidated on child exit, revive, stop and shutdown, so a store holding the lease observes revocation with no notification. Three pre-existing defects fixed along the way: - `killSync()` poisoned `stop()`. It set the same `stopping` flag, so a later `stop()` returned immediately and never escalated to SIGKILL. Terminating and closed are now distinct states. - The auto-revive timer handle was never stored, so it could not be cancelled; it is now tracked and cleared, and the callback re-checks state so nothing can spawn after close. - `stop()` resolved on the child's exit event without proving the PORT was released. It now re-probes under a bounded deadline, and a bind still served after the child exited burns the lease as terminal (`port-release-unproven`) rather than letting a replacement bind over a listener we do not own. Kills continue to go only through the tracked ChildProcess object, so a foreign listener is never signalled. Recovery coalesces by expected generation, and a STALE expected generation restarts nothing -- it returns the already-newer healthy generation instead of queueing behind or triggering a redundant restart. oxigraph-managed.ts now attaches the lease AND the supervisor handoff to the rewritten adapter options. Both are symbol-keyed, so object spread carries them through the storage factory rewrite while JSON.stringify drops them; a test asserts the persisted projection is byte-identical to before and that a JSON round trip yields no lease. Verified by A/B on Windows: control (these changes stashed) 16 failed / 24 passed / 1 skipped; with them 16 failed / 36 passed / 1 skipped. Identical failure set -- 12 new passing tests and zero regressions. The 16 are the pre-existing Windows-environment set (spawn EFTYPE on the extension-less Node stand-in, the deliberately-forced freebsd-x64 PATH fallback, and cgroup/systemd Linux-only cases); the suite is proven on Linux CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
The harness resolves @origintrail-official/dkg-storage through its published dist/ types, so the closure must exist before any typecheck. Omitting the build passed locally -- where a dist is usually left over from a previous run -- and failed only in CI, which starts from a clean checkout. Mirrors the RFC-64 gate0 workflow, which builds its closure for the same reason. Reproduced locally by removing packages/storage/dist and confirming the exact CI error, then confirming the build step clears it. Note the repro needed tsconfig.tsbuildinfo removed too: with it present tsc reports 'Done' while emitting nothing, so dist/index.d.ts never reappears and the failure looks unfixable. CI is unaffected by that, having no incremental state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…eduler work invalidated The comment justified the process-global single-registration invariant by claiming the store scheduler has no store identity, so two managed controllers could not be told apart during a control barrier. That stopped being true in the same PR: the scheduler now carries an opaque storeId and scopes barriers per (storeId, purpose), and its own tests drive two distinct stores with concurrently pending barriers. The invariant is still right, so this is a comment fix rather than a behaviour change -- but a reviewer reading the stack top-to-bottom would have hit a stated justification that the code two files over contradicts. Replaced with the reason that actually holds: there is exactly one daemon-managed child by construction, so a second controller is necessarily wrong about what it owns, and its recovery could stop a child the first is mid-write against. Also records explicitly that enforcement lives here rather than in the scheduler on purpose -- 'who owns the managed store' needs one source of truth, and duplicating it would create a second one to drift. Caught by the scheduler implementation review rather than by a test, since no test can assert that a comment is still true. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…end to end Previously oxigraph-managed.ts attached only the lease, and the adapter requires BOTH a live lease and a supervisor handoff before it advertises the lane. The composition was therefore fail-closed but inert: correct, yet nothing could ever open the lane in the real daemon. This wires the second half. The handoff exists because the two halves of a clean generation swap have different owners. The adapter owns the connection pool and can destroy its own sockets; only the supervisor holds the ChildProcess and can assert PROCESS facts -- that a child exited, that its port was released, that a replacement is the proven listener. Neither can honestly assert the other's half. stopAndProveOwnedChildDead is deliberately NOT stop(). It leaves the supervisor open for a replacement, detaches the child before signalling so the exit it causes cannot be misread as a crash and schedule a restart behind it, and then THROWS if release cannot be proven -- with the lease already terminal at port-release-unproven. stop() keeps its old non-throwing behaviour because it runs from finally blocks and process teardown where a rejection would mask the original error, and because nothing can bind a replacement after close anyway. The handoff has the opposite constraint: its caller is about to bind one. One guard beyond the brief, and it is the important one. The lifecycle mutex is released between the two halves, so an armed backoff or a consumer recoverGeneration() could bind a generation the lane never asked for, over the port the lane believes it has freed -- which would defeat the entire handoff. A handoffPhase flag disarms revive and refuses recovery while retired. If the start half then fails it reaps the unproven child, clears the phase and re-arms ordinary backoff, so a lane that abandons the handoff cannot leave the store permanently childless. Verified independently rather than on report: 16 failed / 41 passed / 1 skipped against 16 / 36 / 1 before, so five new tests pass and the failure set is unchanged (the 16 are the A/B-confirmed Windows environmental set). Four more serial mutants each killed exactly their predicted test, including passing only two arguments here -- which reproduces the inert state this commit fixes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…ive time Found by probing the owned client directly rather than by a test, and it is the dangerous kind of bug: it failed silently by SUCCEEDING. `req.setTimeout()` bounds socket-ACTIVE time only. It does not count time spent waiting for a free socket, and this pool is deliberately maxSockets:1, so a second concurrent call queues behind the first. Measured: a call with a 500 ms timeout resolved SUCCESSFULLY after 3822 ms. Nothing errored, nothing logged -- the caller simply got a success far outside the deadline it was promised, which would blow both the 1,000 ms apply bound and the three-second slice while the node looked healthy. The system-record lane treats its apply timeout as a SAFETY bound, so queue wait has to count against it. The fix needed two steps, and the intermediate state is worth recording because it looked correct and was not. Arming a wall-clock timer that destroys the request produced the right ERROR but the wrong LATENCY: still 3825 ms, because destroying a request that has not yet been assigned a socket does not emit `error` until one arrives. The deadline must bound when the CALLER is released, not merely when teardown starts, so it now settles first and tears down after. Measured again: 503 ms. The socket-level `req.setTimeout` is kept alongside rather than replaced -- it detects a connection that goes quiet mid-exchange, which the wall-clock deadline would only catch at expiry. Adds `managed-http-client-v1.test.ts` covering the queue-wait bound, the already-aborted branch settling rather than hanging (it returns before `req.end()`), the post-destroy socket count being proven zero, and a destroyed client refusing to dispatch. The queue-wait test is mutation-proven: removing the settle-first call turns exactly that test red at 1555 ms, and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
Five findings from the independent resource/operations track, four of them real defects in code added by this PR. **destroyAndSettle awaited before destroying (was the review's BLOCKER).** `Promise.allSettled(inflight)` ran BEFORE `agent.destroy()`, so the method's duration was `max(callerTimeoutMs across inflight) + poll` and the `timeoutMs` parameter did not bound the phase it names. Reviewer measured 3831 ms for a `destroyAndSettle(5000)` with one 4 s request, and no return at all within 8 s against a server dribbling a byte every 100 ms, because the socket-inactivity timer kept resetting. This is step 1 of the enable handoff, so a slow or chatty endpoint could stall the transition, leave the seal uncommitted and hang daemon teardown -- on a healthy-looking endpoint rather than a dead one. Destroying first forces inflight requests to reject: the same scenario now returns in 5 ms with zero sockets. The settle also shares the deadline, so the terminal-failure branch below it is reachable rather than sitting behind an unbounded await. **The negative capability memo latched permanently in three of four layers.** The blob store, graph-set index and agent facade each cached `null` on first absence and never re-probed, while the producer returns undefined during ANY window in which the managed child is not the proven-ready listener -- i.e. every ordinary revive. One probe landing inside a one-second Oxigraph restart would have disabled the lane for the whole process lifetime, silently, recoverable only by a daemon restart. Now only a PRESENT controller is memoized; absence is re-probed. ChangelogStore already forwarded live and is unchanged. **awaitRetiredWork was structurally a no-op.** `destroyClient` nulled `managedClient` and `awaitRetiredWork` then read that same null, so step 3 of a sequence whose own docstring calls it "load-bearing and asserted step by step" could never drain anything. Harmless in enable only because step 1 already awaited settlement -- but actively wrong in disable, which calls `awaitRetiredWork` with no preceding `destroyClient` and so destroyed the LIVE client and left the field pointing at a dead pool bound to the current generation. A retired client is now held in its own field. **maxSockets restated a frozen limit it cites.** Now imports SYSTEM_RECORD_MAX_MATERIALIZER_WRITE_CONCURRENCY rather than hardcoding 1, so a future bump cannot leave the pool pinned with a comment claiming they agree. **Per-write allocation on the always-on guard.** `quads.map((q) => q.graph)` built a throwaway array the length of every write batch -- measured +14.3 ms and ~8 MB transient at 1,000,000 quads. This is the only thing in B2 that runs on every write on every node today, so it now iterates quads directly, matching what ChangelogStore already does. Also removes `excludeInternalGraphsV1`, which had zero production callers: every enumeration surface still filters through `isAtomicGraphReplaceStagingGraph`, so shipping a second unused filter implied a migration that has not happened. Its test is replaced by one asserting the predicate production ACTUALLY calls, with a positive control so a predicate returning true for everything would fail. `isInternalGraphUriV1` and `isEphemeralInternalStagingGraphUriV1` drop off the public barrel for the same reason and stay module-scoped for the tests that pin the reserved/ephemeral partition. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…ceMs The comment claimed the probes were 'spread across stopGraceMs so stop() stays a bounded operation'. The bound is real but it is not stopGraceMs: five attempts at up to readyIntervalMs+1000 each, separated by floor(stopGraceMs/attempts), is ~11.5s, on top of stopGraceMs for the SIGTERM/SIGKILL wait and the bounded listener lookup. Worst case ~22-25s. This matters because stopGraceMs is exactly what an operator would reach for when tuning a shutdown timeout, and a stated bound is what a future reader budgets against. The cost is defensible and the comment now says why: it accrues ONLY while something is still serving our bind after our own child exited, which is the leaked-descendant case where guessing 'released' is worse than waiting. A cheaper probe would buy speed with more false 'released' verdicts, which is the wrong trade for a proof. Also records that there is no durability exposure -- probing starts after the child exits, so RocksDB is already closed and a hard kill at an outer deadline costs a log line, not the store. Comment only; no behaviour change. Raised by the independent resource/operations review, which also confirmed the timing arithmetic and the absence of durability risk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
Two BLOCKERs from the independent protocol/safety review, both demonstrated by an executed exploit rather than argued, and both squarely mine. **Shutdown did not supersede anything.** `runShutdown()` inspected `this.transition` only for `kind === 'shutdown'`, so an in-flight open was neither awaited nor cancelled -- just silently clobbered by overwriting the field, and left running concurrently. Observed: a stalled enable resumed AFTER shutdown completed, set the lane back to `enabled` with activation generation 2, and started a replacement child after shutdown had already proved the old one dead and its port released. The same root cause made a shutdown issued during a disable end at `disabled`, leaving the lane non-terminal and freely re-openable. Shutdown now JOINS the in-flight transition. Superseding something means outliving it, not racing it. **A shut-down lane revived and dispatched a write.** `open()` checked terminal state BEFORE `await this.transition.work` and never re-checked after it, so an open that joined a shutdown proceeded to run the full handoff on a lane the process had already terminated -- and step 5 of the exploit was the one that mattered: `applyVerified` on that session dispatched to the executor and returned `applied`. Every post-await path now re-reads state. TypeScript actively resisted the fix, which is worth recording: it narrows `this.current` and does NOT invalidate the narrowing across an `await`, so it reported the re-checks as impossible comparisons on types with "no overlap". That assumption -- nobody mutates the field while we are suspended -- is exactly the bug. Reads now go through `readState()` so the union stays wide and a future reader is not tempted to delete the guards as dead code. Also from the same review: - **MAJOR: `runEnable` had no post-condition.** It set `enabled` because no step threw, never verifying that `startAndProveCleanGeneration()` actually bound a ready generation; a handoff resolving without binding one produced an "enabled" lane over a child that was not the proven listener. It now asserts a ready, non-terminal snapshot and fails to `unavailable` otherwise. - **MAJOR: the advertising gate checked `terminal` but not `ready`**, despite its own docstring promising exactly the `ready` check. Added. - **MAJOR: the process-global registration was never released on shutdown**, so a replacement controller was unconstructable for the process lifetime. Worse, the adapter called the factory inside an unguarded capability probe, so the refusal propagated out of `getSystemRecordLaneControllerV1?.()` through three decorators -- merely ASKING whether the capability existed could throw. The registration is released on shutdown and the probe now fails closed to `undefined`, because absence is the correct answer to "can I have the lane?". - **MAJOR: unscoped `deleteByPattern` bypassed the reserved-graph guard.** With no `pattern.graph` the guard body (`if (graph)`) is a no-op and the emitted update binds `?g_ctx` across every named graph, reserved ones included. The unbound form now excludes the operation-internal prefix in its WHERE; no legitimate caller targets those graphs, so no real caller's semantics change. The suite drove this machine only SEQUENTIALLY, which is the sole reason both blockers shipped. Four concurrency regression tests added. Both are mutation-proven: restoring the clobbering shutdown reddens the stalled-re-open test, and removing the post-await terminal re-check reddens the dispatch-after-shutdown test -- each exactly one test, nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…d held calls Two HIGH findings from the independent resource/operations review, both reproduced to the reviewer's exact numbers before being fixed. **HIGH 2 — a barrier did not stop untagged traffic, which is 100% of today's store traffic.** The transition IS the Oxigraph stop-and-restart, so for the whole restart window the scheduler kept dispatching legacy queries and writes at full concurrency into a child being SIGTERM'd and then into no listener at all. Measured at maxConcurrent 4: 20 new untagged ops STARTED during the transition; now 0, with all 20 held in queue and completing after it. Untagged entries are gated at selection while a barrier is pending, and barrier readiness waits for untagged inflight derived as `total - tagged`, so the untagged path pays nothing to be waitable. Held work is strictly better than dispatched work here: entries stay queued, keep their existing wait timer, and fail with a typed retryable STORE_SCHEDULER_BUSY rather than transport-level ECONNREFUSED. That is backpressure instead of an error burst, and it is something the caller can actually retry. Barrier readiness is deliberately no longer generation-scoped: one child process serves every generation, so waiting only on the sealed one was wrong for exactly the case the barrier exists to serve. Tagged work for OTHER stores is still not waited on, preserving the two-store no-head-of-line-blocking property. **HIGH 3 — held calls were unbounded, and release turned them into a queue_full burst.** 200 tagged calls under one seal at queueLimits 8 produced 200 held and then 190 rejections AT COMMIT -- callers rejected only after waiting out the entire seal, which is strictly worse than rejecting at admission. Accumulation had no cap: 100,000 held runs measured at 143.6 MB. A held call has been ADMITTED, so it now counts against its lane's queue limit exactly like a queued one. That single accounting change fixes both halves: `queued + held <= limit` is invariant across hold, enqueue and release, so every held call still fits when the seal commits. Now: 192 rejected at admission, 8 held, 8 ok, and ZERO rejected after waiting -- a provable property rather than an empirical one. The ceiling is the queue limit (~385 KB worst case at the measured ~1505 B per held call) rather than caller volume. No timer was added. A timer would break the contract the seal exists to provide and turn a control transition into a source of spurious failures; the barrier commits its seal in a `finally`, so holding forever requires a control-path bug whose blast radius is now bounded by the lane limit. The review also diagnosed WHY HIGH 3 survived testing: the hold test used exactly one call at queueLimits 1, the single arity at which a release cannot overrun, so it could only confirm what it expected. The replacement holds 200 at a limit of 8 and asserts the full outcome distribution split by WHEN rejection happened. Two other tests used the same degenerate arity and were raised. Seven mutants applied serially, presence grepped, all killed. One is worth recording: the first attempt at M17 SURVIVED, but the mutant was invalid -- it added a pump call instead of moving one, so the correct later pump still fired. Rewritten as a genuine move it kills three tests. A surviving mutant that turns out to be a bad mutant is not evidence of coverage. Fast-path invariance is intact: the guard gained one integer term (`taggedQueuedCount > 0 || barriers.length > 0`) and the invariance test still asserts zero admission evaluations on a pure untagged workload, with a tagged positive control. Independent re-benchmarking of three new per-operation costs on the untagged path is outstanding and tracked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
Four findings from the independent execution/QA review. The gate was reporting "PASS: 18 checks" while thirteen of those checks were assertions about literals. **Thirteen verdict rows were constants the generator hardcoded.** Foreign-process signals, old-generation dispatches, stale-facade dispatches, barrier slot-ms, healthy deadline recoveries, indeterminate and recovery latencies -- all emitted as `0` by `run.ts`, all asserted `=== 0` by the verifier. No code path could have made any of them non-zero: the gate never restarts a child, never opens a lane session and never runs a barrier. They are deleted rather than left in place. A verdict that overclaims is worse than a smaller one, because the smaller one does not stop anyone looking. Those rows return when something actually drives them, which is the CAS stack. The sharpest instance: `indeterminateReturnWithinThreeSeconds` was the one check nominally bounding "the write returns in time", and it was asserted against a hardcoded `0`. A request that returned after 3822 ms under a 500 ms deadline -- a real bug found in this PR by probing, not by testing -- would have passed it by construction. **`leakedOwnedSockets` was 0-or-throw.** `destroyAndSettle` loops until the count reaches zero and throws otherwise, so reading it afterwards can only ever be 0 and the check could never report a failure. It is now a PAIR: the socket count is captured BEFORE destroy and must be non-zero, which makes the probe demonstrably live, and the post-destroy zero means something. **The manifest carried a fabricated commit SHA.** `843f5213d` zero-padded to 40 characters is not a commit and resolves to nothing, and an adjacent field claimed "the gate resolves and records the exact 40-character SHA at run time and fails if it cannot" -- which it did neither of. That value was copied verbatim into the artifact CI uploads as evidence, so the published verdict attributed a pass to a commit that does not exist. Replaced with the real SHA 843f521, the false claim removed, and a real check added: every pinned commit must resolve via `git cat-file -e`. Verified to discriminate -- re-injecting the fabricated SHA now fails the gate with `everyManifestCommitResolves: a pinned commit does not exist`. **A crashed run left the previous PASS artifact verifiable.** Found while mutation-testing the above: a malformed manifest crashed the generator, the stale artifact stayed on disk, and the standalone verifier re-verified it and exited 0 -- certifying a run that never completed. `run.ts` now deletes both artifacts before doing anything, and tolerates a UTF-8 BOM rather than crashing on one. The predecessor section is renamed to say what it does: these entries run against the CURRENT binary, and no predecessor is checked out or built. The manifest's role today is to pin WHICH commits must keep the property and to require that each resolves. `manifestIsNonEmptyAndFullyIterated` is documented as structural, its real content being the non-empty term. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…instead of hanging Quiescing untagged work (previous commit) closed an ECONNREFUSED burst but replaced it with a worse failure mode: a caller that issues store work from INSIDE a barrier transition now waits for work that cannot proceed until the transition it is inside returns. Circular wait, and silent. The independent resource review hit it for real -- its own attack harness deadlocked with Node exiting 13 on an unsettled top-level await. A deadlock is harder to diagnose and harder to alert on than the burst it replaced. Exact re-entry detection was considered and REFUSED on measured grounds rather than deferred. `AsyncLocalStorage` latches async_hooks on for the rest of the process after a single `run()`; `disable()` exists but enable/disable churn is itself a deopt event and is version-dependent. That is a permanent process-wide tax on every async operation in the daemon, paid to diagnose a caller bug faster, on a scheduler whose measured default-path cost is +0.003%. Wrong trade. The ambiguity is genuine, not incidental: during a running transition, a held `run()` is EITHER the transition re-entering (deadlock) OR an unrelated caller legitimately waiting. Those are indistinguishable at the call site without ambient context. So the bound compensates for what it cannot prove -- the error names re-entry as the usual cause and carries a `blockedBy` breakdown (untagged inflight, tagged inflight, generations, held runs), putting the diagnosis on the operator's screen even though the scheduler cannot demonstrate it. `StoreControlBarrierTimeoutError` (`STORE_CONTROL_BARRIER_TIMEOUT`), default 60s, env-overridable, per-call override. Deliberately double the child's 30s readyTimeoutMs: this is meant to catch a circular wait, not a slow disk. Recovery differs by phase, and the asymmetry is deliberate: - WAIT phase: the transition never started, so nothing is disrupted. The barrier withdraws completely -- seal released, held runs released, selection woken. Fully recoverable, and this is the phase the real deadlock lands in. - TRANSITION phase: the transition may be part-way through stopping a child. Releasing the seal would admit work into a store that no longer exists, which is HIGH 2 recreated at the worst possible moment. The caller is told and the seal is left in place; if the transition ever settles its own `finally` still cleans up, so this reports without permanently freezing the lane. Three tests, and the third is what stops the bound becoming a nuisance: the real deadlock shape asserting wait-phase withdrawal AND that the store resumes; the re-entrant-transition shape asserting the seal deliberately outlives the rejection; and an ordinary slow transition completing untouched with `barrierTimeouts: 0`, so a bound that fired on every real restart would be caught. Five mutants, serial, all killed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…its value The comment explained the CHOICE of 60s without stating the CONSTRAINT a tuner needs. It now says the relationship to the child's readyTimeoutMs is the constraint rather than the number, names the failure mode of setting it too low -- anything at or below readyTimeoutMs turns every genuinely slow restart into a spurious timeout -- and says why that is worse than no bound at all: it trains operators to ignore the one signal that means a transition actually deadlocked. Also names both tuning surfaces and the raise-together rule. Comment-only, and verified as such rather than assumed: the diff touches no coverage directives or ts-directives, 47/47 tests still pass and the build exits 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…fire The bound's timer is unref'd deliberately -- a safety timer that kept a daemon alive through its own shutdown would be worse than the deadlock it guards. The consequence is not obvious and was found by an independent re-run rather than by reasoning: the bound can only fire while something else holds the event loop open. A daemon always qualifies, since a listening server holds a ref'd handle. A process that has quiesced down to nothing but the deadlock exits silently instead -- the pre-bound behaviour, in the one situation nobody is watching. The practical trap is for tests: a real-timer test of this bound must hold the loop open itself, or the process exits before the timer fires and the run is recorded as a failure with a bare non-zero exit and no output. That is exactly how the reviewer's first re-run presented, and it looks identical to the deadlock the bound was added to fix. The tests here use fake timers and are unaffected. Comment-only; no coverage or ts-directives touched. 47/47 pass, build exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…ase properly
Five findings from the independent review of the supervisor handoff.
**BLOCKER — a synchronous spawn throw stranded the handoff phase permanently.**
`child_process.spawn` throws SYNCHRONOUSLY for EACCES/EFTYPE/E2BIG/EINVAL; the
`.once('error')` handler catches only async failures, so the throw escaped
`startCleanGenerationLocked` and bypassed the only code that clears
`handoffPhase` and re-arms backoff. Measured: 30 backoff periods later, still
one spawn, port unserved, self-heal refusing, and `terminal: false` -- so the
lease reported "momentarily not ready" indefinitely instead of failing closed.
The daemon's triple store was dead for the process lifetime, reachable by a
binary being replaced or re-permissioned under a running daemon. Everything from
`state = 'recovering'` is now inside a try/catch that runs the existing failure
tail and rethrows, covering `spawnChild()`'s own assertion and
`bindProvenGeneration()` throwing on a terminal lease (the same class, milder).
**HIGH — the release proof accepted a probe TIMEOUT as evidence of release.**
`endpointAnswers()` collapsed ECONNREFUSED (positive evidence nothing is bound)
and TimeoutError (no evidence at all) into one `false`. That collapse is
fail-CLOSED in `probeReady()` and correct there; the release proof reused the
same primitive with the OPPOSITE polarity, making it fail-OPEN, and its loop
accepted the FIRST non-answer. Since #2052 is a store-PRESSURE issue, a 1.5s ASK
timeout against a loaded listener is the expected case: the retire resolved in
137ms with ownership non-terminal, which is exactly the state where a
replacement may bind over a live listener.
Fixed at the root rather than patched: `probeBind()` now returns three states --
`serving | refused | inconclusive` -- with refusal detection walking `cause` and
`AggregateError.errors`, since `fetch` wraps the OS error. ECONNRESET is
deliberately NOT proof: something was there. The loop is inverted to retry until
it sees a refusal and return false if it never does. `endpointAnswers()` is
unchanged for readiness, which genuinely wants the collapse.
**HIGH — nothing bounded the retired phase.** Fixed by bounding it rather than
by making a teardown terminal, which is where the reviewer's proposal (mine) was
wrong: the managed child is the DAEMON'S ENTIRE TRIPLE STORE, not the lane's
private one, so a terminal teardown would mean disabling the lane permanently
kills node storage until restart -- strictly worse than the parked state it
fixes. On expiry the phase clears and ordinary supervision resumes; a lane that
returns late is refused by the proven-dead-predecessor check. Fail-closed for
the lane, alive for everyone else.
**MAJOR — a failed retire left the supervisor open and childless.** It now
terminates and closes, so no path can spawn against the unaccounted-for
listener. The residual is real and outside this module: ordinary store traffic
never consults the lease, so the daemon keeps POSTing to that endpoint. A FATAL
log names the condition; closing it properly needs an adapter-side kill switch
and is tracked separately.
**MAJOR — an inert guard removed rather than papered over.** The
`handoffPhase === 'retired'` clause in `scheduleRevive` changed 0 of 57 test
statuses when deleted. Attempts to give it a discriminating test confirmed why:
every `scheduleRevive` caller is unreachable while retired. Replaced with a
comment explaining the invariant. For the same reason a proposed guard in
`reviveLocked`/`spawnChild` was NOT added -- it would have been inert by the
identical argument, and the property now holds by construction: both recovery
tails clear the phase before any spawn.
Four mutants, one test each, serial and grep-verified. One process note worth
keeping: a first attempt turned `try {` into `if (true) {`, leaving a dangling
`catch`; vitest reported "no tests" rather than failing, which is not a green
run but skims as one. A mutant only counts if the suite actually collected
tests. A second mutant was also re-run in isolation because two could each
explain the same red, and its failure message -- the retire wrongly resolving --
confirms the kill is for the right reason.
Verified: 16 failed / 44 passed / 1 skipped, up from 41 passed, with the same 16
Windows-environmental failures and zero regressions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
Merge-readiness reviewReviewed commit: Verdict: not ready to merge. The ownership, capability-discovery, and reserved-state foundations are directionally sound, and the focused suite is green, but the production composition currently advertises an openable live lane without routing the child handoff through the store control barrier. I also reproduced three independent state-machine/policy defects. The first four findings should be fixed before merge. 1. BLOCKER: the live handoff bypasses the scheduler control barrier
I reproduced this with the real Current behavior: sequenceDiagram
participant Q as Ordinary request
participant S as Store scheduler
participant L as System-record lane
participant O as Oxigraph supervisor
participant X as Oxigraph child
Q->>S: start request
S->>X: request remains in flight
L->>O: stop child (no barrier)
O-xX: terminate
X--xQ: transport failure / ambiguous result
Required behavior: sequenceDiagram
participant Q as Store traffic
participant S as Store scheduler
participant L as System-record lane
participant O as Oxigraph supervisor
L->>S: runControlBarrier(storeId, generation)
S-->>L: admission sealed; tagged and untagged work drained
L->>O: stop and prove release
L->>O: start and prove replacement
L->>S: commit generation/epoch transition
S-->>Q: resume against replacement generation
Fix direction: route every enable/disable/shutdown lifecycle transition that can invalidate the client/child through 2. BLOCKER: a generation-raced success returns
|
…lose the namespace guard Four findings from the PR review on b1c1bdf, each reproduced before fixing. **BLOCKER 2 — a generation-raced success returned indeterminate WITHOUT sealing.** The lane sealed only when the EXECUTOR returned indeterminate. When a success was synthesized into indeterminate because the generation changed under the dispatch, it was returned to the caller while the lane stayed `enabled`, so the next apply was admitted: two dispatches, both ambiguous, state `enabled` throughout. The final outcome is now derived first and the seal keyed on IT, however it was reached. Attribution also widened from a generation-string comparison to the whole ownership snapshot. A lease that went not-ready or terminal under the dispatch is equally unable to attribute the response, and comparing only `childGeneration` missed both. Its own regression test pins that: readiness is lost while the generation string is unchanged, and the outcome must still be indeterminate. **HIGH 3 — concurrent shutdowns each ran a teardown.** Both callers passed the "already shutting down?" check while the in-flight transition was still the OPEN, both awaited it, and both ran a teardown -- 4 destroy / 4 stop where 3 of each were expected. Double stop-and-prove-release is not a harmless idempotency assumption at a process-ownership boundary: each one signals a child and asserts a port fact. The fix is one line's PLACEMENT. `runShutdown` is no longer `async`, so the `this.transition` assignment happens synchronously before any await and a second caller joins at the top. Worth recording how that was established, because the first attempt was wrong: a separate `shutdownWork` coalescing field was added and its mutant SURVIVED -- twice, including against the reviewer's exact stalled-open shape. It never changed an outcome, because the synchronous assignment already coalesces. It was removed rather than kept, per the standard adopted earlier in this branch: an inert guard reads as protection and is worse than none. Removing the synchronous assignment instead reddens all three shutdown tests, which is what identifies it as the load-bearing change. **HIGH 4 — unknown names in the internal namespace were writable but hidden.** The guard rejected only the two known-reserved names, so a near-miss or future name such as `...:system-record-v1:future-reserved` was accepted by generic mutation while the prefix-wide enumeration filter hid it -- invisible durable state, writable by anyone who can spell the prefix, and a standing namespace-confusion risk against whatever a later stack reserves for real. It also directly contradicted this module's own docstring, which claimed such names were "not writable". Generic mutation now refuses the WHOLE internal namespace except canonical UUID-shaped ephemeral staging graphs, which stay writable because the atomic-replace builders must be able to drop their own. **LOW 6 — encoding damage, all of it self-inflicted.** Several whole-file PowerShell rewrites in this branch re-encoded UTF-8 as cp1252, corrupting comments in files I touched INCLUDING pre-existing lines. Verified against the base, which had zero mojibake, and the restored pre-existing lines are now byte-identical to it. Repaired with an escape-only script, because the first repair attempt was itself corrupted in transit -- the same mechanism that caused the damage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… barrier BLOCKER 1 from the review on b1c1bdf, and the finding lands hardest because the missing piece was already written: `runControlBarrier` / `sealStoreGeneration` shipped in this stack exported, documented and covered by 25 unit tests, with ZERO production callers. The live handoff stopped the owned child, asserted its port free and bound a replacement while ordinary requests were still in flight on the retired generation -- exactly the window the barrier exists to close. `SystemRecordLaneControllerDepsV1.barrier` is REQUIRED, not optional. An optional barrier is one that gets forgotten, which is the defect this commit fixes; making it mandatory means a future construction site cannot omit it silently. Enable, disable and shutdown each open their own named section. The materializer stays store-agnostic -- it takes an exclusive-section runner, not a scheduler -- and the adapter supplies one backed by the real barrier, using `this` as the opaque store identity and re-reading the lease snapshot's generation per transition rather than capturing the one observed at construction (the lane outlives any single child). Shutdown under a barrier that cannot be acquired: the teardown does NOT run, and that is the safe side. The lane still reaches terminal, so nothing writes through it again, while the child is left alive under the daemon supervisor that still owns it and stops it at process exit. Stopping the child outside a section is the hazard; "could not quiesce" is not a reason to do it anyway. The error propagates, because a shutdown that could not quiesce the store is not a clean one. **Evidence.** The unit tests bracket each transition, so they pin that a section is opened AND that every child-touching step runs inside it. They cannot show that asking for a section does anything, because they inject a pass-through stand-in -- so `system-record-control-barrier-integration-v1.test.ts` drives a real adapter, the real process-global scheduler and the real handoff composition against a genuinely held-open ordinary request. Each half was mutated separately: - adapter barrier -> pass-through: the integration test fails with the reviewer's exact symptom, `['stop','start']` observed while the request is outstanding. All 41 lane unit tests stay GREEN under that mutant, which is the direct demonstration that they could not have caught this. - enable/disable/shutdown wrapping removed one at a time: each reddens its own bracketing test and nothing else. The integration file also carries its own discrimination: a quiesced-store control proving the wait is caused by the in-flight request rather than by the open being async, and a request issued DURING the section that must complete afterwards -- a transition that sealed the store and never resumed it would be a silent permanent outage, which "no stop happened" alone would score as a pass. Storage suite: 90/90 across the B2 files. The full package shows 7-16 failures in `oxigraph-worker-{resilience,respawn}` and one worker-backed factory case; those are 5 s timeouts under ~70 s import cost and reproduce on `main` with none of this stack present (8/11 red there), so they are box contention, not this change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MEDIUM 5 from the review: the gate's workflow header claimed properties of a
separate process and its listen socket, but the gate minted ownership by hand and
injected no-op supervisor methods -- it never restarted a child, never opened a
lane and never ran a barrier. The review's own words for it: "this is why finding
1 passes the advertised live CI gate."
Of the two offered directions -- narrow the claims, or drive the real thing --
this takes the second. Narrowing alone would have left the gate structurally
unable to catch a regression of the very defect it is named after.
The gate now runs ONE real generation handoff first: `startOxigraphServer` from
the CLI daemon owns and spawns the child, its `ownership.lease` and
`supervisorHandoff` travel to the adapter under the symbol key exactly as the
daemon composes them, and the lane transition goes through the process-global
scheduler's control barrier. The only interposition is an observer that records
a reading and delegates straight through.
The probe workload is a 60,000-quad ordinary insert: one scheduler admission, one
HTTP request over the store's own path, and interrupting it is precisely the harm
the barrier prevents.
Ten measured checks, each a fact about processes rather than about a literal:
- the write was STILL RUNNING when the lane was asked to open (without this
the ordering check is vacuous -- a finished write satisfies it regardless)
- zero store requests inflight at the instant the child was stopped
- the in-flight write survived the restart
- the generation advanced; a SECOND OS process was spawned; the first PID is
dead and the last is alive
- the replacement serves the same 60,000 quads (a handoff that silently
produced an empty store would otherwise satisfy everything above)
- the store resumed against the replacement; the lane reports enabled
On the inflight reading: it replaced a wall-clock "stop happened after the write
settled" comparison, which was subtly wrong. The scheduler releases admission
when the store work resolves, and the caller's promise settles a few microtasks
later through `workLifecycle` and `insert`'s tail, so a CORRECT handoff measured
as 0.09 ms too early. The pre-fix build read -49 ms, so any tolerance would have
been chosen to sit between two numbers rather than to mean anything. The inflight
count has no such boundary -- it is the quantity the barrier waits on.
**The gate is proven to discriminate, not merely to pass.** Built with the
adapter's barrier replaced by a pass-through and re-run end to end:
childStoppedOnlyAfterOrdinaryWorkDrained: 1 store request(s) inflight
inflightOrdinaryWriteSurvivedTheHandoff: fetch failed
replacementServesTheSameData: 0/0 quads
-> FAILED (3 of 20 checks)
With the barrier restored: PASS, 20 checks, 3 predecessor entries. That negative
control was not constructed -- the first run of the new section hit a stale
`dist/` predating the fix and reproduced the reviewer's finding unprompted.
The workflow header is also rewritten to enumerate what is proven and, explicitly,
what is not: predecessor entries run against the CURRENT binary rather than each
pinned commit, and no verified apply is dispatched until the CAS lands. The unit
step gains the barrier integration and scheduler admission suites.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
…ally key on The `runControlBarrier` docstring said it "waits for the sealed generation's execution permits to drain". It does not: `isBarrierReady()` waits on this store's whole `taggedInflight`, and the hold is `seals > 0`. Both are generation-blind, deliberately -- the transition is a child-process restart and the child it stops serves every generation. This matters beyond the wording. The PR description carried a "known asymmetry the CAS PR must resolve" section derived from this sentence, warning that a future fix must be an allow-list rather than a not-equal predicate. There is no asymmetry to fix; the sentence was the whole source of it. Left in place it would have invited someone to NARROW the drain to the sealed generation to restore a symmetry that already exists, which is strictly weaker and -- since generations are decimal counters -- trivially bypassable. `sealStoreGeneration` now states what its `generation` argument is: a diagnostic label carried on the seal and reported through `admissionGenerationsInflight`, not an enforcement key. Comment-only, and checked as such: the scheduler suites are unchanged at 47/47. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review addressed — all six findingsReviewed head
1 — the barrierConfirmed exactly as reported:
One decision worth flagging, since it is not what your fix direction implies: shutdown under an unacquirable barrier does not run the teardown. The lane still reaches terminal so nothing writes through it again, and the child is left alive under the supervisor that still owns it and stops it at process exit. Stopping the child outside a section is the hazard; "could not quiesce" is not a reason to do it anyway. The error propagates, because a shutdown that could not quiesce the store is not a clean one. The integration test you asked for is 5 — the gate now discriminatesYou offered two directions; this takes the second. Narrowing the claims alone would have left the gate structurally unable to catch a regression of the defect it is named after. The gate drives That negative control was not constructed. The first run of the new section hit a stale The workflow header now enumerates what is proven and what is not — predecessor entries run against the current binary rather than each pinned commit, and no verified apply is dispatched until the CAS lands. 3 — how the fix was reached, because the first attempt was wrongI first added a separate A stale claim in my own PR description, now correctedThe description carried a "known asymmetry the CAS PR must resolve" section: a generation-scoped drain against a generation-blind hold. Re-read at this head, that was wrong. It came from the scheduler's own docstring, which claimed the barrier "waits for the sealed generation's execution permits to drain". Corrected in Merge gates
On the storage full-suite failures in your verification notes: the |
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
The gate's typecheck went red in CI with a wall of TS2307 and the implicit-any / unknown-catch errors that cascade from the degraded types, while the exact same command passed locally. Cause, and it is the harness's own import graph rather than anything about the packages: `run.ts` now imports `daemon/oxigraph-server` by relative SOURCE path, because the CLI package exports no such subpath. That module imports `./routes/status.js`, which reaches `dkg-agent`, `dkg-node-ui`, `dkg-publisher`, `dkg-chain`, `dkg-mcp` and both adapters. tsc follows all of it, so every one of those needs real `dist/` types -- and the build step only built storage's closure. Locally they were all present from earlier builds, which is precisely why this passed here and only here. Reproduced before fixing rather than reasoned about: renaming `packages/agent/dist` out of the way reproduces the same TS2307 class locally, and restoring it clears them. The wide build was then run end to end and the typecheck re-run clean. The filter now matches what `test:live:system-record-managed-ownership:generate` already builds. That alignment is the point, not a convenience: a typecheck over a narrower closure than the run itself uses can only ever produce a false result in one direction or the other. No added cost either -- the generate step was already running this same wide build seconds later. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
CI follow-up — the gate went red on its own harness closure, now greenMy previous comment reported the live gate passing. It passed locally and then failed in CI on the first run at Cause, and it is worth stating because it is the same class as the finding above. Reproduced before fixing rather than reasoned about: renaming Fixed in
Merge gate 3 is unchanged and still blocking: #2103 is open, so the full CI matrix has not run. Do not merge while this is based on |
Merge-readiness review — round 2Reviewed commit: Verdict: not ready to merge; one lifecycle blocker remains. Findings 1, 2, 4, 5, and 6 from the previous review are materially addressed: production lifecycle calls now use the real scheduler barrier, synthesized indeterminate outcomes seal admission, the internal namespace guard is prefix-wide, the live gate drives a real child handoff, and the encoding damage is repaired. The shutdown fix narrows finding 3, but an older transition can still erase the shutdown intent and allow a later transition to revive the terminal session. 1. BLOCKER: a superseded
|
Round-2 BLOCKER on e8788a8, confirmed and then found to be wider than reported. This does not patch the pointer a third time; it moves what the pointer was carrying onto the state field. **The finding, reproduced against the REAL scheduler.** An older `open()` cleared `this.transition` unconditionally in its `finally`, erasing the shutdown entry that `runShutdown` had installed while joining it. A later `disable` then saw no pointer and a non-terminal state, was admitted, queued behind the shutdown barrier, and wrote `disabled` over the committed terminal state. Downstream, and worse than the review reported: `open()` then RESOLVED, spawned a fresh child after shutdown had proved the old one dead and asserted its port released, and `applyVerified` dispatched and returned `applied`. A second `close('shutdown')` ran a SECOND full teardown -- 4 destroy / 4 stop for one session. The mirror case through `close('disable')`'s own `finally` is equally real. **Root cause, which is one layer under that.** Seven of the nine writes to `this.current` happen AFTER an await and none re-validated that it was still the authoritative transition. Terminality was asserted by READERS (`assertNotTerminal`, `applyVerified`'s ladder) and enforced by no writer, so it could only ever be stale. And shutdown had no representation before it committed: `runEnable`/`runDisable` publish `enabling`/`disabling` synchronously, but `runShutdown` wrote `current` only in its `finally`, so for the whole teardown the lane still read `enabled`. **The change.** Shutdown commits `current = 'shutdown'` SYNCHRONOUSLY at intent, before the method can suspend. Every other write to `current` goes through `commitState`, which refuses to move off a terminal state. That is a change of CARRIER, not a third patch -- and it is why `applyVerified` needed no admission change: a dispatch during the teardown now sees a terminal lane and returns `capability-lost` instead of being admitted. Four consequences that follow, each verified: - `release(entry)` clears the pointer only if it is still ours. The pointer is demoted to a coalescing hint: it decides who joins whom, never whether a transition runs or what state results, so a stale one degrades to a redundant join rather than a wrong transition. - `runShutdown` joins BEFORE checking state. With the latch, checking first would hand every later caller a resolved promise while the child was still being stopped, and they would never learn the teardown failed. - `runEnable` publishes state and activation as ONE commit -- a terminal lane advertising a fresh activation generation is the resurrection the latch exists to prevent. - Two disables queued behind one open ran two `system-record.disable` sections and rotated the epoch twice. One new line fixes it; no design in the review or in my own analysis caught this one. **Three guards DELETED rather than kept**, because under the latch each became a branch that can no longer change an outcome: `close`'s `kind === 'shutdown'` join, the `current = 'shutdown'` write in the teardown's `finally`, and the `if (readState() === 'shutdown') return` inside the teardown -- that last one is now ALWAYS true and would have skipped every teardown. **Evidence.** - 41/41 pre-existing tests pass with ZERO assertion changes. - 12 new interleaving tests. 7 of them FAIL against the shipped model; the other 5 are regression pins for paths the shipped model happens to get right (or for risk this change itself introduces, like the join/state order). I am not claiming 12 discriminators. - Solo-mutant sweep, one guard removed at a time: 11 of 11 killed. - The 11th was earned rather than assumed. `assertNotTerminal()`'s post-join mutant SURVIVED the first sweep with all 52 tests green -- while, without it, a superseded `open()` ran a full destroy/stop/start handoff on an already terminal lane. The guard was load-bearing and the suite could not see the path. The test that now kills it is the last one added. The harness needed upgrading to express any of this: `StallingHandoff` can pause exactly one step, which is why the suite could express "shutdown behind a stalled open" but not "a caller arrives while the TEARDOWN is stalled" -- and the second shape is where both review rounds found their blockers. `GatedHandoff` pauses any named step with several gates live at once, and `track()` makes "has not resolved yet" an assertion rather than a timing hope. Declared behaviour deltas: `state` reads `shutdown` from the instant `close('shutdown')` is called (strictly stronger; both consumers only forward it); a superseded `open()` now rejects `/terminal/` instead of resolving with a terminal session; `close('disable')` during a teardown returns instead of joining; two concurrent disables behind one open run one section. Knowingly stated rather than hidden: `commitState('enabling')` and `commitState('disabling')` are reached with no await between them and their caller's terminal check, so their refusal is currently unreachable. They go through the chokepoint anyway, because the property being relied on is "`commitState` is the only writer of `current`", and a raw write at those two sites reintroduces exactly the await-placement audit that has now been wrong twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ean up on failure
Round-2 findings 2 and 3, plus four "checks that cannot fail" found while
verifying them.
**MEDIUM 2 — no failure-safe cleanup.** The four teardown steps were bare
statements on the SUCCESS tail, so any throw in between leaked a live Oxigraph
child holding an ephemeral port, a temp RocksDB directory, an un-closed store
and — once the capability probe had run — the process-global lane registration.
Six throw sites sit in that window; the dominant one is `lane.open()`, which
rejects on any failed handoff step, so the leak was most likely exactly when the
gate was doing its job. On `ubuntu-latest` a non-detached child is re-parented
rather than reaped and would outlive the job. Resources now register a release
step as they are acquired and a single `finally` unwinds them newest-first, each
caught independently, never masking the original error.
**LOW 3 — a fixed sleep raced against the payload.** Replaced with polling
`getPressureSnapshot()` until the write is observed in flight, bounded at 30 s
and failing the verdict rather than proceeding vacuously. Measured: the 60k-quad
write commits in ~2.1-2.3 s here, so the old 250 ms sample held with ~8x margin
-- but that margin was an accident of the payload size, and at 10k quads the
sample already reads `false`. The witness was also on the wrong side of the same
microtask boundary this file documents elsewhere: `ordinarySettled` flips when
the CALLER's promise settles, a few microtasks after the scheduler releases
admission, so in the marginal regime both halves of the pair could be green and
vacuous at once. It now samples the same counter the other half does.
**The gate now drives the round-2 interleaving, and this is the part that
matters.** My first attempt added terminal-lifecycle checks around a SEQUENTIAL
`close('shutdown')`. I built the pre-fix materializer into `dist/` and re-ran:
all 27 checks still passed. A sequential shutdown on a quiesced lane is correct
even in the broken model, so those checks were regression pins, not
discriminators -- the same defect the review found in the gate one level up.
So the gate now reproduces the defect instead: disable, start a re-open and HOLD
it inside the supervisor's start step, request shutdown while it is held, release
the re-open so its `finally` runs, then issue a disable into that window. Against
the pre-fix build:
laneIsTerminalAfterShutdown: disabled
secondShutdownRanNoSecondTeardown: 2 child stop(s) across both calls
-> FAILED (2 of 27 checks)
`disabled` is the reviewer's exact reported symptom, now produced by the real
supervisor, the real child and the real scheduler rather than by hand. With the
fix: PASS, 27 checks.
Seven new measured checks: terminal after shutdown, shutdown completed cleanly,
exactly one teardown, still one after a second `close('shutdown')`, dispatch
refused, re-open refused, and ZERO children spawned after shutdown -- that last
one is a process fact from the real `spawn`, because a state string can be wrong
and a spawned PID cannot.
**Four checks that could not fail, found while verifying the above.**
1. Probing `full` REGISTERS the process-global controller, after which every
later probe that reaches the factory gets a duplicate-registration refusal the
adapter reports as `undefined`. With `full` probed third, the enabled-changelog
and terminal-ownership checks were satisfied by that backstop rather than by
the guards they name: deleting either guard would not have turned them red.
Every negative case is now probed while the registry is still empty, `full`
goes last, and the remaining coupling is loud -- a negative case that wrongly
advertises takes the registration and reddens `capabilityPresentWhenFullyProven`.
2. The deletion detector was one-shot. Entry 1's `dropGraph` either failed (as it
must) or emptied the graph; entries 2 and 3 then read before=0/after=0 and
PASSED. Reserved state is re-seeded per entry, and the seed count is re-counted
per entry rather than hoisted.
3. Enumeration was measured only on the graph-set-index composition, where two
independent filters hide the internal prefix, so deleting either left the check
green. It now also enumerates on the index-free store, making the always-on
adapter layer independently falsifiable.
4. `verify.ts` cast the parsed artifact straight to the result type and stamped
the CURRENT HEAD onto it. Running verify alone, or a CI cache restoring
`artifacts/`, would re-certify old measurements against a newer commit. The
generator now records `sourceCommit` and the verifier refuses a mismatch or an
unknown schemaVersion.
Also: the pinned-server temp store was `mkdir`'d without a preceding `rm` and
never removed -- 12 had accumulated locally, and a PID collision would seed on
top of leftover reserved quads and fail every predecessor row with a misleading
`seed incomplete`. Now removed before and after.
Predecessor rows carry `executedAgainst: 'current-binary'`. No predecessor is
checked out, built or executed; the caveat previously lived only in prose that
the uploaded evidence does not carry, while the artifact itself said
`<commit>: pass`. The dead `advertisedSystemRecordLane` literal is deleted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
otReviewAgent
left a comment
There was a problem hiding this comment.
Operational Notice: Review Agent could not complete this review.
Business logic reviewer failed: WARNING: failed to clean up stale arg0 temp dirs: Permission denied (os error 13)
Round 2 addressed — all three findings, plus the class under finding 1
1 — confirmed, and worse than reportedReproduced against the real
Negative control, which is what rules out a harness artifact: issuing the disable one step earlier — while the shutdown pointer is still live — ends correctly at I did not patch the pointer a third time. Rounds 1 and 2 are the same root: seven of the nine writes to Shutdown now commits Three guards were deleted because the latch makes each one a branch that can no longer change an outcome: One part of your fix direction does not ship, and I want to be explicit rather than quietly omit it. "Release Relatedly, the write-order premise is inverted: the Evidence. 41/41 pre-existing tests pass with zero assertion changes. 12 new interleaving tests; 7 fail against the shipped model, the other 5 are regression pins (I am not claiming 12 discriminators). Solo-mutant sweep: 11 of 11 guards killed. The 11th was earned — The harness needed upgrading first: 2 and 3 — fixed, and the gate now catches finding 1Cleanup is a resource stack unwound in a single The sleep is replaced by polling The part worth your attention: my first attempt wrapped terminal-lifecycle checks around a sequential The gate now drives the interleaving instead — hold a re-open inside the supervisor's start step, request shutdown, release, then issue a disable into that window. Against the pre-fix build:
Four checks that could not fail, found while verifying the above
Plus: the pinned-server temp store was never removed (12 had accumulated locally; a PID collision would seed on top of leftover quads and fail every predecessor row with a misleading Merge gates
Still standing: do not merge while this is based on |
CI note at
|
Summary
Stack B2 of #2052: the live managed materialization boundary. Default-unused — no producer, provider, requester, protocol handler, advertisement or activation is registered here, and nothing in this PR performs store, epoch, queue, permit or scheduler work unless a daemon supervisor hands over a live ownership lease.
Stacked on #2103 (
feat/2052-system-record-core), which is still open. Base will move totestnet-canaryonce #2103 merges.This PR does not close #2052. Per the plan's ownership table, closure needs Stacks C–F plus the loaded r26/r27 regression.
Scope note — read this first
B2 as specified in the Revision 17 plan is very large. This PR delivers the ownership, capability and lifecycle boundary plus its live evidence, and deliberately stops short of the full-state CAS transaction. Concretely:
SparqlHttpStore.executeSystemRecordApply()currently returns a typeddeferredand dispatches nothing. The lane can be discovered, opened, sealed, disabled and shut down; it cannot yet apply a record.The verified-replacement command construction, bounded
inspectAppliedState, and the exact prior/next subject-union transaction are the next PR in the stack. Splitting here keeps both halves reviewable; the alternative was one PR nobody could review carefully.How it fits together
Capability travels by identity, not by config. Nothing below is reachable from a persisted file.
flowchart LR subgraph daemon["CLI daemon"] SUP["startOxigraphServer\n(lifecycle mutex + state machine)"] CHILD["owned oxigraph child\nPID + listen socket"] SUP -->|"spawn / SIGTERM,\nprove listener owner"| CHILD end subgraph storage["storage package"] CTRL["ownership controller\n(module-private WeakMap)"] LEASE(["lease handle\ndata-less, unique symbol"]) ADAPTER["SparqlHttpStore"] LANE["system-record lane\nsession state machine"] SCHED["StorePriorityScheduler\ncontrol barrier"] POOL["OwnedManagedHttpClient\n(node:http.Agent)"] end SUP --- CTRL CTRL -->|"mints"| LEASE LEASE -->|"symbol-keyed option"| ADAPTER SUP -->|"supervisorHandoff"| ADAPTER ADAPTER -->|"advertises IFF lease live\nAND handoff present\nAND snapshot ready"| LANE LANE -->|"every lifecycle transition"| SCHED LANE -->|"apply (deferred in this PR)"| POOL POOL --> CHILD ADAPTER --> CHILDThe lease carries no data at all — its meaning lives in a module-private
WeakMapkeyed by identity — and travels under aunique symbol, which object spread copies (so it survivesresolveAdapterOptions' rewrite) but JSON cannot produce or round-trip. Mutation authority is a separate controller the supervisor never hands out.The generation handoff, as it now runs
sequenceDiagram participant Q as Ordinary store traffic participant S as StorePriorityScheduler participant L as System-record lane participant A as SparqlHttpStore participant O as Oxigraph supervisor participant X as Owned child Q->>S: run() — admitted, in flight L->>S: runControlBarrier(store, "system-record.enable", gen) S->>S: seal store; HOLD new tagged run() off-queue Q-->>S: in-flight work completes S->>S: quiesced (untagged 0, tagged-for-store 0) S->>L: transition starts in the reserved controller slot L->>A: destroyClient() — retire the owned pool L->>O: stopAndProveOwnedChildDead() O-->>X: SIGTERM, then prove the PORT released L->>A: awaitRetiredWork() — drain the retired pool L->>O: startAndProveCleanGeneration() O->>X: spawn; bind generation ONLY once the child is the proven listener L->>A: rotateMaterializationEpoch() L->>S: transition returns; seal commits S-->>Q: held work resumes against the replacement generationThe order is load-bearing and asserted step by step, not merely as a set: a replacement started before the old child is proven dead would let a request issued over the retired pool reach the new listener.
What lands
Reserved graph policy (
internal-graph-policy.ts). B2's persistent graphs deliberately reuse theurn:dkg:internal:atomic-graph-replace:prefix, because all nine existing filter sites already hide it — predecessors get the downgrade property with no change on their side. That makes "internal" mean two things with opposite lifetimes, so reserved names match exactly, staging names match a canonical UUID shape, and generic mutation refuses everything else in the namespace. The module imports nothing from the system-record barrel becauseindex.tsis on every process's startup path; a test imports the barrel and asserts the literals agree.Unforgeable ownership lease (
managed-oxigraph-ownership-v1-internal.ts). Every signal available today is forgeable from persisted config:managedByDkgis operator-writable andresolveAdapterOptionsactively rewrites it tofalse, whileatomicUpdatesis synthesized astrueby that same function. Hence the data-less, symbol-transported handle described above.Lane session state machine (
system-record-materializer-v1.ts). Owns lifecycle and policy but no bytes; the transaction is performed by an injected executor, which is what makes the plan's pure reference-model fault matrix possible without a live child. Every lifecycle transition runs inside a scheduler control barrier, supplied as a required dependency.Always-on mutation denial. No always-on guard existed — the changelog's is gated on
enabled, which is default-off. The guard now lives on the adapter, the only always-on choke point, and is checked before theatomicUpdatescapability refusal since that flag is synthesized from config.Owned connection pool (
managed-http-client.ts).SparqlHttpStoredispatches through globalfetch, whose sockets belong to Node's process-wide dispatcher, so "destroy the retired generation's client" was unimplementable.node:http.Agentgives the managed lane a pool it owns;undiciis not a dependency here and does not resolve.Explicit per-decorator capability discovery, in all four places — the two storage decorators, the changelog, and the agent's hand-rolled forwarder. Structural
.innerStorewalking would reach past a wrapper whose cache the lane must invalidate, and past the changelog's denial.Scheduler admission (
store-priority-scheduler.ts): generation permits, ordering domains, and one coalesced reserved control barrier — all strictly opt-in, with the existingrun()signature unchanged and every pre-existing test passing unmodified. The hard constraint was the old behaviour: this scheduler is process-global and sits on every store operation, and its admission decision is deliberately entry-independent (canStartreads four integers and never inspects a queued entry). The entire extension therefore hangs off one integer comparison — whiletaggedQueuedCountis zero,nextRunnable()takes the original unconditionalqueue.shift(). That fast path has a witness rather than a comment:admissionEvaluationsincrements before the untagged early return, so it rises if selection ever walks entries the guard should have kept it away from, and a test asserts it stays at zero.Managed child ownership (
daemon/oxigraph-server.ts,oxigraph-managed.ts): one lifecycle mutex and explicit state machine replacing three unlocked closure booleans; a generation bound only after the spawned child is the proven listener owner, never at spawn. Three pre-existing defects fixed on the way —killSync()poisonedstop()so it never escalated to SIGKILL; the auto-revive timer handle was never stored and so could not be cancelled; andstop()resolved on the child's exit event without proving the port was released. A bind still served after the child exited now burns the lease terminal (port-release-unproven) rather than letting a replacement bind over a listener we do not own. Kills still go only through the trackedChildProcess, so a foreign listener is never signalled.Live conformance gate (
devnet/issue-2052-managed-ownership/,.github/workflows/system-record-managed-ownership.yml,pnpm test:live:system-record-managed-ownership). Drives one real generation handoff end to end plus the capability, socket-ownership and predecessor matrices — see below.Two defects this work caught
ChangelogStoresilently swallowed reserved writes. Itsinsert()/delete()strip reserved-graph quads rather than refusing them, so with the changelog enabled a reserved write returned a resolved promise for a write that never happened — a lost update with no signal. Stripping stays correct for the changelog plane and ephemeral staging graphs; persistent state now denies.The live gate failed on a real socket leak in this PR's own code.
OwnedManagedHttpClientreported one live socket afterdestroyAndSettle(), becauseAgent.destroy()tears sockets down asynchronously. Returning there would have let a replacement child bind while a retired keep-alive socket was still open — precisely the stale-generation window the design exists to close. It now polls to zero under a bounded deadline and treats failure as terminal.Merge-readiness review — all six findings addressed
Reviewed at
b1c1bdf6d; fixes atd8b2a2d02. Every finding was validated against the code before being fixed, and every fix is mutation-proven rather than merely green.barrieris a required dep on the lane; enable/disable/shutdown each open a named section; the adapter supplies one backed byrunControlBarrierindeterminatewithout sealingindeterminaterunShutdownis no longerasync, sothis.transitionis assigned synchronously before any awaitgit diffagainst baseFinding 1 in detail
runControlBarrierandsealStoreGenerationshipped in this stack exported, documented and covered by 25 unit tests, with zero production callers. The lane stopped the owned child, asserted its port free and bound a replacement while ordinary requests were still in flight on the retired generation.The materializer stays store-agnostic — it takes an exclusive-section runner, not a scheduler — and the adapter supplies one using
thisas the opaque store identity, re-reading the lease snapshot's generation per transition rather than capturing the one observed at construction (the lane outlives any single child).Shutdown under a barrier that cannot be acquired does not run the teardown, and that is the safe side: the lane still reaches terminal so nothing writes through it again, while the child is left alive under the supervisor that still owns it and stops it at process exit. Stopping the child outside a section is the hazard; "could not quiesce" is not a reason to do it anyway. The error propagates, because a shutdown that could not quiesce the store is not a clean one.
Unit tests bracket each transition and assert every child-touching step ran inside the section — but they inject a pass-through stand-in, so they cannot show that asking for a section does anything. All 41 of them stay green under the pass-through mutant.
system-record-control-barrier-integration-v1.test.tsis the one that fails: real adapter, real process-global scheduler, real handoff composition, genuinely held-open ordinary request.The live gate now discriminates
The gate previously minted ownership by hand and injected no-op supervisor methods, so it never restarted a child, never opened a lane and never ran a barrier — which is exactly why finding 1 passed it. Of the two directions the review offered (narrow the claims, or drive the real thing), this takes the second: narrowing alone would have left the gate structurally unable to catch a regression of the defect it is named after.
It now runs one real handoff first —
startOxigraphServerowns and spawns the child, its lease andsupervisorHandofftravel to the adapter under the symbol key exactly as the daemon composes them — against a 60,000-quad ordinary insert as the probe workload (one scheduler admission, one HTTP request over the store's own path). Ten measured checks: the write was still running when the lane opened; zero store requests inflight at the instant the child was stopped; the write survived; the generation advanced; a second OS process was spawned, the first PID is dead and the last alive; the replacement serves the same 60,000 quads; the store resumed against it; the lane reports enabled.Proven to fail, not merely to pass. Built with the adapter's barrier replaced by a pass-through and re-run end to end:
With the barrier restored:
PASS: 20 checks, 3 predecessor entries. That negative control was not constructed — the first run of the new section hit a staledist/predating the fix and reproduced the reviewer's finding unprompted.One measurement was reworked rather than tuned. The first shape compared wall-clock "stop happened after the write settled", which is subtly wrong: the scheduler releases admission when the store work resolves, and the caller's promise settles a few microtasks later, so a correct handoff measured as 0.09 ms too early. The pre-fix build read −49 ms, so any tolerance would have been chosen to sit between two numbers rather than to mean anything. Reading the scheduler's own inflight count at the instant of the stop has no such boundary — it is the quantity the barrier waits on.
Correction to an earlier revision of this description
A previous revision claimed a barrier asymmetry — a generation-scoped drain against a generation-blind hold — and warned that a future fix must be an allow-list. Re-read at this head, that description was wrong:
isBarrierReady()waits on this store's wholetaggedInflight, and the hold isseals > 0. Both are generation-blind, deliberately, because the transition is a child-process restart and the child it stops serves every generation.runningPermitsfeedsadmissionGenerationsInflightand the timeout blocker report; the seal'sgenerationis a diagnostic label, not an enforcement key. Narrowing either to the sealed generation would be strictly weaker and — generations being decimal counters — trivially bypassable. The scheduler's own docstring said "waits for the sealed generation's execution permits to drain", which is where the claim came from; it is corrected in this PR.The related re-entry note also needs restating now that the barrier has production callers: a transition still must not re-enter
run(), and the three that exist do not — they make supervisor calls, drain the adapter's owned pool, and invalidate a cache synchronously. The bound remains in place for the caller who gets it wrong.Independent review — four earlier tracks
Every finding below was raised by an independent reviewer working read-only in its own worktree pinned to a specific commit, and each is fixed in this branch.
Protocol/safety. Two BLOCKERs, both demonstrated by an executed exploit rather than argued:
shutdownsuperseded nothing while a transition was in flight, so a stalled enable resumed afterwards and started a replacement child after shutdown had proved the old one dead; and anopen()that joined a shutdown never re-checked terminal state, so a shut-down lane revived and dispatched a record write to the executor. Both fixed and mutation-proven. The same track attempted eight forgery attacks on the ownership lease —structuredClone, spread,Object.assign,JSON.parsereviver, same-prototype,Proxy, and two more — and all were rejected; dual-package and worker-thread cases fail closed.Execution/QA. The gate reported "PASS: 18 checks" while 13 were assertions over constants the generator hardcoded to zero. Deleted; it then reported 10, all measured, and now reports 20.
leakedOwnedSocketswas 0-or-throw by construction and is now a before/after pair. The manifest carried a fabricated commit SHA published as CI evidence — fixed, with agit cat-file -echeck proven to catch it. Mutation-testing that fix surfaced a further hole: a crashed run left the previous PASS artifact verifiable, sorun.tsnow clears artifacts first.Resource/operations.
destroyAndSettleawaited before destroying, so itstimeoutMsdid not bound the phase it named (measured 3831 ms; no return at all within 8 s against a dribbling server) — now 5 ms. The negative capability memo latched permanently in 3 of 4 layers, so one probe inside a 1 s restart would have disabled the lane for the process lifetime.awaitRetiredWorkwas a structural no-op. A control barrier did not stop untagged traffic (20 ops started mid-transition → 0), and held calls were unbounded, rejecting 190/200 callers after they had waited out the seal → 0.Supervisor handoff (
036efc6b3). All remediated:child_process.spawnthrows synchronously for EACCES/EFTYPE/E2BIG/EINVAL. The.once('error')handler catches only async failures, so the throw escapedstartCleanGenerationLockedand bypassed the only code that clearshandoffPhaseand re-arms backoff. Measured: 30 backoff periods later, still one spawn, port unserved, self-heal refusing — andterminal: false, so the lease reported "momentarily not ready" indefinitely rather than failing closed.endpointAnswers()collapsed ECONNREFUSED (real evidence) and TimeoutError (no evidence) into onefalse.retiredphase, andrunShutdown()reached the first half without the second, parking the supervisor there indefinitely.handoffPhase === 'retired'clause inscheduleRevivewas inert (deleting it changed 0 of 57 test statuses).The inert guard produced a standard now applied to this branch: every new guard ships with a test that fails when only that guard is removed. It was removed rather than papered over, and a second proposed guard was declined for the same reason. Later in the branch a
shutdownWorkcoalescing field was added, its mutant survived twice, and it was deleted on the same standard.One proposed fix was rejected as worse than the bug. The review (and I) initially proposed making a teardown-retire terminal for the supervisor. That would have been a serious regression: the managed child is the daemon's entire triple store, so disabling the system-record lane would have permanently killed node storage until restart. The phase is bounded instead — fail-closed for the lane, alive for every other consumer.
Measured, not asserted
Default-path cost of the scheduler admission work, re-measured independently after the fix:
+0.003% versus pre-fix, against ±10–20% within-variant spread — three orders of magnitude below the noise floor.
admissionEvaluations = 0on a pure untagged workload held across every round.One number in an earlier revision of this description was wrong and is corrected here: a claimed heap-delta improvement of 19.2 → 13.1 MB was a GC-timing artifact (
heapUsedsampled without forcing GC), not allocation volume. Re-measured as GC count the reduction is real and stable (307 → 260 → 223) but pause time is identical across all three, so the practical benefit is nil.Known deferral
After a retire whose port-release proof fails, the supervisor terminates and cannot spawn again — but ordinary store traffic never consults the ownership lease, so the daemon keeps issuing SPARQL to an endpoint served by a process it could not account for. A FATAL log names the condition and tells the operator to restart.
Closing this properly needs an adapter-side kill switch, deliberately not in this PR: refusing all store traffic is a large change to the daemon's hot path and deserves its own review rather than riding in on a default-unused change.
A failed retire has three causes, and they want different answers:
(c) is a direct consequence of the HIGH fix above. Once "no evidence" stops counting as "evidence of absence", an inconclusive probe is no longer treated as release. A closed loopback port normally RSTs immediately, so this needs the connect or its abort to lose a >1.5 s race under event-loop starvation — and
AbortSignal.timeoutfires from that same starved loop, so starvation often self-cancels. Plausible rather than likely. But #2052 is a store-pressure issue, so the one condition that produces (c) is the condition this feature exists for, and in (c) the FATAL-log-and-restart guidance is telling an operator to restart a node whose port was fine.The coupling underneath it
A failed retire leaves the daemon storeless regardless of any kill switch, and closing the supervisor is not what causes it:
bindReadyGeneration()throws on a terminal lease, so once the lease is terminal no generation can ever bind again. Without the close the supervisor is equally childless, merely less honest about it.That is the same coupling rejected above, surfacing elsewhere: the lease is a lane capability, the child is the daemon's store, and today they share a fate. Burning the lease on an unproven release is correct and should stay irreversible; killing the daemon's storage as a side effect of it is not.
Attempting a revive after a failed retire would in fact be safe — if something really is on the port, the respawned child dies on EADDRINUSE and the ownership proof refuses to adopt the listener, which is an existing, tested guard. So the fix is decoupling lease-terminality from supervisor-liveness, not weakening any guard. The kill switch alone leaves the node storeless; the decoupling alone leaves bad traffic flowing. They want fixing together, and that is a larger design change than this PR.
Test plan
Guards are mutation-proven rather than merely green. Beyond the barrier proofs above: forcing the lease identity check to
truekills exactly the two identity-forgery tests; relaxing the reserved-name check to a prefix match kills the near-miss test. Every mutant was verified applied by grep and then reverted..github/workflows/ci.ymltriggers onpull_request: branches: [main, testnet-canary, release/rc.12, rc17-vm-wip]. This PR is stacked onfeat/2052-system-record-core, which is not in that list, so the rootpnpm testlanes do not run here. The checks currently green are only the path-filtered workflows: protocol evidence, SPARQL scalability lint, the managed-ownership live gate, and the RFC-64 Windows gate.That is expected for a stacked PR and not a defect, but it is a merge gate, not a footnote:
feat/2052-system-record-core.testnet-canary, inspectgit range-diff, change this PR's base totestnet-canary, and let the full CI lanes run.pnpm testmust be green there before merge, per the Revision 17 plan's Full Validation Gate.Known environment caveats
packages/cli/test/oxigraph-*.test.tsis effectively POSIX-only. On Windows it fails for environmental reasons (spawn EFTYPEon the extension-less Node stand-in;No prebuilt Oxigraph 0.5.8 binary for freebsd-x64, which those tests force deliberately; one systemd/uid-only case). Verified on Linux CI only.pnpm --filter @origintrail-official/dkg exec vitest run test/oxigraph-managed.test.ts test/oxigraph-server.test.tsreports "No test files found" with two files but runs fine with one;npx vitestfrompackages/cliruns both. Flagging rather than silently substituting.packages/storagefull-suite failures are load-dependent on Windows, always confined to the embeddedoxigraph-workerpath, which V1 leaves legacy. Established by measurement: the same files fail onmainwith none of this stack present (8 of 11 red inoxigraph-worker-resilience.test.ts), and they fail worse in isolation than in the full run — 5 s test timeouts against ~70 s of import cost. Linux CI is the gate.packages/core/test/system-record-golden-v1fails on a Windows CRLF checkout (core.autocrlf=true, no.gitattributes) because the generator byte-compares against LF. Not a code defect; Linux CI is green.Related
@origintrail-official/dkg-core/system-record-v1barrel so that refactor stays independent🤖 Generated with Claude Code