fix(agent): prevent RS-heal store starvation - #2067
Conversation
8c034ed to
c09cf05
Compare
Jurij89
left a comment
There was a problem hiding this comment.
Merge-readiness assessment
Verdict: request changes. The patch is directionally correct, but it is not merge-ready at c09cf051fbc18573dd1a5f34b6ad4a60b38cf81a. I found three merge-blocking behavioral/lifecycle issues plus one integration-base blocker. The most important architectural problem is that best-effort repair still sits in front of the useful VM work it is meant to yield to.
Reviewed against the PR's actual base 4bd402de89af4b73a200cf120796c3617474edbe, current testnet-canary 31be166b88a9dc3b2282ad1296603342b861a257, issue #2066, and the lifecycle/pressure invariants delivered by #2053.
Blocking findings
1. BLOCKER: RS-heal pressure aborts the primary VM reconcile
executeVmReconcileForCg awaits healStrandedScopedKCs before calling reconcileContextGraph (packages/agent/src/dkg-agent-swm-host.ts:2912-2932). The heal path deliberately rethrows STORE_SCHEDULER_BUSY (:3500-3502, :3524-3532), and there is no boundary catch. A background admission failure therefore terminates an already-admitted live/manual VM job before it processes a single ordinal. The new test at packages/agent/test/rs-heal-stranded-kc.test.ts:298-302 currently codifies the throw, not the required isolation.
This is the inverse of #2066's required behavior and undermines the purpose of the patch: maintenance backpressure suppresses the higher-value sync work. Even when the error does not fire, the default scheduler wait can hold the VM job for ten seconds before the main reconcile starts.
Fix direction: keep this within the existing dispatcher/lifecycle; no new worker or queue is needed. Run the main reconcileContextGraph first, then attempt one bounded RS-heal slice only when the target remains current (preferably when the main result has no immediate trailing work). Convert scheduler-busy into a typed skipped/deferred maintenance result and never into failure of the main result. Do not make the heal fire-and-forget. Add a regression that injects RS-heal busy and proves reconcileContextGraph still runs and its result is returned.
sequenceDiagram
participant D as VM dispatcher
participant H as RS heal
participant S as Store scheduler
participant R as Main VM reconcile
D->>H: admitted live/manual job
H->>S: background store operation
S--xH: STORE_SCHEDULER_BUSY
H--xD: error propagates
Note over D,R: Main reconcile never starts
Recommended sequencing:
sequenceDiagram
participant D as VM dispatcher
participant R as Main VM reconcile
participant H as RS heal
participant S as Store scheduler
D->>R: process bounded VM slice
R-->>D: useful sync result
opt target current and no urgent trailing slice
D->>H: attempt bounded maintenance
H->>S: background store operation
alt admitted
S-->>H: result
H-->>D: page/cursor outcome
else busy
S--xH: STORE_SCHEDULER_BUSY
H-->>D: deferred, main result preserved
end
end
2. BLOCKER: canonical preflight regresses authorized gate-only private graphs
The canonical no-list branch at packages/cli/src/daemon/http-utils.ts:1074-1091 returns 503 when exact existence/writability is positive but policy or authorization is incomplete. However, the existing metadata model treats agent/peer/legacy participant gates as implicit private policy even without an explicit dkg:accessPolicy literal (packages/agent/src/dkg-agent-cg-resolve.ts:749-755). The exact probe only calls the allowlist path when accessPolicy === 'private' (:988-1009).
Consequently, a qualified gate-only graph with a valid allowed caller produces callerAuthorized: undefined and now fails with CONTEXT_GRAPH_VALIDATION_UNAVAILABLE; the previous scoped catalog path could authorize it. This is a supported metadata shape and a user-visible write regression.
Fix direction: extend the bounded point probe so it derives gate-implied privacy and caller authorization using the same semantics as the projection/list path. Do not restore the full catalog fallback. Return explicit reduced decisions from evaluateExactWritePreflight (accept, authoritative reject, non-writable, validation unavailable, unavailable-with-rescue, or bare-name list fallback) instead of passing an optional raw probe through continueToList. Add qualified no-list tests for: explicit public, explicit private authorized/unauthorized, gate-only authorized/unauthorized, unknown, stale/non-writable, healthy-but-incomplete, and unavailable with positive/negative chain rescue.
flowchart TD
A[Write target] --> B{Canonical qualified ID?}
B -- No, bare name --> C[Bounded catalog/name resolution]
B -- Yes --> D[Exact point probe]
D --> E[Declaration + access policy + gate presence]
E --> F[Point authorization check when private or gate-implied private]
F --> G{Explicit reduced outcome}
G -- Accept --> H[Continue write]
G -- Authoritative deny --> I[Fail closed]
G -- Store unavailable --> J[Positive active-public chain rescue]
G -- Incomplete --> K[Retryable 503]
Note[Canonical path never enumerates the full catalog] --- D
3. HIGH, merge-blocking: one RS-heal store read escapes lifecycle cancellation
readMaterializedVersion is called with rsHealStoreOptions('version.readLegacy') but without the current AbortSignal (packages/agent/src/dkg-agent-swm-host.ts:3361-3366); every surrounding store operation spreads signal. The helper at :254-258 is therefore not a complete policy boundary.
This matters after #2053: a queued store read can wait up to the scheduler's default 10 seconds, while VM reconcile shutdown waits 5 seconds. That physical run can outlive rotation/shutdown and force the retirement/quarantine path the previous PR was designed to avoid.
Fix direction: make rsHealStoreOptions(operation, signal) (or a local closure) always return { priority, source, signal }, and use it for every direct store operation and materialization helper. Add a scheduler-backed abort test proving a queued legacy-version read cancels and physically settles before teardown completes.
4. BLOCKER: the PR targets the already-merged feature branch, not testnet-canary
The PR base is still fix/2052-sync-pressure. #2053 was squash-merged into testnet-canary, so this branch history is not an ancestor of current canary (merge-base(testnet-canary, HEAD) is fa9dca15e73a2d0054094b23f688cce20ee8aed1). Simply changing the base would expose the old #2053 commit stack again.
Fix direction: cherry-pick/rebase the single #2067 commit onto current remote testnet-canary, force-update the PR branch safely, and retarget the PR to testnet-canary. Re-run the diff/CI after that operation. The seven intended files do not overlap the post-#2053 canary changes in my check, so this should be mechanically small.
Important follow-ups in this PR
MEDIUM: harden the pager's boundedness and progress proof
The direction is good, but the current test only proves LIMIT when early entries heal and disappear. It does not prove the stated starvation fix. Add the #2066 acceptance case: a full first page of unhealable rows followed by a healable row, then prove advance, later repair, wrap, and retry. Also cover empty/short/full pages and busy-no-advance.
Use the existing safe positive-integer environment parser and a deliberate hard maximum for DKG_RS_HEAL_BATCH_SIZE (packages/agent/src/dkg-agent-base.ts:942-943); Infinity or a very large configured value currently defeats the bounded-work claim. Align the keyset expression (FILTER(STR(?ual)...) with ORDER BY STR(?ual)) and constrain the query to valid IRI candidates so an invalid final binding cannot pin cursor advancement. Extracting cursor mechanics into a focused helper would improve readability, but I would treat that extraction as non-blocking once the behavior is fully tested.
MEDIUM: the patch is count-bounded, not yet resource-bounded
Eight KCs can still mean many root probes and large non-preemptive Blazegraph updates. Background admission protects reserved capacity, but it cannot preempt an admitted expensive query/update. This PR should not add another scheduler or maintenance component; instead, preserve the small page, prioritize the main reconcile as above, and validate the default using the existing W1/store telemetry.
Before merge or as an explicitly recorded rollout gate, compare under the incident-shaped backlog:
- RS-heal operations/pass, duration, busy/deferred count, and cursor progress;
- store queue age/rejections by priority and
agent.swm.rsHeal.*source; - foreground canonical-preflight p95 and 503 count, confirming zero
listContextGraphscalls; - SWM/VM convergence time and dispatcher fairness across CGs;
- Blazegraph CPU/RSS and long-running update duration.
The expected result is not merely fewer RS-heal calls: foreground latency/rejections must improve while the legacy backlog still makes monotonic progress.
LOW/FOLLOW-UP: chain-rescue timeout does not cancel the underlying RPC
rescueWriteTargetWithoutStore uses Promise.race (packages/cli/src/daemon/http-utils.ts:851-876), so timeout returns to the caller but does not stop the chain request. This behavior predates the PR, but canonical store outages now reach it directly and repeated client retries can accumulate unfinished calls. Track bounded single-flight/cancellation or a short positive-proof cache separately; it does not need to expand this repair unless pressure testing shows amplification.
What is correct
- Moving RS-heal reads/writes to the background scheduler lane is the right use of #2053's admission model.
- Finite pages plus a resumable per-CG cursor are materially better than walking the whole historical backlog.
- Canonical qualified IDs should use point validation and must not fall into O(catalog) enumeration.
- Positive active-public chain proof is an appropriate fail-closed outage rescue; private, stale, unknown, and unproven targets must remain denied.
- Extending materialization helpers to carry store options is the right boundary, once cancellation is included consistently.
Validation performed
- Clean detached worktree at
c09cf051fbc18573dd1a5f34b6ad4a60b38cf81a. pnpm run build:packages: passed, 22/22 tasks.- Agent RS-heal focused suites: 3 files, 24/24 tests passed.
- CLI write-preflight resilience suite: 24/24 tests passed.
git diff --check: passed.- GitHub Actions: all required checks currently green.
- The broader local real-daemon write-path fixture exited cleanly before readiness in this environment, so its 13 route cases did not execute locally; the direct resolver cases passed and the corresponding GitHub CI matrix is green.
Overall assessment
The proposal addresses the right two pressure amplifiers, and its basic mechanisms are simpler and more efficient than the current unbounded/catalog-wide behavior. It is not ready to merge because maintenance can still suppress main VM progress, exact validation breaks a supported private authorization shape, one operation violates the lifecycle cancellation contract, and the PR is based on the wrong integration branch. After those are corrected and the missing pressure/progress regressions pass, this should receive another focused review round.
c09cf05 to
19133a2
Compare
|
Addressed the merge-readiness review in
Local evidence:
The pre-existing local deployment-file change was preserved and excluded. GitHub CI has restarted for the rebased head. Please re-review when ready. |
19133a2 to
ce8fbdf
Compare
Jurij89
left a comment
There was a problem hiding this comment.
Merge-readiness assessment, round 2
Verdict: request changes. The update at ce8fbdfea47d8fc3bb002947941f5f7a9173612b resolves three of the prior code blockers and the integration-base blocker, but the gate-aware exact probe still has one merge-blocking cross-source authorization bug. I reproduced both its availability regression and its security consequence with the built production method and a real OxigraphWorkerStore.
Previous blockers
| Previous finding | Status |
|---|---|
| RS-heal busy aborts primary VM reconcile | Resolved: primary slice runs first; maintenance busy is deferred and cannot replace the primary result. |
| Gate-only private canonical writes fail | Partially resolved: same-source gate metadata works, but supported cross-source metadata still fails or can be misclassified. |
| Legacy-version read omits lifecycle cancellation | Resolved: every RS-heal store option now carries the signal; scheduler-backed cancellation coverage passes. |
| PR based on merged feature branch | Resolved: one commit directly on current testnet-canary. |
Blocking finding
BLOCKER: exact preflight does not merge policy authority across metadata sources
The declaration query in packages/agent/src/dkg-agent-cg-resolve.ts:892-940 requires each UNION branch to contain rdf:type dkg:ContextGraph before it can contribute that source's access policy, curator, or gate facts. That is not the repository's established metadata model. ContextGraphMetaProjection independently loads _meta, AGENTS, catalog, and ONTOLOGY facts (packages/agent/src/context-graph-meta-projection.ts:390-423) and then applies a one-way private-policy ratchet across all sources (:682-698). A source does not need to repeat rdf:type for its policy/gate fact to participate.
This leaves two concrete failures:
rdf:typein ONTOLOGY plusdkg:allowedAgentin<cg>/_metaproduces noaccessPolicyorcallerAuthorized, so an authorized member still receives the canonical-path 503.- More seriously,
rdf:type + accessPolicy publicin ONTOLOGY plusaccessPolicy privatein<cg>/_metawithout a duplicate type ignores the private fact. The probe reportspublicandcallerAuthorized: true, so a synced canonical write can fast-accept an otherwise unauthorized caller.
Real-store reproduction against this head returned:
{
"gate": { "accessPolicy": null, "callerAuthorized": null },
"conflict": { "accessPolicy": "public", "callerAuthorized": true },
"allowlistCalls": 0
}The test added for gate-only authorization seeds both the type and gate in ONTOLOGY, so it cannot catch the normal merged-source shape.
Fix direction: use a bounded point projection over the three graph/subject candidates, but collect declaration, policy, curator, and gate presence independently. Preserve the existing precedence exactly:
- any explicit
privatefact across trusted sources wins; - otherwise explicit
publicwins over informational/legacy gates; - otherwise any gate implies legacy private;
- curator and caller authorization use the same merged-source view.
Use a boolean EXISTS/bounded ASK for gate presence rather than returning one row per gate value. The current OPTIONAL gate join can materialize up to the full per-CG allowlist (256 participants) in each duplicated branch merely to derive sawGate, which is unnecessary work in a path intended to stay cheap under store pressure.
flowchart LR
subgraph Current
O1[ONTOLOGY type + public] --> Q1[Branch contributes public]
M1[_meta private or gate, no type] --> X1[Branch discarded]
Q1 --> A1[Incorrect public or incomplete result]
end
subgraph Required
O2[ONTOLOGY facts] --> P[Bounded per-ID fact projection]
G2[AGENTS facts] --> P
M2[_meta facts] --> P
P --> R{Merged precedence}
R -->|Any private| D[Private authorization]
R -->|Else public| U[Public authorization]
R -->|Else gate| D
end
Required regressions:
- type in ONTOLOGY + gate in
_meta, authorized caller: accept without catalog listing; - same shape, unauthorized caller: fail closed without listing;
- public in ONTOLOGY + gate in
_meta: explicit public wins; - public in ONTOLOGY + private in
_meta: private wins and an unauthorized caller is denied; - curator in one source and declaration in another: curator authorization survives;
- no source repeats type except the declaration source.
Non-blocking review items
MEDIUM: make bare-name fallback states mutually exclusive
ExactPreflightDecision.listFallback (packages/cli/src/daemon/http-utils.ts:897-908) still carries correlated booleans and permits impossible states such as deferredReject: true together with exactProbeUnavailable: true. The helper currently constructs valid combinations, so I do not see a present runtime failure, but explicit variants (bareFallback, bareDeferredReject, bareUnavailable with required error text) would complete the stated reduction and remove future branch ambiguity.
Add the missing thrown-probe canonical regression as well. The catch path currently returns unavailableWithRescue correctly, but existing tests only cover a returned storeUnavailable result. This is a coverage gap, not a second blocker.
DEFERABLE: a separate RS-heal service is not required for this fix
The new readRsHealStrandedPage and advanceRsHealCursor helpers, dedicated constants, typed result, and focused tests give paging policy a reasonable boundary without adding another runtime component. Moving this into a separate service/module could be a later maintainability refactor; I would not expand this pressure fix solely for that structural preference.
ROLLOUT: retain the conductor pressure gate
The code is now count-bounded, background-priority, lifecycle-owned, and VM-first. An admitted Blazegraph update remains non-preemptive, so the PR description is correct to retain the incident-shaped conductor run as the operational gate. It should demonstrate lower foreground queue age/rejections and API latency while RS-heal cursor progress remains monotonic.
Validation
- Clean detached review worktree at
ce8fbdfea47d8fc3bb002947941f5f7a9173612b, based directly ontestnet-canary31be166b88a9dc3b2282ad1296603342b861a257. pnpm run build:packages: passed, 22/22 tasks.- Agent recovery and broader VM reconcile suites: 4 files, 149/149 tests passed.
- CLI write-preflight resilience: 30/30 tests passed.
- Cross-source policy reproduction: confirmed with the real built probe and real Oxigraph worker store.
git diff --check: passed; nine intended source/test files only, no local/generated files in the PR diff.- GitHub Actions: all required checks green.
- Local limitation: the separate real-daemon route fixture again exited with code 0 before readiness in this WSL environment, so its 13 route cases were skipped locally; GitHub's corresponding full CI lanes are green.
Overall assessment
The RS-heal architecture and pressure behavior are now in good shape, and the previous lifecycle/branch blockers are resolved. The PR is not merge-ready until exact preflight aggregates policy authority across metadata sources without enumerating gate values. After that focused fix and the cross-source regressions pass, I expect the remaining review surface to be narrow and the conductor run to be the final operational readiness check.
ce8fbdf to
ccb0eaa
Compare
|
@Jurij89 Round 2 is addressed in ccb0eaa.
Validation: package build 22/22; CLI preflight suites 61/61; agent probe/discovery 84/84; RS-heal/VM focused suites 149/149; diff check clean. The unrelated local deployment JSON remains excluded. Please re-review the updated head. |
Jurij89
left a comment
There was a problem hiding this comment.
Merge-readiness assessment, round 3
Verdict: APPROVE. No blocking findings at ccb0eaaa5a30295054c23bbb7d893ae6b81a100f.
Prior blocker closure
- Cross-source authorization is fixed. The exact probe now projects declaration, access policy, curator, and gate presence independently across the three trusted graph/subject pairs. A private policy in any source retains the private-wins ratchet; explicit public wins over legacy gate inference; gate-only graphs remain private. The point query stays bounded and
FILTER EXISTSprevents allowlist fan-out. Seepackages/agent/src/dkg-agent-cg-resolve.ts:892-1010. - Canonical writes no longer fall into the catalog scan. Exact-probe outcomes are fully reduced into mutually exclusive variants; qualified IDs accept, reject, fail closed, or use the bounded active-public rescue without calling
listContextGraphs. Bare-name compatibility retains list/suffix resolution. Seepackages/cli/src/daemon/http-utils.ts:884-1152. - Primary VM work is isolated from maintenance pressure. VM reconcile completes first; urgent trailing work suppresses RS-heal; otherwise one bounded repair page runs as background work. Busy admission becomes a non-throwing deferred result and cannot replace the primary response. See
packages/agent/src/dkg-agent-swm-host.ts:3017-3048. - RS-heal pressure is bounded and cancellable. The hard-capped page, per-CG keyset cursor, background source labels, lifecycle signal propagation, busy-without-cursor-advance behavior, and skipped-row progress address the incident-shaped starvation paths.
Architecture and efficiency
The implementation removes the two harmful amplifiers identified in #2066: unbounded maintenance traversal and O(catalog) fallback for an exact write target. It does so within existing scheduler, reconciliation, metadata projection, and chain-rescue boundaries; it adds no worker, queue, or periodic subsystem. Fail-closed behavior remains intact for unknown, stale, incomplete, and unauthorized private graphs.
I reviewed the remaining yellow note about the RsHealPassResult union. It is non-blocking: the production caller intentionally ignores maintenance status so it cannot affect primary reconciliation, while the typed result makes scheduler-busy deferral directly testable. A later telemetry/cleanup refactor may consume or collapse it without changing this fix.
Validation
pnpm run build:packages: 22/22 tasks passed- CLI write-preflight and route suites under the normal Hardhat-backed config: 61/61 passed
- Collected agent unit coverage for VM reconcile plus decorated RS-heal paths: 131/131 passed
git diff --check origin/testnet-canary...HEAD: passed; review worktree clean- GitHub CI: all required Linux/EVM/package shards and SQLite lifecycle (Windows) passed
A separate local full-agent attempt remained in the repository's Hardhat deployment bootstrap and was stopped without reaching tests; the corresponding complete agent shards are green in CI. The documented Blackbox conductor/load run remains the appropriate operational rollout gate, not a code-merge blocker.
Merge readiness: ready to merge.
Summary
deferred/store-busyresult and cannot erase the main VM result.Related
agentssync fails from every peer; retry pressure pins the single store worker and takes the HTTP API down #1982Behavior
VM reconcile and RS-heal sequencing
sequenceDiagram participant D as VM dispatcher participant R as Primary VM reconcile participant H as RS heal participant S as Store scheduler D->>R: Process bounded VM slice R-->>D: Useful sync result alt Urgent trailing VM work exists D->>D: Queue fair trailing slice else Target current and no urgent work D->>H: Await one bounded maintenance page H->>S: Background operation with lifecycle signal alt Store admits maintenance S-->>H: Result H-->>D: Completed or skipped else Store is busy S--xH: STORE_SCHEDULER_BUSY H-->>D: Deferred end end Note over D,H: Maintenance never suppresses or replaces the primary resultCanonical write validation
flowchart TD A[Write target] --> B{Canonical qualified ID?} B -- No, bare name --> C[Bounded catalog and name resolution] B -- Yes --> D[Exact point probe] D --> E[Cross-source declaration, policy, curator, and gate presence] E --> F[Point authorization when private or gate-implied private] F --> G{Reduced outcome} G -- Accept --> H[Continue write] G -- Unknown, denied, or non-writable --> I[Fail closed] G -- Incomplete --> J[Retryable 503] G -- Store unavailable --> K[Positive active-public chain rescue] K -- Proven --> H K -- Unproven --> JFiles changed
packages/agent/src/dkg-agent-base.tspackages/agent/src/dkg-agent-cg-resolve.tspackages/agent/src/dkg-agent-swm-host.tspackages/publisher/src/metadata.tspackages/agent/test/core-fills-gap.test.tspackages/agent/test/rs-heal-stranded-kc.test.tspackages/agent/test/rs-heal-stranded-kc-decorated.test.tspackages/cli/src/daemon/http-utils.tspackages/cli/test/write-preflight-resilience.test.tsOnly these nine intended files are included; generated and local-only deployment state is excluded.
Validation
pnpm run build:packages— 54 packages in scope; 22/22 build tasks passed.git diff --check origin/testnet-canary— passed.testnet-canary(31be166b8).Rollout gate
The code path is bounded and covered locally. The Blackbox conductor run remains the operational promotion gate: verify complete VM convergence, clean-absence/backoff behavior, foreground fairness, reject growth, API latency, store queue age/rejections by
agent.swm.rsHeal.*source, and Oxigraph/Blazegraph memory and write amplification under the incident-shaped backlog.