test(devnet): add public Context Graph SWM+VM sync proof harness

[branarakic]

What this is

A devnet harness that answers one product question end to end, through the daemon HTTP API a real agent uses — no fixtures, no library shortcuts:

Can any agent subscribing to a public Context Graph reliably converge to both the shared working memory and the finalized verifiable memory corpus?

Covers both public policy cells (open + curated), three receiver shapes (author / live / late), curated authorization negatives, and an opt-in RESILIENCE phase (scale, idempotence, author outage, hold-out).

Why it earns its place

It found three real defects that unit tests and code review missed — two of which had been misdiagnosed twice by reading the code:

1. Markdown KAs replicating to peers with metadata but empty entities/triples (Fix Markdown KA visibility after SWM sharing #1779's bug, verified fixed here).
2. Member→curator SWM permanently dropped on public/curated CGs — the share reported success while the write went nowhere. → PR fix(swm): accept plaintext SWM on public-on-chain agent-gated context graphs #1843
3. Public SWM catch-up never materialized verified snapshots: a node that missed the live gossip stayed empty forever. → PR fix(swm): materialize verified public snapshots on catch-up #1842

All three share a shape that author-side tests cannot see: the write succeeds, the author looks correct, and only the peer is wrong.

Confound controls — each added after a green check turned out to measure nothing

Being explicit here because these were all mistakes made during development, and the assertions exist to stop the next person repeating them:

• Content presence is asserted separately from sync-mechanism progress. A receiver can hold content while the chain-reconcile watermark is stuck at 0. The first VM run "passed" for exactly this reason.
• The decisive receiver is chosen after the publish, from nodes provably outside the storage-ACK set, and must differ from the live receiver's node.
• Corpus counts bind the published subjects and require exact equality. Counting every quad in the CG's graphs sweeps in metadata and passes while content is incomplete — observed 1450 against a 1000-quad target.
• Pre-subscribe baselines are recorded, and a receiver that already held the corpus is reported NOT DECISIVE rather than passing.
• The hold-out proves reconstruction by ordering (the CG is created after the node is stopped), because once catch-up works it converges faster than a baseline read.
• A curated negative asserts the real rule. An earlier version asserted that a non-curator's SWM write must be refused — a rule that does not exist — and would have manufactured a false security finding.

Structural limits it makes visible

Worth knowing when designing any devnet test:

• A devnet below ~5 nodes cannot produce a non-host subscriber. ACK quorum is 3, so every peer must ACK. This hid the VM reconcile defect.
• Public gossip pre-populates every running node regardless of subscription, so a "late" subscriber isn't late. This hid the catch-up defect.
• The hold-out (stop a node before publication, publish, restart via devnet.sh restart-node) is the only construction here that makes corpus reconstruction decidable on one box.

Operational traps documented

• Default devnet ports collide with a locally-running production node (Blazegraph 9999, oxigraph 7878/7879, UI 5173). Override DEVNET_* and DEVNET_DIR.
• restart-node re-derives every port from env. A caller that omits LIBP2P_PORT_BASE silently returns a node on the default libp2p port, where no peer can reach it — while its API still answers 200.
• The outage case is destructive; it now restarts the node it stops, and the health check names the cause instead of surfacing a bare fetch failed.

Usage

node devnet/public-cg-sync-proof/proof.mjs                      # base gate
RESILIENCE=1 NUM_NODES=6 node devnet/public-cg-sync-proof/proof.mjs   # + scale/outage/hold-out
node devnet/public-cg-sync-proof/verify-fixes.mjs               # #1779 / #1780 behaviour

Current result on a clean 6-node devnet: 33/35. The two failures are a known, redundant chain-reconcile watermark defect (documented in FINDINGS.md) — VM is delivered by durable peer sync regardless.

Suggested merge order

Consider merging this before #1842 and #1843, so reviewers can reproduce both bugs before and after the fixes.

🤖 Generated with Claude Code

[otReviewAgent]

🟡 Issue: Fold these ad-hoc runners into the canonical devnet harness instead of duplicating infrastructure

What's wrong
The PR adds a parallel, unregistered devnet test architecture. That increases maintenance surface in exactly the places that tend to drift: auth-token parsing, port derivation, fetch retry behavior, SPARQL binding parsing, polling semantics, and suite discovery. It also leaves a large monolithic proof script outside the normal review and execution path.

Example
A future /api/query response-shape or devnet port-resolution change already has a canonical place to update in _bootstrap/harness.ts; this PR would require separately remembering to update both proof.mjs and verify-fixes.mjs, and neither is covered by the suite manifest guard.

Suggested direction
The code-judo move is to make the new files mostly scenario descriptions and assertions, with HTTP/query/wait/devnet discovery coming from the shared devnet harness. That deletes duplicated mechanics and makes the suite discoverable by the existing manifest checks.

Confidence note
This is based on repository-local conventions: existing devnet suites are wired through Vitest, devnet/suites.json, package scripts, and devnet/_bootstrap/harness.ts. If this directory is intentionally a throwaway forensic artifact rather than maintained devnet coverage, the author should make that boundary explicit.

For Agents
Rework devnet/public-cg-sync-proof as a normal devnet suite: add package.json/vitest.config.ts, wire it into devnet/suites.json and the root script if it is meant to be maintained, and move transport/query/wait/node config helpers behind existing _bootstrap/harness.ts primitives or one small local harness module. Preserve the public/open, public/curated, resilience, and verify-fixes scenarios as scenario-level assertions.

[otReviewAgent]

🟡 Issue: The new scripts duplicate a private devnet harness instead of sharing one boundary

What's wrong
This PR adds two standalone mini-frameworks for the same devnet mechanics. That is exactly the kind of copy-pasted orchestration layer that makes release proof code brittle: the interesting public-CG scenarios are buried under duplicated request wrappers, polling loops, token parsing, result formatting, and agent registration plumbing.

Example
If the daemon auth token parsing, retry policy, JSON parsing, timeout defaults, or port derivation changes, maintainers now have to update the canonical devnet harness plus two private .mjs harnesses in this directory. Those copies already diverge: proof.mjs has CONVERGE_MS, POLL_MS, artifact writing, and stricter empty-token handling; verify-fixes.mjs has fixed 90s waits and returns undefined for an empty token.

Suggested direction
Create one shared harness boundary for this directory, or reuse the existing devnet bootstrap harness, then leave each script with only scenario-specific assertions.

Confidence note
I am treating these scripts as intended to live in the repo rather than as throwaway local artifacts, because the PR commits them under devnet/ with reproducibility docs.

For Agents
Extract a small local module such as devnet/public-cg-sync-proof/harness.mjs, or convert these to Vitest suites that import devnet/_bootstrap/harness.ts patterns. Keep the two scenario scripts behaviorally equivalent, but centralize HTTP calls, token loading, polling, SPARQL result parsing, and result recording.

[otReviewAgent]

🔴 Bug: Markdown visibility check can count metadata as content

What's wrong
The #1779 verification is intended to catch the peer having metadata but zero content triples. Its query is broad enough to match run metadata that includes the same stamp, so it can pass without proving markdown content reached the peer.

Example
If the markdown share again replicated only metadata and no imported heading/body triples, the query can still count the CG name literal FX md ${stamp} and report PEER sees markdown CONTENT triples as PASS.

Suggested direction
Narrow the SPARQL assertion to imported markdown content, for example exact body/heading literals or known imported subjects, and exclude context graph or operation metadata graphs.

Confidence note
This depends on the devnet store exposing the context graph metadata graph to the peer query, which is consistent with the create/subscribe flow but should be confirmed in a run.

For Agents
In verify-fixes.mjs, bind the imported KA subject(s), assertion graph, or exact markdown literals that cannot appear in CG metadata. A regression test should fail when only operation/context metadata exists on the peer and no markdown content triples are present.

[zsculac]

Review findings for head c621ac1.

Git classifies proof.mjs as binary because it contains a literal NUL byte, so GitHub does not provide normal line anchors for that file. The source line references below are from the committed blob.

1. [P1] Require a genuinely non-host receiver before reporting the VM result as a pass (proof.mjs:445-507). When no peer outside the storage-ACK set is available, the script records an informational message but continues with the preselected late receiver and evaluates pass: lateVm.done === true, even when lateWasHost is true. This is guaranteed to be non-decisive with the documented default of four nodes, because the PR itself notes that ACK quorum covers every peer at that size. Require a resolved nonHostNode, require lateWasHost === false, and otherwise record this assertion as non-decisive rather than counting it as a pass.

2. [P1] Make the curated VM negative reach the authorization boundary (proof.mjs:522-580). badKa is intentionally kept out of SWM, then vm/publish is called for it. A precondition failure such as "No quads in shared memory" therefore satisfies refusedHttp || notConfirmed and is reported as proof that curated publication was refused, even though publisher authorization may never have been evaluated. Create a dedicated finalized/SWM-shared asset that is otherwise publishable, invoke publication as the non-curator, and assert the specific authorization status/code rather than accepting every HTTP or non-confirmed failure.

3. [P1] Do not count pre-populated receivers as convergence passes (proof.mjs:655-759). The scale phase detects that a receiver already has the full corpus but still records the subsequent convergence check as a pass. In the outage phase, the supposed pre-subscribe baseline is queried only after subscribe has already returned, and a complete baseline still produces pass: survived.done === true. This contradicts the PR requirement that pre-populated receivers be reported as NOT DECISIVE. Query the outage baseline before subscribing and exclude both checks from boolean pass totals whenever the exact corpus was already present.

4. [P2] Require exact late-SWM convergence (proof.mjs:472-483). The live assertion compares the receiver digest with authorSwm, but the late assertion succeeds on d.count > 0. With the two-quad fixture, receiving only one quad is enough to pass. Compare both count and digest with the author snapshot so the cold-backfill result proves complete content rather than mere presence.

5. [P2] Reconcile the curated SWM negative with the stated specification (proof.mjs:509-560). The PR description says that requiring a non-curator SWM write to be refused is a rule that does not exist, yet this block fails if the explicit share is not refused. The committed reports also describe different semantics. Please identify the authoritative rule and align the harness and reports with it; if only VM publication is curated, the SWM refusal should not be a pass/fail gate.

6. [P2] Verify the #1780 route contract without an ignored author-looking field (verify-fixes.mjs:272-283). The #1780 contract says the seal encodes the author and vm/publish must reject an explicit authorAgentAddress. This request includes agentAddress: member.address, which the route does not use for author selection, and there is no assertion for the required 400 override rejection. Publish successfully with only contextGraphId, then add a separate request using authorAgentAddress and assert the 400 response. That makes auto-resolution and override rejection independently observable.

7. [P2] Keep the main harness reviewable as text (proof.mjs:101-108). digestQuads embeds literal NUL and SOH control bytes as separators. The NUL causes the entire 44.9 KB JavaScript file to appear as Bin 0 -> 44929 bytes, hiding its implementation from the normal PR diff. Spell the same runtime separators as escaped text such as \0 and \x01, or use an unambiguous serialization, so the source remains line-reviewable without changing digest behavior.

8. [P3] Mark the superseded status report as historical (V10.0.9-STATUS.md:1-7,248-280). This document says public-CG sync is not shippable, #1780 is unverified, and resilience has not run, while the PR description and FINDINGS.md present the later opposite conclusion. If this is intentionally retained as an investigation snapshot, label it with the tested commit and the later report that supersedes it; otherwise update it. As committed, readers cannot tell which release conclusion is current.

The existing discussion about integrating these scripts with the canonical devnet harness already covers duplicated infrastructure and retry behavior, so I have not repeated it here.

[lupuszr]

The harness is valuable, and the updated head passes the SPARQL scalability lint locally. It still cannot serve as proof that public-CG SWM sync works for two blocking reasons:

1. proof.mjs:474 can pass with SWM completely absent. This assertion runs after VM publication, calls readSubject without a graph filter, and checks only count > 0; digestQuads also excludes graph identity. A late node that received only durable _verifiable_memory content therefore satisfies the claimed SWM check. Restrict the read to the SWM graph family and require exact count plus digest equality with authorSwm, with a pre-subscribe baseline.

2. proof.mjs:804 defaults the decisive hold-out to five KAs and 100 content quads, which stays on the one-invocation happy path. It does not force the resumed metadata path that the current #1842 head fails. Add a deterministic case that checkpoints between at least two complete KA descriptor sets, resumes from a non-zero metadata offset, and verifies every expected per-KA SWM graph by exact digest rather than aggregate count.

GitHub could not attach these as inline comments because this file contains a literal NUL byte and is rendered as binary. Removing the control byte would also restore reviewable text diffs.

[branarakic]

Markdown visibility check can count metadata as content (RED) — fixed in baaed65. The filter matched any literal carrying the run stamp, which includes the CG's own name literal (FX md <stamp>), so it could report PASS while the peer held metadata and zero imported content — the exact #1779 signature it exists to detect. It now binds distinctive body strings that appear only inside the imported document and excludes the _meta graphs.

Fold into the canonical _bootstrap harness (YELLOW) — agreed in principle, deferred deliberately: these runners were built to be self-contained evidence during an incident-style investigation, and folding them into _bootstrap/harness.ts + the suite manifest is a real port I would rather do as its own PR than as a rider here. Noted as follow-up; the auth-token parsing, port derivation, and polling in these files are already flagged in the file headers as duplication debt.

🤖 Generated with Claude Code

[lupuszr]

Follow-up on the current head after running the combined #1842/#1843/#1844 stack on an isolated six-node devnet (4 core, 2 edge). The normal proof run finished 23/26; an enlarged resilience run with a node stopped before publication and HOLDOUT_KAS=40 finished 32/34. The holdout did reconstruct 800/800 quads in about 3 s, so the public happy path and late corpus arrival are real. However, both public/open and public/curated still left VM reconciliation permanently pending at watermark 0/1, and the first normal run also timed out waiting for the public/open live receiver VM content. More importantly, the harness still cannot prove that SWM and VM are independently complete: readSubject is unfiltered for key VM checks and digestQuads deliberately removes graph identity, so content from one graph family can satisfy the other claim. This is consistent with the separately reproduced #1842 cases where metadata/checkpoint state reports success while an assertion graph is absent. The companion verify-fixes run was also 9/10 (curator publish of a member-shared root returned HTTP 409). Current-head syntax checks pass, but the latest commit introduces the inline SPARQL lint blocker below. Verdict remains REQUEST_CHANGES: the stack demonstrates a useful happy path, but does not prove reliable convergence of both public-CG memory layers.

[lupuszr]

[P1] The latest explanatory block moved the sparql-scan-allow: R2 pragma away from the query it acknowledges, so the current head now fails the repo gate: node scripts/sparql-scale-lint.mjs --diff a6f33e4... baaed650... reports 1 new blocking R2 graph-variable scan here. Keep the acknowledgement immediately adjacent to this query (with the bounded devnet justification), or bind/restructure the graph scan, before merging.

[otReviewAgent]

Operational Notice: Review Agent could not complete this review.

Synthesizer produced only invalid comment anchors.

[Jurij89]

Adversarial review — test(devnet): add public Context Graph SWM+VM sync proof harness

Reviewed against the PR head (c621ac120). The harness is genuinely valuable — the confound controls (content-presence vs sync-mechanism, decisive-receiver-after-publish, exact-count binding) are the right lessons, and the two bugs it found are real. Findings are all low severity, but the first blocks reviewing the harness itself.

---

🟠 LOW–MEDIUM — proof.mjs, the primary deliverable, is committed as a binary blob and cannot be diffed or reviewed

gh pr diff 1844 shows Binary files … proof.mjs differ. Cause: a single raw U+0000 (NUL) byte used as a field delimiter in a template literal at line 105 — schematically:

const norm = quads.map((q) => `${q.p}<NUL>${q.o}`).sort();  // <NUL> = a raw U+0000 byte in the source

git cat-file -s reports 44929 bytes; stripping NUL leaves 44928 — exactly one NUL, at byte offset 4344 (line 105). That one byte flips git's binary detection for the whole 44 KB file, so the harness this PR exists to deliver is unreviewable in the diff and every future edit produces no reviewable change. Fix: write the delimiter as an escape (� / \x00) or use a non-NUL separator (\t, ␟) — identical behaviour, and the file stays text.

---

🟢 LOW — verify-fixes.mjs's "#1780" block actually depends on #1843, and mislabels a #1843 gap as a #1780 regression

The #1780 section shares a member asset on a public/curated (agent-gated) CG and asserts the curator's node receives it and can VM-publish it. On plain main that receive is dropped by the very bug #1843 fixes, so #1780: curator node received the member-shared asset fails and the script process.exit(1)s — attributing to #1780 a failure that is really the absence of #1843. Given the PR suggests merging this harness before #1842/#1843, anyone (or CI) running verify-fixes.mjs in that ordering gets a red run mislabeled as a #1780 regression. Worth either gating the curated-receive assertion on "#1843 present" or relabeling it explicitly as a cross-PR dependency.

---

🟢 LOW — two durable status docs are committed with opposite verdicts, one shipping a "BLOCKER + proposed fix" that #1843 already implements

FINDINGS.md concludes "YES, for both public policy cells … Ship it"; V10.0.9-STATUS.md concludes "not yet shippable" with §3 describing the member→curator drop as an open BLOCKER and a "proposed fix" — which is exactly what #1843 now implements. These read as point-in-time investigation snapshots, but they're committed as permanent repo docs and will be stale/contradictory the moment #1842/#1843 merge. Consider dating them, marking them archival, or trimming the forward-looking status so the repo doesn't carry a "public CGs are broken / here's the fix we haven't written" narrative next to the merged fix.

---

🟢 LOW — #1779 content-presence check can pass on metadata alone (FILTER OR/precedence)

The content-count query filters isLiteral(?o) && CONTAINS(STR(?o), "${stamp}") || CONTAINS(STR(?o), "converge"). && binds tighter than ||, so it evaluates as (isLiteral && CONTAINS_stamp) || CONTAINS_converge — the converge branch runs against any object (including IRIs), unguarded by isLiteral. Since stamp also appears in metadata objects, the check can register content presence from metadata rather than the imported markdown body — the exact "metadata present, content empty" failure mode the harness is built to catch. Parenthesize the intended grouping and keep the isLiteral guard on both branches.

---

🔵 Worth confirming — decisive VM check vs. storage-host fallback

One verifier flagged (and one disputed) that proof.mjs's decisive-VM path can record pass: true against a receiver that is in fact a storage host, in the degenerate case where no provably-non-host node exists. The harness elsewhere reports INFO / NOT DECISIVE for this — worth a quick check that the decisive path can't fall through to a host receiver and score a pass, since that's precisely the false-green the FINDINGS methodology warns about.

---

Method: independent lenses (reviewability / harness-correctness / false-pass) plus two adversarial verifiers per candidate; the NUL-byte finding and the #1843 dependency were re-verified by hand.