Repository navigation
R7+R8: experimental validity layer, governance-graph typing, runtime-graph diff, CI with teeth - #32
Merged
Merged
Conversation
โฆutcome loop Three defects in the cross-match learning path, all found by reading the code โ the local corpus is seven rows, small enough that none of them could surface in a run. queryEpisodicMemory applied LIMIT 10 in SQL and only then scored by keyword, so everything a regime learned before its eleventh match was unreachable no matter how well it matched. Scoring now runs over the regime's whole history behind a cap set far above any realistic corpus. Worse than reported: the ordering was 'timestamp DESC' alone, and CURRENT_TIMESTAMP is second-granular, so memories written in the same second tie and SQLite's order among them is unspecified โ the old cap could just as easily have scanned the OLDEST rows. The AUTOINCREMENT id now breaks the tie. The retriever kept every token longer than one character, so 'the', 'of', 'and' matched nearly every paragraph and ranking degenerated into 'return whatever comes first'. Added an English/Chinese stopword set and a three-character floor, exported as extractKeywords so tests can pin it directly. The skill auditor judged prose plausibility, which measures ~46% against real downstream outcomes โ chance. It now scores three properties that actually predict transfer: whether the skill names a failure mechanism, whether it is actionable at step level, and whether it generalizes a hazardous action. Feeding the judge score into the extractor turned out to be impossible as stated: sedimentation runs at each civ's match end, while judging only happens once every civ has finished, so no score exists yet. Added engine/v5/ skill-outcome.mjs instead โ after judging, the tournament stamps score, rank and provenance into each sedimented skill's frontmatter. Skills from a losing run are the ones most likely to transfer badly, and nothing downstream could weight or retire them without knowing. Best-effort: never fails the tournament. 17 tests, all revert-verified (reverting the LIMIT turns two red, reverting the stopwords turns one red). ci: 321 backend green.
โฆ anything CivAgent claims that differences between 57 historical governance topologies drive differences in governance quality. That claim has never been tested against a null. It is fragile in a specific way: agentCount is 7 for 29 of the 57 regimes and 6-8 for 51 of them, so a ranking that looks like a structural effect could be an agent-count effect. Adds three controls, generated into regimes/_baseline/ and never touching a real regime directory: solo (one agent, no edges), flat-N (same agent count, zero edges, fully parallel) and random-N (same agent count, comparable edge count, edge kinds sampled from the source distribution, connected, no self-loops). Random generation is seeded and reproducible. All three pass validate:regimes, and topology-bearing sources also produce valid topology.json. Reproducibility fix found in review: the artifacts embedded a generatedAt wall clock, so two runs of one seed were not byte-identical. Worse, the test excused it by deleting the field before comparing โ legalising the very non-determinism the seed exists to prevent. The stamp is gone, and the test now also asserts structurally that no timestamp appears in the output: comparing two back-to-back runs cannot catch this on its own, because both land in the same millisecond and compare equal while the artifact is still irreproducible. ci: 333 backend green (29 baseline tests, revert-verified).
Every conclusion CivAgent draws runs through blind judging plus Bradley-Terry, and three known judge biases were neither controlled nor measured. Without that, the rankings โ and any later regression on whether a learned skill helps โ have no statistical footing. Verbosity: transcripts are truncated to a shared budget before the prompt is built, so the score stops partly measuring who wrote more. Truncation is recorded rather than silent. Position: swap passes already averaged the two orderings, which hides the effect; the per-regime forward-vs-swapped delta is now reported, so a systematic seat advantage becomes visible instead of being averaged away. Self-preference: judge providers and civ backends are mapped to model families and the same-family vs cross-family score gap is reported. All of it lands in a new biasReport field. The scores array keeps its exact shape โ the frontend and the stats pipeline both read it, and the Bradley-Terry maths is deliberately untouched this round. ci: 319 backend green (15 calibration tests, fake judges only).
The smoke run that motivated this scored china/tang 10 and its random control 2.5 โ but the control's transcript was the model asking the operator how to configure the experiment. The generator had replaced every office with "Baseline Agent N โ generic governance agent" and cut SOUL.md to a 571-byte disclaimer, so the run varied persona and topology at once and the measured gap was persona presence. Any ranking built on those controls was measuring the wrong thing. Controls now copy SOUL.md verbatim (plus a provenance comment), keep every office id, label and duty from the source, and rewrite only the Decision Flow prose and the mermaid diagram to describe their own edges. The diagram matters: leaving the source's graph in place made a control assert two contradictory wirings and follow whichever it read last, quietly restoring the variable the control exists to remove. Also: - topology.regime is derived from the destination directory, mirroring the validator's own rule, instead of a synthetic name that fails the cross-check the moment a control is staged somewhere runnable. - new --dest flag stages a control at an exact path; without it every control lands three levels deep where that cross-check never fires. - validateRegime accepts a single leading underscore on the region, so _baseline/* and _ablated/* can run as civs at all. They could be generated and validated but never used โ which is how an experiment ends up with no control arm. - the default-path test deleted the whole regimes/_baseline tree on cleanup, so a full `npm test` wiped committed control arms. It now removes only what it created, guarded by a sentinel. - three revert-verified persona-parity tests; a stale hardcoded agent id in an existing test was a silent no-op replace and is now derived. Re-run on cn:doubao: both arms stay in role and produce full governance documents (control 19KB, source 3.4KB digest); control 10 vs source 7.9. The remaining length asymmetry is verbosity bias, which R7-2's judge budget addresses separately. CI: 369 backend + 14 frontend + 57 regimes, all green.
The CI badge was a hardcoded green image. It stayed green whether or not CI passed, which is the worst kind of README claim: the one a reader has no way to check. It now points at the real ci.yml workflow status. Other corrections, all verified against the code rather than memory: - backend count 10 -> 11, and ยง6 no longer says "Ten-Model" in the title while saying "9 backends" in its own closing line. Enumerated instead. - ยง8.4 claimed 9 unit tests; there are 369 backend + 14 frontend. - ยง9.5 claimed a single judge and promised multi-judge as future work. Multi-judge, anonymization, anchored rubric, order swap, verbosity budget and the bias report all exist; the section now describes the residual bias instead of a solved problem. Added: - ยง2.1.1 places the project against "graph engineering", the term that spread in mid-2026, and states the two caveats that bound every claim in the file: rankings are over declared topologies, and controls are only valid while persona is held constant. - ยง3.5 documents the R5-R7 validity layer. - ยง9.6 states plainly that no declared edge has been verified as traversed, so a regime declaring an unused veto edge is behaviourally flat while scored as checked. - ยง8.1 documents the model-free path, so someone without an API key can still run something. All four commands in it were executed before being written down. - ยง12.1 now mentions topology.json and warns that a prose IDENTITY compiles to zero agents and passes silently. Also gitignores .ccb/, the project-local agent runtime.
โฆuter) Nodes were untyped: every node was implicitly an LLM-backed office, so the graph could not express the difference between a reviewer that may object and a rule that always fires. `kind` is optional and defaults to agent, so all 57 existing regimes are unchanged. Validator constraints, each with a stated consequence: an unknown kind is rejected rather than silently read as agent (which would under-count gates); a non-agent kind cannot hold a content-producing functional_role; a router may not emit review/veto edges (that is judgment, not dispatch); a gate or checkpoint with no inbound edge is unreachable. Metrics gain gate_count and checkpoint_count, deliberately separate from checks_cycles. Review note: the first pass labelled menxia and silijian as gates. Both are LLM reviewers exercising judgment, and "gate" is defined in this very schema as "no model discretion". Labelling them would have made gate_count report a hard mechanical veto where the regime has a discretionary one. They are agents; only shangshu keeps an annotation (router), on the strength of its SOUL text. No current regime declares a deterministic gate, and the test now asserts that โ the genuinely deterministic [VETO] sniffer lives in engine/mechanisms/veto.mjs, attached to an agent node rather than modelled as one.
โฆeclared one Every ranking this project produces is over a declared topology; nothing checked whether a declared edge was ever traversed. reconstructRuntimeGraph rebuilds the actual call graph from the events.jsonl span tree, and diffTopology reports unexercised edges, undeclared edges and per-edge exercise counts. Both tolerate truncated streams: a SIGKILLed match has no match_end, and orphan or duplicate spans are reported rather than thrown on. The finding is more useful than the feature. Declared nodes are offices (zhongshu, menxia); runtime actors are regimes (china/tang), because run-v5.mjs emits every turn with actor=regime โ Claude Code runs all offices in one process and the output is not tagged per agent. The two id spaces do not overlap at all, so per-office edge verification is not merely unimplemented, it is blocked on instrumentation. unexercised_ratio is therefore null when the id spaces do not match. It would otherwise read 1.000 for every regime in every match: a number that looks like a measurement, is not one, and is exactly the kind of value that ends up in a results table. civagent runtime-graph <matchId> [--diff] [--json], match id validated against the same SAFE_MATCH_ID contract replay.mjs uses.
CI ran lint, unit tests and regime validation but never exercised a full path through the system, and someone cloning the repo without an API key could run nothing at all โ every headline command needs a model backend. examples/example-regime is a small invented regime (4 offices, clearly labelled as an example so it cannot contaminate the 57-regime corpus) with a markdown role table, which is also the format the parser requires: a prose IDENTITY compiles to zero agents and passes silently. examples/smoke.mjs chains validate -> metrics -> generate a flat control -> validate the control, entirely on pure functions and local data. It writes to a temp dir with a finally-cleanup, so a crash cannot leave a 58th regime behind. Verified red as well as green: adding an edge to a nonexistent node makes smoke exit 1 with the offending edge named. Backend CI now runs on Node 20 and 22 (engines raised to >=20; 18 is EOL and CI was already on 20), so the minimum supported line and the current LTS are both exercised.
โฆ events Re-recording a tournament id stacked rows in match_results and episodic_memory, silently inflating the win counts Bradley-Terry reads. A UNIQUE(tournament_id, match_id) plus INSERT OR IGNORE makes a re-record a no-op, the original row wins, and a skipped count is warned rather than swallowed so a retry storm stays visible. Existing databases get the constraint via CREATE UNIQUE INDEX IF NOT EXISTS, and a failure to create it (pre-existing duplicates) is reported instead of aborting boot. The pre-existing test asserted the stacking as correct behaviour, which is how the defect survived; it now asserts the fixed contract. Review note: the first pass put UNIQUE(match_id, regime, event_type) on the whole episodic_memory table and swallowed collisions with INSERT OR IGNORE in /api/history/sync. That silently discarded the second veto_triggered event of a match. Two vetoes are two facts, not a duplicate โ and for a project whose subject is checks and balances, that is the most damaging row to drop. Uniqueness is now a PARTIAL index scoped to match_end, the one event recordTournamentResult writes exactly once per (match, regime); /sync keeps a plain INSERT, since its clear-then-insert already gives re-sync idempotency. /sync also counted insert attempts rather than rows written, so it reported more imported events than a client could read back.
README: 369 -> 421 backend tests, Node >=20, smoke in the CI sequence, and runtime-graph in the CLI reference. ยง9.6 previously said the declared-vs- exercised comparison had not shipped; it has, and what it established is that the comparison cannot be made at office level at all, because the event stream tags actors by regime. The section now says that rather than implying the gap is merely unimplemented. ESLint: .ccb/ holds the CCB agent runtime โ git worktrees and vendored provider code checked out inside the repo. It is gitignored, but ESLint does not read .gitignore, so lint:backend was reporting third-party files as this project's errors and taking minutes to do it.
P1 โ skill outcome stamping crossed tournaments.
skillsForMatch matched on the last six characters of the match id. A
tournament child id is `${tournamentId}__${regime with / as -}`, so every
china/tang child ever run ends in "a-tang": stamping one tournament
overwrote outcome_score/rank/total/of on skills learned in unrelated
earlier tournaments, teaching the corpus that a losing pattern had won.
Attribution now reads the full match id from the provenance banner
writeSkillFile already writes. The old test used a made-up match-id shape
(`T-1__china-tang-abc123`) and stayed green against the broken path; the
fixtures now use the real shape, and a new test pins that stamping one
tournament leaves another's skills untouched.
P2 โ controls leaked the source wiring for non-mermaid diagrams.
rewriteMermaid only matched ```mermaid fences. china/ming draws its
dual-track chart as ASCII in a plain fence, so the Ming flat control
declared zero edges in topology.json while still showing the model
็ๅธ โ ๅ
้/ๅธ็คผ็ โ ๅ
ญ้จ. The model reads IDENTITY.md, not topology.json,
so that control isolated nothing. Any fenced diagram is now replaced, and
the control's Decision Flow says explicitly that the historical procedure
described above it is background and the flow below governs.
P2 โ the constraint migration did not migrate.
CREATE UNIQUE INDEX raises on a table that already violates it, and the
catch logged and continued, leaving INSERT OR IGNORE nothing to ignore
against. It now collapses pre-existing duplicates (earliest row wins,
matching OR IGNORE semantics) and retries, and reports loudly if
uniqueness still cannot be enforced. It also drops the superseded
table-wide ux_episodic_memory_mre index, which would otherwise make a
plain INSERT of a match's second veto fail. Both paths tested.
P2 โ judge events hung off a span that was never opened.
judgeSpanId was generated but no event carried it as its own span_id, so
runtime-graph reconstructed the whole judging step as orphans. A
judging_start event now opens the span. Judge-event assertions are scoped
to scoring events; the integration test additionally checks end to end
that every pass parents to the opener.
P2 โ the diff could not read control regimes.
loadDeclaredTopology hardcoded china|global, excluding _baseline/* and
_ablated/* โ the control arms this comparison exists to evaluate. It now
uses the same contract as civ-memory.mjs::validateRegime.
P3 โ dropped the shangshu "router" label.
regime-to-cc still compiles it into a model-backed subagent with a prompt,
while the schema defines router as making no model call and producing no
content. No regime now carries a kind annotation, and a test asserts it:
the typing mechanism stays, the unearned labels do not.
CI: 427 backend + 14 frontend + 57 regimes + smoke, green.
Five regimes and three scenarios leave enough freedom to narrate almost any result after the fact. The predictions โ including the one that would count against the project's own proposition, a ranking identical across all three scenarios โ are committed before the first match runs.
Five regimes x three scenarios, all on cn:doubao. Pre-registered at 7841c96. P1 held: the winner reorders across scenarios (ming / tang / qin), with tang moving from last to first. P2 held on two of three. P3 failed on S3, where the spread between first and last (1.25) is smaller than the position effect the same judge showed between its own two passes (1.67) โ that cell produced no measurement at all, so P2's failure there is not interpretable either. The finding that outranks the predictions: score tracks transcript length (Spearman +1.00 on S1, +0.70 on S2), and the three shortest transcripts are not short answers but summaries of work that was never captured. china/tang's S1 transcript contains no policy, only a claim that an edict 'was already drafted, debated through two rounds of Menxia veto ... and dispatched' โ none of which is in the stream. It scored last. Three of fifteen cells are affected, and each is the lowest score in its scenario. So E1 cannot separate the two worlds it was built to separate: a third explanation, whether the answer was captured at all, dominates both. Transcript capture is now the blocker, ahead of control arms and repeat cells. Also fixes a defect the experiment exposed: judge() computed biasReport and the manifest writer dropped it, so every tournament ever run recorded a ranking with no way to check it against the judge's own noise โ the exact question P3 asks. Persisted, with the assertion added to the manifest contract test and revert-verified. S1-S3 predate the fix; their position effects are recomputed by hand from the per-pass JSON.
The backend was spawned as `claude --agents <json> -p "<task>"`. The default output format for `-p` is `text`, which prints exactly one thing: the coordinator's final assistant message. A regime's offices are Claude Code subagents, so every word they exchanged went to the subagent channel and was discarded. This is what the E1 pilot was actually measuring. Three of fifteen transcripts contained no policy at all, each scored last in its scenario, and score tracked transcript length with Spearman +1.00 on the one scenario where the spread exceeded the judge's noise. The tang cell's transcript claimed an edict "was already drafted, debated through two rounds of Menxia veto ... and dispatched" and I read that as the judge scoring a claim instead of work. The claim was true. The work was happening; the harness was throwing it away. Backend now runs with --output-format stream-json --verbose, and engine/v5/stream-json.mjs renders the stream back to plain text. A delegation carries input.subagent_type and the reply carries parent_tool_use_id, so turns are attributed per office (china/tang#menxia). Turn events keep their shape, so the judge, cleanTranscript and the dashboard are unchanged. Verified on a real run, same regime and scenario as the failing E1 cell: 579 captured characters -> 72,828; 3 turn events -> 60; all three departments attributed, with the Chancellery's three rounds of fengbo review present, including substantive historical criticism of the draft. --- and a second defect that the first fix exposed --- With office prose finally reaching the mechanism engine, the first verification run was SIGKILLed mid-draft. checkVeto matched the bare substring ้ฉณๅ, and the Secretariat had merely *described* the Chancellery's power to amend and return documents. All three mechanisms shared the defect: checkEdict matched bare ่ฏไนฆ, the commonest noun in a Tang governance document, and firing EDICT grants vetoImmunity โ so a checks-and-balances regime would have silently lost the behaviour under study. checkImpeach matched an unbracketed ๅผนๅพ๏ผX. Only [VETO] is ever taught to agents as a marker; 33 of the 57 regime files contain ้ฉณๅ/่ฏไนฆ/ๅฃๆจ as ordinary vocabulary. All three now require bracketed markers, with a regression test that feeds a regime's own description of its constitution through the engine and asserts nothing fires. Both defects were dormant only because the transcripts were empty. E1 is superseded: its scores came from transcripts missing most of what the regimes did, and the three lowest are the three worst-affected cells. The pre-registration stands and should be re-run unchanged. CI: 442 backend + 14 frontend + 57 regimes + smoke.
โฆutruns the data Now that transcript capture attributes turns per office (china/tang#menxia), the runtime graph reconstructs at office level: which offices spoke, how often, and the coordinator's dispatch sequence, parsed from the [โ office] lines. The investigation result matters more than the feature. Typed office-to-office edges โ the thing topology.json actually declares โ still cannot be observed. Claude Code subagents do not call each other; the coordinator relays every hand-off, so `menxia โ zhongshu [veto]` is physically indistinguishable from `coordinator โ menxia` followed by `coordinator โ zhongshu`. No edge is inferred: inferred.topology_edges is empty by design, and the comparability gate refuses to treat a coincidental edge-kind match as evidence. Review fix on top of that work: the diff still filled unexercised_edges with all 15 declared edges and total_unexercised with 15 while comparable was false. A consumer reading the count without also reading `comparable` concludes none of Tang's edges were exercised โ and in the very match this was verified against, the Chancellery reviewed the Secretariat's draft three times, each time using the word ๅฐ้ฉณ. Those edges were exercised; they are unobservable. A count that looks like a measurement and is not one is the defect this module already rejected at the ratio level, one level down. When comparable is false, unexercised_edges, total_unexercised and edge_exercise_counts are now null, and the same list is published as unobservable_edges / total_unobservable. Per-edge runtimeCount is withheld too: a zero for an edge nobody could observe reads as "never fired". The CLI says "Unobservable: 15 of 15 declared edges (not a count of non-exercise)". Old matches without office actors still degrade to regime granularity. CI: 457 backend + 14 frontend + 57 regimes + smoke.
R10 was orchestrated by Codex across six cluster agents. Verified independently rather than accepted on report. R10-1 (the one that mattered): the judge read `readMatchText(matchId, 6000)`, whose implementation is `text.slice(-6000)` โ a blind tail. Fine when transcripts were 1-20KB; after the stream-json capture fix a Tang match is 72,830 characters, and I measured what survived: zero of the Secretariat's four segments, zero of the Chancellery's four, zero of the Department of State Affairs' three. Only two coordinator fragments. The three rounds of fengbo review โ the entire behavioural signal of a checks-and-balances regime โ were outside the judge's view. Selection is now actor-stratified: on the same match, 3/4, 3/4 and 3/3 office segments reach the judge. Per-civ metadata records originalLength, selectedLength, postAnonymizationLength, judgeViewLength and omittedContentChars (67,867 on that match), so the omission is stated rather than implied. R10-2 was deliberately NOT implemented, and the reasoning is right: caused_by_span_id alone cannot establish a typed officeโoffice edge, and "one observed handoff upgrades the whole graph to observable" does not follow โ absent edges may be unexercised or merely uncaptured. unexercised_ratio keeps returning null. Declining to ship a measurement is the correct outcome here. R10-3 stages five seeded random controls and a dry-run-by-default experiment orchestrator that refuses --execute without an operator-supplied AFP estimate, since CivAgent cannot measure AFP itself. R10-4 renders office badges with old-actor fallback. R10-5 declined a new API endpoint with a reason. Review finding I fixed: nothing pinned the strategy the judge actually uses. Flipping tournament.mjs back to "tail" left all 462 tests green โ the unit was tested (selectTranscript samples across actors) and the wiring was not. That is the seventh time this repo has had a suite that passes while the thing under study is broken. test/judge-transcript-wiring.test.mjs now drives judge() over a synthetic three-office stream and asserts every office's marker reaches the prompt; it goes red under "tail". CI: 464 backend + 17 frontend + 57 regimes + smoke.
โฆnning The orchestrator fixed the design but registered no claims. C1 names the outcome that would count against the project's own proposition โ random wiring winning or tying as often as it loses โ and requires it be reported as plainly as the favourable one. C2 keeps pairs inside the judge's own noise band out of the tally. The E1 smoke datapoint where the control beat the source 10 to 7.9 is recorded here so it cannot be quietly forgotten.
โฆn a null Three tournaments, five source/control pairs each, cn:doubao throughout, --no-skill everywhere, anonymized and order-swapped. Predictions registered at 495fadd before the first execute. Scored against the pre-registered rules: historical wiring wins 0, random rewiring wins 1, unresolved 5, excluded 9. Every apparent source win in the raw table came from a pair where at least one arm never delegated to any office. That is the actual finding. In 13 of 30 arms the coordinator answered the task itself and no topology โ historical or scrambled โ was exercised at all. This is not a capture failure: raw-log fallback was 0/30 and every arm used actor-stratified selection. The offices were simply never invoked. Delegation turns out to be scenario-driven, not constitution-driven: plague 0/10 arms failed to delegate, border-city 6/10, militarization 8/10. Whether a regime runs its own constitution depends on the task text. And it costs almost nothing not to: single-actor arms averaged 7.37 against 7.56 for multi-actor arms, which is a plausible reason the models often skip it. So the obstacle has moved rather than cleared. Before, the harness discarded the deliberation; that is fixed. Now the regimes frequently do not deliberate, and nothing in the design noticed until the pairs were tallied. Delegation has to become a measured precondition at run time โ the dispatch count is already in runtime-graph's observed block โ and the project has to decide whether delegation is enforced or optional, because those are different experiments. Transcript length spanned 2,321 to 441,914 characters, a 190x spread, median 16,209. No claim that topology affects performance is supported by this run. Neither is the opposite.
E1-control could not answer either question: 13 of 30 arms never called a single office, so neither the historical nor the scrambled topology was exercised, and delegation turned out to be driven by the scenario (plague 10/10 delegated, militarization 2/10) rather than by the constitution. Source and control arms failed to delegate at exactly the same rate, 7/15 each. R11 builds the instrument rather than picking a side. The coordinator first states, with tools off, which offices it would call โ that plan is B's dependent variable. Enforcement then requires the offices that are targets of incoming edges in that arm's own topology, and only post-enforcement output is eligible for A. Verified independently rather than accepted on report: - Enforcement is symmetric. requiredOffices comes from officesWithIncomingEdges on each arm's own topology; there is no branch on _baseline anywhere in dispatch-plan.mjs or plan-diff.mjs. The control is forced to run its scrambled wiring exactly as the source runs its historical one. - plan-diff invents no composite score. Membership, order and omission are supported; directed_edge_alignment, edge_kind_alignment, multiedge_coverage and edge_exercise are named in an explicit unsupported_dimensions map with a reason each. - Participation is a three-state observation (observed / not_observed / unknown), not an execution rate. - The wiring is pinned, not just the units: cutting the enforcement injection out of run-v5 turns the R11 wiring tests red. That was last round's failure mode and it did not recur. Review addition: the plan question is itself an intervention, and E2 did not say so. Asking "which offices would you call" raises the salience of delegation, so B measures stated intent when asked, not spontaneous behaviour. E2's plan-stage adoption rate is therefore NOT comparable to E1-control's 13/30 baseline, which was collected with no plan question. The source-control contrast inside E2 survives, since both arms get the identical prompt, but the absolute level is inflated by an unknown amount โ and the gap against E1 is a lower bound on the observer effect, to be reported rather than read as regimes adopting their topologies more often than E1 suggested. Registered in E2-preregistration.md before any E2 run, along with the decision not to add a no-plan control arm. Standing limit, unchanged: enforcement proves the coordinator called the offices a topology names. It does not prove any command/review/info/veto edge executed. E2 is designed (3 scenarios x 5 repeats x 10 arms = 150 arms) and NOT run. CI: 472 backend + 19 frontend + 57 regimes + smoke.
Bilingual pair in the repo's standard shape: two single-language files with a toggle badge row at the top of each (current language green, the other grey and linking across), matching README.md section for section โ 14 numbered sections, same tables, same figures. Both pages updated to the state after R8โR11: - test counts 421 -> 472 backend, 14 -> 19 frontend - ยง3.5 gains transcript capture, actor-stratified transcript selection for judging, and the dispatch plan / equal-roster enforcement - new ยง9.7 records what the two pilots actually showed, including the result that counts against this project's own proposition: scored against pre-registered rules, historical wiring won 0 resolved pairs, and 13 of 30 arms never called a single office, so neither topology was exercised - release history gains the R8-R11 row The Chinese page is a translation of the same claims, not a softened summary โ the limitations sections (ยง9.1-ยง9.7) carry the same numbers and the same admissions as the English.
Pre-merge review of PR #32. Replacing planPrompt with the literal string "noop" โ so the coordinator is asked nothing at all โ left all 472 tests green. buildPlanPrompt's non-inducement wording was asserted in isolation while nothing checked that the wording reached the model. That hole sits on B's entire measurement. Under it the plan call could ask nothing, or ask a leading question, and E2 would still collect numbers labelled 'voluntary adoption' and generated from neither. The run-v5 wiring test now asserts against the captured plan-phase invocation: the adoption question is present, the non-inducement sentences are present, the office roster is present, and no explicit demand to use all offices is. Both failure modes revert-verified โ cutting the prompt reddens it, and rewriting the wording into 'You must call all available offices' reddens it twice. Seventh recurrence of unit-tested-but-unwired in this repo; the check that found it is now part of the pre-merge routine.
Codex pre-merge review, P1, and it is my own regression. R10 correctly stopped
the mechanisms from matching bare words like ้ฉณๅ and ่ฏไนฆ โ but that is only
half the contract. engine/regime-to-cc.mjs was still compiling every
veto-bearing office with "reject it explicitly by stating \"VETO:\"", while
checkVeto now accepts only [VETO] / [ๅฐ้ฉณ].
So every checks-and-balances regime carried a veto power no agent could
exercise. Tang's menxia and xingbu were both taught the inert form. Verified
directly: the compiled prompt teaches "VETO:", and checkVeto("VETO: legality
defect") returns false โ no event, no abort, the match records as a normal
completion.
All 472 tests stayed green because the mechanism tests hand-write the bracketed
marker. A test that supplies its own correct input cannot notice that the
production input is wrong. The new test takes the marker out of a really
compiled Tang prompt and feeds THAT to checkVeto, for every office granted
[CONSTITUTIONAL VETO POWER]. Revert-verified: restoring the old wording
reddens it.
Also corrects a published number. The single-actor count in E1-control is
14 of 30, not 13 โ 6 + 0 + 8 across the three scenarios. 13 appeared in
E1-control-results.md, both READMEs and E2-preregistration.md, and contradicted
the 7/15 + 7/15 split stated in the same paragraph.
Not merging. Three P1s from the same review remain open: mechanisms accept
markers from any text with no provenance or role authorisation (a tool result
quoting [VETO] can SIGKILL a match); dispatch and participation are inferred
from forgeable "[โ office]" text rather than from the structured tool_use the
renderer already sees; and baseline --dest can write outside the regimes tree.
Codex pre-merge review, P1. Every generator unconditionally overwrites IDENTITY.md, SOUL.md, metadata.json and topology.json at the destination, and --dest went straight through unchecked. Reproduced against the real corpus: civagent baseline china/tang --type solo --dest regimes/china/tang destroyed the Tang regime in one command, IDENTITY.md 1465 -> 526 bytes. `--dest ../../anywhere` left the repository entirely. --dest is now confined to regimes/_baseline/, and even inside it refuses a non-empty directory that is not itself a generated control (checked via the _baseline marker in metadata.json), while re-running a control the generator produced stays idempotent. The existing test named 'real regime files are never touched' only exercised a temp outRoot and never the CLI's destDir, so it could not see any of this. Four new tests, all revert-verified: removing the guard block reddens three of them.
โฆfrom text Codex pre-merge review, P1 x2, one root cause. StreamRenderer held the structured evidence โ tool_use, tool id, subagent_type, message role โ and run-v5 flattened it to text before the mechanism engine, enforcement and runtime-graph each regex-guessed the structure back. Two exploits followed, both reproduced: - A user-role tool_result containing "[VETO]" โ external document text, or a prompt injection โ reached MechanismEngine.process() and SIGKILLed the match while emitting a forged veto_triggered. "[EDICT]" was worse: it grants vetoImmunity, so any untrusted content could silently disable the veto mechanism for the rest of the run. - A model that called no subagent at all could print "[โ zhongshu]" and "[โ menxia]" in ordinary prose and obtain enforcement_passed plus participation_observed with dispatchCount 2 and officeTurnCount 0 โ the exact precondition E2 uses to decide which arms enter the quality comparison. The stream-json capture fix is what widened this: subagent and tool-result content that used to be discarded now flows through the same scanner. Turn events now carry additive structured fields (message_role, content_items, tool_uses). processStructured() gates on actor presence, assistant role and absence of tool_result. Dispatch evidence comes only from tool_use items with a real subagent_type. runtime-graph prefers tool_uses and falls back to text only for legacy events that predate the field. Review additions on top of the agent's work: - dispatchOfficesFromText is deleted rather than left exported. It had no callers but manufacturing evidence is exactly what it did, and an exported helper that turns two printed strings into proof of delegation is a trap. - A test pins that run-v5 calls only processStructured. The gated and ungated entry points both exist, so a future edit could drop back to the unguarded one while every gate test kept passing โ they call processStructured directly and would not notice. Revert-verified: swapping run-v5 back to process(text) reddens it. CI: 502 backend + 19 frontend + 57 regimes + smoke.
โฆmoke The first version of the guard required --dest to live under regimes/_baseline/. That is not the property worth enforcing, and CI caught it: examples/smoke.mjs legitimately stages a control beside an example regime inside a disposable sandbox (projectRoot = an OS temp dir), so the smoke went red on both Node matrix legs while every backend unit test stayed green. A guard that is too strict looks exactly like this โ green units, red end-to-end. The property that actually matters is: never leave the project's regimes tree, and never write over the source regime the control is derived from. The guard now checks those two directly, plus the existing refusal to clobber a non-empty directory that is not itself a generated control. `civagent baseline china/tang --type solo --dest regimes/china/tang` is still rejected โ now with a message that names the real reason rather than a path convention. Note for anyone revert-verifying this: removing the guard block and running the suite actually executes the destructive case against regimes/china/tang. It happened twice here and was restored from git both times. Verify it in a scratch clone, not the working tree. CI: 502 backend + 19 frontend + 57 regimes + smoke.
LeoLin990405
added a commit
that referenced
this pull request
Aug 6, 2026
โฆgraph diff, CI with teeth (#32) Stacks R7 through R12 onto main: the experimental-validity layer, governance-graph typing, runtime-graph reconstruction, transcript capture and selection, dispatch plan and enforcement, and the provenance fixes from the pre-merge review. Merged after an independent pre-merge review returned "do not merge" with four P1s. All four are fixed and revert-verified: - The compiler taught reviewers to emit "VETO:" while the detector, after an earlier fix, accepted only "[VETO]". Every checks-and-balances regime had a veto power no agent could exercise, and 472 tests stayed green because the mechanism tests hand-write the correct marker instead of taking it from a real compiled prompt. - Mechanisms had no provenance or role authorization: a user-role tool_result containing "[VETO]" could SIGKILL a match, and "[EDICT]" could grant vetoImmunity to any untrusted content. - Dispatch and participation were inferred from forgeable text: printing "[โ zhongshu]" without calling any subagent produced enforcement_passed and participation_observed. - `baseline --dest` could overwrite a source regime or escape the repository; reproduced by destroying regimes/china/tang in one command. Also corrected a published number: 14 of 30 arms never delegated, not 13. The figure appeared in both READMEs and the E1-control results while the same paragraph said 7/15 per side, which sums to 14. CI on the merge commit: 502 backend + 19 frontend + 57 regimes + end-to-end smoke, on Node 20 and 22. Standing limits, unchanged and documented in README ยง9.5โยง9.7: rankings are over declared topologies; enforcement proves the coordinator called the offices a topology names, not that any typed edge executed; and no claim that governance topology affects multi-agent performance is supported yet, nor the opposite.
LeoLin990405
added a commit
that referenced
this pull request
Aug 7, 2026
P1 โ skill outcome stamping crossed tournaments.
skillsForMatch matched on the last six characters of the match id. A
tournament child id is `${tournamentId}__${regime with / as -}`, so every
china/tang child ever run ends in "a-tang": stamping one tournament
overwrote outcome_score/rank/total/of on skills learned in unrelated
earlier tournaments, teaching the corpus that a losing pattern had won.
Attribution now reads the full match id from the provenance banner
writeSkillFile already writes. The old test used a made-up match-id shape
(`T-1__china-tang-abc123`) and stayed green against the broken path; the
fixtures now use the real shape, and a new test pins that stamping one
tournament leaves another's skills untouched.
P2 โ controls leaked the source wiring for non-mermaid diagrams.
rewriteMermaid only matched ```mermaid fences. china/ming draws its
dual-track chart as ASCII in a plain fence, so the Ming flat control
declared zero edges in topology.json while still showing the model
็ๅธ โ ๅ
้/ๅธ็คผ็ โ ๅ
ญ้จ. The model reads IDENTITY.md, not topology.json,
so that control isolated nothing. Any fenced diagram is now replaced, and
the control's Decision Flow says explicitly that the historical procedure
described above it is background and the flow below governs.
P2 โ the constraint migration did not migrate.
CREATE UNIQUE INDEX raises on a table that already violates it, and the
catch logged and continued, leaving INSERT OR IGNORE nothing to ignore
against. It now collapses pre-existing duplicates (earliest row wins,
matching OR IGNORE semantics) and retries, and reports loudly if
uniqueness still cannot be enforced. It also drops the superseded
table-wide ux_episodic_memory_mre index, which would otherwise make a
plain INSERT of a match's second veto fail. Both paths tested.
P2 โ judge events hung off a span that was never opened.
judgeSpanId was generated but no event carried it as its own span_id, so
runtime-graph reconstructed the whole judging step as orphans. A
judging_start event now opens the span. Judge-event assertions are scoped
to scoring events; the integration test additionally checks end to end
that every pass parents to the opener.
P2 โ the diff could not read control regimes.
loadDeclaredTopology hardcoded china|global, excluding _baseline/* and
_ablated/* โ the control arms this comparison exists to evaluate. It now
uses the same contract as civ-memory.mjs::validateRegime.
P3 โ dropped the shangshu "router" label.
regime-to-cc still compiles it into a model-backed subagent with a prompt,
while the schema defines router as making no model call and producing no
content. No regime now carries a kind annotation, and a test asserts it:
the typing mechanism stays, the unearned labels do not.
CI: 427 backend + 14 frontend + 57 regimes + smoke, green.
LeoLin990405
added a commit
that referenced
this pull request
Aug 7, 2026
Pre-merge review of PR #32. Replacing planPrompt with the literal string "noop" โ so the coordinator is asked nothing at all โ left all 472 tests green. buildPlanPrompt's non-inducement wording was asserted in isolation while nothing checked that the wording reached the model. That hole sits on B's entire measurement. Under it the plan call could ask nothing, or ask a leading question, and E2 would still collect numbers labelled 'voluntary adoption' and generated from neither. The run-v5 wiring test now asserts against the captured plan-phase invocation: the adoption question is present, the non-inducement sentences are present, the office roster is present, and no explicit demand to use all offices is. Both failure modes revert-verified โ cutting the prompt reddens it, and rewriting the wording into 'You must call all available offices' reddens it twice. Seventh recurrence of unit-tested-but-unwired in this repo; the check that found it is now part of the pre-merge routine.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacks R7 (experimental validity) and R8 (graph engineering foundations + hardening) onto main.
Why this branch exists
R7 started from a smoke run that scored
china/tang10 and its random control 2.5. The control's transcript turned out to be the model asking the operator how to configure the experiment: the generator had replaced every office with "Baseline Agent N" and cutSOUL.mdto a 571-byte disclaimer, so the run varied persona and topology at once. The number was confident and meaningless. Most of what follows is aimed at making that class of error visible rather than plausible.R7 โ experimental validity
SOUL.mdis copied verbatim, office ids/labels/duties are preserved, and only the decision-flow prose and the mermaid diagram are rewritten. Leaving the source diagram in place made a control assert two contradictory wirings and follow whichever it read last.topology.regimeis derived from the destination directory (mirroring the validator's own rule);--deststages a control at an exact runnable path;validateRegimeaccepts a single leading underscore so_baseline/*can actually run as a civ โ previously a control could be generated and validated but never used.biasReportwith per-provider variance, same-family vs cross-family gap, and position effect.CURRENT_TIMESTAMPis second-granular).R8 โ graph engineering foundations + hardening
agent/gate/checkpoint/router), optional and defaulting toagent, plusgate_count/checkpoint_countkept deliberately separate fromchecks_cycles. No current regime declares a deterministic gate and the tests assert that: the Chancellery and the Directorate of Ceremonial are LLM reviewers, and the genuinely deterministic[VETO]sniffer lives inengine/mechanisms/veto.mjsattached to an agent node.civagent runtime-graph <matchId> --diffreconstructs the actual call graph from the event span tree and compares it to the declaration. Its main result is a negative one: declared nodes are offices, runtime actors are regimes, and the id spaces do not overlap, so per-office edge verification is blocked on instrumentation rather than merely unimplemented.unexercised_ratioisnullin that case instead of the1.000it would otherwise report for every regime in every match.episodic_memoryis a partial index scoped tomatch_endโ a table-wideUNIQUE(match_id, regime, event_type)would have discarded the secondveto_triggeredevent of a match, which for this project is the most damaging row to lose.examples/+npm run smokeโ a zero-API-key end-to-end path (validate โ metrics โ control โ validate) that was verified red as well as green.Verification
npm run cigreen on the integrated branch: 421 backend + 14 frontend + 57 regimes + smoke.Every regression test in this branch was revert-verified โ undo the fix, watch it go red, restore. That discipline caught four tests that passed for the wrong reason, including one that asserted duplicate row-stacking as correct behaviour and had locked the defect in permanently.
Known limits, stated in the README rather than hidden