From aa33995199ebc158c0b900fdd93101bfc49ba8d3 Mon Sep 17 00:00:00 2001 From: Dana Burks Date: Sun, 6 Sep 2026 21:39:17 -0700 Subject: [PATCH 1/2] [bench] docs: add CHANGELOG.md; docstrings describe the code as it is Docstrings argued with an imagined judge and cited ledger receipts by hash; comments written to persuade the Oracle rot faster than comments written for a maintainer, and there was no changelog, so the history had nowhere else to live. CHANGELOG.md, new, covers 1.0.0 (2026-04-21), 1.1.0 (2026-07-12), 2.0.0 (2026-07-24), and 2.1.0 as unreleased, reconstructed from the tags and merge history; every 2.1.0 entry names something that has merged. Docstrings and comments across the source now describe current behavior. The chain module no longer narrates the array freeze, LedgerReadError states its invariant rather than the old bug, the relocation paragraph in resolve_ledger_path with its genesis hash is gone (CLAUDE.md keeps that record), the hook drops its ledger-entry citation and describes the fallback as it is, the three stage normalizers drop the dated receipts that motivated each repair and say what happens if the drift is left unrepaired, the sanitation module states the published-copy rule as a property rather than a story, the migration module and command describe clone states, and the snapshot module explains why a deep copy will not do. One date remains in source, at two sites in retire.py, both operational identifiers of the frozen 2026-07-24 anchor: the fixed value of the anchor's tool string, and the archive-path shape the auditor must keep reading beside the shape new retirements write. The other two uses were narration and are now stated as rules. Audit across all source, tests excluded: no ledger hash citations, no history phrasing, dates only at those two sites. No behavior changed. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0122bFBUvdtXT245QFoYDkMf --- CHANGELOG.md | 152 +++++++++++++++++++++++++++++++++++++++++ cli/commands.py | 4 +- hooks/pre-tool-use.py | 32 ++++----- ledger/chain.py | 34 ++++----- ledger/migrate.py | 8 +-- ledger/retire.py | 33 +++++---- ledger/sanitize.py | 26 +++---- pipeline/challenger.py | 18 ++--- pipeline/defender.py | 35 +++++----- pipeline/oracle.py | 7 +- pipeline/snapshot.py | 10 +-- utils/diff.py | 2 +- 12 files changed, 249 insertions(+), 112 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..c290cf9 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,152 @@ +# Changelog + +All notable changes to Bench. The format follows Keep a Changelog, and the +project follows semantic versioning. Dates are the merge dates of the release +tags. Every change listed here landed through Bench's own governance pipeline +and carries a receipt in the operational ledger. + +## [2.1.0] - unreleased + +The theme is making the claim survive a skeptic: cheaper verdicts, a real +installer, honest numbers, and tests for the properties that used to live in +comments. + +### Added + +- `pip install` gives a `bench` command; `bench install --project PATH` writes + the hook settings with an absolute path, appends `/.bench/` to the project's + gitignore, and sets the commit guard; `bench uninstall` reverses exactly what + install wrote, from a receipt it keeps at `.bench/install.json`. +- Per-stage wall time recorded beside token counts in every entry; the + dashboard shows median and p90 latency per week and the README quotes the + ledger's own median tokens and seconds per edit. +- Prompt caching of the constitution and repository context on the anthropic + provider; the claude_code provider lifts the constitution into its system + prompt file so the CLI reads it back from cache. Cache reads are priced at + the cached rate in `cli stats` and the viewer. +- A recorded model override per stage: `BENCH_CHALLENGER_MODEL`, + `BENCH_DEFENDER_MODEL`, and `BENCH_ORACLE_MODEL`, with the model actually + used written into each stage's record and filterable on the dashboard. +- A derived tip cache beside `entries/` so an append reads and validates only + the entries it links to. Never authoritative: the auditor ignores it, and a + stale, missing, or doubtful cache falls back to the full scan and rebuilds. +- An append lock so two sessions on one machine cannot fork the chain, held by + retirement across its whole mutation window. Appends wait as long as the + holder lives; retirement's own acquisition is bounded and refuses loudly. +- A ruff and strict mypy gate in CI over every source package and the hook. +- Tests that run the hook as a real subprocess and pin its host contract: + exact allow and deny JSON on stdout, exit code zero always. +- Tests that a prompt-injected diff cannot move the verdict: offline placement + proofs for every stage and provider, and an opt-in live run that expects a + VETO. +- A per-project constitution layer: a governed project's `bench.json` may add + `P-` constraints and raise core severities, never remove or redefine a core + constraint; `cli constitution` prints the merged result and its sources. +- Chain retirement under C-008's single bounded exception, with archive + verification before removal, an anchor entry opening the successor, and + `cli audit-retirement`. +- Published-copy sanitation records and their audit, for the case where a + chain's content must be removed from copies that retirement cannot reach. +- A dashboard in the viewer: weekly veto and pipeline-error rates, verdicts by + C-007 scope, constraint citations, token cost, and seconds per stage, all + computed by the same helpers `cli stats` uses. +- A migration command for clones that predate the private ledger location. + +### Changed + +- The operational ledger lives at `/.bench/` for every governed + project, Bench included, and is never a tracked artifact: an entry records + the full diff body of the change it governs. The legacy array segment is + frozen and new entries are written one per file, so two branches that both + appended merge as a union and the next governed edit reconciles the fork. +- Constitution v7 splits each constraint into the rule the models read and + commentary they do not; prompts carry the rule only. +- Cosmetic drift in judge output (an empty remediation on PASS, a blank + advisory, a severity written as WARNING, a finding index written as a + string) is repaired and recorded on the entry instead of failing closed. A + missing or unknown verdict, or a citation outside the constitution, still + fails closed. +- The subprocess bypass is bound to a per-call nonce the provider records on + disk; a bare `BENCH_SUBPROCESS=1` or a guessed token is governed normally. +- Every subprocess call carries a timeout, enforced by a test that scans the + source tree. +- The CLI parses arguments with argparse; `--help` lists every command and its + flags, and a bad flag is a usage error rather than silence. +- Diffs are bounded by a character budget as well as a line budget, so a few + very long lines cannot carry an unbounded payload to the models. +- The Oracle's per-stage timeout in this repo's settings is 300 seconds, so a + slow judge on a constitutionally heavy diff is not mistaken for a strict one. +- Path normalization has one implementation in `utils.diff`, one deliberate + fallback copy in the hook pinned equal to it by a test, and the dead + path-traversal placeholder is gone. +- The viewer file is created owner-only from its first byte on Windows as well + as POSIX. +- README statements that had drifted from the code were corrected and pinned + by tests. +- Docstrings and comments describe what the code does now. Release history + lives in this file; the one date that remains in source names the frozen + anchor entry whose shape the retirement auditor must keep reading. + +## [2.0.0] - 2026-07-24 + +### Added + +- Global governance: the hook can be registered in the user's settings and + govern every project on the machine, with verdicts routed to the ledger of + the project being governed. +- `SECURITY.md` and a documented governance boundary: files written through + Bash or MCP tools never reach the hook and carry no verdict. + +### Changed + +- Governance fails closed. A pipeline error, a judge that cannot be reached, or + a response that cannot be validated denies the change instead of allowing + it, and the ledger entry records the error as a pipeline error rather than + a ruling. +- Stage models moved to Sonnet 5 for the Challenger and Defender and Opus 4.8 + for the Oracle, with `utils/api.py` the single source of truth for model ids. +- C-004 type safety tightened to the precision the constitution states. + +## [1.1.0] - 2026-07-12 + +### Added + +- The claude_code provider: stages run through the local `claude` CLI in + headless mode on the user's subscription, with the stage system prompt kept + at system priority over the untrusted diff and a reentrancy guard for the + judge subprocess. +- A test suite and CI workflow across supported Python versions. +- Centralized governance logging. +- `utils/stats.py` as the single source of truth for ledger statistics, used + by `cli stats` and the viewer alike. + +### Changed + +- The hook emits the documented `BENCH VETO [C-XXX]` reason format and surfaces + Oracle advisories on PASS. +- Paths resolve against the repository root derived from the source file + location, not the working directory, so in-repo edits from a subdirectory + are governed as in-repo. +- The openai SDK is a soft dependency, installed only for the OpenRouter + provider; the anthropic SDK has a pinned lower bound. + +### Fixed + +- Response parsing is guarded, silent catches log, and verification is + anchored to the ledger meta pin. +- The viewer's temporary file is no longer removed before the browser loads it. +- Hardening against the findings of a security audit of the pipeline. + +## [1.0.0] - 2026-04-21 + +### Added + +- The governance pipeline: a PreToolUse hook intercepts Write, Edit, and + MultiEdit, and a Challenger, a Defender, and an Oracle rule on the change + before it lands. The Oracle's PASS or VETO is binding. +- The constitution, `bench.json`, with constraints C-001 through C-008 and a + version field. +- The hash-chained, append-only ledger: every verdict is an entry whose hash + covers its fields and links to the previous entry. +- The CLI: `verify`, `ledger`, `stats`, `constitution`, and `viewer`, a + self-contained HTML browser for the ledger. diff --git a/cli/commands.py b/cli/commands.py index 92e5cea..a54a7e2 100644 --- a/cli/commands.py +++ b/cli/commands.py @@ -817,8 +817,8 @@ def cmd_migrate_ledger() -> int: """Populate this clone's private chain from the pre-migration location. Only needed once, and only by a clone that existed before the ledger - became private. Checking out that switch makes git delete the formerly - tracked chain under ``ledger/``, and nothing can repopulate ``.bench/`` + became private. Checking out that switch makes git delete the chain such + a clone tracks under ``ledger/``, and nothing can repopulate ``.bench/`` from git because it is ignored by design. Without this the clone would resolve to an empty chain and silently open a fresh GENESIS. """ diff --git a/hooks/pre-tool-use.py b/hooks/pre-tool-use.py index d19a2f6..594760e 100644 --- a/hooks/pre-tool-use.py +++ b/hooks/pre-tool-use.py @@ -43,10 +43,10 @@ traceback.print_exc(file=sys.stderr) run_governance_pipeline = None # type: ignore[assignment] -# utils.diff (ledger entry 8bc4a3671a13, verdict PASS) provides the -# hardened extractor: binary detection, truncation with governance-critical -# line preservation, and change_type labeling. Imported in its own try -# so the fallback path below can engage independently of pipeline import. +# utils.diff provides the hardened extractor: binary detection, truncation +# with governance-critical line preservation, and change_type labeling. +# Imported in its own try so the fallback path below can engage +# independently of pipeline import. try: from utils.diff import build_diff_info as _build_diff_info_hardened except Exception as _diff_e: # pragma: no cover: degrades to the inline fallback below @@ -91,20 +91,16 @@ def extract_diff_info(tool_name: str, tool_input: dict[str, Any]) -> dict[str, A governance on them so the pipeline can decide what to do, but we don't fabricate fields. - Global governance behavior (differs from prior fallback): when a - file path resolves outside the Bench repo root (_REPO_ROOT), the - fallback no longer returns the sentinel ``[PATH_TRAVERSAL_BLOCKED]``. - Instead it delegates to ``_fallback_normalize_to_cwd`` (defined above - in this module) which normalizes the path relative to CWD (the - governed project's root). This enables governance of edits in - external projects without over-blocking. A ``_path_normalized_external`` - flag is set on the returned dict so pipeline stages (Challenger, - Defender, Oracle) can see that the path originated outside the - Bench repo and was normalized via CWD heuristics. - - C-005 test coverage: both CWD-normalization branches (cross-drive - ValueError and escapes-repo-root) are covered by - TestFallbackExternalNormalization in tests/test_hook.py. + A file path that resolves outside the Bench repo root (``_REPO_ROOT``) + is normalized by ``_fallback_normalize_path`` relative to the working + directory, which Claude Code sets to the governed project's root, and + the returned dict carries ``_path_normalized_external`` so the pipeline + stages can see that the path came from outside the repo. + + Test coverage, both in tests/test_hook.py: TestFallbackExternalNormalization + covers the cross-drive and escapes-repo-root branches, and + FallbackAgreesWithPrimaryTests pins the fallback equal to + ``utils.diff._normalize_path`` on the same inputs. """ if _build_diff_info_hardened is not None: return _build_diff_info_hardened(tool_name, tool_input) diff --git a/ledger/chain.py b/ledger/chain.py index 2f4c959..82acb15 100644 --- a/ledger/chain.py +++ b/ledger/chain.py @@ -11,14 +11,12 @@ beside it as a permanent pin on that segment's tip and count. New entries are written one per file to ``entries/.json``. -That split is what lets two branches record verdicts independently. A single -array rewritten in full on every append gave two branches divergent chains that -could not be merged — interleaving breaks the links, rebasing rewrites hashes — -so the array was frozen rather than migrated, because C-008 forbids moving or -rewriting an existing entry and a file that is never written cannot conflict. -``previous_hash`` accordingly holds a string (legacy, one parent) or a sorted -list, and an append names every current tip, so a fork left by a merge is -reconciled by the next governed edit. +The split is what lets two branches record verdicts independently. A file +that is never written cannot conflict, and an entry file named by its own +hash merges as a union, so two branches that both appended combine without +rewriting any entry, which C-008 forbids. ``previous_hash`` holds a string +(legacy, one parent) or a sorted list, and an append names every current +tip, so a fork left by a merge is reconciled by the next governed edit. Writes are atomic via ``os.replace`` on a same-directory temp file, so a crash mid-write cannot leave a half-written file on disk. @@ -92,10 +90,10 @@ class LedgerReadError(RuntimeError): """Raised when existing ledger content cannot be read or would be lost. - Fail-closed on the write path (C-001, C-008). A corrupt ledger previously - read as an empty chain, which made the next append restart from GENESIS and + Fail-closed on the write path (C-001, C-008). A corrupt ledger must not + read as an empty chain: the next append would restart from GENESIS and overwrite the damaged file. Refusing to proceed preserves the evidence - instead of destroying it; the caller in ``pipeline.runner`` already logs and + instead of destroying it; the caller in ``pipeline.runner`` logs and returns the verdict without a receipt rather than blocking the developer. """ @@ -134,14 +132,6 @@ def resolve_ledger_path() -> str: therefore never a tracked artifact of the repository it governs, and dogfooding does not earn an exception to that. - Bench's existing chain was relocated to that path before this branch - removed the special case: the legacy segment, ``ledger-meta.json``, and - every entry file were copied unchanged, and ``verify_chain`` reports VALID - at the new location with genesis hash 4e98fb41 unchanged. No entry was - modified, reordered, or removed, so continuity is preserved and no - retirement was triggered, which matters because a storage-location change - is not a permitted C-008 retirement trigger. - Claude Code invokes hooks with the governed project as the working directory. That is the same assumption ``utils.diff`` already relies on when normalizing paths that fall outside the Bench repo. @@ -486,9 +476,9 @@ def _entry_file_names(entries_dir: Path) -> list[str]: a tip the cache never links to; ``_FOLD_CASE`` carries pathlib's rule. Names are not checked for being regular files, which would cost a stat per name: ``glob`` does not check either, so a stray directory with the - suffix is treated exactly as before. Its appearance changes the listing, - the scan refuses it as unreadable, and the auditor reports it. It is - never a tip, so the cache never links to it. + suffix is handled by the scan, not the cache. Its appearance changes the + listing, the scan refuses it as unreadable, and the auditor reports it. + It is never a tip, so the cache never links to it. """ if not entries_dir.is_dir(): return [] diff --git a/ledger/migrate.py b/ledger/migrate.py index 8823197..caa83cb 100644 --- a/ledger/migrate.py +++ b/ledger/migrate.py @@ -1,9 +1,9 @@ """One-time upgrade path for clones created before the ledger went private. -Bench used to keep its own chain as tracked files under ``ledger/``. That -exemption is gone: every governed project, Bench included, now keeps its -chain in a gitignored ``.bench/``. The commit that made the switch untracks -the old paths, so pulling it makes git delete ``ledger/bench-ledger.json``, +Such a clone holds Bench's own chain as tracked files under ``ledger/``, +while every governed project, Bench included, keeps its chain in a +gitignored ``.bench/``. The commit that made the switch untracks the old +paths, so pulling it makes git delete ``ledger/bench-ledger.json``, ``ledger/ledger-meta.json``, and ``ledger/entries/`` from the working tree. Nothing in git can repopulate ``.bench/``, because ``.bench/`` is ignored by diff --git a/ledger/retire.py b/ledger/retire.py index c946f64..4cebc3d 100644 --- a/ledger/retire.py +++ b/ledger/retire.py @@ -54,7 +54,11 @@ from utils.project import project_root ANCHOR_TOOL: str = "ChainRetirement" -"""``change.tool`` on an anchor entry, matching the 2026-07-24 reference.""" +"""``change.tool`` on an anchor entry. + +Fixed to the value the frozen 2026-07-24 anchor carries, so every anchor in +a chain is recognised by one string. +""" _LOCK_WAIT_SECONDS: float = 30.0 """How long a retirement waits for the chain's append lock before refusing. @@ -327,8 +331,9 @@ def build_anchor_summary( ) -> dict: """Assemble the anchor's ``change.diff_summary``. - The field set is the one the conforming 2026-07-24 anchor established, which - is in turn C-008(c)'s enumeration plus the bookkeeping an auditor needs. + The field set is C-008(c)'s enumeration plus the bookkeeping an auditor + needs, and it is the set ``validate_anchor_summary`` checks on every + anchor, the frozen one already in the chain included. ``predecessor_first_entry`` and ``predecessor_last_entry`` hold timestamp *values*, not hashes or indices, because that is what C-008(c) names. """ @@ -540,10 +545,9 @@ def _project_relative(ledger_path: str) -> str: relative: str = os.path.relpath( os.path.realpath(ledger_path), str(project_root()) ) - # Forward slashes, matching the 2026-07-24 reference anchor's - # "ledger/bench-ledger.json". The ledger is read on whatever platform - # audits it, so the recorded evidence should not carry the separator of - # the machine that happened to write it. + # Forward slashes, as every anchor records the path. The ledger is + # read on whatever platform audits it, so the recorded evidence should + # not carry the separator of the machine that happened to write it. return relative.replace(os.sep, "/") except (OSError, ValueError) as exc: # Windows: a different drive from the project has no relative form. @@ -886,13 +890,14 @@ def audit_retirement( if archive_ledger_path: resolved: str = archive_ledger_path elif recorded.suffix == ".json": - # The 2026-07-24 retirement archived a single JSON array and recorded - # that file directly. Retirements written by execute_retirement record - # the archive *directory*, because a chain is now up to three segments. - # Both are legitimate values for C-008(c)'s "verbatim archive path", so - # the auditor reads either rather than only the shape it writes. - # Keyed on the suffix, not on is_file, so a missing archive is reported - # as a failed verification rather than as a confusing path. + # An anchor's archive_path is one of two shapes. The frozen 2026-07-24 + # anchor records a single JSON array file directly; every anchor + # execute_retirement writes records the archive *directory*, because a + # chain is up to three segments. Both are legitimate values for + # C-008(c)'s "verbatim archive path", so the auditor reads either + # rather than only the shape it writes. Keyed on the suffix, not on + # is_file, so a missing archive is reported as a failed verification + # rather than as a confusing path. resolved = str(recorded) else: resolved = str(recorded / Path(resolve_ledger_path()).name) diff --git a/ledger/sanitize.py b/ledger/sanitize.py index 525f4e6..b7fc229 100644 --- a/ledger/sanitize.py +++ b/ledger/sanitize.py @@ -6,19 +6,16 @@ it for writing, and is invalid by definition if the chain fails ``verify_chain`` afterwards. What it concerns is a different object. -Before the ledger became private, Bench committed its chain to a public -repository, and git retained those commits. Those git objects are a -*publication* of ledger data, not the ledger: nothing reads them, nothing -appends to them, and `resolve_ledger_path()` has never pointed at them. -Deleting a published copy is therefore outside C-008's immutability scope -in the same way that deleting a printed copy of a document is not an edit -to the document. - -Retirement cannot reach them. The chain retired on 2026-07-24 was retired -because it held unpublishable third-party content, and its anchor records -that 264 of its entries had already been published. Retirement archived the -chain and opened a successor; the published copies stayed exactly where -they were. +A published copy is a different object from the ledger. Git objects in a +repository that tracked a chain are a *publication* of ledger data, not the +ledger: nothing reads them, nothing appends to them, and +`resolve_ledger_path()` does not point at them. Deleting a published copy is +therefore outside C-008's immutability scope in the same way that deleting a +printed copy of a document is not an edit to the document. + +Retirement cannot reach them. It archives a chain and opens a successor, and +any published copies of the retired chain stay exactly where they were, +which is the case sanitation exists for. Being outside C-008's scope is not a licence to act unaccountably, which is the whole reason this module exists. A sanitation is recorded in the live @@ -26,8 +23,7 @@ was removed and confirm the chain it spared is intact. Whole files only. Sanitation never edits, reorders, or partially rewrites an -entry in any copy. That operation is what C-008 forbids without exception, -and Bench correctly vetoed it when it was first attempted here. +entry in any copy. That operation is what C-008 forbids without exception. Two deliberate differences from retirement's evidence requirements: diff --git a/pipeline/challenger.py b/pipeline/challenger.py index 858d98b..e301cef 100644 --- a/pipeline/challenger.py +++ b/pipeline/challenger.py @@ -211,19 +211,19 @@ def _build_user_content(diff_info: dict) -> str: def _normalize_challenger_response(response: dict[str, Any]) -> list[str]: """Repair one cosmetic drift in a Challenger response in place; return notes. - A finding severity of "WARNING" becomes "CONCERN". The operational - ledger recorded that shape as INVALID_CHALLENGER_RESPONSE on 2026-08-22 - and 2026-09-05 (UTC): each a fail-closed VETO recorded as a pipeline - error, not a ruling, on an edit the Challenger had examined and - reported findings for. The finding reaches the Oracle with its - constraint, location, evidence, and reasoning untouched. + A finding severity of "WARNING" becomes "CONCERN". WARNING is the word + C-005 uses for its own severity, echoed back where the schema wants + VIOLATION, CONCERN, or OBSERVATION; left as written it fails validation, + and an edit the Challenger examined and reported findings for is denied + as a pipeline error rather than ruled on. The finding reaches the Oracle + with its constraint, location, evidence, and reasoning untouched. Nothing else is repaired. A finding without evidence, a location, a constraint, or reasoning, any other severity, an unknown status, and a non-list findings are left for _validate_challenger_response to fail - closed exactly as before: the normalizer never synthesizes a field the - schema requires. run_challenger records the returned notes on the - result as ``_normalized`` so the ledger entry shows the repair. + closed: the normalizer never synthesizes a field the schema requires. + run_challenger records the returned notes on the result as + ``_normalized`` so the ledger entry shows the repair. tests/test_challenger.py NormalizeChallengerResponseTests covers the repair, the untouched clean response, and each fail-closed case end to end through run_challenger. diff --git a/pipeline/defender.py b/pipeline/defender.py index 1495a73..5ed0804 100644 --- a/pipeline/defender.py +++ b/pipeline/defender.py @@ -211,16 +211,15 @@ def _build_user_content(diff_info: dict, challenger_result: dict) -> str: ) -# Cosmetic drift the operational ledger has recorded in otherwise sound -# Defender responses (INVALID_DEFENDER_RESPONSE on 2026-07-31, twice, and -# 2026-08-04, twice), each a fail-closed VETO recorded as a pipeline error, -# not a ruling. The positions here are the two the system prompt above -# names as the mistakes to avoid, and each is a plain word for agreeing -# with a finding, which is what CONCEDE means. No alias maps to REBUT or -# MITIGATE, and a word that could mean disagreement is never guessed at: -# CONFIRM_CLEAR, which the ledger also recorded, is the schema's top-level -# status for a clear assessment and inside a rebuttal can mean the code is -# clear of the finding, so it stays fail-closed. +# Cosmetic drift in otherwise sound Defender responses: a rebuttal position +# written as a plain word for agreeing with a finding, which is what CONCEDE +# means. Left as written, the response fails validation and the edit is +# denied as a pipeline error rather than ruled on. The two positions here +# are the two the system prompt above names as the mistakes to avoid. No +# alias maps to REBUT or MITIGATE, and a word that could mean disagreement +# is never guessed at: CONFIRM_CLEAR is the schema's top-level status for a +# clear assessment and inside a rebuttal can mean the code is clear of the +# finding, so it stays fail-closed. _POSITION_ALIASES: dict[str, str] = { "CONFIRM": "CONCEDE", "AGREE": "CONCEDE", @@ -235,16 +234,16 @@ def _normalize_defender_response( Two repairs, neither changing the argument made: a ``finding_index`` given as a digit string becomes the integer, but only when that integer names one of the Challenger's findings, so a rebuttal to a finding that - does not exist fails closed as it always has; and a position in - _POSITION_ALIASES becomes CONCEDE. Any other position, a non-numeric, - negative, or out-of-range index, and a missing argument or summary are - left for _validate_defender_response to fail closed, exactly as before. + does not exist still fails closed; and a position in _POSITION_ALIASES + becomes CONCEDE. Any other position, a non-numeric, negative, or + out-of-range index, and a missing argument or summary are left for + _validate_defender_response to fail closed. This does not weaken enforcement. A rebuttal's position is an input to the Oracle, which reads the argument text and rules on the merits; a - response the validator rejected for spelling CONCEDE as CONFIRM was - never adjudicated at all, and the VETO it produced was a pipeline - error. Mapping an agreement word to CONCEDE sends the Oracle the same + response the validator rejects for spelling CONCEDE as CONFIRM is never + adjudicated at all, and the VETO it produces is a pipeline error. + Mapping an agreement word to CONCEDE sends the Oracle the same argument with the position the schema meant, whether the response also carries a genuine REBUT or not. @@ -276,7 +275,7 @@ def _normalize_defender_response( # superscripts and circled digits that int() rejects, int() refuses # very long digit strings, and an exception here would skip the # PIPELINE_ERROR receipt the validator writes. An index past the - # findings list stays a string and fails closed as it always has. + # findings list stays a string and fails closed. if ( isinstance(finding_index, str) and finding_index.strip().isascii() diff --git a/pipeline/oracle.py b/pipeline/oracle.py index 1d17409..161aed4 100644 --- a/pipeline/oracle.py +++ b/pipeline/oracle.py @@ -245,10 +245,9 @@ def _build_user_content( ) -# Cosmetic drift in otherwise sound Oracle responses. A missing confidence -# is the one the ledger has recorded (every invalid Oracle response so far); -# the remediation placeholders and blank advisories are the drift the -# schema invites, since it asks for null on PASS and an optional list. +# Cosmetic drift in otherwise sound Oracle responses: a missing confidence, +# and the remediation placeholders and blank advisories the schema invites, +# since it asks for null on PASS and an optional list. _DEFAULT_CONFIDENCE: str = "LOW" _REMEDIATION_PLACEHOLDERS: frozenset[str] = frozenset( {"", "null", "none", "n/a", "na", "not applicable"} diff --git a/pipeline/snapshot.py b/pipeline/snapshot.py index 244aa63..9cb31f3 100644 --- a/pipeline/snapshot.py +++ b/pipeline/snapshot.py @@ -3,11 +3,11 @@ Each stage repairs a short list of cosmetic drift in its judge's JSON before validating it (the _normalize_*_response functions), and on a validation failure the ledger records what the judge actually wrote, not the repaired -copy. That record used to be a deep copy, which recursed into every field -and raised RecursionError on an unknown field nested a few hundred levels -deep, turning an otherwise valid response into a pipeline error even -though the validators tolerate unknown fields and the ledger can serialize -far deeper than a deep copy can walk. +copy. A deep copy would not do: it recurses into every field and raises +RecursionError on an unknown field nested a few hundred levels deep, turning +an otherwise valid response into a pipeline error even though the validators +tolerate unknown fields and the ledger can serialize far deeper than a deep +copy can walk. The normalizers only ever write to the top level of the response and to the dicts inside one list (findings, rebuttals). So the snapshot copies diff --git a/utils/diff.py b/utils/diff.py index 2470764..7c5e0c3 100644 --- a/utils/diff.py +++ b/utils/diff.py @@ -149,7 +149,7 @@ def build_diff_info(tool_name: str, tool_input: dict) -> dict[str, Any]: * Write -> change_type="create", addition-only formatted_diff * Edit -> change_type="modify", old/new strings (possibly truncated) * MultiEdit -> change_type="multi_modify", edits list (possibly truncated) - * anything else -> empty dict (preserves prior hook behavior) + * anything else -> empty dict (no fields to extract) Any embedded binary content anywhere in the payload collapses the whole dict to a metadata-only representation — raw bytes never From f54c97783a854ce0755d844a4278234996a00af8 Mon Sep 17 00:00:00 2001 From: Dana Burks Date: Sun, 6 Sep 2026 21:51:59 -0700 Subject: [PATCH 2/2] [bench] docs: qualify four changelog claims; drop the stale branch-merge property Codex review on #64, four findings, all wording that overstated. The receipts claim is qualified: verdicts have been recorded since partway through 1.0.0, and the earliest work predates that. The installer line carries the existing-hook caveat the README already states: the guard is installed as the pre-commit hook only when the project has none. The ledger line says what an entry holds, the change up to the diff budget with binary and out-of-project content reduced to metadata, rather than "the full diff body". And the branch-union claim is gone: the ledger is untracked, so a git merge never touches it, and a fork arises only when entry files reach one directory from more than one source. That property predates the private ledger, and this PR had restated it in the chain module docstring and the README carried it too; both now say that a chain with two tips, however it came to have them, is reconciled by the next governed edit. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0122bFBUvdtXT245QFoYDkMf --- CHANGELOG.md | 20 +++++++++++++------- README.md | 4 ++-- ledger/chain.py | 12 ++++++------ 3 files changed, 21 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c290cf9..2a0aa7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,8 +2,9 @@ All notable changes to Bench. The format follows Keep a Changelog, and the project follows semantic versioning. Dates are the merge dates of the release -tags. Every change listed here landed through Bench's own governance pipeline -and carries a receipt in the operational ledger. +tags. Since the runner began recording verdicts, partway through 1.0.0, every +change has landed through Bench's own governance pipeline and carries a +receipt in the operational ledger; the earliest 1.0.0 work predates that. ## [2.1.0] - unreleased @@ -15,8 +16,10 @@ comments. - `pip install` gives a `bench` command; `bench install --project PATH` writes the hook settings with an absolute path, appends `/.bench/` to the project's - gitignore, and sets the commit guard; `bench uninstall` reverses exactly what - install wrote, from a receipt it keeps at `.bench/install.json`. + gitignore, and installs the ledger commit guard as the project's pre-commit + hook when it has none, otherwise reporting the existing hook and how to + call the guard from it; `bench uninstall` reverses exactly what install + wrote, from a receipt it keeps at `.bench/install.json`. - Per-stage wall time recorded beside token counts in every entry; the dashboard shows median and p90 latency per week and the README quotes the ledger's own median tokens and seconds per edit. @@ -56,9 +59,12 @@ comments. - The operational ledger lives at `/.bench/` for every governed project, Bench included, and is never a tracked artifact: an entry records - the full diff body of the change it governs. The legacy array segment is - frozen and new entries are written one per file, so two branches that both - appended merge as a union and the next governed edit reconciles the fork. + the change it governs, up to the diff budget and with binary and + out-of-project content reduced to metadata, so a committed ledger publishes + source. The legacy array segment is frozen and new entries are written one + per file, each named by its hash, so no entry is ever rewritten; an append + names every current tip, so a chain that comes to have two tips is + reconciled by the next governed edit. - Constitution v7 splits each constraint into the rule the models read and commentary they do not; prompts carry the rule only. - Cosmetic drift in judge output (an empty remediation on PASS, a blank diff --git a/README.md b/README.md index d8e1a38..2bc2be3 100644 --- a/README.md +++ b/README.md @@ -357,13 +357,13 @@ The corollary is worth stating: keeping a ledger out of git also removes git as ### Ledger Format -The ledger is stored in two segments, and the split is what lets two branches record verdicts independently. +The ledger is stored in two segments: a frozen legacy array, and one file per new entry. `bench-ledger.json` is the **frozen** legacy segment: a hash-chained JSON array that every reader still reads and nothing writes. `ledger-meta.json` is frozen with it, as a permanent pin on that segment's tip hash and entry count. Every new entry is written to its own file at `/entries/.json`. That shape exists because the original single array was rewritten in full on every append. Two branches appending produced divergent chains with no valid resolution: interleaving breaks the hash links, and rebasing rewrites hashes, which C-008 forbids. The array was frozen rather than migrated for the same reason — C-008 permits no moving or rewriting of an existing entry, and a file that is never written can never conflict. -`previous_hash` therefore holds either a string (legacy entries, one parent) or a sorted list of parent hashes. An append names **every current tip**, so after a `git merge` leaves two heads, the next governed edit names both and the fork reconciles itself. There is no merge command: the reconciliation is an ordinary governed, hash-linked entry. +`previous_hash` therefore holds either a string (legacy entries, one parent) or a sorted list of parent hashes. An append names **every current tip**, so a chain that comes to have two heads, however it came to have them, is reconciled by the next governed edit, which names both. The ledger is untracked, so a `git merge` never touches it; a fork arises only when entry files reach one directory from more than one source. There is no merge command: the reconciliation is an ordinary governed, hash-linked entry. Beside `entries/` sits `tip-cache.json`, a derived record of the current tips. It lets an append read and validate only the entries it will link to instead of re-reading the whole chain, so what an append reads does not grow with the chain's length; only a directory listing still does, and that is two orders of magnitude cheaper than reading the chain. It is never authoritative: the cache is keyed to a digest of the entries listing, so any file added or removed outside the writer invalidates it, a missing or doubtful cache falls back to the full scan and is rebuilt from it, and `python -m cli verify` does not read it at all. Deleting it is always safe. diff --git a/ledger/chain.py b/ledger/chain.py index 82acb15..ad7da35 100644 --- a/ledger/chain.py +++ b/ledger/chain.py @@ -11,12 +11,12 @@ beside it as a permanent pin on that segment's tip and count. New entries are written one per file to ``entries/.json``. -The split is what lets two branches record verdicts independently. A file -that is never written cannot conflict, and an entry file named by its own -hash merges as a union, so two branches that both appended combine without -rewriting any entry, which C-008 forbids. ``previous_hash`` holds a string -(legacy, one parent) or a sorted list, and an append names every current -tip, so a fork left by a merge is reconciled by the next governed edit. +A file that is never written cannot conflict, and an entry file named by +its own hash is never rewritten, which is what C-008 forbids; a chain +assembled from more than one source combines as a union of files. +``previous_hash`` holds a string (legacy, one parent) or a sorted list, and +an append names every current tip, so a chain that comes to have two tips, +however it came to have them, is reconciled by the next governed edit. Writes are atomic via ``os.replace`` on a same-directory temp file, so a crash mid-write cannot leave a half-written file on disk.