diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..2a0aa7a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,158 @@ +# Changelog + +All notable changes to Bench. The format follows Keep a Changelog, and the +project follows semantic versioning. Dates are the merge dates of the release +tags. Since the runner began recording verdicts, partway through 1.0.0, every +change has landed through Bench's own governance pipeline and carries a +receipt in the operational ledger; the earliest 1.0.0 work predates that. + +## [2.1.0] - unreleased + +The theme is making the claim survive a skeptic: cheaper verdicts, a real +installer, honest numbers, and tests for the properties that used to live in +comments. + +### Added + +- `pip install` gives a `bench` command; `bench install --project PATH` writes + the hook settings with an absolute path, appends `/.bench/` to the project's + gitignore, and installs the ledger commit guard as the project's pre-commit + hook when it has none, otherwise reporting the existing hook and how to + call the guard from it; `bench uninstall` reverses exactly what install + wrote, from a receipt it keeps at `.bench/install.json`. +- Per-stage wall time recorded beside token counts in every entry; the + dashboard shows median and p90 latency per week and the README quotes the + ledger's own median tokens and seconds per edit. +- Prompt caching of the constitution and repository context on the anthropic + provider; the claude_code provider lifts the constitution into its system + prompt file so the CLI reads it back from cache. Cache reads are priced at + the cached rate in `cli stats` and the viewer. +- A recorded model override per stage: `BENCH_CHALLENGER_MODEL`, + `BENCH_DEFENDER_MODEL`, and `BENCH_ORACLE_MODEL`, with the model actually + used written into each stage's record and filterable on the dashboard. +- A derived tip cache beside `entries/` so an append reads and validates only + the entries it links to. Never authoritative: the auditor ignores it, and a + stale, missing, or doubtful cache falls back to the full scan and rebuilds. +- An append lock so two sessions on one machine cannot fork the chain, held by + retirement across its whole mutation window. Appends wait as long as the + holder lives; retirement's own acquisition is bounded and refuses loudly. +- A ruff and strict mypy gate in CI over every source package and the hook. +- Tests that run the hook as a real subprocess and pin its host contract: + exact allow and deny JSON on stdout, exit code zero always. +- Tests that a prompt-injected diff cannot move the verdict: offline placement + proofs for every stage and provider, and an opt-in live run that expects a + VETO. +- A per-project constitution layer: a governed project's `bench.json` may add + `P-` constraints and raise core severities, never remove or redefine a core + constraint; `cli constitution` prints the merged result and its sources. +- Chain retirement under C-008's single bounded exception, with archive + verification before removal, an anchor entry opening the successor, and + `cli audit-retirement`. +- Published-copy sanitation records and their audit, for the case where a + chain's content must be removed from copies that retirement cannot reach. +- A dashboard in the viewer: weekly veto and pipeline-error rates, verdicts by + C-007 scope, constraint citations, token cost, and seconds per stage, all + computed by the same helpers `cli stats` uses. +- A migration command for clones that predate the private ledger location. + +### Changed + +- The operational ledger lives at `/.bench/` for every governed + project, Bench included, and is never a tracked artifact: an entry records + the change it governs, up to the diff budget and with binary and + out-of-project content reduced to metadata, so a committed ledger publishes + source. The legacy array segment is frozen and new entries are written one + per file, each named by its hash, so no entry is ever rewritten; an append + names every current tip, so a chain that comes to have two tips is + reconciled by the next governed edit. +- Constitution v7 splits each constraint into the rule the models read and + commentary they do not; prompts carry the rule only. +- Cosmetic drift in judge output (an empty remediation on PASS, a blank + advisory, a severity written as WARNING, a finding index written as a + string) is repaired and recorded on the entry instead of failing closed. A + missing or unknown verdict, or a citation outside the constitution, still + fails closed. +- The subprocess bypass is bound to a per-call nonce the provider records on + disk; a bare `BENCH_SUBPROCESS=1` or a guessed token is governed normally. +- Every subprocess call carries a timeout, enforced by a test that scans the + source tree. +- The CLI parses arguments with argparse; `--help` lists every command and its + flags, and a bad flag is a usage error rather than silence. +- Diffs are bounded by a character budget as well as a line budget, so a few + very long lines cannot carry an unbounded payload to the models. +- The Oracle's per-stage timeout in this repo's settings is 300 seconds, so a + slow judge on a constitutionally heavy diff is not mistaken for a strict one. +- Path normalization has one implementation in `utils.diff`, one deliberate + fallback copy in the hook pinned equal to it by a test, and the dead + path-traversal placeholder is gone. +- The viewer file is created owner-only from its first byte on Windows as well + as POSIX. +- README statements that had drifted from the code were corrected and pinned + by tests. +- Docstrings and comments describe what the code does now. Release history + lives in this file; the one date that remains in source names the frozen + anchor entry whose shape the retirement auditor must keep reading. + +## [2.0.0] - 2026-07-24 + +### Added + +- Global governance: the hook can be registered in the user's settings and + govern every project on the machine, with verdicts routed to the ledger of + the project being governed. +- `SECURITY.md` and a documented governance boundary: files written through + Bash or MCP tools never reach the hook and carry no verdict. + +### Changed + +- Governance fails closed. A pipeline error, a judge that cannot be reached, or + a response that cannot be validated denies the change instead of allowing + it, and the ledger entry records the error as a pipeline error rather than + a ruling. +- Stage models moved to Sonnet 5 for the Challenger and Defender and Opus 4.8 + for the Oracle, with `utils/api.py` the single source of truth for model ids. +- C-004 type safety tightened to the precision the constitution states. + +## [1.1.0] - 2026-07-12 + +### Added + +- The claude_code provider: stages run through the local `claude` CLI in + headless mode on the user's subscription, with the stage system prompt kept + at system priority over the untrusted diff and a reentrancy guard for the + judge subprocess. +- A test suite and CI workflow across supported Python versions. +- Centralized governance logging. +- `utils/stats.py` as the single source of truth for ledger statistics, used + by `cli stats` and the viewer alike. + +### Changed + +- The hook emits the documented `BENCH VETO [C-XXX]` reason format and surfaces + Oracle advisories on PASS. +- Paths resolve against the repository root derived from the source file + location, not the working directory, so in-repo edits from a subdirectory + are governed as in-repo. +- The openai SDK is a soft dependency, installed only for the OpenRouter + provider; the anthropic SDK has a pinned lower bound. + +### Fixed + +- Response parsing is guarded, silent catches log, and verification is + anchored to the ledger meta pin. +- The viewer's temporary file is no longer removed before the browser loads it. +- Hardening against the findings of a security audit of the pipeline. + +## [1.0.0] - 2026-04-21 + +### Added + +- The governance pipeline: a PreToolUse hook intercepts Write, Edit, and + MultiEdit, and a Challenger, a Defender, and an Oracle rule on the change + before it lands. The Oracle's PASS or VETO is binding. +- The constitution, `bench.json`, with constraints C-001 through C-008 and a + version field. +- The hash-chained, append-only ledger: every verdict is an entry whose hash + covers its fields and links to the previous entry. +- The CLI: `verify`, `ledger`, `stats`, `constitution`, and `viewer`, a + self-contained HTML browser for the ledger. diff --git a/README.md b/README.md index d8e1a38..2bc2be3 100644 --- a/README.md +++ b/README.md @@ -357,13 +357,13 @@ The corollary is worth stating: keeping a ledger out of git also removes git as ### Ledger Format -The ledger is stored in two segments, and the split is what lets two branches record verdicts independently. +The ledger is stored in two segments: a frozen legacy array, and one file per new entry. `bench-ledger.json` is the **frozen** legacy segment: a hash-chained JSON array that every reader still reads and nothing writes. `ledger-meta.json` is frozen with it, as a permanent pin on that segment's tip hash and entry count. Every new entry is written to its own file at `/entries/.json`. That shape exists because the original single array was rewritten in full on every append. Two branches appending produced divergent chains with no valid resolution: interleaving breaks the hash links, and rebasing rewrites hashes, which C-008 forbids. The array was frozen rather than migrated for the same reason — C-008 permits no moving or rewriting of an existing entry, and a file that is never written can never conflict. -`previous_hash` therefore holds either a string (legacy entries, one parent) or a sorted list of parent hashes. An append names **every current tip**, so after a `git merge` leaves two heads, the next governed edit names both and the fork reconciles itself. There is no merge command: the reconciliation is an ordinary governed, hash-linked entry. +`previous_hash` therefore holds either a string (legacy entries, one parent) or a sorted list of parent hashes. An append names **every current tip**, so a chain that comes to have two heads, however it came to have them, is reconciled by the next governed edit, which names both. The ledger is untracked, so a `git merge` never touches it; a fork arises only when entry files reach one directory from more than one source. There is no merge command: the reconciliation is an ordinary governed, hash-linked entry. Beside `entries/` sits `tip-cache.json`, a derived record of the current tips. It lets an append read and validate only the entries it will link to instead of re-reading the whole chain, so what an append reads does not grow with the chain's length; only a directory listing still does, and that is two orders of magnitude cheaper than reading the chain. It is never authoritative: the cache is keyed to a digest of the entries listing, so any file added or removed outside the writer invalidates it, a missing or doubtful cache falls back to the full scan and is rebuilt from it, and `python -m cli verify` does not read it at all. Deleting it is always safe. diff --git a/cli/commands.py b/cli/commands.py index 92e5cea..a54a7e2 100644 --- a/cli/commands.py +++ b/cli/commands.py @@ -817,8 +817,8 @@ def cmd_migrate_ledger() -> int: """Populate this clone's private chain from the pre-migration location. Only needed once, and only by a clone that existed before the ledger - became private. Checking out that switch makes git delete the formerly - tracked chain under ``ledger/``, and nothing can repopulate ``.bench/`` + became private. Checking out that switch makes git delete the chain such + a clone tracks under ``ledger/``, and nothing can repopulate ``.bench/`` from git because it is ignored by design. Without this the clone would resolve to an empty chain and silently open a fresh GENESIS. """ diff --git a/hooks/pre-tool-use.py b/hooks/pre-tool-use.py index d19a2f6..594760e 100644 --- a/hooks/pre-tool-use.py +++ b/hooks/pre-tool-use.py @@ -43,10 +43,10 @@ traceback.print_exc(file=sys.stderr) run_governance_pipeline = None # type: ignore[assignment] -# utils.diff (ledger entry 8bc4a3671a13, verdict PASS) provides the -# hardened extractor: binary detection, truncation with governance-critical -# line preservation, and change_type labeling. Imported in its own try -# so the fallback path below can engage independently of pipeline import. +# utils.diff provides the hardened extractor: binary detection, truncation +# with governance-critical line preservation, and change_type labeling. +# Imported in its own try so the fallback path below can engage +# independently of pipeline import. try: from utils.diff import build_diff_info as _build_diff_info_hardened except Exception as _diff_e: # pragma: no cover: degrades to the inline fallback below @@ -91,20 +91,16 @@ def extract_diff_info(tool_name: str, tool_input: dict[str, Any]) -> dict[str, A governance on them so the pipeline can decide what to do, but we don't fabricate fields. - Global governance behavior (differs from prior fallback): when a - file path resolves outside the Bench repo root (_REPO_ROOT), the - fallback no longer returns the sentinel ``[PATH_TRAVERSAL_BLOCKED]``. - Instead it delegates to ``_fallback_normalize_to_cwd`` (defined above - in this module) which normalizes the path relative to CWD (the - governed project's root). This enables governance of edits in - external projects without over-blocking. A ``_path_normalized_external`` - flag is set on the returned dict so pipeline stages (Challenger, - Defender, Oracle) can see that the path originated outside the - Bench repo and was normalized via CWD heuristics. - - C-005 test coverage: both CWD-normalization branches (cross-drive - ValueError and escapes-repo-root) are covered by - TestFallbackExternalNormalization in tests/test_hook.py. + A file path that resolves outside the Bench repo root (``_REPO_ROOT``) + is normalized by ``_fallback_normalize_path`` relative to the working + directory, which Claude Code sets to the governed project's root, and + the returned dict carries ``_path_normalized_external`` so the pipeline + stages can see that the path came from outside the repo. + + Test coverage, both in tests/test_hook.py: TestFallbackExternalNormalization + covers the cross-drive and escapes-repo-root branches, and + FallbackAgreesWithPrimaryTests pins the fallback equal to + ``utils.diff._normalize_path`` on the same inputs. """ if _build_diff_info_hardened is not None: return _build_diff_info_hardened(tool_name, tool_input) diff --git a/ledger/chain.py b/ledger/chain.py index 2f4c959..ad7da35 100644 --- a/ledger/chain.py +++ b/ledger/chain.py @@ -11,14 +11,12 @@ beside it as a permanent pin on that segment's tip and count. New entries are written one per file to ``entries/.json``. -That split is what lets two branches record verdicts independently. A single -array rewritten in full on every append gave two branches divergent chains that -could not be merged — interleaving breaks the links, rebasing rewrites hashes — -so the array was frozen rather than migrated, because C-008 forbids moving or -rewriting an existing entry and a file that is never written cannot conflict. -``previous_hash`` accordingly holds a string (legacy, one parent) or a sorted -list, and an append names every current tip, so a fork left by a merge is -reconciled by the next governed edit. +A file that is never written cannot conflict, and an entry file named by +its own hash is never rewritten, which is what C-008 forbids; a chain +assembled from more than one source combines as a union of files. +``previous_hash`` holds a string (legacy, one parent) or a sorted list, and +an append names every current tip, so a chain that comes to have two tips, +however it came to have them, is reconciled by the next governed edit. Writes are atomic via ``os.replace`` on a same-directory temp file, so a crash mid-write cannot leave a half-written file on disk. @@ -92,10 +90,10 @@ class LedgerReadError(RuntimeError): """Raised when existing ledger content cannot be read or would be lost. - Fail-closed on the write path (C-001, C-008). A corrupt ledger previously - read as an empty chain, which made the next append restart from GENESIS and + Fail-closed on the write path (C-001, C-008). A corrupt ledger must not + read as an empty chain: the next append would restart from GENESIS and overwrite the damaged file. Refusing to proceed preserves the evidence - instead of destroying it; the caller in ``pipeline.runner`` already logs and + instead of destroying it; the caller in ``pipeline.runner`` logs and returns the verdict without a receipt rather than blocking the developer. """ @@ -134,14 +132,6 @@ def resolve_ledger_path() -> str: therefore never a tracked artifact of the repository it governs, and dogfooding does not earn an exception to that. - Bench's existing chain was relocated to that path before this branch - removed the special case: the legacy segment, ``ledger-meta.json``, and - every entry file were copied unchanged, and ``verify_chain`` reports VALID - at the new location with genesis hash 4e98fb41 unchanged. No entry was - modified, reordered, or removed, so continuity is preserved and no - retirement was triggered, which matters because a storage-location change - is not a permitted C-008 retirement trigger. - Claude Code invokes hooks with the governed project as the working directory. That is the same assumption ``utils.diff`` already relies on when normalizing paths that fall outside the Bench repo. @@ -486,9 +476,9 @@ def _entry_file_names(entries_dir: Path) -> list[str]: a tip the cache never links to; ``_FOLD_CASE`` carries pathlib's rule. Names are not checked for being regular files, which would cost a stat per name: ``glob`` does not check either, so a stray directory with the - suffix is treated exactly as before. Its appearance changes the listing, - the scan refuses it as unreadable, and the auditor reports it. It is - never a tip, so the cache never links to it. + suffix is handled by the scan, not the cache. Its appearance changes the + listing, the scan refuses it as unreadable, and the auditor reports it. + It is never a tip, so the cache never links to it. """ if not entries_dir.is_dir(): return [] diff --git a/ledger/migrate.py b/ledger/migrate.py index 8823197..caa83cb 100644 --- a/ledger/migrate.py +++ b/ledger/migrate.py @@ -1,9 +1,9 @@ """One-time upgrade path for clones created before the ledger went private. -Bench used to keep its own chain as tracked files under ``ledger/``. That -exemption is gone: every governed project, Bench included, now keeps its -chain in a gitignored ``.bench/``. The commit that made the switch untracks -the old paths, so pulling it makes git delete ``ledger/bench-ledger.json``, +Such a clone holds Bench's own chain as tracked files under ``ledger/``, +while every governed project, Bench included, keeps its chain in a +gitignored ``.bench/``. The commit that made the switch untracks the old +paths, so pulling it makes git delete ``ledger/bench-ledger.json``, ``ledger/ledger-meta.json``, and ``ledger/entries/`` from the working tree. Nothing in git can repopulate ``.bench/``, because ``.bench/`` is ignored by diff --git a/ledger/retire.py b/ledger/retire.py index c946f64..4cebc3d 100644 --- a/ledger/retire.py +++ b/ledger/retire.py @@ -54,7 +54,11 @@ from utils.project import project_root ANCHOR_TOOL: str = "ChainRetirement" -"""``change.tool`` on an anchor entry, matching the 2026-07-24 reference.""" +"""``change.tool`` on an anchor entry. + +Fixed to the value the frozen 2026-07-24 anchor carries, so every anchor in +a chain is recognised by one string. +""" _LOCK_WAIT_SECONDS: float = 30.0 """How long a retirement waits for the chain's append lock before refusing. @@ -327,8 +331,9 @@ def build_anchor_summary( ) -> dict: """Assemble the anchor's ``change.diff_summary``. - The field set is the one the conforming 2026-07-24 anchor established, which - is in turn C-008(c)'s enumeration plus the bookkeeping an auditor needs. + The field set is C-008(c)'s enumeration plus the bookkeeping an auditor + needs, and it is the set ``validate_anchor_summary`` checks on every + anchor, the frozen one already in the chain included. ``predecessor_first_entry`` and ``predecessor_last_entry`` hold timestamp *values*, not hashes or indices, because that is what C-008(c) names. """ @@ -540,10 +545,9 @@ def _project_relative(ledger_path: str) -> str: relative: str = os.path.relpath( os.path.realpath(ledger_path), str(project_root()) ) - # Forward slashes, matching the 2026-07-24 reference anchor's - # "ledger/bench-ledger.json". The ledger is read on whatever platform - # audits it, so the recorded evidence should not carry the separator of - # the machine that happened to write it. + # Forward slashes, as every anchor records the path. The ledger is + # read on whatever platform audits it, so the recorded evidence should + # not carry the separator of the machine that happened to write it. return relative.replace(os.sep, "/") except (OSError, ValueError) as exc: # Windows: a different drive from the project has no relative form. @@ -886,13 +890,14 @@ def audit_retirement( if archive_ledger_path: resolved: str = archive_ledger_path elif recorded.suffix == ".json": - # The 2026-07-24 retirement archived a single JSON array and recorded - # that file directly. Retirements written by execute_retirement record - # the archive *directory*, because a chain is now up to three segments. - # Both are legitimate values for C-008(c)'s "verbatim archive path", so - # the auditor reads either rather than only the shape it writes. - # Keyed on the suffix, not on is_file, so a missing archive is reported - # as a failed verification rather than as a confusing path. + # An anchor's archive_path is one of two shapes. The frozen 2026-07-24 + # anchor records a single JSON array file directly; every anchor + # execute_retirement writes records the archive *directory*, because a + # chain is up to three segments. Both are legitimate values for + # C-008(c)'s "verbatim archive path", so the auditor reads either + # rather than only the shape it writes. Keyed on the suffix, not on + # is_file, so a missing archive is reported as a failed verification + # rather than as a confusing path. resolved = str(recorded) else: resolved = str(recorded / Path(resolve_ledger_path()).name) diff --git a/ledger/sanitize.py b/ledger/sanitize.py index 525f4e6..b7fc229 100644 --- a/ledger/sanitize.py +++ b/ledger/sanitize.py @@ -6,19 +6,16 @@ it for writing, and is invalid by definition if the chain fails ``verify_chain`` afterwards. What it concerns is a different object. -Before the ledger became private, Bench committed its chain to a public -repository, and git retained those commits. Those git objects are a -*publication* of ledger data, not the ledger: nothing reads them, nothing -appends to them, and `resolve_ledger_path()` has never pointed at them. -Deleting a published copy is therefore outside C-008's immutability scope -in the same way that deleting a printed copy of a document is not an edit -to the document. - -Retirement cannot reach them. The chain retired on 2026-07-24 was retired -because it held unpublishable third-party content, and its anchor records -that 264 of its entries had already been published. Retirement archived the -chain and opened a successor; the published copies stayed exactly where -they were. +A published copy is a different object from the ledger. Git objects in a +repository that tracked a chain are a *publication* of ledger data, not the +ledger: nothing reads them, nothing appends to them, and +`resolve_ledger_path()` does not point at them. Deleting a published copy is +therefore outside C-008's immutability scope in the same way that deleting a +printed copy of a document is not an edit to the document. + +Retirement cannot reach them. It archives a chain and opens a successor, and +any published copies of the retired chain stay exactly where they were, +which is the case sanitation exists for. Being outside C-008's scope is not a licence to act unaccountably, which is the whole reason this module exists. A sanitation is recorded in the live @@ -26,8 +23,7 @@ was removed and confirm the chain it spared is intact. Whole files only. Sanitation never edits, reorders, or partially rewrites an -entry in any copy. That operation is what C-008 forbids without exception, -and Bench correctly vetoed it when it was first attempted here. +entry in any copy. That operation is what C-008 forbids without exception. Two deliberate differences from retirement's evidence requirements: diff --git a/pipeline/challenger.py b/pipeline/challenger.py index 858d98b..e301cef 100644 --- a/pipeline/challenger.py +++ b/pipeline/challenger.py @@ -211,19 +211,19 @@ def _build_user_content(diff_info: dict) -> str: def _normalize_challenger_response(response: dict[str, Any]) -> list[str]: """Repair one cosmetic drift in a Challenger response in place; return notes. - A finding severity of "WARNING" becomes "CONCERN". The operational - ledger recorded that shape as INVALID_CHALLENGER_RESPONSE on 2026-08-22 - and 2026-09-05 (UTC): each a fail-closed VETO recorded as a pipeline - error, not a ruling, on an edit the Challenger had examined and - reported findings for. The finding reaches the Oracle with its - constraint, location, evidence, and reasoning untouched. + A finding severity of "WARNING" becomes "CONCERN". WARNING is the word + C-005 uses for its own severity, echoed back where the schema wants + VIOLATION, CONCERN, or OBSERVATION; left as written it fails validation, + and an edit the Challenger examined and reported findings for is denied + as a pipeline error rather than ruled on. The finding reaches the Oracle + with its constraint, location, evidence, and reasoning untouched. Nothing else is repaired. A finding without evidence, a location, a constraint, or reasoning, any other severity, an unknown status, and a non-list findings are left for _validate_challenger_response to fail - closed exactly as before: the normalizer never synthesizes a field the - schema requires. run_challenger records the returned notes on the - result as ``_normalized`` so the ledger entry shows the repair. + closed: the normalizer never synthesizes a field the schema requires. + run_challenger records the returned notes on the result as + ``_normalized`` so the ledger entry shows the repair. tests/test_challenger.py NormalizeChallengerResponseTests covers the repair, the untouched clean response, and each fail-closed case end to end through run_challenger. diff --git a/pipeline/defender.py b/pipeline/defender.py index 1495a73..5ed0804 100644 --- a/pipeline/defender.py +++ b/pipeline/defender.py @@ -211,16 +211,15 @@ def _build_user_content(diff_info: dict, challenger_result: dict) -> str: ) -# Cosmetic drift the operational ledger has recorded in otherwise sound -# Defender responses (INVALID_DEFENDER_RESPONSE on 2026-07-31, twice, and -# 2026-08-04, twice), each a fail-closed VETO recorded as a pipeline error, -# not a ruling. The positions here are the two the system prompt above -# names as the mistakes to avoid, and each is a plain word for agreeing -# with a finding, which is what CONCEDE means. No alias maps to REBUT or -# MITIGATE, and a word that could mean disagreement is never guessed at: -# CONFIRM_CLEAR, which the ledger also recorded, is the schema's top-level -# status for a clear assessment and inside a rebuttal can mean the code is -# clear of the finding, so it stays fail-closed. +# Cosmetic drift in otherwise sound Defender responses: a rebuttal position +# written as a plain word for agreeing with a finding, which is what CONCEDE +# means. Left as written, the response fails validation and the edit is +# denied as a pipeline error rather than ruled on. The two positions here +# are the two the system prompt above names as the mistakes to avoid. No +# alias maps to REBUT or MITIGATE, and a word that could mean disagreement +# is never guessed at: CONFIRM_CLEAR is the schema's top-level status for a +# clear assessment and inside a rebuttal can mean the code is clear of the +# finding, so it stays fail-closed. _POSITION_ALIASES: dict[str, str] = { "CONFIRM": "CONCEDE", "AGREE": "CONCEDE", @@ -235,16 +234,16 @@ def _normalize_defender_response( Two repairs, neither changing the argument made: a ``finding_index`` given as a digit string becomes the integer, but only when that integer names one of the Challenger's findings, so a rebuttal to a finding that - does not exist fails closed as it always has; and a position in - _POSITION_ALIASES becomes CONCEDE. Any other position, a non-numeric, - negative, or out-of-range index, and a missing argument or summary are - left for _validate_defender_response to fail closed, exactly as before. + does not exist still fails closed; and a position in _POSITION_ALIASES + becomes CONCEDE. Any other position, a non-numeric, negative, or + out-of-range index, and a missing argument or summary are left for + _validate_defender_response to fail closed. This does not weaken enforcement. A rebuttal's position is an input to the Oracle, which reads the argument text and rules on the merits; a - response the validator rejected for spelling CONCEDE as CONFIRM was - never adjudicated at all, and the VETO it produced was a pipeline - error. Mapping an agreement word to CONCEDE sends the Oracle the same + response the validator rejects for spelling CONCEDE as CONFIRM is never + adjudicated at all, and the VETO it produces is a pipeline error. + Mapping an agreement word to CONCEDE sends the Oracle the same argument with the position the schema meant, whether the response also carries a genuine REBUT or not. @@ -276,7 +275,7 @@ def _normalize_defender_response( # superscripts and circled digits that int() rejects, int() refuses # very long digit strings, and an exception here would skip the # PIPELINE_ERROR receipt the validator writes. An index past the - # findings list stays a string and fails closed as it always has. + # findings list stays a string and fails closed. if ( isinstance(finding_index, str) and finding_index.strip().isascii() diff --git a/pipeline/oracle.py b/pipeline/oracle.py index 1d17409..161aed4 100644 --- a/pipeline/oracle.py +++ b/pipeline/oracle.py @@ -245,10 +245,9 @@ def _build_user_content( ) -# Cosmetic drift in otherwise sound Oracle responses. A missing confidence -# is the one the ledger has recorded (every invalid Oracle response so far); -# the remediation placeholders and blank advisories are the drift the -# schema invites, since it asks for null on PASS and an optional list. +# Cosmetic drift in otherwise sound Oracle responses: a missing confidence, +# and the remediation placeholders and blank advisories the schema invites, +# since it asks for null on PASS and an optional list. _DEFAULT_CONFIDENCE: str = "LOW" _REMEDIATION_PLACEHOLDERS: frozenset[str] = frozenset( {"", "null", "none", "n/a", "na", "not applicable"} diff --git a/pipeline/snapshot.py b/pipeline/snapshot.py index 244aa63..9cb31f3 100644 --- a/pipeline/snapshot.py +++ b/pipeline/snapshot.py @@ -3,11 +3,11 @@ Each stage repairs a short list of cosmetic drift in its judge's JSON before validating it (the _normalize_*_response functions), and on a validation failure the ledger records what the judge actually wrote, not the repaired -copy. That record used to be a deep copy, which recursed into every field -and raised RecursionError on an unknown field nested a few hundred levels -deep, turning an otherwise valid response into a pipeline error even -though the validators tolerate unknown fields and the ledger can serialize -far deeper than a deep copy can walk. +copy. A deep copy would not do: it recurses into every field and raises +RecursionError on an unknown field nested a few hundred levels deep, turning +an otherwise valid response into a pipeline error even though the validators +tolerate unknown fields and the ledger can serialize far deeper than a deep +copy can walk. The normalizers only ever write to the top level of the response and to the dicts inside one list (findings, rebuttals). So the snapshot copies diff --git a/utils/diff.py b/utils/diff.py index 2470764..7c5e0c3 100644 --- a/utils/diff.py +++ b/utils/diff.py @@ -149,7 +149,7 @@ def build_diff_info(tool_name: str, tool_input: dict) -> dict[str, Any]: * Write -> change_type="create", addition-only formatted_diff * Edit -> change_type="modify", old/new strings (possibly truncated) * MultiEdit -> change_type="multi_modify", edits list (possibly truncated) - * anything else -> empty dict (preserves prior hook behavior) + * anything else -> empty dict (no fields to extract) Any embedded binary content anywhere in the payload collapses the whole dict to a metadata-only representation — raw bytes never