feat(test): gate contract claims against the built binary (#907)

[drawmeanelephant]

Agent Completion Report

• Status: complete

• Branch and Worktree:

  • Branch: feat/contract-claims-gate
  • Worktree: ./ (primary checkout)

• Commit and PR:

  • Commit: 300c1245 (head; 791e1a0e is the gate, 300c1245 the fragment rename)
  • Target PR / Branch: main (this PR is feat(test): gate contract claims against the built binary (#907) #997)

• Linked Issues (auto-close convention):

  Refs Cooklang name-termination: contract's {-adjacency guard is not enforced (@salt into the {bowl} misparses) #907

  This PR does not close the issue. It records the divergence as a
  known-divergence claim and makes it fail the moment the upstream fix lands;
  the defect itself is still open and its fix lives in Oliver.

• Changed Files:

  • test/contract-claims.txt — the claim registry (six records)
  • test/contract-claims.checks.sh — the named black-box checks
  • scripts/test-contract-claims.sh — the gate runner
  • build.zig — test-contract-claims step, wired into test
  • test/README.md — registry documentation and the three failure conditions
  • docs/changelog.d/997-contract-claims-gate.md — changelog fragment

• Preserved Unrelated Files:

  • Nothing pre-existing was dirty when this work started. The stash@{0} entry
    belonging to fix/issue-close-declared-intent and the
    t3code/cli-reference-current branch are untouched. Local main was not
    used or moved. The separate rendered-search title fix is on its own branch
    and is not included here; this branch is verified green without it.

• Implementation Summary:

  • A contract can state a load-bearing rule that nothing executes, and the
    implementation then diverges silently. The Cooklang adapter contract says
    the { closing a multiword name must touch the name, and spells out the
    exact failure if it does not — an unrelated braced word later in a sentence
    being absorbed into the name and the prose between them deleted. The pinned
    parser scans to the first { on the line and had never enforced it.
  • test/contract-claims.txt pairs a verbatim quote from a normative contract
    with a black-box check against the installed binary. Three failure
    conditions: the quote no longer appears in the contract (prose drift), an
    enforced claim is violated, or a recorded known-divergence no longer
    reproduces.
  • The third condition is the ratchet that was missing: a known-divergence
    passes while the bug is present, so the upstream fix fails the gate and
    forces the record to be reclassified instead of being left stale. Neither
    direction can rot quietly.
  • Ships six records: five enforced (legacy parentEntry / parent_entry
    rejection, EPARENTMISSING for an absent parent, EUSAGE exit 2, pure I/O
    exit 3, and the version pin reusing scripts/test-version-pin.sh verbatim
    via check: script:<path>), plus the Cooklang divergence.
  • Checks drive the installed binary and never re-implement compiler logic: a
    check that re-derived what the compiler does would pass while the compiler
    was wrong. The parentEntry check asserts the absence of EPARENTMISSING,
    which is what makes it a test of "no silent alias" rather than of "this
    corpus fails".
  • --audit lists contracts with no quoted tether, framed as a worklist: 52 of
    57 today. Untethered is not untested — publication-claims.md, scanner.md,
    watch-mode.md and others carry dedicated test steps of their own. An
    earlier draft of that output said "no executable claim", which overclaimed,
    and was corrected before landing.

• Known Gaps:

  • The registry holds six of the normative rules; the audit list is the
    remaining work, not a completed survey.
  • rendered-search.md is deliberately left untethered. Its title rule is not
    reachable black-box — the compiler CLI exposes no search flag — and tethering
    it would have required either a nested build or asserting structure and
    calling it behavior. Its own unit test still guards it.
  • The gate adds roughly a second of process spawns per run (six records, each
    one or two boris invocations). Acceptable now; a registry of dozens of
    records would want batching.

• Exact Commands Run:

  1. zig build test
  2. zig build test-contract-claims
  3. bash scripts/test-contract-claims.sh --audit
  4. zig fmt --check build.zig
  5. git diff --check origin/main...feat/contract-claims-gate

• Exact Gate Results:

  • zig build test: pass — exit 0 on this branch alone, with the new
    contract-claims step running inside it and src/search_index.zig still at
    its original revision
  • zig build test-contract-claims: pass — 5 enforced, 1 known divergence(s), 6 record(s) total
  • bash scripts/test-contract-claims.sh --audit: pass — 52 of 57 normative contracts have no record in this registry
  • Prose drift proved to bite: changing one contract word (exit 2 → exit 9
    in the EUSAGE row of docs/contracts/diagnostics.md) made the gate exit 1,
    name cli.usage-error-exit-2, and print the stale quote; the other two
    records in that file stayed green. Restored, git diff clean.
  • Enforced divergence proved to bite: making one check assert an exit code
    the binary does not produce made the gate exit 1 with
    the implementation does not honor this contract claim, naming the record,
    contract, and check. Restored.
  • zig fmt --check build.zig: pass
  • git diff --check origin/main...feat/contract-claims-gate: clean

• Determinism Result:

  • Deterministic. The gate reads fixed registry and contract bytes and runs
    fixed corpora in a per-run scratch tree under .zig-cache/contract-claims,
    which is removed on exit. No content enumeration order is relied on; the
    audit walk sorts by filename through the shell glob.

• Generated Artifacts:

  • .zig-cache/contract-claims/ scratch trees only (gitignored, removed on
    exit). No dist/, rag/, or source-rag/ output was committed.

• Blockers and Next Card:

  • Blockers: None
  • Next Card: Tether the highest-risk untethered contracts next — html-output.md,
    validation.md, scanner.md — and, separately, land the Cooklang adjacency
    guard upstream so the recorded divergence can be flipped to enforced.

[itoqa]

Commit: 300c124: 12 test cases ran, 2 failed ❌, 9 passed ✅, 1 additional finding ⚠️.

Summary

The run covers normal build and contract-validation flows, version and artifact consistency, expected failure handling, audit behavior, and adversarial registry cases such as missing or duplicate rules, along with a recipe-parsing edge case. Core happy paths pass, but the safeguards around validating the validation rules themselves are not fully reliable.

Merge with caution — the PR introduces medium-severity weaknesses in the validation gate that can misclassify unavailable checks and accept duplicate coverage, reducing confidence that CI will detect incomplete or misleading contract coverage. The unrelated recipe-parsing issue is a flag for later rather than a merge driver.

Tests run by Ito

Result	Severity	Type	Description
❌		General	The check could not run the recipe probe, but the gate presented that execution failure as proof that the Cooklang problem was resolved. A failed probe must not be treated as a successful resolution.
❌		Rev	The gate accepted two identical claim records and counted both of them as coverage.
✅	—	General	The aggregate command ran the installed artifact and the contract gate, and it kept the existing checks in the build. A retained documentation-link check still failed in every run, and the controlled failure was correctly reported instead of being hidden.
✅	—	General	Removing the version-check script makes the contract gate fail with a clear error, and restoring the script makes the full check pass again.
✅	—	General	The aggregate build rebuilt and installed the Boris binary before the contract gate ran. The command later stopped on 31 broken documentation links, which is separate from the ordering this test checks.
✅	—	Build	The focused build step installed Boris before running the contract checks, and all six checks completed successfully.
✅	—	Frontmatter	The compiler accepted the valid parent case and correctly rejected legacy parent keys, missing parents, unknown options, and missing roots with the expected diagnostics and exit codes.
✅	—	Gate	The contract check validated all six registry records, including the quoted contract text, required sections, and executable checks. Five enforced checks passed, and the documented Cooklang difference was still reproduced as expected.
✅	—	Rev	Audit mode completed successfully after all six registered checks passed. It listed 52 of 57 normative contracts without a quoted registry check as an informational worklist and exited with status 0.
✅	—	Test	The aggregate build ran the new contract gate and retained the existing test dependencies. It ended with a separate documentation-link failure that was already present, not with a failure in the added gate or build wiring.
✅	—	Version	The version check passed for supported queries and all tested artifact sets. A tampered manifest id was rejected, and the documented version pin matched the compiler.
⚠️		Cooklang	The ingredient is returned with the name salt into the and the original amount bowl. The space before {bowl} should end the ingredient name rather than allowing a later braced word to be absorbed into it.

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Cooklang names absorb later braces

• Severity: Medium
• Description: The ingredient is returned with the name salt into the and the original amount bowl. The space before {bowl} should end the ingredient name rather than allowing a later braced word to be absorbed into it.
• Impact: Recipes using this spacing can show incorrect ingredient data and lose words from a cooking step. Authors can work around it by putting the opening brace directly after the ingredient name.
• Steps to Reproduce:
  1. Create a Cooklang recipe containing Add @salt into the {bowl} and stir..
  2. Run the recipe-scale command for that recipe with factor 2 and Cooklang mode enabled.
  3. Inspect the ingredient object in the JSON output.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: The normative contract at docs/contracts/cooklang-compatibility.md:91-102 requires a multiword Cooklang name to close with a { that touches the name, and explains that @salt into the {bowl} must not absorb bowl into the name or delete the intervening prose. The recorded black-box check in test/contract-claims.checks.sh:160-168 creates exactly that input and invokes the installed Boris binary through recipe-scale; at lines 179-194 it treats name: salt into the and original: bowl as the reproduced divergence. The captured output matches both assertions. Boris does not implement the parser locally: src/cooklang_seam.zig:1-11 states that the .cook body is delegated to oliver.cooklang.parse, so the defect is in the pinned upstream parser path rather than in the PR's newly added gate. The smallest practical fix is to update the pinned Oliver parser (or repin to a version containing its adjacency guard), then change the known-divergence record to assert the correct parse; the contract-claim gate will make that transition fail until the record is updated.

Evidence Package

• text/COOKLANG-1/COOKLANG-1-cooklang-divergence.txt — 2 KB

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

Unavailable recipe probe is reported as resolved

What failed: The check could not run the recipe probe, but the gate presented that execution failure as proof that the Cooklang problem was resolved. A failed probe must not be treated as a successful resolution.

Impact · Steps · Stub / mock · Analysis · Why this is likely a bug

• Severity: Medium
• Impact: CI can report that the known Cooklang problem is fixed when the recipe probe did not run. This can mislead maintainers and block or misdirect release checks until the result is reviewed.
• Steps to Reproduce:
  1. Make the recipe-scale command unavailable and run the contract-claim gate with the Cooklang record still marked as a known divergence.
  2. Observe that the probe reports it exited with an error, then the gate prints that the divergence no longer reproduces and exits with a failed record.
  3. Restore the command and run the same check again; it reports the known misparse still reproducing.
• Stub / mock content: The run used a disposable unavailable-oracle setup for the first probe, then restored the command for comparison. No application mocks or route stubs were used.
• Code Analysis: The production repository path is the newly added shell contract gate. In test/contract-claims.checks.sh, contract_check_cooklang_name_termination_adjacency() runs the installed binary at lines 164-172, captures its exit status, and returns nonzero at lines 174-177 when recipe-scale cannot execute. In scripts/test-contract-claims.sh, finish_record() stores that nonzero result as check_rc=1 at lines 167-201, then the known-divergence branch at lines 215-224 treats every nonzero check_rc as the divergence no longer reproducing and emits the resolution instructions. There is no distinction between an observable contract-violating result and an unavailable oracle. The targeted fix is to propagate an execution-error outcome separately and only enter the resolution branch when the probe completed and its output contradicted the known divergence.
• Why this is likely a bug: The recorded unavailable run explicitly says the divergence could not be probed, while the same restored run reports the documented misparse still reproducing. The source confirms that the check returns failure for inability to execute, and the gate then assigns that failure the semantic meaning of a resolved divergence. That is a false regression result in the new test gate, not a setup-only failure: it can make CI claim that an upstream parser defect disappeared when no successful parser observation was made. A small result-state distinction in the check/gate is sufficient; no parser rewrite is needed.

Relevant code

test/contract-claims.checks.sh:164-177

contract_check_cooklang_name_termination_adjacency() {
  ...
  out="$("$BORIS" recipe-scale --input "$corpus/in" --id adjacency --factor 2 --cooklang 2>&1)"
  rc=$?
  ...
  if [[ "$rc" -ne 0 ]]; then
    cc_detail "recipe-scale exited $rc, so the divergence could not be probed:"
    ...
    return 1
  fi

scripts/test-contract-claims.sh:203-224

known-divergence)
  if [[ "$check_rc" -eq 0 ]]; then
    warn "$r_id (known divergence $r_issue — still reproduces)"
    ...
  else
    failmsg "$r_id (known-divergence) — the divergence no longer reproduces"
    ...
    failed=$((failed + 1))
  fi

Evidence Package

• text/BF-FAIL-3/BF-FAIL-3-unavailable-oracle-classification.txt — 3 KB

Copy prompt for an agent

Ito QA identified the following failure during automated PR testing. Please investigate and propose a fix.

**Medium severity — Unavailable recipe probe is reported as resolved**

**What failed:** The check could not run the recipe probe, but the gate presented that execution failure as proof that the Cooklang problem was resolved. A failed probe must not be treated as a successful resolution.

- **Impact:** CI can report that the known Cooklang problem is fixed when the recipe probe did not run. This can mislead maintainers and block or misdirect release checks until the result is reviewed.
- **Steps to reproduce:**
  1. Make the recipe-scale command unavailable and run the contract-claim gate with the Cooklang record still marked as a known divergence.
  2. Observe that the probe reports it exited with an error, then the gate prints that the divergence no longer reproduces and exits with a failed record.
  3. Restore the command and run the same check again; it reports the known misparse still reproducing.
- **Stub / mock content:** The run used a disposable unavailable-oracle setup for the first probe, then restored the command for comparison. No application mocks or route stubs were used.
- **Code analysis:** The production repository path is the newly added shell contract gate. In test/contract-claims.checks.sh, contract_check_cooklang_name_termination_adjacency() runs the installed binary at lines 164-172, captures its exit status, and returns nonzero at lines 174-177 when recipe-scale cannot execute. In scripts/test-contract-claims.sh, finish_record() stores that nonzero result as check_rc=1 at lines 167-201, then the known-divergence branch at lines 215-224 treats every nonzero check_rc as the divergence no longer reproducing and emits the resolution instructions. There is no distinction between an observable contract-violating result and an unavailable oracle. The targeted fix is to propagate an execution-error outcome separately and only enter the resolution branch when the probe completed and its output contradicted the known divergence.
- **Why this is likely a bug:** The recorded unavailable run explicitly says the divergence could not be probed, while the same restored run reports the documented misparse still reproducing. The source confirms that the check returns failure for inability to execute, and the gate then assigns that failure the semantic meaning of a resolved divergence. That is a false regression result in the new test gate, not a setup-only failure: it can make CI claim that an upstream parser defect disappeared when no successful parser observation was made. A small result-state distinction in the check/gate is sufficient; no parser rewrite is needed.

**Relevant code:**

`test/contract-claims.checks.sh:164-177`

~~~bash
contract_check_cooklang_name_termination_adjacency() {
  ...
  out="$("$BORIS" recipe-scale --input "$corpus/in" --id adjacency --factor 2 --cooklang 2>&1)"
  rc=$?
  ...
  if [[ "$rc" -ne 0 ]]; then
    cc_detail "recipe-scale exited $rc, so the divergence could not be probed:"
    ...
    return 1
  fi
~~~

`scripts/test-contract-claims.sh:203-224`

~~~bash
known-divergence)
  if [[ "$check_rc" -eq 0 ]]; then
    warn "$r_id (known divergence $r_issue — still reproduces)"
    ...
  else
    failmsg "$r_id (known-divergence) — the divergence no longer reproduces"
    ...
    failed=$((failed + 1))
  fi
~~~

[itoqa]

Duplicate registry entries pass the gate

What failed: The gate accepted two identical claim records and counted both of them as coverage.

Impact · Steps · Stub / mock · Analysis · Why this is likely a bug

• Severity: Medium
• Impact: A registry with duplicate claim IDs can pass the gate and count the same claim twice, hiding missing contract coverage from maintainers. This can let an incomplete contract set merge until the duplicate is removed.
• Steps to Reproduce:
  1. From the repository root, copy the complete frontmatter.parent-legacy-keys-rejected record in test/contract-claims.txt and append the copy as a second record with the same claim ID.
  2. Run bash scripts/test-contract-claims.sh with the installed Boris binary available.
  3. Check the output and exit status: the duplicate record runs twice, the summary reports seven total records, and the command exits 0 instead of reporting a duplicate-ID error.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: The PR-added runner initializes per-record fields and aggregate counters at scripts/test-contract-claims.sh:91-95, but it has no collection of previously seen claim IDs. In the parser at lines 232-253, every claim id: line calls finish_record for the preceding record and then assigns the new ID; the same ID is therefore treated as a new independent record. finish_record validates the fields, tethered quote, and executable check, but never compares r_id with earlier IDs. The exact duplicate mutation consequently reaches the normal success path and increments records/enforced again. The smallest fix is to keep a Bash-3.2-compatible list of accepted IDs (or check each new ID against a newline-delimited seen-ID string) and increment failed plus emit a duplicate-ID diagnostic before dispatching the duplicate record; then return a nonzero result for the registry.
• Why this is likely a bug: The registry format describes each claim ID as a stable address for one record, and the gate is specifically intended to prevent missing contract coverage from being hidden by bad registry data. The exact duplicate run demonstrates that this is not only a theoretical gap: the command exits successfully and reports the same claim twice. A focused uniqueness check in the newly added parser is sufficient; no broader redesign is needed.

Relevant code

scripts/test-contract-claims.sh:91-95

r_id=""; r_contract=""; r_section=""; r_status=""; r_issue=""; r_quote=""; r_check=""
records=0
enforced=0
divergent=0
failed=0

scripts/test-contract-claims.sh:232-253

while IFS= read -r line || [[ -n "$line" ]]; do
  case "$line" in
    'claim id: '*)
      finish_record
      r_id="${line#claim id: }"
      ;;
    ...
  esac
done < "$REGISTRY"
finish_record

Evidence Package

• text/REV-7/REV-7-duplicate-claim-evidence.txt — 3 KB

Copy prompt for an agent

Ito QA identified the following failure during automated PR testing. Please investigate and propose a fix.

**Medium severity — Duplicate registry entries pass the gate**

**What failed:** The gate accepted two identical claim records and counted both of them as coverage.

- **Impact:** A registry with duplicate claim IDs can pass the gate and count the same claim twice, hiding missing contract coverage from maintainers. This can let an incomplete contract set merge until the duplicate is removed.
- **Steps to reproduce:**
  1. From the repository root, copy the complete frontmatter.parent-legacy-keys-rejected record in test/contract-claims.txt and append the copy as a second record with the same claim ID.
  2. Run bash scripts/test-contract-claims.sh with the installed Boris binary available.
  3. Check the output and exit status: the duplicate record runs twice, the summary reports seven total records, and the command exits 0 instead of reporting a duplicate-ID error.
- **Stub / mock content:** No stubs, mocks, or bypasses were applied for this test in the recorded run.
- **Code analysis:** The PR-added runner initializes per-record fields and aggregate counters at scripts/test-contract-claims.sh:91-95, but it has no collection of previously seen claim IDs. In the parser at lines 232-253, every `claim id:` line calls finish_record for the preceding record and then assigns the new ID; the same ID is therefore treated as a new independent record. finish_record validates the fields, tethered quote, and executable check, but never compares r_id with earlier IDs. The exact duplicate mutation consequently reaches the normal success path and increments records/enforced again. The smallest fix is to keep a Bash-3.2-compatible list of accepted IDs (or check each new ID against a newline-delimited seen-ID string) and increment failed plus emit a duplicate-ID diagnostic before dispatching the duplicate record; then return a nonzero result for the registry.
- **Why this is likely a bug:** The registry format describes each claim ID as a stable address for one record, and the gate is specifically intended to prevent missing contract coverage from being hidden by bad registry data. The exact duplicate run demonstrates that this is not only a theoretical gap: the command exits successfully and reports the same claim twice. A focused uniqueness check in the newly added parser is sufficient; no broader redesign is needed.

**Relevant code:**

`scripts/test-contract-claims.sh:91-95`

~~~bash
r_id=""; r_contract=""; r_section=""; r_status=""; r_issue=""; r_quote=""; r_check=""
records=0
enforced=0
divergent=0
failed=0
~~~

`scripts/test-contract-claims.sh:232-253`

~~~bash
while IFS= read -r line || [[ -n "$line" ]]; do
  case "$line" in
    'claim id: '*)
      finish_record
      r_id="${line#claim id: }"
      ;;
    ...
  esac
done < "$REGISTRY"
finish_record
~~~