Skip to content

docs(verification): record what CI does not prove about harness adapters - #57

Open
sbracewell64 wants to merge 1 commit into
mainfrom
fm/cfvc-lane-a-residual-uncertainties
Open

docs(verification): record what CI does not prove about harness adapters#57
sbracewell64 wants to merge 1 commit into
mainfrom
fm/cfvc-lane-a-residual-uncertainties

Conversation

@sbracewell64

@sbracewell64 sbracewell64 commented Aug 7, 2026

Copy link
Copy Markdown
Owner

What this changes

One commit, one file: docs/verification/runtime-backends.md.

That file is the classified maintainer-verification record for runtime backends.
It already stated that the portable regression is CI-enforced while the real-harness drift guard is opt-in, but it did not state the consequence of that split.
This records the consequence, the dated measurement behind it, and bounded output showing how the guard accounts for harnesses it could not check.

Why

This resolves item 4 of the CFVC Lane A residual-uncertainty register: do the four uninstalled harness adapters (opencode, pi-signed, grok, kimi) still work?

Measured, not assumed:

  • No CI job installs any harness binary. .github/workflows/ci.yml installs ShellCheck, tasks-axi, Herdr, Treehouse, tmux, jq, python3, curl and tar, and nothing else.
  • All nine members of the live-harness-optin family are selected into CI lanes and execute on every run, but gate-skip on an opt-in environment variable and exit 0 without testing anything. Measured 2026-08-07: FM_TEST_SUMMARY total=9 failed=0 skipped_gate=9, every member logging expected_gate_skip=optin-env and gate_skip=true.
  • Only the Herdr lane passes --fail-on-gate-skip, so nothing converts those skips into failures.
  • Negative control, because a skip proved by absence proves nothing: with the switch on and the binary still absent the same tests fail loudly (not ok - opencode not found, exit 1). The exit-0 seen in CI is therefore a real gate skip, not a vacuous pass.

The consequence is that every per-harness row in that record rests on the dated operator runs alone and is never refreshed by CI.

Check state, disclosed

This PR is not green. 12 of 14 checks pass; 2 fail, and neither failure is caused by this change.

1. Behavior portable serial 2 — pre-existing red on the fork trunk.

The failing assertion is not ok - first inheritance transaction never reached its blocked write, in tests/fm-remote-secondmate-lifecycle-e2e.test.sh.

Evidence that this branch did not cause it:

  • git diff --stat ed376cf HEAD is exactly one file, docs/verification/runtime-backends.md. Every script under bin/ and every file under tests/, including the failing test, is byte-identical to the base.
  • The last ten CI runs on this fork's main all conclude failure, including ed376cf itself, which is this branch's base.
  • The assertion reproduces locally on this branch with the identical message, which given the byte-identity above is the base's failure, not a new one.

This is a separate defect on the fork trunk and needs its own work item. It is not in this change's scope.

2. PR must be raised via no-mistakes — structural consequence of the venue.

That check requires the PR body to carry the git push no-mistakes signature and errors with This PR was not raised through no-mistakes.

This PR was opened by hand because the pipeline raised its PR against upstream kunchenguid/firstmate:main, which is the wrong venue for a branch based on the fork trunk. That upstream PR (#1888) has been closed. When it was measured, its head 2494ccce had 0 check-runs and 0 workflow runs and was mergeable=false, mergeable_state=dirty.

Against this fork the same head is mergeable=true and runs 14 real checks. The venue was the problem; the branch was never unmergeable.

On the pipeline's own outcome. The no-mistakes run for this branch (01KZE20T4Y027D307KKG6MZKTT) reports outcome: passed, but its ci step was skipped, so that outcome carries no CI evidence of its own. Read the checks on this PR directly rather than inferring green from it. An empty or skipped check set is not a pass, which is the rule this repository already encodes in ed376cf and, as it happens, the subject of this change.

The rebase step was also skipped, deliberately. This branch is based on the fork trunk, which is 46 commits ahead of upstream. Rebasing would replay that fork landing queue into contribution history, which is not wanted.

Test and gate evidence

What ran, and its result:

  • review completed.
  • test completed. No test command is configured for this repository, so the pipeline agent validated the change by reproducing the measurements the new documentation records, including running the negative control.
  • document completed with 0 unresolved items and no agent changes to commit.
  • lint passed: bin/fm-lint.sh, ShellCheck 0.11.0 against the pinned 0.11.0.
  • bin/fm-doc-audience-check.sh passes locally: ok surfaces=72 local_links=213.
  • In CI on this PR: Lint shell scripts, Test coverage guard, Repo invariants, Windows launcher bridge, Stock macOS Bash snapshot compatibility, Behavior tests (Herdr), Behavior portable parallel 1 and 2, and Behavior portable serial 1, 3 and 4 all pass.

The change touches no shell script, so the lint result reflects an unchanged script surface.

Base and scope

  • Base: main on this fork, at ed376cf.
  • Head: fm/cfvc-lane-a-residual-uncertainties at 2494ccce, ahead 1, behind 0.
  • Diff: one commit, one file.

What was deliberately not done

  • Recorded versions in the existing 2026-08-03 table were not updated, even though this machine measurably carries pi 0.81.1 against a recorded 0.82.0 and claude 2.1.224 against a recorded 2.1.220. Those rows are version-scoped to the operator run that produced them and are not false. Rewriting them without re-verifying the whole table on a machine carrying all seven harnesses would be fabrication.
  • No live end-to-end test was added for pi-signed or kimi, which lack one while opencode and grok each have one. That is a real gap, and it is recorded as one. Writing the test requires the binary to prove it against, and the coding guidelines forbid landing a harness-dependent check that was never run against the real harness. No opencode, pi-signed, grok or kimi binary exists on this machine.
  • The policy is not restated, only the evidence. That an absent harness must be reported rather than passed over is owned by .agents/skills/firstmate-coding-guidelines/SKILL.md. Duplicating it here would violate the one-owner rule.
  • No AGENTS.md change. Nothing here is needed by every session on every turn.

The other three uncertainties

Items 2, 3 and 5 of the same register were also resolved in this task and correctly produce no change to this repository:

  • Item 2, the stale ownership catalog. Verified read-only on the real platform clone: byte-current, PASS. The defect had already been fixed by platform commit 5f39690 on 2026-08-03, three days before it was reported. Negative control: deleting exactly the two relevant rows in a scratch export makes the same verifier FAIL, exit 1.
  • Item 3, the fork/upstream bin/ divergence. Measured: 48 of 49 modified files differ in executable lines, 1 is comment-only. It decomposes into a small set of upstream feature families the fork has not taken plus fork-original work upstream has not taken, not 48 independent semantic differences. This is input to the separate reconciliation work, and the fork-versus-upstream posture is private fleet state rather than shared tracked material, so it belongs in the task report and not in tracked docs.
  • Item 5, run_pm10_replication.py. It is the repro command for a completed, fenced evidence record and should be neither certified nor retired. The real defect is platform-side: the ownership catalog derives is_entry_point from a filename suffix while that same source file argues at length against name-based derivation, misclassifying seven modules. That is platform code, outside this repository.

Resolving CFVC Lane A residual uncertainty 4: whether the four
uninstalled adapters (opencode, pi-signed, grok, kimi) still work.

Measured: no CI job installs a harness binary, so all nine members of
the live-harness-optin family execute on every CI run and gate-skip on
an opt-in environment variable, exiting 0 without testing anything
(FM_TEST_SUMMARY total=9 failed=0 skipped_gate=9, 2026-08-07). With the
switch on and the binary absent they fail loudly instead, so the exit-0
seen in CI is a real gate skip rather than a vacuous pass.

The record stated the CI-versus-opt-in split without stating its
consequence: every per-harness row rests on the dated operator runs
alone. It now says so, and carries bounded output from a Linux run
carrying three of the seven, showing how the guard accounts for the
adapters it could not check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant