docs(verification): record what CI does not prove about harness adapters - #57
Open
sbracewell64 wants to merge 1 commit into
Open
docs(verification): record what CI does not prove about harness adapters#57sbracewell64 wants to merge 1 commit into
sbracewell64 wants to merge 1 commit into
Conversation
Resolving CFVC Lane A residual uncertainty 4: whether the four uninstalled adapters (opencode, pi-signed, grok, kimi) still work. Measured: no CI job installs a harness binary, so all nine members of the live-harness-optin family execute on every CI run and gate-skip on an opt-in environment variable, exiting 0 without testing anything (FM_TEST_SUMMARY total=9 failed=0 skipped_gate=9, 2026-08-07). With the switch on and the binary absent they fail loudly instead, so the exit-0 seen in CI is a real gate skip rather than a vacuous pass. The record stated the CI-versus-opt-in split without stating its consequence: every per-harness row rests on the dated operator runs alone. It now says so, and carries bounded output from a Linux run carrying three of the seven, showing how the guard accounts for the adapters it could not check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
One commit, one file:
docs/verification/runtime-backends.md.That file is the classified maintainer-verification record for runtime backends.
It already stated that the portable regression is CI-enforced while the real-harness drift guard is opt-in, but it did not state the consequence of that split.
This records the consequence, the dated measurement behind it, and bounded output showing how the guard accounts for harnesses it could not check.
Why
This resolves item 4 of the CFVC Lane A residual-uncertainty register: do the four uninstalled harness adapters (
opencode,pi-signed,grok,kimi) still work?Measured, not assumed:
.github/workflows/ci.ymlinstalls ShellCheck,tasks-axi, Herdr, Treehouse, tmux, jq, python3, curl and tar, and nothing else.live-harness-optinfamily are selected into CI lanes and execute on every run, but gate-skip on an opt-in environment variable and exit 0 without testing anything. Measured 2026-08-07:FM_TEST_SUMMARY total=9 failed=0 skipped_gate=9, every member loggingexpected_gate_skip=optin-envandgate_skip=true.--fail-on-gate-skip, so nothing converts those skips into failures.not ok - opencode not found, exit 1). The exit-0 seen in CI is therefore a real gate skip, not a vacuous pass.The consequence is that every per-harness row in that record rests on the dated operator runs alone and is never refreshed by CI.
Check state, disclosed
This PR is not green. 12 of 14 checks pass; 2 fail, and neither failure is caused by this change.
1.
Behavior portable serial 2— pre-existing red on the fork trunk.The failing assertion is
not ok - first inheritance transaction never reached its blocked write, intests/fm-remote-secondmate-lifecycle-e2e.test.sh.Evidence that this branch did not cause it:
git diff --stat ed376cf HEADis exactly one file,docs/verification/runtime-backends.md. Every script underbin/and every file undertests/, including the failing test, is byte-identical to the base.mainall concludefailure, includinged376cfitself, which is this branch's base.This is a separate defect on the fork trunk and needs its own work item. It is not in this change's scope.
2.
PR must be raised via no-mistakes— structural consequence of the venue.That check requires the PR body to carry the
git push no-mistakessignature and errors withThis PR was not raised through no-mistakes.This PR was opened by hand because the pipeline raised its PR against upstream
kunchenguid/firstmate:main, which is the wrong venue for a branch based on the fork trunk. That upstream PR (#1888) has been closed. When it was measured, its head2494cccehad 0 check-runs and 0 workflow runs and wasmergeable=false, mergeable_state=dirty.Against this fork the same head is
mergeable=trueand runs 14 real checks. The venue was the problem; the branch was never unmergeable.On the pipeline's own outcome. The
no-mistakesrun for this branch (01KZE20T4Y027D307KKG6MZKTT) reportsoutcome: passed, but itscistep was skipped, so that outcome carries no CI evidence of its own. Read the checks on this PR directly rather than inferring green from it. An empty or skipped check set is not a pass, which is the rule this repository already encodes ined376cfand, as it happens, the subject of this change.The
rebasestep was also skipped, deliberately. This branch is based on the fork trunk, which is 46 commits ahead of upstream. Rebasing would replay that fork landing queue into contribution history, which is not wanted.Test and gate evidence
What ran, and its result:
reviewcompleted.testcompleted. No test command is configured for this repository, so the pipeline agent validated the change by reproducing the measurements the new documentation records, including running the negative control.documentcompleted with 0 unresolved items and no agent changes to commit.lintpassed:bin/fm-lint.sh, ShellCheck 0.11.0 against the pinned 0.11.0.bin/fm-doc-audience-check.shpasses locally:ok surfaces=72 local_links=213.Lint shell scripts,Test coverage guard,Repo invariants,Windows launcher bridge,Stock macOS Bash snapshot compatibility,Behavior tests (Herdr),Behavior portable parallel 1and2, andBehavior portable serial 1,3and4all pass.The change touches no shell script, so the lint result reflects an unchanged script surface.
Base and scope
mainon this fork, ated376cf.fm/cfvc-lane-a-residual-uncertaintiesat2494ccce, ahead 1, behind 0.What was deliberately not done
pi 0.81.1against a recorded0.82.0andclaude 2.1.224against a recorded2.1.220. Those rows are version-scoped to the operator run that produced them and are not false. Rewriting them without re-verifying the whole table on a machine carrying all seven harnesses would be fabrication.pi-signedorkimi, which lack one whileopencodeandgrokeach have one. That is a real gap, and it is recorded as one. Writing the test requires the binary to prove it against, and the coding guidelines forbid landing a harness-dependent check that was never run against the real harness. Noopencode,pi-signed,grokorkimibinary exists on this machine..agents/skills/firstmate-coding-guidelines/SKILL.md. Duplicating it here would violate the one-owner rule.AGENTS.mdchange. Nothing here is needed by every session on every turn.The other three uncertainties
Items 2, 3 and 5 of the same register were also resolved in this task and correctly produce no change to this repository:
5f39690on 2026-08-03, three days before it was reported. Negative control: deleting exactly the two relevant rows in a scratch export makes the same verifier FAIL, exit 1.bin/divergence. Measured: 48 of 49 modified files differ in executable lines, 1 is comment-only. It decomposes into a small set of upstream feature families the fork has not taken plus fork-original work upstream has not taken, not 48 independent semantic differences. This is input to the separate reconciliation work, and the fork-versus-upstream posture is private fleet state rather than shared tracked material, so it belongs in the task report and not in tracked docs.run_pm10_replication.py. It is the repro command for a completed, fenced evidence record and should be neither certified nor retired. The real defect is platform-side: the ownership catalog derivesis_entry_pointfrom a filename suffix while that same source file argues at length against name-based derivation, misclassifying seven modules. That is platform code, outside this repository.