fix(ci): the nightly control installs its tools before it moves to the tag (#794) - #797
Merged
Merged
Conversation
…e tag (#794) `runtime-proof.yml`'s stacks job runs twice: once on main, once with the working tree moved to the last release tag, as a positive control — the thing that tells "our night is red" from "the world is red". #794 moved every workflow download onto `tools/ci/fetch.sh`. The control checks out v0.13.0 before it installs anything, so the steps that install Incus and Terraform looked for that helper inside a tag cut before it existed: tools/ci/fetch.sh: No such file or directory Process completed with exit code 127 Thirty-five seconds in, before a single stack was applied, on 36150807823. No pull request could have caught it: the stacks job only runs on the nightly schedule and on dispatch, and the control leg carries `continue-on-error`, so it does not even redden the job it belongs to. It was found by reading a run. THE FIX IS THE ORDER, and it is also the better control. Everything that provisions the RUNNER now happens before the move — Go, Incus, OVN, Terraform — and everything that is the PRODUCT happens after it: the binary, the images, the suites. A witness should differ from its subject in one thing, the code under test, not in how its runner was built. Running the tag's own installation steps mixed the runtime under test with the harness that fetched it, and this is what made that mixing visible. `Install Terraform` moves up with the other tools for the same reason; nothing else changes order, and the step that carries the gate is untouched. What holds it: TestTheControlNeverRunsARepositoryScriptInsideTheTag asserts that no `tools/ci/` script runs after the detach, and — the accepting half — that one does run before it, so a job that simply lost its control cannot pass by being empty. tools/falsify/specs/the-control-runs-on-its-own-tools.json carries two mutations: a CI script placed after the detach, and the detach moved back before the tools. Both compile and both redden the test. `mise run prepush` green. Assisted-by: Claude Code (claude-opus-5)
stephrobert
deleted the
fix/794-the-control-installs-before-it-moves
branch
September 25, 2026 16:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by reading run 36150807823, the dispatch that proved #125's fix. The runtime legs went green; the stacks control leg died in thirty-five seconds:
What broke it
runtime-proof.yml's stacks job runs twice. The second leg is a positive control: the same gate with the working tree moved to the last release tag, so a red night can be told apart from a red world.The move is
git checkout --detach v0.13.0, and it ran second — before Go, before Incus, before Terraform. So every later step read files as that tag had them. #794 moved all workflow downloads ontotools/ci/fetch.sh, which exists onmainand not in a tag cut before it, and the control stopped measuring anything at all.Nothing on a pull request could have caught it. The stacks job only runs on the nightly schedule and on dispatch, and the control leg carries
continue-on-error, so its failure does not even redden the job it belongs to. It was found by reading a run — which is the reading this PR replaces with a test.That is the shape this repository keeps paying for: an instrument that lies before its subject does. A control that cannot run is not a neutral loss, it is a green nobody can interpret, and the four-quadrant reading in the
verdictjob depends on it.The fix is the order, and it is also a better control
Everything that provisions the runner now runs before the move — Go, Incus, OVN, Terraform. Everything that is the product runs after it — the binary, the images, the suites.
A witness should differ from its subject in one thing, the code under test, and not in how its runner was built. Running the tag's own installation steps mixed the runtime under test with the harness that fetched it; this defect is what made that mixing visible. The step that carries the gate is untouched, and
Install Terraformmoves up with the other tools for the same reason.What holds it
TestTheControlNeverRunsARepositoryScriptInsideTheTag: notools/ci/script after the detach — and, the accepting half, at least one before it, so a job that simply lost its control cannot pass by being empty. It anchors on thegit checkout --detachcommand rather than on a step title, so renaming the step cannot retire the check.tools/falsify/specs/the-control-runs-on-its-own-tools.json, two mutations: a CI script placed after the detach, and the detach moved back before the tools, which is the defect itself. Both compile, both redden the test.mise run prepushgreen.Still open, separately
The same run also failed
stacks (incus-ovn)on the main leg:That is a different failure — the control died before applying any stack, this one applied them — and it is being measured rather than guessed at: the gate plays three passes precisely because the defect class it hunts is intermittent, so one red is not a verdict. Not folded in here.
🤖 Generated with Claude Code