Skip to content

fix(ci): the nightly control installs its tools before it moves to the tag (#794) - #797

Merged
stephrobert merged 1 commit into
mainfrom
fix/794-the-control-installs-before-it-moves
Sep 25, 2026
Merged

stephrobert merged 1 commit into
mainfrom
fix/794-the-control-installs-before-it-moves

Conversation

@stephrobert

Copy link
Copy Markdown
Owner

Found by reading run 36150807823, the dispatch that proved #125's fix. The runtime legs went green; the stacks control leg died in thirty-five seconds:

tools/ci/fetch.sh: No such file or directory
Process completed with exit code 127

What broke it

runtime-proof.yml's stacks job runs twice. The second leg is a positive control: the same gate with the working tree moved to the last release tag, so a red night can be told apart from a red world.

The move is git checkout --detach v0.13.0, and it ran second — before Go, before Incus, before Terraform. So every later step read files as that tag had them. #794 moved all workflow downloads onto tools/ci/fetch.sh, which exists on main and not in a tag cut before it, and the control stopped measuring anything at all.

Nothing on a pull request could have caught it. The stacks job only runs on the nightly schedule and on dispatch, and the control leg carries continue-on-error, so its failure does not even redden the job it belongs to. It was found by reading a run — which is the reading this PR replaces with a test.

That is the shape this repository keeps paying for: an instrument that lies before its subject does. A control that cannot run is not a neutral loss, it is a green nobody can interpret, and the four-quadrant reading in the verdict job depends on it.

The fix is the order, and it is also a better control

Everything that provisions the runner now runs before the move — Go, Incus, OVN, Terraform. Everything that is the product runs after it — the binary, the images, the suites.

A witness should differ from its subject in one thing, the code under test, and not in how its runner was built. Running the tag's own installation steps mixed the runtime under test with the harness that fetched it; this defect is what made that mixing visible. The step that carries the gate is untouched, and Install Terraform moves up with the other tools for the same reason.

What holds it

  • TestTheControlNeverRunsARepositoryScriptInsideTheTag: no tools/ci/ script after the detach — and, the accepting half, at least one before it, so a job that simply lost its control cannot pass by being empty. It anchors on the git checkout --detach command rather than on a step title, so renaming the step cannot retire the check.
  • tools/falsify/specs/the-control-runs-on-its-own-tools.json, two mutations: a CI script placed after the detach, and the detach moved back before the tools, which is the defect itself. Both compile, both redden the test.
  • mise run prepush green.

Still open, separately

The same run also failed stacks (incus-ovn) on the main leg:

FAIL: outscale: a rule of platform-app's group opens 8080 and platform-web-a
does not reach 10.50.2.5:8080

That is a different failure — the control died before applying any stack, this one applied them — and it is being measured rather than guessed at: the gate plays three passes precisely because the defect class it hunts is intermittent, so one red is not a verdict. Not folded in here.

🤖 Generated with Claude Code

…e tag (#794)

`runtime-proof.yml`'s stacks job runs twice: once on main, once with the working
tree moved to the last release tag, as a positive control — the thing that tells
"our night is red" from "the world is red".

#794 moved every workflow download onto `tools/ci/fetch.sh`. The control checks
out v0.13.0 before it installs anything, so the steps that install Incus and
Terraform looked for that helper inside a tag cut before it existed:

  tools/ci/fetch.sh: No such file or directory
  Process completed with exit code 127

Thirty-five seconds in, before a single stack was applied, on 36150807823. No
pull request could have caught it: the stacks job only runs on the nightly
schedule and on dispatch, and the control leg carries `continue-on-error`, so it
does not even redden the job it belongs to. It was found by reading a run.

THE FIX IS THE ORDER, and it is also the better control. Everything that
provisions the RUNNER now happens before the move — Go, Incus, OVN, Terraform —
and everything that is the PRODUCT happens after it: the binary, the images, the
suites. A witness should differ from its subject in one thing, the code under
test, not in how its runner was built. Running the tag's own installation steps
mixed the runtime under test with the harness that fetched it, and this is what
made that mixing visible.

`Install Terraform` moves up with the other tools for the same reason; nothing
else changes order, and the step that carries the gate is untouched.

What holds it: TestTheControlNeverRunsARepositoryScriptInsideTheTag asserts that
no `tools/ci/` script runs after the detach, and — the accepting half — that one
does run before it, so a job that simply lost its control cannot pass by being
empty. tools/falsify/specs/the-control-runs-on-its-own-tools.json carries two
mutations: a CI script placed after the detach, and the detach moved back before
the tools. Both compile and both redden the test.

`mise run prepush` green.

Assisted-by: Claude Code (claude-opus-5)
@stephrobert
stephrobert merged commit 3f1e404 into main Sep 25, 2026
38 checks passed
@stephrobert
stephrobert deleted the fix/794-the-control-installs-before-it-moves branch September 25, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant