From cb927af89588dc416e46a25f50c9cd853709c85f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20ROBERT?= Date: Fri, 25 Sep 2026 17:32:37 +0200 Subject: [PATCH] fix(ci): the nightly control installs its tools before it moves to the tag (#794) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `runtime-proof.yml`'s stacks job runs twice: once on main, once with the working tree moved to the last release tag, as a positive control — the thing that tells "our night is red" from "the world is red". #794 moved every workflow download onto `tools/ci/fetch.sh`. The control checks out v0.13.0 before it installs anything, so the steps that install Incus and Terraform looked for that helper inside a tag cut before it existed: tools/ci/fetch.sh: No such file or directory Process completed with exit code 127 Thirty-five seconds in, before a single stack was applied, on 36150807823. No pull request could have caught it: the stacks job only runs on the nightly schedule and on dispatch, and the control leg carries `continue-on-error`, so it does not even redden the job it belongs to. It was found by reading a run. THE FIX IS THE ORDER, and it is also the better control. Everything that provisions the RUNNER now happens before the move — Go, Incus, OVN, Terraform — and everything that is the PRODUCT happens after it: the binary, the images, the suites. A witness should differ from its subject in one thing, the code under test, not in how its runner was built. Running the tag's own installation steps mixed the runtime under test with the harness that fetched it, and this is what made that mixing visible. `Install Terraform` moves up with the other tools for the same reason; nothing else changes order, and the step that carries the gate is untouched. What holds it: TestTheControlNeverRunsARepositoryScriptInsideTheTag asserts that no `tools/ci/` script runs after the detach, and — the accepting half — that one does run before it, so a job that simply lost its control cannot pass by being empty. tools/falsify/specs/the-control-runs-on-its-own-tools.json carries two mutations: a CI script placed after the detach, and the detach moved back before the tools. Both compile and both redden the test. `mise run prepush` green. Assisted-by: Claude Code (claude-opus-5) --- .github/workflows/runtime-proof.yml | 98 +++++++++++------- tools/ci/control_leg_test.go | 99 +++++++++++++++++++ .../the-control-runs-on-its-own-tools.json | 19 ++++ 3 files changed, 178 insertions(+), 38 deletions(-) create mode 100644 tools/ci/control_leg_test.go create mode 100644 tools/falsify/specs/the-control-runs-on-its-own-tools.json diff --git a/.github/workflows/runtime-proof.yml b/.github/workflows/runtime-proof.yml index 8bb23d3..ff5a7ef 100644 --- a/.github/workflows/runtime-proof.yml +++ b/.github/workflows/runtime-proof.yml @@ -617,26 +617,6 @@ jobs: # and cannot be wrong. fetch-depth: 0 - # The control's whole subject, resolved here so the log says which tag it - # is about. A control that did not name its own subject would be a second - # green nobody can interpret. - # - # It moves the working tree to the tag, SUITES INCLUDED, and that is - # deliberate: the question is "does the thing we shipped still work in - # today's world", and a harness fix that only exists on main is not part of - # what we shipped. The consequence has to be said out loud, because it is - # the one that would otherwise teach "the control is always red": a defect - # already fixed on main can still redden the control until the next - # release carries the fix. - - name: Move to the last release, and say which - if: matrix.control - run: | - set -euo pipefail - tag="$(git describe --tags --abbrev=0 --match 'v*')" - echo "control: the same gate at ${tag}, suites included" >> "$GITHUB_STEP_SUMMARY" - echo "CONTROL_TAG=${tag}" >> "$GITHUB_ENV" - git checkout --detach "${tag}" - - name: Setup Go uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: @@ -708,6 +688,66 @@ jobs: - name: Keep the runner's Docker firewall out of the verdict run: sudo iptables -P FORWARD ACCEPT + # `feint up` drives Terraform, which is what applies both stacks. Same + # pinned version and upstream checksums as conformance.yml — one client, + # three workflows, one truth. No provider CLI is installed: this gate + # speaks curl and jq to the emulator and reads the machines with `incus`. + # + # Installed BEFORE the control moves to the tag, with the other tools, and + # that position is the fix for #794's regression — see the step below. + - name: Install Terraform + env: + TERRAFORM_VERSION: '1.13.3' + run: | + set -euo pipefail + workdir="$(mktemp -d)" + base="https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}" + asset="terraform_${TERRAFORM_VERSION}_linux_amd64.zip" + tools/ci/fetch.sh "${base}/${asset}" "${workdir}/${asset}" + tools/ci/fetch.sh "${base}/terraform_${TERRAFORM_VERSION}_SHA256SUMS" "${workdir}/sums" + (cd "${workdir}" && grep " ${asset}$" sums | sha256sum -c -) + sudo unzip -o "${workdir}/${asset}" terraform -d /usr/local/bin + terraform version + + # The control's whole subject, resolved here so the log says which tag it + # is about. A control that did not name its own subject would be a second + # green nobody can interpret. + # + # It moves the working tree to the tag, SUITES INCLUDED, and that is + # deliberate: the question is "does the thing we shipped still work in + # today's world", and a harness fix that only exists on main is not part of + # what we shipped. The consequence has to be said out loud, because it is + # the one that would otherwise teach "the control is always red": a defect + # already fixed on main can still redden the control until the next + # release carries the fix. + # + # WHY IT SITS HERE AND NOT BEFORE THE TOOLS (#794). It used to run second, + # so every later step ran inside the tag's tree — including the ones that + # install Incus and Terraform. On 2026-09-25 #794 moved those downloads onto + # `tools/ci/fetch.sh`, a file that exists on main and not in v0.13.0, and + # the control died with `tools/ci/fetch.sh: No such file or directory`, + # exit 127, thirty-five seconds in, before a single stack. No pull request + # could have caught it: this job only exists on the nightly schedule. + # + # The position is also the better control. A witness should differ from its + # subject in ONE thing, the code under test — not in how its runner was + # built. Installing the tools from main for both legs is what makes the two + # comparable; running the tag's own installation steps mixed the runtime + # under test with the harness that fetched it. + # + # So: everything that provisions the RUNNER runs before this line, and + # everything that is the PRODUCT — the binary, the images, the suites — + # runs after it. TestTheControlNeverRunsARepositoryScriptInsideTheTag + # fails without it. + - name: Move to the last release, and say which + if: matrix.control + run: | + set -euo pipefail + tag="$(git describe --tags --abbrev=0 --match 'v*')" + echo "control: the same gate at ${tag}, suites included" >> "$GITHUB_STEP_SUMMARY" + echo "CONTROL_TAG=${tag}" >> "$GITHUB_ENV" + git checkout --detach "${tag}" + - name: Build feint and diagnose the runtime run: | set -euo pipefail @@ -791,24 +831,6 @@ jobs: - name: Build the machine images the stacks boot run: sudo ./feint images --vm incus-ovn - # `feint up` drives Terraform, which is what applies both stacks. Same - # pinned version and upstream checksums as conformance.yml — one client, - # three workflows, one truth. No provider CLI is installed: this gate - # speaks curl and jq to the emulator and reads the machines with `incus`. - - name: Install Terraform - env: - TERRAFORM_VERSION: '1.13.3' - run: | - set -euo pipefail - workdir="$(mktemp -d)" - base="https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}" - asset="terraform_${TERRAFORM_VERSION}_linux_amd64.zip" - tools/ci/fetch.sh "${base}/${asset}" "${workdir}/${asset}" - tools/ci/fetch.sh "${base}/terraform_${TERRAFORM_VERSION}_SHA256SUMS" "${workdir}/sums" - (cd "${workdir}" && grep " ${asset}$" sums | sha256sum -c -) - sudo unzip -o "${workdir}/${asset}" terraform -d /usr/local/bin - terraform version - # The gate itself. FEINT_VM is exported rather than left to the default so # the mode is announced with the provenance a reader can check against # this job's own name — the property tools/runtime-mode.sh holds and #574 diff --git a/tools/ci/control_leg_test.go b/tools/ci/control_leg_test.go new file mode 100644 index 0000000..ff60c01 --- /dev/null +++ b/tools/ci/control_leg_test.go @@ -0,0 +1,99 @@ +package ci + +import ( + "strings" + "testing" +) + +// The positive control must not run the repository's CI scripts from inside the +// tag it checked out (#794, found by #125's dispatch run). +// +// `runtime-proof.yml`'s stacks job runs twice: once on main, and once with the +// working tree moved to the last release tag, as a positive control — the thing +// that tells "our night is red" from "the world is red". The move is a +// `git checkout --detach`, so every step after it reads files as that tag had +// them. +// +// On 2026-09-25, #794 moved every workflow download onto `tools/ci/fetch.sh`. +// That file exists on main and not in v0.13.0, so the control died on +// +// tools/ci/fetch.sh: No such file or directory +// Process completed with exit code 127 +// +// thirty-five seconds in, before a single stack was applied. Nothing on a pull +// request could have caught it: the stacks job only runs on the nightly +// schedule and on dispatch, and the control leg carries `continue-on-error`, so +// the failure does not even redden the job it belongs to — it was found by +// reading the run, which is the reading this test replaces. +// +// The rule it holds: everything that provisions the RUNNER runs before the +// move, everything that is the PRODUCT runs after it. `tools/ci/` is the CI +// harness, which lives on main by definition; `tools/conformance/` and the rest +// are the product, and running the tag's copy of those is the control's whole +// point. +func TestTheControlNeverRunsARepositoryScriptInsideTheTag(t *testing.T) { + workflow := readWorkflow(t, "runtime-proof.yml") + + job := jobBlock(workflow, "stacks") + if job == "" { + t.Fatal("runtime-proof.yml has no stacks job: this test names the wrong one") + } + + // The move itself, by the command that performs it rather than by the step's + // title: a renamed step must not silently retire this check. + cut := strings.Index(job, "git checkout --detach") + if cut < 0 { + t.Fatal("the stacks job no longer detaches onto a tag: either the control " + + "leg is gone, or it moved somewhere this test cannot see it") + } + + // The accepting half first, so a job that simply stopped having a control + // cannot pass by being empty. + if !strings.Contains(job[:cut], "tools/ci/fetch.sh") { + t.Error("no CI helper runs before the control moves to the tag: the tools are " + + "meant to be installed from main, for both legs, so that the witness differs " + + "from its subject in the code alone") + } + + for _, line := range strings.Split(job[cut:], "\n") { + if strings.Contains(line, "tools/ci/") { + t.Errorf("the control runs a CI script after checking out the tag, and a tag that "+ + "predates that script fails with exit 127 before it measures anything:\n %s", + strings.TrimSpace(line)) + } + } +} + +// jobBlock answers one job's body, from its key to the next job's. +// +// Jobs are the two-space keys under `jobs:`, so the next one at that exact +// indentation ends this one. Bounded on purpose: asserting over the whole file +// would forbid `tools/ci/` in jobs that never check out a tag, which is most of +// them and is perfectly correct there. +func jobBlock(workflow, name string) string { + start := strings.Index(workflow, "\n "+name+":\n") + if start < 0 { + return "" + } + rest := workflow[start+1:] + for offset := 0; ; { + next := strings.Index(rest[offset:], "\n ") + if next < 0 { + return workflow[start:] + } + at := offset + next + len("\n ") + // A job key, not a deeper line: one more space means it is inside this job. + if at < len(rest) && rest[at] != ' ' && strings.Contains(lineAt(rest, at), ":") { + return rest[:offset+next] + } + offset = at + } +} + +// lineAt answers the line starting at an index. +func lineAt(s string, at int) string { + if end := strings.IndexByte(s[at:], '\n'); end >= 0 { + return s[at : at+end] + } + return s[at:] +} diff --git a/tools/falsify/specs/the-control-runs-on-its-own-tools.json b/tools/falsify/specs/the-control-runs-on-its-own-tools.json new file mode 100644 index 0000000..e37d2b2 --- /dev/null +++ b/tools/falsify/specs/the-control-runs-on-its-own-tools.json @@ -0,0 +1,19 @@ +{ + "package": "./tools/ci/", + "mutations": [ + { + "label": "a CI script runs after the control checked out the tag, which is how #794 killed the positive control with exit 127 thirty-five seconds in, before a single stack (#125)", + "file": ".github/workflows/runtime-proof.yml", + "find": " run: sudo ./feint images --vm incus-ovn", + "replace": " run: tools/ci/fetch.sh https://example.invalid/image /dev/null && sudo ./feint images --vm incus-ovn", + "test": "TestTheControlNeverRunsARepositoryScriptInsideTheTag" + }, + { + "label": "the control moves to the tag before the tools are installed, so the witness differs from its subject in how its runner was built and not only in the code under test (#125)", + "file": ".github/workflows/runtime-proof.yml", + "find": " tools/ci/fetch.sh https://pkgs.zabbly.com/key.asc \"${workdir}/zabbly.asc\"\n fpr=\"$(gpg --show-keys --with-colons \"${workdir}/zabbly.asc\" | awk -F: '/^fpr/ {print $10; exit}')\"\n if [ \"${fpr}\" != \"${ZABBLY_FPR}\" ]; then\n echo \"unexpected Zabbly key fingerprint: ${fpr}\" >&2\n exit 1\n fi\n sudo install -D -m 0644 \"${workdir}/zabbly.asc\" /etc/apt/keyrings/zabbly.asc\n # shellcheck disable=SC1091\n codename=\"$(. /etc/os-release && echo \"${VERSION_CODENAME}\")\"\n arch=\"$(dpkg --print-architecture)\"\n sudo tee /etc/apt/sources.list.d/zabbly-incus.sources >/dev/null <&2\n exit 1\n fi\n sudo install -D -m 0644 \"${workdir}/zabbly.asc\" /etc/apt/keyrings/zabbly.asc\n # shellcheck disable=SC1091\n codename=\"$(. /etc/os-release && echo \"${VERSION_CODENAME}\")\"\n arch=\"$(dpkg --print-architecture)\"\n sudo tee /etc/apt/sources.list.d/zabbly-incus.sources >/dev/null <