Skip to content

fix(ci): the nightly control installs its tools before it moves to the tag (#794) #797

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stephrobert merged 1 commit into from
Sep 25, 2026
Merged

fix(ci): the nightly control installs its tools before it moves to the tag (#794) #797

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 60 additions & 38 deletions .github/workflows/runtime-proof.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -617,26 +617,6 @@ jobs:
# and cannot be wrong.
fetch-depth: 0

# The control's whole subject, resolved here so the log says which tag it
# is about. A control that did not name its own subject would be a second
# green nobody can interpret.
#
# It moves the working tree to the tag, SUITES INCLUDED, and that is
# deliberate: the question is "does the thing we shipped still work in
# today's world", and a harness fix that only exists on main is not part of
# what we shipped. The consequence has to be said out loud, because it is
# the one that would otherwise teach "the control is always red": a defect
# already fixed on main can still redden the control until the next
# release carries the fix.
- name: Move to the last release, and say which
if: matrix.control
run: |
set -euo pipefail
tag="$(git describe --tags --abbrev=0 --match 'v*')"
echo "control: the same gate at ${tag}, suites included" >> "$GITHUB_STEP_SUMMARY"
echo "CONTROL_TAG=${tag}" >> "$GITHUB_ENV"
git checkout --detach "${tag}"

- name: Setup Go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
with:
Expand Down Expand Up @@ -708,6 +688,66 @@ jobs:
- name: Keep the runner's Docker firewall out of the verdict
run: sudo iptables -P FORWARD ACCEPT

# `feint up` drives Terraform, which is what applies both stacks. Same
# pinned version and upstream checksums as conformance.yml — one client,
# three workflows, one truth. No provider CLI is installed: this gate
# speaks curl and jq to the emulator and reads the machines with `incus`.
#
# Installed BEFORE the control moves to the tag, with the other tools, and
# that position is the fix for #794's regression — see the step below.
- name: Install Terraform
env:
TERRAFORM_VERSION: '1.13.3'
run: |
set -euo pipefail
workdir="$(mktemp -d)"
base="https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}"
asset="terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
tools/ci/fetch.sh "${base}/${asset}" "${workdir}/${asset}"
tools/ci/fetch.sh "${base}/terraform_${TERRAFORM_VERSION}_SHA256SUMS" "${workdir}/sums"
(cd "${workdir}" && grep " ${asset}$" sums | sha256sum -c -)
sudo unzip -o "${workdir}/${asset}" terraform -d /usr/local/bin
terraform version

# The control's whole subject, resolved here so the log says which tag it
# is about. A control that did not name its own subject would be a second
# green nobody can interpret.
#
# It moves the working tree to the tag, SUITES INCLUDED, and that is
# deliberate: the question is "does the thing we shipped still work in
# today's world", and a harness fix that only exists on main is not part of
# what we shipped. The consequence has to be said out loud, because it is
# the one that would otherwise teach "the control is always red": a defect
# already fixed on main can still redden the control until the next
# release carries the fix.
#
# WHY IT SITS HERE AND NOT BEFORE THE TOOLS (#794). It used to run second,
# so every later step ran inside the tag's tree — including the ones that
# install Incus and Terraform. On 2026-09-25 #794 moved those downloads onto
# `tools/ci/fetch.sh`, a file that exists on main and not in v0.13.0, and
# the control died with `tools/ci/fetch.sh: No such file or directory`,
# exit 127, thirty-five seconds in, before a single stack. No pull request
# could have caught it: this job only exists on the nightly schedule.
#
# The position is also the better control. A witness should differ from its
# subject in ONE thing, the code under test — not in how its runner was
# built. Installing the tools from main for both legs is what makes the two
# comparable; running the tag's own installation steps mixed the runtime
# under test with the harness that fetched it.
#
# So: everything that provisions the RUNNER runs before this line, and
# everything that is the PRODUCT — the binary, the images, the suites —
# runs after it. TestTheControlNeverRunsARepositoryScriptInsideTheTag
# fails without it.
- name: Move to the last release, and say which
if: matrix.control
run: |
set -euo pipefail
tag="$(git describe --tags --abbrev=0 --match 'v*')"
echo "control: the same gate at ${tag}, suites included" >> "$GITHUB_STEP_SUMMARY"
echo "CONTROL_TAG=${tag}" >> "$GITHUB_ENV"
git checkout --detach "${tag}"

- name: Build feint and diagnose the runtime
run: |
set -euo pipefail
Expand Down Expand Up @@ -791,24 +831,6 @@ jobs:
- name: Build the machine images the stacks boot
run: sudo ./feint images --vm incus-ovn

# `feint up` drives Terraform, which is what applies both stacks. Same
# pinned version and upstream checksums as conformance.yml — one client,
# three workflows, one truth. No provider CLI is installed: this gate
# speaks curl and jq to the emulator and reads the machines with `incus`.
- name: Install Terraform
env:
TERRAFORM_VERSION: '1.13.3'
run: |
set -euo pipefail
workdir="$(mktemp -d)"
base="https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}"
asset="terraform_${TERRAFORM_VERSION}_linux_amd64.zip"
tools/ci/fetch.sh "${base}/${asset}" "${workdir}/${asset}"
tools/ci/fetch.sh "${base}/terraform_${TERRAFORM_VERSION}_SHA256SUMS" "${workdir}/sums"
(cd "${workdir}" && grep " ${asset}$" sums | sha256sum -c -)
sudo unzip -o "${workdir}/${asset}" terraform -d /usr/local/bin
terraform version

# The gate itself. FEINT_VM is exported rather than left to the default so
# the mode is announced with the provenance a reader can check against
# this job's own name — the property tools/runtime-mode.sh holds and #574
Expand Down
99 changes: 99 additions & 0 deletions tools/ci/control_leg_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
package ci

import (
"strings"
"testing"
)

// The positive control must not run the repository's CI scripts from inside the
// tag it checked out (#794, found by #125's dispatch run).
//
// `runtime-proof.yml`'s stacks job runs twice: once on main, and once with the
// working tree moved to the last release tag, as a positive control — the thing
// that tells "our night is red" from "the world is red". The move is a
// `git checkout --detach`, so every step after it reads files as that tag had
// them.
//
// On 2026-09-25, #794 moved every workflow download onto `tools/ci/fetch.sh`.
// That file exists on main and not in v0.13.0, so the control died on
//
// tools/ci/fetch.sh: No such file or directory
// Process completed with exit code 127
//
// thirty-five seconds in, before a single stack was applied. Nothing on a pull
// request could have caught it: the stacks job only runs on the nightly
// schedule and on dispatch, and the control leg carries `continue-on-error`, so
// the failure does not even redden the job it belongs to — it was found by
// reading the run, which is the reading this test replaces.
//
// The rule it holds: everything that provisions the RUNNER runs before the
// move, everything that is the PRODUCT runs after it. `tools/ci/` is the CI
// harness, which lives on main by definition; `tools/conformance/` and the rest
// are the product, and running the tag's copy of those is the control's whole
// point.
func TestTheControlNeverRunsARepositoryScriptInsideTheTag(t *testing.T) {
workflow := readWorkflow(t, "runtime-proof.yml")

job := jobBlock(workflow, "stacks")
if job == "" {
t.Fatal("runtime-proof.yml has no stacks job: this test names the wrong one")
}

// The move itself, by the command that performs it rather than by the step's
// title: a renamed step must not silently retire this check.
cut := strings.Index(job, "git checkout --detach")
if cut < 0 {
t.Fatal("the stacks job no longer detaches onto a tag: either the control " +
"leg is gone, or it moved somewhere this test cannot see it")
}

// The accepting half first, so a job that simply stopped having a control
// cannot pass by being empty.
if !strings.Contains(job[:cut], "tools/ci/fetch.sh") {
t.Error("no CI helper runs before the control moves to the tag: the tools are " +
"meant to be installed from main, for both legs, so that the witness differs " +
"from its subject in the code alone")
}

for _, line := range strings.Split(job[cut:], "\n") {
if strings.Contains(line, "tools/ci/") {
t.Errorf("the control runs a CI script after checking out the tag, and a tag that "+
"predates that script fails with exit 127 before it measures anything:\n %s",
strings.TrimSpace(line))
}
}
}

// jobBlock answers one job's body, from its key to the next job's.
//
// Jobs are the two-space keys under `jobs:`, so the next one at that exact
// indentation ends this one. Bounded on purpose: asserting over the whole file
// would forbid `tools/ci/` in jobs that never check out a tag, which is most of
// them and is perfectly correct there.
func jobBlock(workflow, name string) string {
start := strings.Index(workflow, "\n "+name+":\n")
if start < 0 {
return ""
}
rest := workflow[start+1:]
for offset := 0; ; {
next := strings.Index(rest[offset:], "\n ")
if next < 0 {
return workflow[start:]
}
at := offset + next + len("\n ")
// A job key, not a deeper line: one more space means it is inside this job.
if at < len(rest) && rest[at] != ' ' && strings.Contains(lineAt(rest, at), ":") {
return rest[:offset+next]
}
offset = at
}
}

// lineAt answers the line starting at an index.
func lineAt(s string, at int) string {
if end := strings.IndexByte(s[at:], '\n'); end >= 0 {
return s[at : at+end]
}
return s[at:]
}
19 changes: 19 additions & 0 deletions tools/falsify/specs/the-control-runs-on-its-own-tools.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"package": "./tools/ci/",
"mutations": [
{
"label": "a CI script runs after the control checked out the tag, which is how #794 killed the positive control with exit 127 thirty-five seconds in, before a single stack (#125)",
"file": ".github/workflows/runtime-proof.yml",
"find": " run: sudo ./feint images --vm incus-ovn",
"replace": " run: tools/ci/fetch.sh https://example.invalid/image /dev/null && sudo ./feint images --vm incus-ovn",
"test": "TestTheControlNeverRunsARepositoryScriptInsideTheTag"
},
{
"label": "the control moves to the tag before the tools are installed, so the witness differs from its subject in how its runner was built and not only in the code under test (#125)",
"file": ".github/workflows/runtime-proof.yml",
"find": " tools/ci/fetch.sh https://pkgs.zabbly.com/key.asc \"${workdir}/zabbly.asc\"\n fpr=\"$(gpg --show-keys --with-colons \"${workdir}/zabbly.asc\" | awk -F: '/^fpr/ {print $10; exit}')\"\n if [ \"${fpr}\" != \"${ZABBLY_FPR}\" ]; then\n echo \"unexpected Zabbly key fingerprint: ${fpr}\" >&2\n exit 1\n fi\n sudo install -D -m 0644 \"${workdir}/zabbly.asc\" /etc/apt/keyrings/zabbly.asc\n # shellcheck disable=SC1091\n codename=\"$(. /etc/os-release && echo \"${VERSION_CODENAME}\")\"\n arch=\"$(dpkg --print-architecture)\"\n sudo tee /etc/apt/sources.list.d/zabbly-incus.sources >/dev/null <<EOF\n Enabled: yes\n Types: deb\n URIs: https://pkgs.zabbly.com/incus/stable\n Suites: ${codename}\n Components: main\n Architectures: ${arch}\n Signed-By: /etc/apt/keyrings/zabbly.asc\n EOF\n sudo apt-get update\n sudo apt-get install -y --no-install-recommends incus incus-client netcat-openbsd\n sudo incus --version\n\n - name: Install and wire OVN",
"replace": " tools/ci/fetch.sh https://pkgs.zabbly.com/key.asc \"${workdir}/zabbly.asc\"\n fpr=\"$(gpg --show-keys --with-colons \"${workdir}/zabbly.asc\" | awk -F: '/^fpr/ {print $10; exit}')\"\n if [ \"${fpr}\" != \"${ZABBLY_FPR}\" ]; then\n echo \"unexpected Zabbly key fingerprint: ${fpr}\" >&2\n exit 1\n fi\n sudo install -D -m 0644 \"${workdir}/zabbly.asc\" /etc/apt/keyrings/zabbly.asc\n # shellcheck disable=SC1091\n codename=\"$(. /etc/os-release && echo \"${VERSION_CODENAME}\")\"\n arch=\"$(dpkg --print-architecture)\"\n sudo tee /etc/apt/sources.list.d/zabbly-incus.sources >/dev/null <<EOF\n Enabled: yes\n Types: deb\n URIs: https://pkgs.zabbly.com/incus/stable\n Suites: ${codename}\n Components: main\n Architectures: ${arch}\n Signed-By: /etc/apt/keyrings/zabbly.asc\n EOF\n sudo apt-get update\n sudo apt-get install -y --no-install-recommends incus incus-client netcat-openbsd\n sudo incus --version\n\n - name: Move the control to the tag too early\n if: matrix.control\n run: git checkout --detach \"$(git describe --tags --abbrev=0 --match 'v*')\"\n\n - name: Install and wire OVN",
"test": "TestTheControlNeverRunsARepositoryScriptInsideTheTag"
}
]
}
Loading