Skip to content

ci: reproducibility covers every shipped element, shared CI scripts, doc drift - #403

Merged
jeefy merged 7 commits into
mainfrom
ci/gha-c1-repro-scripts-docs
Oct 5, 2026
Merged

jeefy merged 7 commits into
mainfrom
ci/gha-c1-repro-scripts-docs

Conversation

@jeefy

@jeefy jeefy commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Review findings M4, L2, L3 and the doc-drift list from the GitHub Actions review. The kernel-cache work (M5) is in a separate PR.

M4: reproducibility rebuilds everything that ships

  • FINAL_ASSEMBLY now also has oci/nvidia-open-595-sysext.bst, oci/nvidia-container-toolkit-sysext.bst, oci/bluefin-server-sbom.bst (it ships as bluefin-server_<ver>.spdx.json) and homelab/homelab-templates.bst (the .bu/.ign release assets). Before this, the second build reused their cached artifacts, so they were never compared.
  • tests/unit/test_reproducibility_workflow.py builds the shipped set from two sources: every dependency oci/bluefin-server-image.bst stages at a location, plus every oci/* target of just validate. The test fails if FINAL_ASSEMBLY misses any of them. Checked: on the old workflow it names exactly the four missing elements.
  • Why reproducibility.yml has 0 runs: this is expected, not a bug. The workflow landed Tue 2026-09-29 (build: reproducible image assembly, key-set safety, patch docs #281). Its only trigger is cron: '0 9 * * 1', so the first run is due Mon 2026-10-05 09:00 UTC. Other scheduled workflows in this repo do fire.

L2: runner disk setup

  • scripts/ci-runner-disk.sh replaces the three inline copies (two in build.yml, one in reproducibility.yml). It uses ln -sfn, so running it twice no longer nests a symlink inside /mnt/buildstream.
  • The test runs the script twice against a stub sudo. Another test makes sure no workflow inlines graphroot/ln -sf again, and that every job that builds calls the script.

L3: one FSDK version parser and one tracker PR script

  • check-release-version.py gains two options:
    • --print-fsdk, which the Justfile's fsdk_version now uses.
    • --fix, which track-junctions now uses: it syncs installer-version and then runs the same check.
  • No grep/sed parser of the junction is left; a test checks this.
  • .github/scripts/propose-pr.sh is the create-or-update-PR step for both trackers. It pushes nothing when:
    • the paths already match HEAD, or
    • an open PR's branch already holds the same content (compared by content, not title). This stops track-junctions force-pushing a new commit, and a new FSDK build, every day.
  • track-binaries keeps its existing behaviour of not reopening a closed title, via --skip-closed-title.
  • It pushes with --force-with-lease on the branch state it saw. A failed gh or ls-remote stops the run before any push.
  • Tested with a real git checkout against a bare remote and a stub gh.

Doc drift

  • ci-tooling.md#build-time-and-caches now holds the one build-time figure: the build job on main has a median of ~50 min and a p90 of ~2 h (56 runs, 2026-09-24..10-04), and the kernel takes 1 h 43 min when uncached. build.yml, kernel-cache.sh and image-build-needed.py link to it instead of repeating their own (conflicting) numbers.
  • Other fixes:
    • just is installed in six steps across four workflows, and tests/unit/test_ci_tool_pins.py checks that all pins are the same.
    • Removed the gnome-build-meta mention.
    • Fixed the indentation of the Core Process list.
    • The Justfile comment now says "pinned by commit-named tag".

Gates

just validate, pytest (1103 passed; also in a detached worktree without files/boot-keys), bats, docs-checks, shellcheck, actionlint (only the existing SC2129), and zizmor (same 7 medium + 1 high as main, no new findings).

Found while checking: both tracker workflows currently fail at "Get mergeraptor token" with Not Found - get-a-repository-installation. The mergeraptor App does not look installed on this repo. That is a settings matter and is not changed here.

jeefy and others added 4 commits October 4, 2026 20:06
…isk setup

reproducibility.yml's FINAL_ASSEMBLY missed the NVIDIA driver and Container Toolkit sysexts, the SBOM and the homelab templates, so their cached artifacts were reused by the second build and never compared. tests/unit/test_ci_workflows.py derives the shipped set from what oci/bluefin-server-image.bst stages plus the oci/* targets of `just validate` and fails when FINAL_ASSEMBLY misses one.

The three copies of the runner disk setup (two in build.yml, one in reproducibility.yml) had drifted between `ln -sf` and `ln -sfn`; scripts/ci-runner-disk.sh replaces them with the idempotent `-sfn`.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Justfile's fsdk_version and track-junctions.yml each re-parsed the junction ref with grep/sed. check-release-version.py gains --print-fsdk (the Justfile) and --fix (track-junctions: sync installer-version, then check). Also correct the Justfile comment: the bst2 image is pinned by a commit-named tag, not by SHA.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
.github/scripts/propose-pr.sh commits the given paths onto the tracker branch, force-pushes with a lease and creates or edits the PR. It pushes nothing when an open PR's branch already holds the same content, which stops track-junctions from force-pushing a fresh commit (and a full FSDK build) every day; track-binaries keeps not reopening a closed title via --skip-closed-title.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
One canonical figure in ci-tooling.md (build job on main: median ~50 min, p90 ~2 h over 56 runs; uncached kernel 1 h 43 min) that build.yml, kernel-cache.sh and image-build-needed.py link to. Also: the just install sites are six steps in four workflows, the old track-refs job carried freedesktop-sdk junction bumps (there is no gnome-build-meta junction), and the Core Process list is no longer mis-indented.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

tests/unit/test_ci_workflows.py also lands in #401 with other contracts (including the single just version), so the reproducibility coverage and the runner disk tests move to test_reproducibility_workflow.py and test_ci_runner_disk.py.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at a72b225 (correctness, security, intent-alignment, style, docs-currency).

correctness

  • tests/unit/test_propose_pr.py:114 (test_opens_a_pull_request_on_a_new_branch) asserts the author is github-actions[bot], but propose-pr.sh:77-79 sets identity with git -c user.name/user.email, which GIT_AUTHOR_NAME/GIT_COMMITTER_* env vars override. The env fixture (test_propose_pr.py:62-71) inherits os.environ, so on any machine that exports those vars the test fails ('reviewer' != 'github-actions[bot]' here; passes with them unset, 13/13). Either strip/set GIT_AUTHOR_*/GIT_COMMITTER_* in the fixture (as the git() helper at :38-40 already does) or export them in the script so the bot identity holds regardless of the caller's environment. CI runners don't export these, so this is a dev-machine flake rather than a CI failure.

intent-alignment

  • PR body: "just is installed in six steps across four workflows, and a test checks that all pins are the same." No such test is in the diff — the only just@1.58.0 references added are workflow steps and docs/skills/ci-tooling.md:56-59; tests/unit/ at head has no test reading the just@ pin (checked test_release_workflow.py, test_ci_runner_disk.py, test_reproducibility_workflow.py). Either add it or drop the claim. (Body also names the new test file tests/unit/test_ci_workflows.py; it is tests/unit/test_reproducibility_workflow.py.)

Verified: all 14 FINAL_ASSEMBLY entries are location: deps of elements/oci/bluefin-server-image.bst:60-104 and every just validate oci/* target (Justfile:73) is covered; bluefin-server-sbom.bst stamps creationInfo.created from fixed SOURCE_DATE_EPOCH (element description), so adding it to the reproducibility set is sound; --force-with-lease=refs/heads/B: with empty expect correctly requires the branch not to exist (propose-pr.sh:80).

No findings from: security, style, docs-currency.

Confidence: 3/5 (needs attention) — 2 perspectives requested changes

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

jeefy and others added 2 commits October 4, 2026 23:24
Keeps #402's tracker changes (trackers environment, token minted after the tracking steps and narrowed, persist-credentials: false) and moves its one-off extraheader push into propose-pr.sh, which both trackers now call.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add tests/unit/test_ci_tool_pins.py asserting every workflow installs the
same just pin, as the ci-tooling doc describes. Strip inherited
GIT_AUTHOR_*/GIT_COMMITTER_* from the propose-pr test env so the bot
author assertion does not depend on the caller's environment.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>
@jeefy
jeefy merged commit 41b5315 into main Oct 5, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/scanner Filed or owned by the scanner agent.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant