Repository navigation
ci: track patch releases of upstream binaries (Kubernetes, containerd, runc, CNI, k0s, ORAS) - #272
Merged
Merged
Conversation
… sha256 Upstream binaries are pinned by a version in include/*.yml (or the Justfile) and a sha256 in elements/*.bst (or a workflow). A version bump without its checksums is a broken BuildStream source ref, so the two have to move in one change. `check` reports the newest release inside each pinned MAJOR.MINOR series: Kubernetes from dl.k8s.io stable-X.Y.txt, cri-tools, containerd, runc, CNI plugins, k0s and ORAS from GitHub releases, skipping drafts, prereleases and releases whose pinned assets are not attached yet. `apply` moves one component and rewrites every pin derived from it, taking each sha256 from the checksum file upstream publishes and confirming it against the downloaded asset before writing anything. `apply --version` makes a manual minor bump with the same verification. Refs are written in place rather than with `bst source track`, which re-downloads every source of the element, trusts whatever bytes it gets, and re-indents every list in elements/k0s/k0s-bin.bst. Against live upstream both produce identical refs for kubelet, kubeadm, kubectl 1.34.12, containerd 2.1.9, runc 1.3.6 and k0s 1.36.4+k0s.1. Tests fake urllib.request.urlopen (no network) and include a contract test that reads the real repository files. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The kubeadm payload now follows the release series of the cluster's nodes rather than their exact patch, so security patches can land without a hand-written PR. test_versions_stay_on_the_cluster_nodes_series asserts the series (Kubernetes and cri-tools 1.34, containerd 2.1, runc 1.3, CNI plugins 1.1) instead of one release each; moving a series stays a deliberate edit. Each version in include/kubeadm.yml now sits under its own comment line. The tracker opens one pull request per component, and git reports a conflict when two changes touch adjacent lines, so with the old layout merging the containerd bump would have conflicted the runc one. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
test_k0s_version.py asserted k0s 1.36.4+k0s.0 and its sha256, so any k0s patch release failed the unit tests. It now asserts the 1.36 series and a sha256-shaped ref; check-k0s-version.py still guards how the version is derived. include/k0s.yml's derivation table is labelled as an example for v1.36.4+k0s.0, so it stays true after the atoms move. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Kubernetes, cri-tools, containerd, runc, CNI plugins, k0s and ORAS are pinned by version and sha256 in files Renovate cannot update together, so security patch releases needed hand-written PRs. track-binaries.yml runs daily after track-junctions and on dispatch, never on pull_request. `check` lists the components with a newer patch in their pinned series; `propose` applies each one in its own matrix leg and opens or updates one PR per component on auto/track-binaries/<component>. An open PR with the same title is left alone; a closed or reverted one with the same title is not reopened. GITHUB_TOKEN stays read-only. Pushes and PRs use the mergeraptor app token, narrowed to contents and pull-requests, plus workflows for the ORAS leg because its pins live in build.yml. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
kubeadm-sysext.md: the sysext runs the Kubernetes minor of the cluster it joins, independently of k0s's bundled Kubernetes; kubeadm join must match the control plane's minor and kubelet must not be newer than the API server. Patches arrive from track-binaries.yml; a minor bump is track-binaries.py apply for kubernetes and cri-tools, the pause image, containerd's support table and the series in the unit test. k0s-sysext.md: where the k0s patch and minor bumps now come from. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
5 tasks done
castrojo
approved these changes
Sep 29, 2026
castrojo
left a comment
Contributor
There was a problem hiding this comment.
Reviewed at 29045d2. I read the full diff, ran the tracker against the live upstreams and checked the output by hand.
What I verified:
- Checksums are really checked.
apply()(track-binaries.py:472-505) fetches the checksum file upstream publishes next to each asset ({url}.sha256,.sha256sum, runc'srunc.sha256sum, k0ssha256sums.txt), downloads the asset itself, and raises unless the two sha256 values agree. Nothing is computed locally and trusted without checking.published_sha256()rejects a checksum file that lists the asset more than once or not at all. - Series boundary.
_candidates()keeps only non-draft, non-prereleasev*tags whose_series()equals the pinned MAJOR.MINOR, and only once every pinned asset (and its sums file) is attached to the release. Kubernetes usesstable-<series>.txtand refuses an answer outside the series. A minor bump stays a manualapply --version. - Live run.
python3 .github/scripts/track-binaries.py checkagainst current upstreams reports kubernetes 1.34.3→1.34.12, containerd 2.1.5→2.1.9, runc 1.3.3→1.3.6, k0s 1.36.4+k0s.0→+k0s.1, and cri-tools, cni-plugins and oras current.apply runcrewrote exactlyinclude/kubeadm.yml(runc-version) and the matchingref:inelements/kubeadm/kubeadm-bin.bstto3f3921db…023c. Downloadingrunc.amd64v1.3.6 myself and runningsha256sumgives the same value, and so does upstream'srunc.sha256sum. I reverted the change afterwards. - Workflow. Top-level
permissions: contents: read. The check job'sGITHUB_TOKENis used only for release listings. The propose job's mergeraptor app token is narrowed tocontents/pull-requests: write, withworkflows: writeonly in theorasmatrix leg, since ORAS pins live inbuild.yml. It runs onschedule/workflow_dispatchonly and checks outmain, so no PR-controlled code runs with the app token. A closed PR with the same title is not reopened, and a failedgh pr liststops the step instead of reading as "no PR". - Renovate overlap. None.
renovate.jsonandcheck-renovate-seriestrack the annotated BuildStream junctions, not thesekind: remote/tarbinary pins. - Refactor is value-neutral. The
include/kubeadm.ymlandinclude/k0s.ymledits only move comments; every version value is unchanged. - Local.
pytest tests/unit: 398 passed.bats tests/unit: pass.actionlint: clean apart from my local actionlint not knowing theubuntu-26.04label. - CI on this head. build, boot-test, unit and docs all pass (runs 36517392498, 36517392483, 36517392500).
This was referenced Sep 29, 2026
Collaborator
Author
|
Heads-up from #279: #279 adds arm64 source pins to elements/k0s/k0s-bin.bst and elements/kubeadm/kubeadm-bin.bst under an |
This was referenced Sep 29, 2026
jeefy
added a commit
that referenced
this pull request
Sep 29, 2026
…#288) #279 added arm64 sources under `arch == "aarch64"` in k0s-bin.bst and kubeadm-bin.bst after #272's tracker landed. The tracker already treats every `url:` line carrying a component's marker as a pin, so it rewrites the arm64 refs too, but nothing held it to that. Behavioural tests (mocked HTTP) now do: a runc and a k0s bump rewrite both architectures' refs; a release missing the arm64 asset is not a candidate; a missing arm64 asset or checksum fails the component and writes nothing; a no-op run leaves the files byte-identical; the real elements pin arm64 for every amd64 source. The docstring and the tracker docs say so. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The upstream binaries in the kubeadm sysext (kubelet/kubeadm/kubectl, crictl, containerd static, runc, CNI plugins), k0s, and ORAS (Justfile
oras_imageplus the setup-orasurl/checksuminbuild.yml) are pinned by a version plus a sha256. Renovate can't see any of them, so every security patch release needs a hand-written PR. Today's pins are behind: Kubernetes is 1.34.3 while 1.34.12 is out, containerd 2.1.5 (2.1.9 out), runc 1.3.3 (1.3.6 out), and k0s 1.36.4+k0s.0 (+k0s.1 out).What this adds
.github/scripts/track-binaries.py(stdlib only)checkfinds the newest release in each pinnedMAJOR.MINORseries. Kubernetes comes fromdl.k8s.io/release/stable-X.Y.txt; the rest come from GitHub releases, skipping drafts, prereleases and releases whose pinned assets aren't attached yet.apply <component> [--version V]moves one component and rewrites every sha256 pin that depends on it. Each sha256 comes from the checksum file upstream publishes (*.sha256,*.sha256sum, runc's PGP-signedrunc.sha256sum, k0ssha256sums.txt, ORASchecksums.txt). It must also match the sha256 of the downloaded asset, or nothing is written.apply --versionis also how you do a manual minor bump, with the same verification..github/workflows/track-binaries.ymlfollows thetrack-junctions.ymlpattern: daily at 08:30 UTC plusworkflow_dispatch, never onpull_request.checkbuilds a matrix.proposeopens or updates one PR per component onauto/track-binaries/<component>.GITHUB_TOKENstays read-only. The mergeraptor app token is narrowed tocontents+pull-requests, plusworkflowsfor the ORAS leg only, because its pins live inbuild.yml.kubeadm joinmust match the control plane's minor, and kubelet must never be newer than the API server. k0s bundles its own Kubernetes for a separate cluster, so its minor moves independently. This is documented indocs/skills/kubeadm-sysext.md(canonical), with a pointer ink0s-sysext.mdand a workflow row inci-tooling.md.test_kubeadm_sysext.pychecks 1.34 / 1.34 / 2.1 / 1.3 / 1.1, andtest_k0s_version.pychecks 1.36 plus a sha256-shaped ref. Before this change, every automated patch PR would have failed them. A series change is still a deliberate edit.include/kubeadm.ymlputs a comment line above each version. With one PR per component, adjacent version lines made git conflict when two bumps landed one after another; I checked that with a local merge.Why not Renovate or
bst source trackinclude/*.ymlor the Justfile and the digests inelements/*.bstor a workflow. One Kubernetes version feeds three dl.k8s.io checksums, and no datasource computes those. k0s's version is split into two atoms.postUpgradeTasksare blocked on the hosted app.bst source track: I dry-ran it under the shared lock.remote/tardo support tracking, but it re-downloads every source of the element, not just the bumped one's. It also takes whatever bytes it gets as the new ref and warns "Potential man-in-the-middle attack!". And it re-indents every list inelements/k0s/k0s-bin.bst, producing a 24-line diff for a one-line ref change.Verification
checkagainst live upstream today:applyfor those four in a scratch copy, with real downloads: the 6 refs it wrote are identical to the refsbst source trackcomputed for the same versions.apply --version <current>for cri-tools, CNI and ORAS is a byte-for-byte no-op, so the checksum mapping reproduces today's pins. The bst dry run was reverted and none of these bumps are in this PR; the tracker proposes them after merge.pytest tests/unit398 passed (27 new, no network), plus a mutation check that removing the mapping boundary, the token scoping, the hash check, the prerelease filter or the asset-presence check each fails a test.bats tests/unit74/74,docs-checks.pypass,just validatepass, and actionlint is clean except the existingubuntu-26.04label warning.Notes
build.yml's comment "pin 1.3.4 (matching the Justfile's oras_image)" restates the version. I leftbuild.ymluntouched because ci(release): dry-run the publish path on pull requests; attest provenance and publish an SBOM #269 edits those steps; the tracker updates every setup-orasurl/checksumpair (including ci(release): dry-run the publish path on pull requests; attest provenance and publish an SBOM #269's second one) and lists leftover mentions of the old version in the PR body.test_track_binaries.pyfail until its component entry goes too.