Skip to content

ci: track patch releases of upstream binaries (Kubernetes, containerd, runc, CNI, k0s, ORAS) - #272

Merged
jeefy merged 5 commits into
projectbluefin:mainfrom
jeefy:ci/track-upstream-binaries
Sep 29, 2026
Merged

jeefy merged 5 commits into
projectbluefin:mainfrom
jeefy:ci/track-upstream-binaries

Conversation

@jeefy

@jeefy jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Problem

The upstream binaries in the kubeadm sysext (kubelet/kubeadm/kubectl, crictl, containerd static, runc, CNI plugins), k0s, and ORAS (Justfile oras_image plus the setup-oras url/checksum in build.yml) are pinned by a version plus a sha256. Renovate can't see any of them, so every security patch release needs a hand-written PR. Today's pins are behind: Kubernetes is 1.34.3 while 1.34.12 is out, containerd 2.1.5 (2.1.9 out), runc 1.3.3 (1.3.6 out), and k0s 1.36.4+k0s.0 (+k0s.1 out).

What this adds

  • .github/scripts/track-binaries.py (stdlib only)
    • check finds the newest release in each pinned MAJOR.MINOR series. Kubernetes comes from dl.k8s.io/release/stable-X.Y.txt; the rest come from GitHub releases, skipping drafts, prereleases and releases whose pinned assets aren't attached yet.
    • apply <component> [--version V] moves one component and rewrites every sha256 pin that depends on it. Each sha256 comes from the checksum file upstream publishes (*.sha256, *.sha256sum, runc's PGP-signed runc.sha256sum, k0s sha256sums.txt, ORAS checksums.txt). It must also match the sha256 of the downloaded asset, or nothing is written.
    • apply --version is also how you do a manual minor bump, with the same verification.
  • .github/workflows/track-binaries.yml follows the track-junctions.yml pattern: daily at 08:30 UTC plus workflow_dispatch, never on pull_request.
    • check builds a matrix. propose opens or updates one PR per component on auto/track-binaries/<component>.
    • An open PR with the same title is left alone, so there's no churn and no rebuild. A closed or reverted one with the same title is not reopened.
    • GITHUB_TOKEN stays read-only. The mergeraptor app token is narrowed to contents + pull-requests, plus workflows for the ORAS leg only, because its pins live in build.yml.
  • Policy: patches are automatic, minors are manual. The kubeadm payload follows the minor of the cluster it joins. kubeadm join must match the control plane's minor, and kubelet must never be newer than the API server. k0s bundles its own Kubernetes for a separate cluster, so its minor moves independently. This is documented in docs/skills/kubeadm-sysext.md (canonical), with a pointer in k0s-sysext.md and a workflow row in ci-tooling.md.
  • Tests that pinned exact releases now pin the series. test_kubeadm_sysext.py checks 1.34 / 1.34 / 2.1 / 1.3 / 1.1, and test_k0s_version.py checks 1.36 plus a sha256-shaped ref. Before this change, every automated patch PR would have failed them. A series change is still a deliberate edit.
  • include/kubeadm.yml puts a comment line above each version. With one PR per component, adjacent version lines made git conflict when two bumps landed one after another; I checked that with a local merge.

Why not Renovate or bst source track

  • Renovate: a regex manager needs the version and its digest in one match. Here the version lives in include/*.yml or the Justfile and the digests in elements/*.bst or a workflow. One Kubernetes version feeds three dl.k8s.io checksums, and no datasource computes those. k0s's version is split into two atoms. postUpgradeTasks are blocked on the hosted app.
  • bst source track: I dry-ran it under the shared lock. remote/tar do support tracking, but it re-downloads every source of the element, not just the bumped one's. It also takes whatever bytes it gets as the new ref and warns "Potential man-in-the-middle attack!". And it re-indents every list in elements/k0s/k0s-bin.bst, producing a 24-line diff for a one-line ref change.
  • BuildStream still checks each ref against the bytes when this PR's build fetches the sources.

Verification

  • check against live upstream today:
    component    series current        latest         state
    kubernetes   1.34   1.34.3         1.34.12        update
    cri-tools    1.34   1.34.0         1.34.0         current
    containerd   2.1    2.1.5          2.1.9          update
    runc         1.3    1.3.3          1.3.6          update
    cni-plugins  1.1    1.1.1          1.1.1          current
    k0s          1.36   1.36.4+k0s.0   1.36.4+k0s.1   update
    oras         1.3    1.3.4          1.3.4          current
    
  • apply for those four in a scratch copy, with real downloads: the 6 refs it wrote are identical to the refs bst source track computed for the same versions. apply --version <current> for cri-tools, CNI and ORAS is a byte-for-byte no-op, so the checksum mapping reproduces today's pins. The bst dry run was reverted and none of these bumps are in this PR; the tracker proposes them after merge.
  • pytest tests/unit 398 passed (27 new, no network), plus a mutation check that removing the mapping boundary, the token scoping, the hash check, the prerelease filter or the asset-presence check each fails a test. bats tests/unit 74/74, docs-checks.py pass, just validate pass, and actionlint is clean except the existing ubuntu-26.04 label warning.

Notes

jeefy and others added 5 commits September 28, 2026 22:28
… sha256

Upstream binaries are pinned by a version in include/*.yml (or the
Justfile) and a sha256 in elements/*.bst (or a workflow). A version
bump without its checksums is a broken BuildStream source ref, so the
two have to move in one change.

`check` reports the newest release inside each pinned MAJOR.MINOR
series: Kubernetes from dl.k8s.io stable-X.Y.txt, cri-tools,
containerd, runc, CNI plugins, k0s and ORAS from GitHub releases,
skipping drafts, prereleases and releases whose pinned assets are not
attached yet. `apply` moves one component and rewrites every pin
derived from it, taking each sha256 from the checksum file upstream
publishes and confirming it against the downloaded asset before
writing anything. `apply --version` makes a manual minor bump with the
same verification.

Refs are written in place rather than with `bst source track`, which
re-downloads every source of the element, trusts whatever bytes it
gets, and re-indents every list in elements/k0s/k0s-bin.bst. Against
live upstream both produce identical refs for kubelet, kubeadm,
kubectl 1.34.12, containerd 2.1.9, runc 1.3.6 and k0s 1.36.4+k0s.1.

Tests fake urllib.request.urlopen (no network) and include a contract
test that reads the real repository files.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The kubeadm payload now follows the release series of the cluster's
nodes rather than their exact patch, so security patches can land
without a hand-written PR. test_versions_stay_on_the_cluster_nodes_series
asserts the series (Kubernetes and cri-tools 1.34, containerd 2.1,
runc 1.3, CNI plugins 1.1) instead of one release each; moving a series
stays a deliberate edit.

Each version in include/kubeadm.yml now sits under its own comment
line. The tracker opens one pull request per component, and git
reports a conflict when two changes touch adjacent lines, so with the
old layout merging the containerd bump would have conflicted the runc
one.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
test_k0s_version.py asserted k0s 1.36.4+k0s.0 and its sha256, so any
k0s patch release failed the unit tests. It now asserts the 1.36 series
and a sha256-shaped ref; check-k0s-version.py still guards how the
version is derived.

include/k0s.yml's derivation table is labelled as an example for
v1.36.4+k0s.0, so it stays true after the atoms move.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Kubernetes, cri-tools, containerd, runc, CNI plugins, k0s and ORAS are
pinned by version and sha256 in files Renovate cannot update together,
so security patch releases needed hand-written PRs.

track-binaries.yml runs daily after track-junctions and on dispatch,
never on pull_request. `check` lists the components with a newer patch
in their pinned series; `propose` applies each one in its own matrix
leg and opens or updates one PR per component on
auto/track-binaries/<component>. An open PR with the same title is
left alone; a closed or reverted one with the same title is not
reopened.

GITHUB_TOKEN stays read-only. Pushes and PRs use the mergeraptor app
token, narrowed to contents and pull-requests, plus workflows for the
ORAS leg because its pins live in build.yml.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
kubeadm-sysext.md: the sysext runs the Kubernetes minor of the cluster
it joins, independently of k0s's bundled Kubernetes; kubeadm join must
match the control plane's minor and kubelet must not be newer than the
API server. Patches arrive from track-binaries.yml; a minor bump is
track-binaries.py apply for kubernetes and cri-tools, the pause image,
containerd's support table and the series in the unit test.

k0s-sysext.md: where the k0s patch and minor bumps now come from.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy jeefy mentioned this pull request Sep 29, 2026
5 tasks done

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 29045d2. I read the full diff, ran the tracker against the live upstreams and checked the output by hand.

What I verified:

  • Checksums are really checked. apply() (track-binaries.py:472-505) fetches the checksum file upstream publishes next to each asset ({url}.sha256, .sha256sum, runc's runc.sha256sum, k0s sha256sums.txt), downloads the asset itself, and raises unless the two sha256 values agree. Nothing is computed locally and trusted without checking. published_sha256() rejects a checksum file that lists the asset more than once or not at all.
  • Series boundary. _candidates() keeps only non-draft, non-prerelease v* tags whose _series() equals the pinned MAJOR.MINOR, and only once every pinned asset (and its sums file) is attached to the release. Kubernetes uses stable-<series>.txt and refuses an answer outside the series. A minor bump stays a manual apply --version.
  • Live run. python3 .github/scripts/track-binaries.py check against current upstreams reports kubernetes 1.34.3→1.34.12, containerd 2.1.5→2.1.9, runc 1.3.3→1.3.6, k0s 1.36.4+k0s.0→+k0s.1, and cri-tools, cni-plugins and oras current. apply runc rewrote exactly include/kubeadm.yml (runc-version) and the matching ref: in elements/kubeadm/kubeadm-bin.bst to 3f3921db…023c. Downloading runc.amd64 v1.3.6 myself and running sha256sum gives the same value, and so does upstream's runc.sha256sum. I reverted the change afterwards.
  • Workflow. Top-level permissions: contents: read. The check job's GITHUB_TOKEN is used only for release listings. The propose job's mergeraptor app token is narrowed to contents/pull-requests: write, with workflows: write only in the oras matrix leg, since ORAS pins live in build.yml. It runs on schedule/workflow_dispatch only and checks out main, so no PR-controlled code runs with the app token. A closed PR with the same title is not reopened, and a failed gh pr list stops the step instead of reading as "no PR".
  • Renovate overlap. None. renovate.json and check-renovate-series track the annotated BuildStream junctions, not these kind: remote/tar binary pins.
  • Refactor is value-neutral. The include/kubeadm.yml and include/k0s.yml edits only move comments; every version value is unchanged.
  • Local. pytest tests/unit: 398 passed. bats tests/unit: pass. actionlint: clean apart from my local actionlint not knowing the ubuntu-26.04 label.
  • CI on this head. build, boot-test, unit and docs all pass (runs 36517392498, 36517392483, 36517392500).

@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Heads-up from #279: #279 adds arm64 source pins to elements/k0s/k0s-bin.bst and elements/kubeadm/kubeadm-bin.bst under an arch == aarch64 conditional. Whichever merges second should extend track-binaries.py to refresh those arm64 refs too; otherwise the first automated bump leaves them stale (x86_64 is unaffected).

@jeefy
jeefy merged commit fd34c54 into projectbluefin:main Sep 29, 2026
5 checks passed
jeefy added a commit that referenced this pull request Sep 29, 2026
…#288)

#279 added arm64 sources under `arch == "aarch64"` in k0s-bin.bst and
kubeadm-bin.bst after #272's tracker landed. The tracker already treats
every `url:` line carrying a component's marker as a pin, so it rewrites
the arm64 refs too, but nothing held it to that. Behavioural tests (mocked
HTTP) now do: a runc and a k0s bump rewrite both architectures' refs; a
release missing the arm64 asset is not a candidate; a missing arm64 asset
or checksum fails the component and writes nothing; a no-op run leaves
the files byte-identical; the real elements pin arm64 for every amd64
source. The docstring and the tracker docs say so.

Assisted-by: Claude Opus 5.5 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants