ci(build-kernel): fix the dispatch defaults that make every manual build the expensive one (TIN-4065) - #94
Merged
Jess Sullivan (Jesssullivan) merged 2 commits intoAug 29, 2026
Conversation
…ild the expensive one (TIN-4065) Two workflow_dispatch defaults on build-kernel.yml were set the expensive way, and W2's proof-build wave has to dispatch this workflow twice. use_ccache: 'false' -> 'true' The ccache restore/save steps are gated on `github.event_name != 'workflow_dispatch' || inputs.use_ccache == 'true'`, so push and tag builds always use ccache and only manual dispatch had to opt in. Defaulted off, every manual build is a cold build unless someone remembers a dropdown. Dispatch now matches push; opt out for a deliberately cold build. variant: 'both' -> 'generic' The `Check variant filter` step runs inside each matrix job and max-parallel is 1, so `both` does not fan out -- it serialises two builds observed at 2h16m-3h24m each. Tag builds still build both regardless of this input (the filter only applies to workflow_dispatch), so release behaviour is unchanged. This only changes what a manual build does by default, to the one variant the fleet actually runs. Evidence this is the live footgun and not a hypothetical: workflow_dispatch run 32695394106 on 2026-08-24 ran build-rpm (generic) and build-rpm (rt, 6.19.3-rt1) back to back. Also sharpens three input descriptions: kernel_version notes that the 6.19.5 default is on an EOL line while 6.18.y and 7.1.y are the maintained ones, variant notes that `both` roughly doubles wall clock, and rt_version notes that it is only consulted for the rt variant. The kernel_version default value is deliberately NOT changed -- picking 6.18 or 7.1 is the operator's W2 decision, not a CI default's. Not in scope: the -j4 cap in the spec, the tinyland-dind scratch-volume question, and dispatching the W2 builds themselves. Linear: TIN-4065
…che across bases (TIN-4065) Review follow-up on the dispatch-defaults change. The defaults themselves (variant=generic, use_ccache=true) are unchanged and still right; what was wrong was the evidence attached to them and one real gap in the cache keys. variant comment: the quoted "2h16m-3h24m" range came from two generic-only runs, not from runs that built both variants. The three dispatches that actually built both (23221273296, 25202336642, 25609434372) ran 2h06m57s + 2h06m28s, 2h48m27s + 2h46m49s, and 2h53m41s + 1h46m46s -- a per-variant range of 1h47m-2h54m. The old ceiling also understated current cold builds: the generic job of tag run 32764070756 took 4h06m24s. "The fleet does not run RT" was too flat for a committed justification. RT was deliberately dispatched in 5 of the 13 successful dispatches that built anything, README calls it an experimental lane, site/ ships an RT installer, and site/docs/honey.md records a successful RT boot on honey with /sys/kernel/realtime = 1. The narrow true statement is that no fleet host currently boots RT and 7.1.x RT is parked per base-anchor.yml. ccache restore-keys: every key embedded kversion, so a dispatch on a new base (the 6.18 LTS control and 7.1.x candidate) matched no prefix and went fully cold while still paying ~2.85GB to save. Adds a version-agnostic ccache-<variant>- key so a new-base build can seed off an existing tree. ccache is content-addressed, so cross-version seeding is a hit-rate question, never a correctness one. Cache scope is unchanged and still binds: all 19 live cache entries put ccache on refs/tags/v6.19.5-xr12 and none on refs/heads/xr/main, so a branch dispatch still cannot read a tag build's ccache. rt_version description: notes that the 6.19.3-rt1 default is pinned to the 6.19 line, since the kernel_version note now steers operators at 6.18.y/7.1.y and xr/source-sync.md pins the last proven RT lane at v7.0.1-rt2. A variant=rt kernel_version=7.1.3 dispatch would otherwise silently pair a 7.1.3 tree with a 6.19.3 RT patch. README: documents use_ccache (absent entirely), corrects "warm ~1h vs cold ~2h" to the measured 43m40s warm vs 4h06m cold, corrects "save-always: true" to the explicit actions/cache steps the workflow actually uses, and drops the false claim that the two variants share a ccache -- the directory and the key are both per-variant. Release behaviour is unchanged: the variant filter still no-ops for non-dispatch events and tag builds still build both variants. Linear: TIN-4065
Jess Sullivan (Jesssullivan)
marked this pull request as ready for review
August 29, 2026 05:08
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two
workflow_dispatchdefaults onbuild-kernel.ymlwere set the expensive way. W2 has to dispatch this workflow twice (7.1.x candidate, 6.18 LTS control), so it is worth fixing before those dispatches rather than after.use_ccache: 'false'→'true'The ccache restore/save steps are gated on:
Push and tag builds always use ccache; only manual dispatch had to opt in. Defaulted off, every manual build is a cold build unless someone remembers a dropdown. Dispatch now matches push behaviour; opt out when you specifically want a cold build.
This is the well-evidenced half. On tag run 32764070756, the rt job built 4h06m24s cold on attempt 2 and 43m40s warm on attempt 3 after restoring 2.85 GB — about 5.6x. The spec is engineered for it:
kernel-xr.spec:360-364exportsKBUILD_BUILD_TIMESTAMP=''precisely because "without these, ccache gets 0% hits".variant: 'both'→'generic'Check variant filterruns inside each matrix job andmax-parallel: 1, sobothdoes not fan out — it serialises two builds. Tag builds still build both regardless of this input (the filter only applies toworkflow_dispatch), so release behaviour is unchanged.What the run history actually says
An earlier draft of this PR called run
32695394106a live instance of thebothdefault firing. That was wrong, and the run refutes it. In that run thebuild-rpm (rt, 6.19.3-rt1)job skipped every step fromParse version from tagthroughUpload RPM artifactsand finished in 8 seconds; only generic built (3h36m06s).Check variant filtersetsskip=trueonly when the selection is neitherbothnor the current variant, so that run was dispatchedvariant=generic— two job rows in the UI, one build. What32695394106does show isRestore ccache=skipped/Save ccache=skipped: it is evidence for theuse_ccachehalf, not thevarianthalf.Full census of the 13 successful
workflow_dispatchruns that actually built something: 8 generic-only, 2 rt-only, 3both(23221273296,25202336642,25609434372). Thebothdefault was overridden 10 of 13 times (77%), and the last dispatch that actually paid forbothwas25609434372on 2026-05-09 — not four days ago.README.mdcorroborates: all three documented invocations pass-f variant=explicitly.So the
variantflip is a small, correct tidy — it makes the default match what operators pick anyway — not a fix for a recurring live burn. It is kept on that (weaker) merit.ccache restore-keys: the W2 dispatches would have restored nothing
The
use_ccacheflip alone does not deliver its benefit to the two W2 dispatches, because all three restore-keys embedded${{ steps.version.outputs.kversion }}. Two independent misses stacked:7.1.3or6.18.x, so a new-base build matched nothing.refs/heads/refs/tags/v6.19.5-xr12; zero ccache entries exist onrefs/heads/xr/main, where a dispatch runs. (The write-scope split is directly observed —sources-6.19.5-exists twice under one key on two different refs.)Net effect without a fix: each W2 dispatch restores nothing, builds fully cold, and still pays ~2.85 GB and ~3.5 min to save.
This PR adds a version-agnostic fourth restore-key so a new-base build can seed off an existing tree:
ccache-${{ matrix.variant.name }}-ccache is content-addressed, so cross-version seeding is a hit-rate question, never a correctness one. The scope half is not fixed by this and is not fixable from the keys — a branch dispatch still cannot read a tag build's ccache, so the first build on a new base is cold regardless; the benefit starts from the second dispatch on
xr/main. This also changes how a new-version tag build seeds (speed only, never output).Worth flagging for after the W2 dispatches: at 6.99 GB across 19 entries, two ~2.85 GB saves land near GitHub's 10 GB default repo cache budget, and LRU eviction would reach
ccache-generic-6.19.5-…— the entry the nextv6.19.5-xr13tag build wants. Considergh cache deleteon the candidate-base entries once W2 is done. That is not idle: attempt 2's rt job ran 6h02m08s and was cancelled againsttimeout-minutes: 360after its build succeeded, losing the artifacts and forcing a third attempt.Also
Three input descriptions sharpened.
kernel_versionnotes that the6.19.5default sits on an EOL line while 6.18.y and 7.1.y are the maintained ones.variantnow carries measured numbers from the runs that actually built both variants — 1h47m–2h54m per variant — instead of the previous2h16m-3h24m, which was taken from two generic-only runs and also understated current cold builds (4h06m).rt_versionnotes that its6.19.3-rt1default is pinned to the 6.19 line. This matters now that thekernel_versionnote steers operators at 6.18.y/7.1.y: avariant=rt kernel_version=7.1.3dispatch would otherwise silently pair a 7.1.3 tree with a 6.19.3 RT patch, andxr/source-sync.mdpins the last proven RT lane atv7.0.1-rt2.The
variantcomment no longer claims "the fleet does not run RT". RT was deliberately dispatched in 5 of those 13 runs,README.mdcalls it an experimental lane,site/publishes an RT installer, andsite/docs/honey.mdrecords an RT boot on honey with/sys/kernel/realtime= 1. The accurate statement — no fleet host currently boots RT, and 7.1.x RT is parked perbase-anchor.yml— is what the comment says now, so nobody reads it as licence to drop the rt matrix entry.README.mdis updated in the same commit: it documented this exact surface and never mentioneduse_ccache. It also claimedsave-always: true(the workflow uses explicitactions/cachesteps withif: always()), quoted "warm ~1h vs cold ~2h" (measured: 43m40s vs 4h06m, with cold builds ranging 3.5–6h), and said both variants "share ccache" — the directory and the key are both per-variant.The
kernel_versiondefault value is deliberately not changed. Picking 6.18 or 7.1 is the operator's W2 decision; a CI default should not quietly make it.What this does not do
This does not produce the two W2 RPMs. That needs an actual dispatch of a multi-hour build per candidate, which is an operator action on a night when runner caps are tight — and TIN-4065's DONE bar is explicitly "an RPM, not a dry-run".
Also out of scope: the
-j4cap atxr/specs/kernel-xr.spec:355-358(consumed at:366), and thetinyland-dindscratch-volume question.Verification
actionlint 1.7.12reports the same 3 findings before and after this change, all pre-existing and outside the diff (unknown labeltinyland-dind; SC2001/SC2129 in the version-parse script). YAML parses with all 5 inputs and both defaults intact.git diff --checkclean.Re-verification against the ticket
The ticket's premise "the last successful
build-kernel.ymlrun was 2026-07-09" is stale. There are two successful runs since:32695394106(dispatch,xr/main, 2026-08-24) and32764070756(tagv6.19.5-xr12, 2026-08-24, generic + rt + release all green, artifacts live). The workflow works; what is missing is a build against either candidate base.Note this workflow will not run on this PR (it triggers on tags and dispatch only).
Linear: TIN-4065