Skip to content

ci(build-kernel): fix the dispatch defaults that make every manual build the expensive one (TIN-4065) - #94

Merged
Jess Sullivan (Jesssullivan) merged 2 commits into
xr/mainfrom
jess/tin-4065-dispatch-defaults
Aug 29, 2026
Merged

Jess Sullivan (Jesssullivan) merged 2 commits into
xr/mainfrom
jess/tin-4065-dispatch-defaults

Conversation

@Jesssullivan

@Jesssullivan Jess Sullivan (Jesssullivan) commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

What

Two workflow_dispatch defaults on build-kernel.yml were set the expensive way. W2 has to dispatch this workflow twice (7.1.x candidate, 6.18 LTS control), so it is worth fixing before those dispatches rather than after.

use_ccache: 'false''true'

The ccache restore/save steps are gated on:

if: ... && (github.event_name != 'workflow_dispatch' || inputs.use_ccache == 'true')

Push and tag builds always use ccache; only manual dispatch had to opt in. Defaulted off, every manual build is a cold build unless someone remembers a dropdown. Dispatch now matches push behaviour; opt out when you specifically want a cold build.

This is the well-evidenced half. On tag run 32764070756, the rt job built 4h06m24s cold on attempt 2 and 43m40s warm on attempt 3 after restoring 2.85 GB — about 5.6x. The spec is engineered for it: kernel-xr.spec:360-364 exports KBUILD_BUILD_TIMESTAMP='' precisely because "without these, ccache gets 0% hits".

variant: 'both''generic'

Check variant filter runs inside each matrix job and max-parallel: 1, so both does not fan out — it serialises two builds. Tag builds still build both regardless of this input (the filter only applies to workflow_dispatch), so release behaviour is unchanged.

What the run history actually says

An earlier draft of this PR called run 32695394106 a live instance of the both default firing. That was wrong, and the run refutes it. In that run the build-rpm (rt, 6.19.3-rt1) job skipped every step from Parse version from tag through Upload RPM artifacts and finished in 8 seconds; only generic built (3h36m06s). Check variant filter sets skip=true only when the selection is neither both nor the current variant, so that run was dispatched variant=generic — two job rows in the UI, one build. What 32695394106 does show is Restore ccache=skipped / Save ccache=skipped: it is evidence for the use_ccache half, not the variant half.

Full census of the 13 successful workflow_dispatch runs that actually built something: 8 generic-only, 2 rt-only, 3 both (23221273296, 25202336642, 25609434372). The both default was overridden 10 of 13 times (77%), and the last dispatch that actually paid for both was 25609434372 on 2026-05-09 — not four days ago. README.md corroborates: all three documented invocations pass -f variant= explicitly.

So the variant flip is a small, correct tidy — it makes the default match what operators pick anyway — not a fix for a recurring live burn. It is kept on that (weaker) merit.

ccache restore-keys: the W2 dispatches would have restored nothing

The use_ccache flip alone does not deliver its benefit to the two W2 dispatches, because all three restore-keys embedded ${{ steps.version.outputs.kversion }}. Two independent misses stacked:

  • Version. No key prefix matches 7.1.3 or 6.18.x, so a new-base build matched nothing.
  • Scope. Of the 19 live cache entries, both ccache entries sit on refs/heads/refs/tags/v6.19.5-xr12; zero ccache entries exist on refs/heads/xr/main, where a dispatch runs. (The write-scope split is directly observed — sources-6.19.5- exists twice under one key on two different refs.)

Net effect without a fix: each W2 dispatch restores nothing, builds fully cold, and still pays ~2.85 GB and ~3.5 min to save.

This PR adds a version-agnostic fourth restore-key so a new-base build can seed off an existing tree:

    ccache-${{ matrix.variant.name }}-

ccache is content-addressed, so cross-version seeding is a hit-rate question, never a correctness one. The scope half is not fixed by this and is not fixable from the keys — a branch dispatch still cannot read a tag build's ccache, so the first build on a new base is cold regardless; the benefit starts from the second dispatch on xr/main. This also changes how a new-version tag build seeds (speed only, never output).

Worth flagging for after the W2 dispatches: at 6.99 GB across 19 entries, two ~2.85 GB saves land near GitHub's 10 GB default repo cache budget, and LRU eviction would reach ccache-generic-6.19.5-… — the entry the next v6.19.5-xr13 tag build wants. Consider gh cache delete on the candidate-base entries once W2 is done. That is not idle: attempt 2's rt job ran 6h02m08s and was cancelled against timeout-minutes: 360 after its build succeeded, losing the artifacts and forcing a third attempt.

Also

Three input descriptions sharpened. kernel_version notes that the 6.19.5 default sits on an EOL line while 6.18.y and 7.1.y are the maintained ones. variant now carries measured numbers from the runs that actually built both variants — 1h47m–2h54m per variant — instead of the previous 2h16m-3h24m, which was taken from two generic-only runs and also understated current cold builds (4h06m).

rt_version notes that its 6.19.3-rt1 default is pinned to the 6.19 line. This matters now that the kernel_version note steers operators at 6.18.y/7.1.y: a variant=rt kernel_version=7.1.3 dispatch would otherwise silently pair a 7.1.3 tree with a 6.19.3 RT patch, and xr/source-sync.md pins the last proven RT lane at v7.0.1-rt2.

The variant comment no longer claims "the fleet does not run RT". RT was deliberately dispatched in 5 of those 13 runs, README.md calls it an experimental lane, site/ publishes an RT installer, and site/docs/honey.md records an RT boot on honey with /sys/kernel/realtime = 1. The accurate statement — no fleet host currently boots RT, and 7.1.x RT is parked per base-anchor.yml — is what the comment says now, so nobody reads it as licence to drop the rt matrix entry.

README.md is updated in the same commit: it documented this exact surface and never mentioned use_ccache. It also claimed save-always: true (the workflow uses explicit actions/cache steps with if: always()), quoted "warm ~1h vs cold ~2h" (measured: 43m40s vs 4h06m, with cold builds ranging 3.5–6h), and said both variants "share ccache" — the directory and the key are both per-variant.

The kernel_version default value is deliberately not changed. Picking 6.18 or 7.1 is the operator's W2 decision; a CI default should not quietly make it.

What this does not do

This does not produce the two W2 RPMs. That needs an actual dispatch of a multi-hour build per candidate, which is an operator action on a night when runner caps are tight — and TIN-4065's DONE bar is explicitly "an RPM, not a dry-run".

Also out of scope: the -j4 cap at xr/specs/kernel-xr.spec:355-358 (consumed at :366), and the tinyland-dind scratch-volume question.

Verification

actionlint 1.7.12 reports the same 3 findings before and after this change, all pre-existing and outside the diff (unknown label tinyland-dind; SC2001/SC2129 in the version-parse script). YAML parses with all 5 inputs and both defaults intact. git diff --check clean.

Re-verification against the ticket

The ticket's premise "the last successful build-kernel.yml run was 2026-07-09" is stale. There are two successful runs since: 32695394106 (dispatch, xr/main, 2026-08-24) and 32764070756 (tag v6.19.5-xr12, 2026-08-24, generic + rt + release all green, artifacts live). The workflow works; what is missing is a build against either candidate base.

Note this workflow will not run on this PR (it triggers on tags and dispatch only).

Linear: TIN-4065

…ild the expensive one (TIN-4065)

Two workflow_dispatch defaults on build-kernel.yml were set the expensive way,
and W2's proof-build wave has to dispatch this workflow twice.

use_ccache: 'false' -> 'true'
  The ccache restore/save steps are gated on
  `github.event_name != 'workflow_dispatch' || inputs.use_ccache == 'true'`, so
  push and tag builds always use ccache and only manual dispatch had to opt in.
  Defaulted off, every manual build is a cold build unless someone remembers a
  dropdown. Dispatch now matches push; opt out for a deliberately cold build.

variant: 'both' -> 'generic'
  The `Check variant filter` step runs inside each matrix job and max-parallel
  is 1, so `both` does not fan out -- it serialises two builds observed at
  2h16m-3h24m each. Tag builds still build both regardless of this input
  (the filter only applies to workflow_dispatch), so release behaviour is
  unchanged. This only changes what a manual build does by default, to the one
  variant the fleet actually runs.

Evidence this is the live footgun and not a hypothetical: workflow_dispatch run
32695394106 on 2026-08-24 ran build-rpm (generic) and build-rpm (rt, 6.19.3-rt1)
back to back.

Also sharpens three input descriptions: kernel_version notes that the 6.19.5
default is on an EOL line while 6.18.y and 7.1.y are the maintained ones,
variant notes that `both` roughly doubles wall clock, and rt_version notes that
it is only consulted for the rt variant. The kernel_version default value is
deliberately NOT changed -- picking 6.18 or 7.1 is the operator's W2 decision,
not a CI default's.

Not in scope: the -j4 cap in the spec, the tinyland-dind scratch-volume
question, and dispatching the W2 builds themselves.

Linear: TIN-4065
@linear-code

linear-code Bot commented Aug 29, 2026

Copy link
Copy Markdown

TIN-4065

…che across bases (TIN-4065)

Review follow-up on the dispatch-defaults change. The defaults themselves
(variant=generic, use_ccache=true) are unchanged and still right; what was
wrong was the evidence attached to them and one real gap in the cache keys.

variant comment: the quoted "2h16m-3h24m" range came from two generic-only
runs, not from runs that built both variants. The three dispatches that
actually built both (23221273296, 25202336642, 25609434372) ran 2h06m57s +
2h06m28s, 2h48m27s + 2h46m49s, and 2h53m41s + 1h46m46s -- a per-variant range
of 1h47m-2h54m. The old ceiling also understated current cold builds: the
generic job of tag run 32764070756 took 4h06m24s.

"The fleet does not run RT" was too flat for a committed justification. RT was
deliberately dispatched in 5 of the 13 successful dispatches that built
anything, README calls it an experimental lane, site/ ships an RT installer,
and site/docs/honey.md records a successful RT boot on honey with
/sys/kernel/realtime = 1. The narrow true statement is that no fleet host
currently boots RT and 7.1.x RT is parked per base-anchor.yml.

ccache restore-keys: every key embedded kversion, so a dispatch on a new base
(the 6.18 LTS control and 7.1.x candidate) matched no prefix and went fully
cold while still paying ~2.85GB to save. Adds a version-agnostic
ccache-<variant>- key so a new-base build can seed off an existing tree. ccache
is content-addressed, so cross-version seeding is a hit-rate question, never a
correctness one. Cache scope is unchanged and still binds: all 19 live cache
entries put ccache on refs/tags/v6.19.5-xr12 and none on refs/heads/xr/main, so
a branch dispatch still cannot read a tag build's ccache.

rt_version description: notes that the 6.19.3-rt1 default is pinned to the 6.19
line, since the kernel_version note now steers operators at 6.18.y/7.1.y and
xr/source-sync.md pins the last proven RT lane at v7.0.1-rt2. A
variant=rt kernel_version=7.1.3 dispatch would otherwise silently pair a 7.1.3
tree with a 6.19.3 RT patch.

README: documents use_ccache (absent entirely), corrects "warm ~1h vs cold ~2h"
to the measured 43m40s warm vs 4h06m cold, corrects "save-always: true" to the
explicit actions/cache steps the workflow actually uses, and drops the false
claim that the two variants share a ccache -- the directory and the key are
both per-variant.

Release behaviour is unchanged: the variant filter still no-ops for non-dispatch
events and tag builds still build both variants.

Linear: TIN-4065
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant