ci: extract the repeated ccache and apt scaffolding from native_release

[leehack]

Remaining work from #49, after #83 took the CUDA version and Android CPU variant lists. The issue's opening line is stale: native_release.yml is 1470 lines on main, not 1242; this brings it to 1365.

native_release.yml is workflow_dispatch-only, so PR CI never runs it. Nothing here was verified by reading the diff — each extraction was executed and compared against origin/main, as recorded below.

1. ccache scaffolding, 4 lanes

.github/actions/setup-ccache (restore + prime) and .github/actions/ccache-stats (stats + summary). Two actions, not one: the trio brackets the build step, and a composite action cannot contribute a post: step.

Before (android):

- name: Restore ccache
  uses: actions/cache@v6
  with:
    path: ${{ env.CCACHE_DIR }}
    key: ccache-android-${{ matrix.android_abi }}-${{ matrix.backend }}-${{ github.run_id }}
    restore-keys: |
      ccache-android-${{ matrix.android_abi }}-${{ matrix.backend }}-
      ccache-android-${{ matrix.android_abi }}-
- name: Prime ccache
  run: |
    mkdir -p "${CCACHE_DIR}"
    ccache --max-size "${CCACHE_MAXSIZE}"
    ccache --zero-stats

After:

- name: Set up ccache
  uses: ./.github/actions/setup-ccache
  with:
    cache-dir: ${{ env.CCACHE_DIR }}
    max-size: ${{ env.CCACHE_MAXSIZE }}
    key-prefix: ccache-android-${{ matrix.android_abi }}-${{ matrix.backend }}
    extra-restore-keys: ccache-android-${{ matrix.android_abi }}-

Proof. A script loaded origin/main's workflow and this branch's, expanded the composite action's key/restore-keys/path against each call site's inputs, and compared: cache action version, path, key, restore-key list, prime script, stats script, summary script, both if: always() conditions, the position of setup before / stats after Build, and the remaining step list. All identical for all four lanes, plus all 15 concrete matrix expansions:

ccache-android-arm64-v8a-vulkan-<RUN_ID>  restore: ccache-android-arm64-v8a-vulkan-, ccache-android-arm64-v8a-
ccache-apple-ios-sim-x86_64-<RUN_ID>      restore: ccache-apple-ios-sim-x86_64-
ccache-linux-arm64-blas-<RUN_ID>          restore: ccache-linux-arm64-blas-, ccache-linux-arm64-
ccache-linux-x64-hip-<RUN_ID>             restore: ccache-linux-x64-hip-

The empty extra-restore-keys line on the apple and hip lanes is dropped, not passed as a wildcard. In actions/cache@v6 source, restoreImpl.ts reads the input as utils.getInputAsArray(Inputs.RestoreKeys), and getInputAsArray is .split("\n").map(s => …trim()).filter(x => x !== "") — the blank line is filtered out.

The shell: the composite steps get

Composite steps must name a shell, so the extracted scripts carry shell: bash where the inline originals carried nothing. Grounded in runner v2.337.0 — the version that ran this workflow, per line 1 of the log for run 34969208714:

• ScriptHandlerHelpers._defaultArguments maps "bash" to --noprofile --norc -e -o pipefail {0} and "sh" to -e {0}.
• ScriptHandler with no shell: sets shellCommand = "sh" and resolves the path as which("bash") ?? which("sh"). For a step the runner executes directly that lands on the bash binary invoked with -e {0}; a step inside a container: gets the unresolved sh, which the log prints verbatim as shell: sh -e {0}.

That same run's log shows exactly this split. 79 steps report shell: /usr/bin/bash -e {0} and 34 report shell: /bin/bash -e {0} — all plain run: steps. 23 report a --noprofile --norc -e -o pipefail shell (11 /usr/bin/bash, 6 /bin/bash on macOS, 5 Git\bin\bash.EXE on Windows, 1 bare bash in the ROCm container), and every one of those 23 belongs to a step whose script opens set -euo pipefail — the single shell: bash step of ./.github/actions/checkout-llama-ref, whose 8 call sites expand to 23 job instances. That is origin/main's only local composite action. The run declares 177 shells in all; the balance is 33 PowerShell steps on the Windows lanes and 8 shell: sh -e {0}, every one of those 8 in build-linux-hip.

Named delta: build-linux-hip moves from dash to bash

This is the one behaviour change in the PR, and it lands on a workflow_dispatch path PR CI never runs.

For build-android, build-apple and build-linux the binary is unchanged — their inline steps already resolved to bash — and the only additions are --noprofile/--norc, inert for a non-interactive script read from a file, and -o pipefail. build-linux-hip is the exception: it runs in container: rocm/dev-ubuntu-22.04:6.1.2, where /bin/sh is dash. The run log reports shell: sh -e {0} for all 8 of its no-shell: steps; three of those are the ones extracted here — mkdir -p "${CCACHE_DIR}", ccache --show-stats || true, and the { … } >> "$GITHUB_STEP_SUMMARY" block. Under the composite actions those three become bash --noprofile --norc -e -o pipefail {0}: a different interpreter, plus pipefail.

Keeping dash for that lane alone would mean an expression-valued shell: in the composite (shell: ${{ inputs.shell }}). That is not adopted here: an unevaluated expression would break the composite in all four ccache lanes rather than changing the shell in one, and that risk asymmetry did not justify the change. Instead the move is disclosed, and both halves of it were checked by running them:

• bash exists in the image. docker run --rm --platform linux/amd64 rocm/dev-ubuntu-22.04:6.1.2 sh -c 'which bash; bash --version | head -1; readlink -f /bin/sh' prints /usr/bin/bash, GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu), /usr/bin/dash. The step cannot fail for want of a shell.
• pipefail cannot bite. It changes only the exit status of a pipeline, and none of the three scripts contains one. grep -n '|' over both new action files returns five lines, not two: three block-scalar indicators (setup-ccache:25 restore-keys: |, setup-ccache:33 run: |, ccache-stats:17 run: |) and two logical-OR operators inside extracted script text (ccache-stats:12 and ccache-stats:21, both ccache --show-stats || true). || is not a pipe.
• The scripts behave the same either way. All three were run inside that image, with ccache installed, under sh -e and under bash --noprofile --norc -e -o pipefail. Exit status, combined stdout/stderr, and the bytes written to $GITHUB_STEP_SUMMARY were identical for all three.

tests/test_native_release_dedup.py::CcacheStepShellTests pins the result: every lane's container: and the default shell its steps had before extraction, the single shell the extracted steps declare, and an assertion that no extracted script grows a pipeline — the thing that would make -o pipefail load-bearing. Each of those four assertions was confirmed to fail under a deliberate mutation.

The nested actions/cache save still runs

Traced through the same runner v2.337.0 source, not release notes:

1. ActionRunner.RunAsync — when the nested action's handlerData.HasPost and the stage is Pre or Main, it builds an ActionRunner with Stage = ActionRunStage.Post, names it $"Post {this.DisplayName}", and calls ExecutionContext.RegisterPostJobStep.
2. ExecutionContext.RegisterPostJobStep — when the context IsEmbedded (the step sits inside a composite), it records Root.EmbeddedStepsWithPostRegistered[action.Id] = condition and returns.
3. CompositeActionHandler.RunAsync at ActionRunStage.Post — walks Data.PostSteps and runs exactly those whose id is in EmbeddedStepsWithPostRegistered, restoring the recorded condition. CompositeActionExecutionData.HasPost is PostSteps.Count > 0, so Set up ccache itself registers a post step.

The one local-action restriction on that path is pre, not post: ActionRunner warns `pre` execution is not supported for local action and has no post equivalent.

Windows sccache stays separate. It shares no scaffolding with the ccache lanes: setup is a third-party action (mozilla-actions/sccache-action), storage is the GHA cache service via SCCACHE_GHA_ENABLED rather than actions/cache over a directory, there is no --max-size/--zero-stats priming and no stats summary, and it is gated on matrix.arch == 'x64' with an arm64 step that clears the launcher instead. Folding it in would mean one action with two disjoint bodies.

2. apt retry helpers

tools/apt_retry.sh, sourced by the two Install build deps steps, which run after checkout. It picks sudo by uid, which is what let the hosted-runner copy and the ROCm container copy merge:

_apt_get() {
  if [ "$(id -u)" -eq 0 ]; then
    apt-get -o Acquire::Retries=3 "$@"
  else
    sudo apt-get -o Acquire::Retries=3 "$@"
  fi
}

How many declarations, exactly

grep -c on .github/workflows/native_release.yml:

	apt_get_update() {	apt_get_install() {	total
origin/main	2	3	5
this branch	1	1	2

The five on main sit in three steps: build-linux → Install build deps (2), build-linux-hip → Install build deps (1), build-linux-hip → Install Git (2). Three declarations go; the two in Install Git stay, because that step runs before actions/checkout, so tools/ does not exist yet and neither a sourced script nor a local composite action is reachable. That step is byte-identical to origin/main — 26 lines each side, md5 ca597d1ce748c92901d10c030740c619, against origin/main at e56add1:

git show origin/main:.github/workflows/native_release.yml | sed -n '659,684p' | openssl dgst -md5
sed -n '577,602p' .github/workflows/native_release.yml | openssl dgst -md5

The android lane's plain sudo apt-get update && sudo apt-get install -y ninja-build ccache is left alone — wiring it to the retry helper would add behaviour, not preserve it.

build-linux, all 5 matrix rows

Both versions of the step were extracted from YAML, matrix-expanded, and executed in ubuntu:24.04 as uid 1001 under bash -e — the shell and user the run log shows for this lane — with apt-get/sudo/dpkg/tee/sleep/python3/nvcc stubbed on PATH to log argv and DEBIAN_FRONTEND. All five logs are byte-identical before and after. Example, linux/arm64/vulkan:

sudo dpkg --add-architecture arm64 [DEBIAN_FRONTEND=<unset>]
sudo apt-get -o Acquire::Retries=3 update [DEBIAN_FRONTEND=<unset>]
sudo apt-get -o Acquire::Retries=3 install -y ccache ninja-build pkg-config make [DEBIAN_FRONTEND=<unset>]
sudo apt-get -o Acquire::Retries=3 install -y gcc-aarch64-linux-gnu g++-aarch64-linux-gnu qemu-user [DEBIAN_FRONTEND=<unset>]
sudo apt-get -o Acquire::Retries=3 install -y libvulkan-dev glslc [DEBIAN_FRONTEND=<unset>]
sudo apt-get -o Acquire::Retries=3 install -y libvulkan-dev:arm64 [DEBIAN_FRONTEND=<unset>]

The two arm64 rows also had the file they pipe into sudo tee captured: byte-identical before and after, 664 bytes, md5 3a2ff1e0fe891d31dc1a913b9f0c5849.

Textually, that step is byte-identical to origin/main from the apt_get_update call to the end of the step — 42 lines, md5 47955c66265077baec9988dde4fbe73b on both sides. Only the head of the step changed.

build-linux-hip runs under sh, not bash

The run log reports shell: sh -e {0} for this job's plain run: steps. That follows from the source above: inside a container validateShellOnHost is false, the runner skips the which("bash") lookup, and the default stays the literal sh — the image's /bin/sh, which on the rocm/dev-ubuntu-22.04:6.1.2 base is dash.

So tools/apt_retry.sh is held to POSIX, not bash. shellcheck is clean under -s sh, -s bash and -s dash, and this lane's equivalence run below was executed under sh -e in ubuntu:22.04.

Named delta: DEBIAN_FRONTEND on the hip lane

An earlier revision of this branch moved DEBIAN_FRONTEND=noninteractive from the apt-get command prefix to the step's env:. That exported it for the whole step, including python3 -m pip install --upgrade cmake and cmake --version. Reverted — it is now a prefix on the call, so only the helper's apt-get sees it and tools/apt_retry.sh stays frontend-agnostic (the hosted build-linux lane must not get it):

. tools/apt_retry.sh
DEBIAN_FRONTEND=noninteractive apt_get_install build-essential binutils ccache ...
python3 -m pip install --upgrade cmake

Reverting rather than documenting, because the prefix form is exactly equivalent where it counts and costs nothing. Executed in ubuntu:24.04 under sh -e as root, before vs after, with stubs logging argv and DEBIAN_FRONTEND — identical:

apt-get -o Acquire::Retries=3 install -y build-essential binutils ccache ... [DEBIAN_FRONTEND=noninteractive]
python3 -m pip install --upgrade cmake [DEBIAN_FRONTEND=<unset>]
cmake --version [DEBIAN_FRONTEND=<unset>]

A prefixed assignment on a function call could have leaked into the rest of the step under POSIX rules; it does not, in either shell that can run this step — checked explicitly under both dash and bash.

One residual, stated plainly. With every apt-get forced to fail, both versions produce the same ladder — 3 attempts, sleep 15, sleep 30, exit 1 — and every apt-get sees DEBIAN_FRONTEND=noninteractive in both. The only difference in the whole comparison is that the ladder's two sleep calls now also inherit DEBIAN_FRONTEND=noninteractive, where on main they saw it unset. sleep(1) reads no such variable. This is the entire behaviour delta of item 2.

3. ubuntu.sources divergence

tools/docker/ubuntu.sources hardcoded noble; the workflow emitted the same three stanzas from a printf heredoc with ${VERSION_CODENAME}. Both now render tools/docker/ubuntu.sources.template from their own /etc/os-release, so the builder image follows its base image instead of a pinned codename.

Proof. Three renderings, all 664 bytes, all md5 3a2ff1e0fe891d31dc1a913b9f0c5849, all byte-identical to origin/main's committed tools/docker/ubuntu.sources:

• the workflow's old inline heredoc, captured from the sudo tee stub in the build-linux arm64 runs above;
• the workflow's new sed … ubuntu.sources.template, captured the same way in the same runs;
• a real docker build on ubuntu:24.04 with the Dockerfile's COPY + RUN . /etc/os-release && sed …, then cat out of the built image.

4. Inline bash validators — not done

Left out deliberately. The android validator reaches into NDK tool discovery, mktemp/trap cleanup and matrix conditionals; porting it to tools/validate_*.py could not be proven equivalent by execution the way items 1-3 were, and the only way to exercise it is a release dispatch. Items 1-3 are the issue's explicit suggested fixes; this one is the trailing aside.

Checks

• python3 -m unittest discover -s tests -p 'test_*.py' — 157 pass (137 on origin/main; 20 new).
• python3 -m unittest discover -s tests -p '*_test.py' — 18 pass.
• actionlint — 26 findings on origin/main, 26 here, all in native_release.yml (pre-existing shellcheck noise on the PowerShell steps); the diff of findings with line/column stripped is empty. It does read both new composite actions and validate every input name at all 8 call sites — misspelling key-prefix at one call site produces both input "key-prefixx" is not defined in action "Setup ccache" and missing input "key-prefix" which is required.
• shellcheck tools/apt_retry.sh — clean under -s sh, -s bash and -s dash.
• The tests were mutation-checked. Seven injected regressions (wrong key prefix, dropped restore key, wrong summary label, unsourced helper, broken composite key, Dockerfile sed drift, re-hardcoded codename) each fail at least one test. Three more, added with the per-lane assertions:
  • swap key-prefix between build-linux and build-linux-hip — the old set-comparing test passed this; the per-lane test fails it.
  • swap label between build-android and build-apple — likewise.
  • put DEBIAN_FRONTEND back in the hip step's env: — fails test_container_step_installs_noninteractively_without_exporting_it.