Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,8 @@
# line-ending rule; the line above still applies to them.
*.patch -whitespace
*.diff -whitespace

# Third-party licence texts beside LICENSE (LICENSE-graphviz, LICENSE-expat)
# are kept byte for byte as their projects publish them, trailing whitespace
# included: a stripped copy is no longer the text the pinned archive carries.
LICENSE-* -whitespace
43 changes: 41 additions & 2 deletions .github/actions/cache/action.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: Job caches
description: >
The caches a canonical job keeps: hermetic's package store and, with go, the Go build and module caches, plus the linter's with linter-cache. Called three times per job, each mode once: attach (before setup-hermetic), restore (right after setup-hermetic, before the job runs a tool) and save (after the job's last command). On Windows they live in one disk image (windows-cache-image): attach restores and attaches it, since hermetic's root is inside it, restore does nothing, and save detaches and saves it. On Linux and macOS attach does nothing, restore restores the store and the Go caches as files, and save saves them. Every save is on the default branch only, read from the event (a scheduled run is on it by definition), never assumed by name, and only on a miss: a cache a pull request run saves is scoped to that pull request and restorable by nothing else, and a key is never overwritten (the cache budget, caches.yaml). The Go caches and the image accumulate, so their keys carry the UTC day and the default branch refreshes them at most once a day; the store is fully determined by its pins. What save needs, attach and restore leave in the job's environment, so save cannot use another key or path list, and a mode called twice in one job fails.
The caches a canonical job keeps: hermetic's package store, with go the Go build and module caches, plus the linter's with linter-cache, and in a repository that pins a Rust toolchain (rust-toolchain.toml) what the Rust lanes install under LIMEN_BUILD: rustup with the toolchain, cargo's caches, zig's. Called three times per job, each mode once: attach (before setup-hermetic), restore (right after setup-hermetic, before the job runs a tool) and save (after the job's last command). On Windows they live in one disk image (windows-cache-image): attach restores and attaches it, since hermetic's root is inside it, restore does nothing, and save detaches and saves it. On Linux and macOS attach does nothing, restore restores the store and the Go caches as files, and save saves them. Every save is on the default branch only, read from the event (a scheduled run is on it by definition), never assumed by name, and only on a miss: a cache a pull request run saves is scoped to that pull request and restorable by nothing else, and a key is never overwritten (the cache budget, caches.yaml). The Go caches and the image accumulate, so their keys carry the UTC day and the default branch refreshes them at most once a day; the store is fully determined by its pins. What save needs, attach and restore leave in the job's environment, so save cannot use another key or path list, and a mode called twice in one job fails.
inputs:
mode:
description: attach, restore or save.
Expand Down Expand Up @@ -37,7 +37,7 @@ runs:
with:
mode: attach
prefix: win-image-${{ runner.os }}-${{ runner.arch }}-
hash: ${{ hashFiles('.hermetic/hermetic.yaml', '.hermetic/aqua-checksums.json', '.limen/hermetic.yaml', '.limen/hermetic-registry.yaml', '**/go.sum') }}
hash: ${{ hashFiles('.hermetic/hermetic.yaml', '.hermetic/aqua-checksums.json', '.limen/hermetic.yaml', '.limen/hermetic-registry.yaml', '**/go.sum', 'rust-toolchain.toml', '**/Cargo.lock') }}

- if: inputs.mode == 'attach' && runner.os == 'Windows'
shell: bash
Expand Down Expand Up @@ -102,6 +102,31 @@ runs:
${{ steps.paths.outputs.prefix }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.paths.outputs.version }}-
${{ steps.paths.outputs.prefix }}-${{ runner.os }}-${{ runner.arch }}-

# The Rust toolchain and its caches, in a repository that pins one: what
# rust_env installs under LIMEN_BUILD (book/rust.md), rustup with the
# toolchain and its targets, cargo's registry and git caches, zig's and
# cargo-zigbuild's. No tool to run for the paths: LIMEN_BUILD is
# setup-hermetic's export. Keyed on the toolchain file and every
# Cargo.lock, plus the day: cargo's caches accumulate like Go's.
- id: rust-day
if: inputs.mode == 'restore' && runner.os != 'Windows' && hashFiles('rust-toolchain.toml') != ''
shell: bash
run: echo "day=$(date -u +%Y%m%d)" >> "$GITHUB_OUTPUT"

- id: rust
if: inputs.mode == 'restore' && runner.os != 'Windows' && hashFiles('rust-toolchain.toml') != ''
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
${{ env.LIMEN_BUILD }}/rustup
${{ env.LIMEN_BUILD }}/cargo
${{ env.LIMEN_BUILD }}/zig
key: rust-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('rust-toolchain.toml') }}-${{ hashFiles('**/Cargo.lock') }}-${{ steps.rust-day.outputs.day }}
restore-keys: |
rust-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('rust-toolchain.toml') }}-${{ hashFiles('**/Cargo.lock') }}-
rust-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('rust-toolchain.toml') }}-
rust-${{ runner.os }}-${{ runner.arch }}-

- if: inputs.mode == 'restore' && runner.os != 'Windows'
shell: bash
env:
Expand All @@ -112,6 +137,8 @@ runs:
BUILD: ${{ steps.paths.outputs.build }}
MOD: ${{ steps.paths.outputs.mod }}
LINT: ${{ steps.paths.outputs.lint }}
RUST_KEY: ${{ steps.rust.outputs.cache-primary-key }}
RUST_HIT: ${{ steps.rust.outputs.cache-hit }}
run: |
{
echo "LIMEN_CACHE_AQUA_KEY=$AQUA_KEY"
Expand All @@ -123,6 +150,8 @@ runs:
if [ -n "$MOD" ]; then echo "$MOD"; fi
if [ -n "$LINT" ]; then echo "$LINT"; fi
echo "LIMEN_CACHE_GO_PATHS_END"
echo "LIMEN_CACHE_RUST_KEY=$RUST_KEY"
echo "LIMEN_CACHE_RUST_HIT=$RUST_HIT"
} >> "$GITHUB_ENV"

# --- save: the default branch only, on a miss ---
Expand All @@ -138,6 +167,16 @@ runs:
path: ${{ env.LIMEN_CACHE_GO_PATHS }}
key: ${{ env.LIMEN_CACHE_GO_KEY }}

# The same three paths restore listed: LIMEN_BUILD is the same export.
- if: inputs.mode == 'save' && runner.os != 'Windows' && (github.event_name == 'schedule' || github.ref == format('refs/heads/{0}', github.event.repository.default_branch)) && env.LIMEN_CACHE_RUST_KEY != '' && env.LIMEN_CACHE_RUST_HIT != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: |
${{ env.LIMEN_BUILD }}/rustup
${{ env.LIMEN_BUILD }}/cargo
${{ env.LIMEN_BUILD }}/zig
key: ${{ env.LIMEN_CACHE_RUST_KEY }}

- if: inputs.mode == 'save' && runner.os == 'Windows' && (github.event_name == 'schedule' || github.ref == format('refs/heads/{0}', github.event.repository.default_branch)) && env.LIMEN_CACHE_IMAGE_KEY != '' && env.LIMEN_CACHE_IMAGE_HIT != 'true'
uses: ./.github/actions/windows-cache-image
with:
Expand Down
70 changes: 42 additions & 28 deletions .github/actions/setup-hermetic/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ runs:
# version. One digest covers every platform: the asset is verified
# against checksums.txt, and checksums.txt against this.
# renovate: depName=forkcloser/hermetic
HERMETIC_VERSION: v0.2.2
HERMETIC_CHECKSUMS_SHA256: da752591bf17d7fc5310c97b3bc47764fd876be387f099bc812c94790aee592a
# The job's own token, for `gh attestation verify` only, here and not
# exported: a read of GitHub's attestations API, before any
# branch-declared binary runs.
HERMETIC_VERSION: v0.2.3
HERMETIC_CHECKSUMS_SHA256: c7eb09712dc3d756bead96ae1286b53de275134d0012601ef37f57ec793f9431
# The job's own token, for gh alone (the release download and
# `gh attestation verify`), here and not exported: reads of GitHub's
# API, before any branch-declared binary runs.
GH_TOKEN: ${{ github.token }}
run: |
# Pin the root explicitly: hermetic's own default differs per
Expand Down Expand Up @@ -71,32 +71,46 @@ runs:
echo "hermetic ${HERMETIC_VERSION} already in the root (the cache image)"
else
tmp="$(mktemp -d)"
base="https://github.com/forkcloser/hermetic/releases/download/${HERMETIC_VERSION}"
# Retries for transient runner failures. Windows curl (schannel)
# refuses a handshake when the CA's revocation server is unreachable
# (exit 35, seen repeatedly): best-effort revocation still rejects a
# known-revoked certificate, and the checksums below are the real
# guard. schannel-only flag, so windows-only.
revoke=()
[ "$RUNNER_OS" != Windows ] || revoke=(--ssl-revoke-best-effort)
curl --proto '=https' --tlsv1.2 -fsSL --retry 5 --retry-delay 3 --retry-all-errors "${revoke[@]}" \
-o "${tmp}/checksums.txt" "${base}/checksums.txt"
# Through the releases API with the job's token, by gh: the release
# page's download URL is an anonymous fetch, which GitHub throttles
# per runner IP under load (a 504 on every lane of a run, through
# curl's own retries), and a token sent there is routed, not checked.
# gh retries nothing itself; --clobber so a retry starts over. A
# pattern no asset matches is not an error to gh while another
# matches, so each file is checked for.
attempt=0
until gh release download "${HERMETIC_VERSION}" --repo forkcloser/hermetic --dir "${tmp}" --clobber \
--pattern checksums.txt --pattern "${asset}"; do
attempt=$((attempt + 1))
[ "$attempt" -lt 5 ] || { echo "::error::hermetic ${HERMETIC_VERSION}: checksums.txt or ${asset} could not be downloaded"; exit 1; }
sleep 3
done
for f in checksums.txt "${asset}"; do
[ -f "${tmp}/${f}" ] || { echo "::error::hermetic ${HERMETIC_VERSION} has no release asset ${f}"; exit 1; }
done
# No single digest tool exists everywhere: linux and windows git-bash
# ship coreutils sha256sum, macOS ships perl shasum.
if command -v sha256sum >/dev/null 2>&1; then
echo "${HERMETIC_CHECKSUMS_SHA256} ${tmp}/checksums.txt" | sha256sum -c -
else
echo "${HERMETIC_CHECKSUMS_SHA256} ${tmp}/checksums.txt" | shasum -a 256 -c -
fi
curl --proto '=https' --tlsv1.2 -fsSL --retry 5 --retry-delay 3 --retry-all-errors "${revoke[@]}" \
-o "${tmp}/${asset}" "${base}/${asset}"
# ship coreutils sha256sum, macOS ships perl shasum. A mismatch names
# the computed digest and the size beside the expected one: a stale
# pin (the previous release's hash, which a Renovate bump carries
# until the checksum workflow refreshes it) and a wrong download read
# differently off the log.
verify_sha256() {
local expected="$1" file="$2" actual
if command -v sha256sum >/dev/null 2>&1; then
actual="$(sha256sum "$file" | cut -d' ' -f1)"
else
actual="$(shasum -a 256 "$file" | cut -d' ' -f1)"
fi
if [ "$actual" != "$expected" ]; then
echo "::error::sha256 mismatch for $(basename "$file") ($(wc -c <"$file" | tr -d ' ') bytes): expected ${expected}, computed ${actual}"
exit 1
fi
echo "sha256 ok: $(basename "$file")"
}
verify_sha256 "${HERMETIC_CHECKSUMS_SHA256}" "${tmp}/checksums.txt"
expected="$(awk -v a="$asset" '$2 == a {print $1}' "${tmp}/checksums.txt")"
[ -n "$expected" ] || { echo "::error::checksums.txt of hermetic ${HERMETIC_VERSION} does not list ${asset}"; exit 1; }
if command -v sha256sum >/dev/null 2>&1; then
echo "${expected} ${tmp}/${asset}" | sha256sum -c -
else
echo "${expected} ${tmp}/${asset}" | shasum -a 256 -c -
fi
verify_sha256 "${expected}" "${tmp}/${asset}"
# GitHub's build provenance, on every runner through gh: the asset was
# built by hermetic's release workflow from the tagged commit.
gh attestation verify "${tmp}/${asset}" --repo forkcloser/hermetic
Expand Down
13 changes: 13 additions & 0 deletions .github/actions/windows-cache-image/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,16 @@ runs:
$build = ($env:GITHUB_WORKSPACE -replace '\\', '/') + '/.build'
New-Item -ItemType Directory -Force -Path C:\vcache\golangci, $build | Out-Null
New-Item -ItemType Junction -Path "$build/cache" -Target C:\vcache\golangci | Out-Null
# The Rust lanes' state the same way, in a repository that pins a
# toolchain: rustup, cargo and zig under LIMEN_BUILD are junctions
# into the image (book/rust.md); rustup and cargo take a junction
# as the directory it is.
if (Test-Path "$env:GITHUB_WORKSPACE/rust-toolchain.toml") {
foreach ($d in 'rustup', 'cargo', 'zig') {
New-Item -ItemType Directory -Force -Path "C:\vcache\$d" | Out-Null
New-Item -ItemType Junction -Path "$build/$d" -Target "C:\vcache\$d" | Out-Null
}
}
"LIMEN_BUILD=$build" >> $env:GITHUB_ENV
"GOCACHE=C:\vcache\go-build" >> $env:GITHUB_ENV
"GOMODCACHE=C:\vcache\mod" >> $env:GITHUB_ENV
Expand All @@ -98,6 +108,9 @@ runs:
run: |
$ErrorActionPreference = 'Stop'
Remove-Item "$env:LIMEN_BUILD/cache" -Force
foreach ($d in 'rustup', 'cargo', 'zig') {
if (Test-Path "$env:LIMEN_BUILD/$d") { Remove-Item "$env:LIMEN_BUILD/$d" -Force }
}
Dismount-DiskImage -ImagePath C:\vhd\cache.vhdx | Out-Null

- if: inputs.mode == 'save'
Expand Down
22 changes: 10 additions & 12 deletions .github/release.yml
Original file line number Diff line number Diff line change
@@ -1,22 +1,20 @@
# DO NOT EDIT MANUALLY: content-pinned by limen, reset on drift.
#
# How GitHub groups a release's notes, which are the titles of the pull
# requests merged since the previous tag (goreleaser's `use: github-native`).
# A pull request lands in the FIRST category it matches, so the catch-all
# must exclude what a later category claims.
# How GitHub writes a release's notes, the titles of the pull requests merged
# since the previous tag (goreleaser's `use: github-native`). Excluded labels
# and authors are left out of the notes; a pull request lands in the FIRST
# category it matches.
changelog:
exclude:
labels:
- dependencies
- internal
authors:
- renovate[bot]
categories:
- title: Breaking changes
labels:
- breaking
- title: Changes
labels:
- '*'
exclude:
labels:
- dependencies
authors:
- renovate[bot]
- title: Dependencies
labels:
- '*'
53 changes: 5 additions & 48 deletions .github/workflows/caches.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# DO NOT EDIT MANUALLY: content-pinned by limen, reset on drift.
# DO NOT EDIT MANUALLY: content-pinned by limen, reset on drift; the one line
# Renovate moves is the `uses:` sha.
#
# The cache budget. GitHub keeps 10 GB of Actions caches per repository and
# evicts the least recently used beyond it, the current entries included, so
Expand All @@ -9,8 +10,8 @@
# eviction (and the caches that accumulate, Go's and the fuzz corpus, start
# one a day). This keeps one generation per key on the default branch: after
# each ci run there, and once a day, every entry under a canonical prefix that
# is not the newest of its family is deleted. GitHub's own token, `gh` from
# the runner image; no checkout, no third-party actions.
# is not the newest of its family is deleted. The steps are limen's reusable
# workflow.
name: caches

on:
Expand All @@ -31,53 +32,9 @@ permissions: {}
concurrency:
group: ${{ github.workflow }}

defaults:
run:
shell: bash

jobs:
prune:
runs-on: ubuntu-24.04
timeout-minutes: 10
uses: farcloser/limen/.github/workflows/limen-caches.yaml@56ef4e8a97ae7666a759b353a94be77e0ea05d12 # v0.26.0
permissions:
# Listing and deleting caches is an Actions write.
actions: write
steps:
- name: One generation per key on the default branch
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
run: |
# The default branch's entries, newest first, the branch read from
# the repository, never assumed by name (a scheduled run has no
# event naming it). Listed before the loop so a refused token or a
# failed call fails the step: fed through a process
# substitution, the loop would read nothing and pass, and the prune
# would silently not happen. A family is a key without its UTC day
# and content hash (and, for the Go caches, without the Go version,
# so a toolchain bump retires the old version's entry too), plus the
# entry's version: GitHub tells entries apart by key and version,
# and the version follows the cached paths, so two entries caching
# different paths under one key are two families, never each
# other's older generation. The first of each family is kept, the
# rest deleted. Only the canonical prefixes; whatever else is in the
# store is not ours to judge.
default_branch=$(gh api "repos/$GH_REPO" --jq .default_branch)
entries=$(gh cache list --ref "refs/heads/$default_branch" --limit 500 --sort created_at --order desc \
--json id,key,version,sizeInBytes --jq '.[] | "\(.id)\t\(.key)\t\(.version)\t\(.sizeInBytes)"')
kept=
while IFS=$'\t' read -r id key version size; do
case "$key" in
aqua-pkgs-*|go-*|fuzz-corpus-*|win-image-*) ;;
*) continue ;;
esac
family=$(printf '%s' "$key" | sed -E 's/-[0-9]{8}$//; s/-[0-9a-f]{64}$//; s/-+$//; s/^(go-(lint-)?[^-]+-[^-]+)-go[0-9][^-]*$/\1/')
family="$family $version"
if printf '%s' "$kept" | grep -qxF -- "$family"; then
printf 'delete %s (%d MB)\n' "$key" "$((size / 1000000))"
gh cache delete "$id"
else
kept="$kept$family"$'\n'
printf 'keep %s\n' "$key"
fi
done <<< "$entries"
2 changes: 1 addition & 1 deletion .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ concurrency:

jobs:
limen:
uses: ./.github/workflows/limen-verify.yaml
uses: farcloser/limen/.github/workflows/limen-verify.yaml@56ef4e8a97ae7666a759b353a94be77e0ea05d12 # v0.26.0
permissions:
contents: read
# The binary links Virtualization.framework through cgo and clones disks
Expand Down
Loading
Loading