diff --git a/.github/actions/setup-env/action.yml b/.github/actions/setup-env/action.yml index 42d9c6b6..e7b1b76b 100644 --- a/.github/actions/setup-env/action.yml +++ b/.github/actions/setup-env/action.yml @@ -14,29 +14,33 @@ # # The cache inventory (full architecture: .github/workflows/README.md): # -# 1. Go module + build cache keyed on every go.sum, `gobuild-v2-` key -# family. `restore-keys` keeps the build cache warm across go.sum -# bumps — avoids setup-go's exact-key-only cache (actions/setup-go#357). -# `go-cache-suffix` partitions the cache per job: the unit, -# integration, and e2e jobs compile with different flags -# (-race/-cover/-coverpkg/-tags), so a shared entry would only ever -# be warm for whichever job saved it. Cross-suffix restore-keys -# still share the (identical) module cache on a cold start. -# 2. golangci-lint binary + analysis cache keyed on Makefile + +# 1. Go modules, `gomod-v1-` key family, keyed on every go.mod + go.sum +# and NOT partitioned per job — ~/go/pkg/mod is a pure function of +# those files, so one entry serves every Go job, stored once rather +# than per flavor (~1.6 GB on disk; 0.48 GB as a stored archive on a +# cold save, drifting up as superseded module versions accumulate +# through the restore -> save chain). +# 2. Go build objects, `gobuild-v3-` key family, same key inputs, +# partitioned by `go-cache-suffix`: the unit, integration and e2e jobs +# compile with different flags (-race/-cover/-coverpkg/-tags), so a +# shared entry would only ever be warm for whichever job saved it. +# `restore-keys` keeps it warm across dependency bumps — avoids +# setup-go's exact-key-only cache (actions/setup-go#357). +# 3. golangci-lint binary + analysis cache keyed on Makefile + # .golangci.yml. Analysis cache is the win (~10s warm vs ~90s). # Only the lint job needs it. -# 3. pnpm store (path resolved at runtime) keyed on the root +# 4. pnpm store (path resolved at runtime) keyed on the root # lockfile — the pnpm workspace projects share one lockfile + # store. Path is dynamic because pnpm's documented default # ~/.local/share/pnpm/store only applies when $HOME and the project # tree share a mount; on some runners it falls back to a # workspace-relative path. Hard-coding the default silently fails # the save with a Path Validation Error. -# 4. Playwright browser cache (~/.cache/ms-playwright) keyed on the +# 5. Playwright browser cache (~/.cache/ms-playwright) keyed on the # root lockfile. ~130 MB Chromium download otherwise re-fetched # every docs build (rehype-mermaid renders via headless Chrome). # Only the docs-build job needs it. See #132. -# 5. Astro content-collection cache (docs/.astro/) keyed on the root +# 6. Astro content-collection cache (docs/.astro/) keyed on the root # lockfile + astro.config.mjs. Speeds up warm `astro check` / # `astro build` — unchanged content skips the parse + transform # pipeline. See #132. @@ -48,7 +52,7 @@ inputs: description: "Set up the Go toolchain + the Go module/build cache" default: "true" go-cache-suffix: - description: "Per-job Go build-cache partition, e.g. '-unit' (different jobs compile with different flags). Empty = the shared default key." + description: "REQUIRED when `go` is true (the default) — the action fails the job without one. Per-job Go build-cache partition, e.g. '-unit' (different jobs compile with different flags); an empty suffix yields a restore-key that prefix-matches every other flavor's entry, the cross-flavor restore the gomod-v1/gobuild-v3 split exists to avoid. A job that doesn't build Go passes `go: \"false\"` instead. The shared part of the cache is gomod-v1, which needs no suffix. The default stays empty because a composite action's `required:` isn't enforced by the runner — the guard step is." default: "" golangci: description: "Cache the golangci-lint binary + analysis cache (lint job only)" @@ -75,26 +79,93 @@ outputs: runs: using: composite steps: - # Key version (v2): bumped when the cache's expected CONTENTS change - # shape — v2 added the go toolchain itself (~/go/pkg/mod/golang.org/ - # toolchain, see the GOTOOLCHAIN note below). Saves only happen on an - # exact-key miss, so without a bump the pre-change entry would - # exact-hit forever and the new content would never be saved. + # Fail loudly on the one way to misuse the split. An empty suffix on a Go + # job yields restore-key `gobuild-v3--go-`, which prefix-matches every + # flavor's entry AND publish-dev's `-release` one — restoring + # cross-compiled or wrong-flag objects nothing can hit, then saving an + # extra entry against the sizing policy. Documented in the input + # description and the README's "Adding a job" checklist; enforced here + # so a new job can't inherit the empty default silently. + - name: Require a go-cache-suffix on Go jobs + if: ${{ inputs.go == 'true' && inputs.go-cache-suffix == '' }} + shell: bash + run: | + echo "::error title=setup-env::go-cache-suffix is required when go is true (the default)." \ + "An empty suffix cross-matches every other flavor's build cache." \ + "If this job builds Go: pass a fresh suffix for new compile flags, or an" \ + "existing flavor's if it compiles identically. If it does not build Go:" \ + "pass go: \"false\". See .github/workflows/README.md." + exit 1 + + # The module cache is UNSUFFIXED on purpose. ~/go/pkg/mod is a pure + # function of go.mod + go.sum — byte-identical for every compile + # flavor — so folding it into the suffixed build cache stored that tree + # five times over, once per -lint/-unit/-integration/-e2e-cov/-cov job: + # five entries of ~0.9-1.2 GB stored each, ~5.2 GB per generation. + # Against a 10 GB repo cap that means two live generations overflowed + # it, and GitHub started LRU-evicting warm entries mid-run. Split out, + # it is stored once and a measured generation is 1.05 GB — 0.48 module + # plus 0.57 across the five build entries (#443). Sizes here are + # stored-archive bytes / 2^30, the unit the README's usage check + # prints. + # + # All Go jobs miss this key together on a dependency bump and all try to + # save; the backend keeps the first and the rest log a benign "already + # exists" (same trade-off as the rest of the inventory, see header). + - uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + if: ${{ inputs.go == 'true' }} + id: gomod-cache + with: + path: ~/go/pkg/mod + # go.mod is in the key, not just go.sum, because the GOTOOLCHAIN=auto + # toolchain rides in ~/go/pkg/mod/golang.org/toolchain (no setup-go) + # and go.sum records no entry for it. Raising go.mod's `go` directive + # therefore changes which toolchain belongs in this cache while + # leaving go.sum untouched — a go.sum-only key would exact-hit, and + # because saves fire only on an exact-key MISS the freshly fetched + # toolchain would never be saved, re-downloading on every run. + key: gomod-v1-${{ runner.os }}-${{ hashFiles('**/go.mod', '**/go.sum') }} + # A stale generation is still worth restoring: a bump moves a handful + # of modules, so most of the tree is reusable and `go mod download` + # fetches only the delta. + restore-keys: | + gomod-v1-${{ runner.os }}- + + # Build objects only. These genuinely differ per job — unit, + # integration and e2e compile with different flags + # (-race/-cover/-coverpkg/-tags) — so a shared entry would only ever be + # warm for whichever job saved it last. + # + # Key version (v3): bumped because the cache's CONTENTS changed shape + # (~/go/pkg/mod moved out, above). Saves only happen on an exact-key + # miss, so without a bump the old v2 entry would exact-hit forever and + # the new, smaller content would never be saved. - uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 if: ${{ inputs.go == 'true' }} id: gobuild-cache with: - path: | - ~/go/pkg/mod - ~/.cache/go-build - key: gobuild-v2-${{ runner.os }}-go${{ inputs.go-cache-suffix }}-${{ hashFiles('**/go.sum') }} - # Same-suffix prefix first (this job's flavor across go.sum bumps), - # then the bare prefix as a cold-start fallback — it matches any - # other job's suffixed entry, which still carries the shared module - # cache even if its build objects don't apply. + path: ~/.cache/go-build + # go.mod belongs in this key for its own reason, not just symmetry + # with gomod-v1: the compiler's build ID feeds every action hash, so + # a toolchain bump invalidates every object in here. Keyed on go.sum + # alone that bump exact-hits, nothing restored is reusable, and the + # freshly built objects are never saved (saves fire only on a miss) + # — so every run recompiles cold *and* restores objects nothing can + # hit, until some unrelated go.sum change rotates the key. + # + # This closes the go-directive half only. GOTOOLCHAIN=auto uses the + # LOCAL toolchain whenever it satisfies the directive, so a runner + # image bumping its bundled Go swaps the compiler with neither + # go.mod nor go.sum moving — same pathology, past both keys. If you + # see unexplained cold build caches, that's the cause: rotate the + # v prefix to force a save. + key: gobuild-v3-${{ runner.os }}-go${{ inputs.go-cache-suffix }}-${{ hashFiles('**/go.mod', '**/go.sum') }} + # Same-suffix only. The bare-prefix fallback the v2 key carried + # existed to pick up the shared module cache from another job's + # entry; that job is now gomod-v1's, and another flavor's build + # objects are not reusable here. restore-keys: | - gobuild-v2-${{ runner.os }}-go${{ inputs.go-cache-suffix }}- - gobuild-v2-${{ runner.os }}-go- + gobuild-v3-${{ runner.os }}-go${{ inputs.go-cache-suffix }}- - uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 if: ${{ inputs.golangci == 'true' }} diff --git a/.github/workflows/README.md b/.github/workflows/README.md index e6867cd8..cc5c4d62 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -119,13 +119,20 @@ Break one of these knowingly or not at all. runs the PR tree with no secrets beyond a read-mostly `GITHUB_TOKEN`. Fork PRs: secrets are absent and `docs-preview` skips itself. -6. **Caches are owned end-to-end by `setup-env`** +6. **`ci.yml`'s caches are owned end-to-end by `setup-env`** ([.github/actions/setup-env](../actions/setup-env/action.yml)): each cache is a nested `actions/cache` step that restores inline and saves automatically at job end on an exact-key miss. No save steps in `ci.yml`. Trade-offs accepted: failed jobs don't save (restore-keys cushion the next run), and concurrent same-key misses produce benign - "already exists" warnings. + "already exists" warnings. **Two cache steps live outside it**, both in + `publish-dev.yml`, because that workflow doesn't use `setup-env` at all + (it runs GoReleaser, not the test suites): a bare `actions/cache` owning + the release build cache, and an `actions/cache/restore` that *reads* + `gomod-v1` and owns nothing. Those two are the only `actions/cache*` + steps outside the composite — keep it that way. A workflow that needs + the shared module tree reads it restore-only; writing it belongs to the + `ci.yml` jobs that run a full `go mod download`. ## Coverage publishing @@ -183,11 +190,54 @@ Queue settings live in the `main branch protection` ruleset's | Cache | Key | Saved by | Notes | |---|---|---|---| -| Go modules + build | `gobuild-v2--go-` | every Go job (own suffix) | Suffix partitions by compile flavor (`-lint`, `-unit`, `-integration`, `-e2e-cov`, `-cov`). v2 = the GOTOOLCHAIN=auto toolchain rides in `~/go/pkg/mod` (no setup-go). | +| Go modules | `gomod-v1--` | every `ci.yml` Go job via `setup-env` (shared) | `~/go/pkg/mod`, **unsuffixed** — a pure function of `go.mod` + `go.sum`, so one entry serves every job (~1.6 GB on disk; 0.48 GB stored on a cold save, drifting up as superseded versions accumulate). The GOTOOLCHAIN=auto toolchain rides in here too (no setup-go), which is why `go.mod` is in the key — a `go` directive bump changes the required toolchain without touching `go.sum`. | +| Go build objects | `gobuild-v3--go-` | every `ci.yml` Go job via `setup-env` (own suffix) | `~/.cache/go-build` only — 25–152 MB stored per flavor. Suffix partitions by compile flavor (`-lint`, `-unit`, `-integration`, `-e2e-cov`, `-cov`), which compile with different flags. `go.mod` is in the key for its own reason — the compiler's build ID keys every object, so a toolchain bump invalidates all of them. | | golangci binary + analysis | `golangci--` | lint | Analysis cache: ~10s warm vs ~90s. `.bin` also carries shellcheck + actionlint. | | pnpm store | `pnpm--` | any node job on miss | Store path resolved from pnpm at runtime. docs-build prunes before its save on a key rotation. | | Playwright Chromium | `playwright--` | docs-build | rehype-mermaid renders via headless Chrome at docs build. | | Astro content collections | `astro--` | lint / docs-build | Warm `astro check`/`build` skip unchanged content. | +| Go build objects (release) | `gobuild-v3--go-release-` | publish-dev (hand-rolled, not `setup-env`) | `~/.cache/go-build` from GoReleaser's 8-target cross-compile (~0.5 GB). Same family and key inputs as the CI flavors, `-release` suffix because cross-compiled objects share nothing with the native-only ones. Worth ≈2.5–7 min on every push to main (mean delta ≈4.8 min). | +| Go modules (release read) | `gomod-v1--` | nobody — **restore-only** | `publish-dev` reads `ci.yml`'s shared entry from `main`'s scope via `actions/cache/restore`, so its cross-compile isn't slowed by a cold module tree. No post-step save, so 0 GB of budget and no risk of a partial write to the shared key. | +| CodeQL DB + deps | `codeql-dependencies-*`, `codeql-overlay-base-database-*` | GHAS default setup | **Not ours** — minted by GitHub's default CodeQL setup, not by any workflow in this repo, and not configurable here. ~0.4 GB. Listed so the budget arithmetic below is honest. | + +Deliberately **not** cached: `actions/setup-go`'s bundled cache +(`cache: false` in `publish-dev.yml`, `release.yml` and +`goreleaser-validate.yml`) — for different reasons per job. + +It stores `~/go/pkg/mod` **and** `~/.cache/go-build` under one entry +(~1 GB stored), keyed on the root `go.mod` — setup-go hashed `go.sum` +through v6.2.0 and `go.mod` from v6.3.0, see +[actions/setup-go#705](https://github.com/actions/setup-go/pull/705) — so +roughly half of it re-stores the module tree `gomod-v1` already keeps once. +`publish-dev.yml` opts out of that entry and caches the half that pays for +itself on its own key (`gobuild-v3--go-release-`, +~0.5 GB): its GoReleaser step takes 36–246 s warm versus 401–446 s cold, so +dropping the build objects outright would cost roughly 2.5–7 minutes on +every push to main (mean delta ≈4.8 min across those runs). Those timings were measured with setup-go's bundled entry, which +also held `~/go/pkg/mod` — so `publish-dev` additionally *restores* (never +saves) `gomod-v1` from `main`'s scope, keeping the module tree warm too. +Without that restore the job would re-download ~112 MB per push and land +above the warm range this table quotes. + +`release.yml` keeps the plain opt-out — no re-cache. After this change +nothing mints a `setup-go-*` key at all, so turning its bundled cache back +on would be a cold miss *and* a fresh ~1 GB save rather than a hit. What is +warm is `publish-dev`'s `gobuild-v3--go-release-` entry, which a tag run +could restore from the default branch's scope — but a tagged release is rare +and not latency-sensitive, so it isn't worth a hand-rolled restore step. + +Re-enabling the bundled cache there would be strictly negative, not merely +unhelpful: cache writes are scoped to the ref that made them, so a save +from `refs/tags/v1.0.0` can never be read by `refs/tags/v1.0.1`, by `main`, +or by a PR — only by a re-run of that same tag. It would be a ~1 GB entry +per release that nothing but a retry can ever read. If release wall-clock +ever does matter, the lever is `actions/cache/restore` on +`publish-dev`'s key: restore-only, so it reads `main`'s warm entry and never +writes a tag-scoped one. + +`goreleaser-validate.yml` opts out on its own grounds: its `--single-target` +snapshot is fast enough that the post-step save costs more than a cold +`go mod download`. Key-versioning policy: bump the `v` prefix whenever the cache's expected *contents* change shape — saves only fire on an exact-key miss, @@ -195,6 +245,54 @@ so without a bump the old entry exact-hits forever and the new content is never captured. Keep the old prefixes as transitional restore-keys, then delete them once main has saved the new version. +**Exception — a rotation that *narrows* `path:` carries no transitional +restore-key.** The old archive still contains the paths you just removed, +so restoring it would re-materialize exactly the content the rotation was +meant to stop storing (and, for `~/go/pkg/mod`, extract 0444 module files +over an already-restored tree). Drop the old prefix and purge the stale +entries instead — they hold budget the new keys need. `gobuild-v3` is the +worked example: it kept only its own same-suffix prefix. + +Purge **after** the rotation is on `main`, not before — until then `main` +still restores the old keys, so an early delete just forces a cold +repopulate of caches you are about to abandon: + +```bash +gh api repos/Wave-RF/WaveHouse/actions/caches --paginate \ + -q '.actions_caches[]|select(.key|startswith(""))|.id' \ + | xargs -I{} gh api -X DELETE repos/Wave-RF/WaveHouse/actions/caches/{} +``` + +Include every family the rotation orphans, not just the renamed one — e.g. +turning on `cache: false` strands that job's `setup-go-*` entry too. + +**Sizing policy — the repo cache budget is 10 GB, hard.** Past it GitHub +LRU-evicts, so warm entries disappear mid-run and builds silently get +slower. Budget for **two live generations**: a `go.mod`/`go.sum` or +lockfile bump mints a whole new set while the previous one is still warm, +so the steady state is ~2× a single generation. That is why `~/go/pkg/mod` +is cached **once** (`gomod-v1`) rather than folded into each suffixed +build cache — doing the latter stored the module tree five times over, +five entries of ~0.9-1.2 GB each, ~5.2 GB per generation, and #438's 24-module +bump pushed the repo to 10.53 GB +([#443](https://github.com/Wave-RF/WaveHouse/issues/443)). + +Steady state after the split is roughly 5 GB of the 10 — two generations +of `gomod-v1` + the five `gobuild-v3` flavors + the release build cache, +plus the node-side caches and CodeQL. Before adding a cache or widening an +existing `path:`, check the current footprint and confirm two generations +still fit: + +```bash +gh api repos/Wave-RF/WaveHouse/actions/cache/usage \ + -q '"\(.active_caches_size_in_bytes/1073741824*100|round/100)GB / 10GB"' +gh api repos/Wave-RF/WaveHouse/actions/caches --paginate \ + -q '.actions_caches[]|"\(.size_in_bytes)\t\(.key)"' | sort -rn | head +``` + +Never add a per-job copy of content that is a pure function of a lockfile +— key it once, unsuffixed, and let every job share it. + ## Timing (steady state, full pipeline) The non-gating **Timing summary** job writes a per-job wall-clock table @@ -242,8 +340,24 @@ suite's wall-clock becomes a problem again, start here: 2. Gate on the change set via `needs: changes` + `if:` on its outputs — never with workflow-level `paths` filters (they'd orphan the required check, invariant 1). -3. Use `setup-env` with a fresh `go-cache-suffix` if it compiles Go with - new flags; never add cache save steps (invariant 6). +3. Use `setup-env`. Three rules come with it: + - **Pass a `go-cache-suffix` if the job compiles Go** — a fresh one for + new flags, an existing flavor's if it compiles identically. Never + empty: the resulting `gobuild-v3--go-` restore-key prefix-matches + every flavor's entry (the cross-flavor restore the split exists to + avoid) and mints an extra build entry against the sizing policy above. + `setup-env` fails the job outright whenever `go` is true — which is + the **default** — and no suffix is passed, so a new job can't inherit + the empty default silently. A job that doesn't build Go passes + `go: "false"` instead, as the docs jobs do. + - **Make sure the job's make target reaches `go-mod-download`.** Every + `ci.yml` Go job races to save the shared unsuffixed `gomod-v1`, so a + job that only fetches the modules it happens to import can store a + partial tree that then exact-hits for everyone until the next + rotation. This is why `cov` carries the prerequisite. + - **Never add cache save steps to `ci.yml`** (invariant 6). A workflow + outside it that needs a cache hand-rolls one, as `publish-dev.yml` + does. 4. Need a build product / data from another job? Upload it as an artifact there, then either `needs` the producer + `download-artifact` (simple, but serializes this job's setup behind the producer), or — when this diff --git a/.github/workflows/publish-dev.yml b/.github/workflows/publish-dev.yml index 97497b15..a09e6002 100644 --- a/.github/workflows/publish-dev.yml +++ b/.github/workflows/publish-dev.yml @@ -50,6 +50,53 @@ jobs: - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version-file: "go.mod" + # Opt out of setup-go's bundled cache and cache the half that + # earns its keep, below. The bundled one stores ~/go/pkg/mod AND + # ~/.cache/go-build under one entry (~1 GB stored), keyed on the + # root go.mod — setup-go hashed go.sum through v6.2.0 and go.mod + # from v6.3.0 (actions/setup-go#705). So roughly half of it + # re-stores the module tree ci.yml already keeps once as + # `gomod-v1` — the duplication #443 is about. + cache: false + + # The other half is the reason this job is fast, so cache it on its + # own. GoReleaser cross-compiles 8 targets here (4 goos × 2 goarch), + # and the last 20 runs split cleanly on whether these objects were + # warm: GoReleaser finished in 36-246s with them, 401-446s without. + # (Measured on setup-go's bundled entry, which carried this same + # ~/.cache/go-build tree — this key is new here.) Dropping them + # outright would cost roughly 2.5-7 min per push to main (the envelope + # between those two clusters; mean delta ~4.8 min). + # + # Release-scoped suffix: these objects are cross-compiled for 8 + # GOOS/GOARCH pairs and share nothing with ci.yml's native-only + # flavors, so `-release` keeps both sides from restoring bytes the + # other can't use. Same `gobuild-v3` family and key inputs as + # setup-env's (see .github/workflows/README.md); ~0.5 GB rather than + # the ~1 GB the bundled cache held. + - uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: ~/.cache/go-build + key: gobuild-v3-${{ runner.os }}-go-release-${{ hashFiles('**/go.mod', '**/go.sum') }} + restore-keys: | + gobuild-v3-${{ runner.os }}-go-release- + + # And read — never write — ci.yml's shared module tree, so dropping + # setup-go's bundled cache doesn't leave this job re-downloading ~112 MB + # of modules on every push. This workflow runs on main, the same scope + # ci.yml saves `gomod-v1` into, so the entry is there to hit. + # + # restore, not cache: a full actions/cache would add a post-step save, + # and this job has no business writing the entry every ci.yml Go job + # shares — it never runs `go mod download` for the full graph (see the + # cov note in the Makefile for why a partial save there is corrosive). + # Restore-only writes nothing, so it costs 0 GB of the budget. + - uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 + with: + path: ~/go/pkg/mod + key: gomod-v1-${{ runner.os }}-${{ hashFiles('**/go.mod', '**/go.sum') }} + restore-keys: | + gomod-v1-${{ runner.os }}- # dockers_v2 builds the linux/amd64+arm64 manifest via `docker buildx`; # the GitHub-hosted runner's default docker driver can't build diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 9a313b46..74b1ebb2 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -24,6 +24,29 @@ jobs: - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version-file: "go.mod" + # Opting out keeps a ~1 GB archive — the module tree plus this + # job's cross-compile objects — out of the 10 GB repo budget. + # (That entry is keyed on the root go.mod, not go.sum: setup-go + # hashed go.sum through v6.2.0 and go.mod from v6.3.0 — + # actions/setup-go#705.) + # + # Turning it back on here would be strictly negative. Nothing + # mints that key any more — publish-dev.yml and + # goreleaser-validate.yml opt out too, and ci.yml runs no + # setup-go — so the restore is a guaranteed miss. Worse, cache + # writes are scoped to the ref that made them: a save from + # refs/tags/v1.0.0 can never be read by refs/tags/v1.0.1, by main, + # or by a PR — only by a re-run of that same tag. It would be a + # ~1 GB entry per release that nothing but a retry can ever read. + # + # What IS warm is publish-dev's gobuild-v3--go-release- entry, + # restorable from the default branch's scope. A tagged release is + # rare and not latency-sensitive, so it isn't worth a hand-rolled + # restore step here — but that's the lever if it ever is. + # + # Same cache: false as publish-dev.yml and goreleaser-validate.yml; + # different follow-up (publish-dev re-caches the useful half). #443. + cache: false # dockers_v2 builds the linux/amd64+arm64 manifest via `docker buildx`; # the GitHub-hosted runner's default docker driver can't build diff --git a/CHANGELOG.md b/CHANGELOG.md index 3db0d718..d1594f03 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **Go module cache stored once instead of once per compile flavor** (`.github/actions/setup-env/action.yml`, `.github/workflows/README.md`, `.github/workflows/publish-dev.yml`, `.github/workflows/release.yml`, `Makefile`): closes [#443](https://github.com/Wave-RF/WaveHouse/issues/443). `setup-env` cached `~/go/pkg/mod` together with `~/.cache/go-build` under a key partitioned by `go-cache-suffix`, but the module cache is a pure function of `go.mod` + `go.sum` and byte-identical for every flavor — so that tree was stored five times over (`-lint`, `-unit`, `-integration`, `-e2e-cov`, `-cov`) — five entries of ~0.9-1.2 GB stored each, ~5.2 GB per generation (the tree is ~1.6 GB on disk; 0.48 GB as a stored archive on a cold save, drifting up as superseded versions accumulate). Two live generations is the steady state (a bump mints a new set while the previous is still warm), so the repo sat near GitHub's hard 10 GB cache cap; the 24-module go-deps bump ([#438](https://github.com/Wave-RF/WaveHouse/pull/438)) tipped it to 10.53 GB and GitHub began LRU-evicting warm entries mid-run. The one cache is now two: `gomod-v1--` on `~/go/pkg/mod`, **unsuffixed** and shared by every `ci.yml` Go job that goes through `setup-env`, and `gobuild-v3--go-` on `~/.cache/go-build` only, still per flavor. Measured after the split: **1.05 GB per generation** (0.48 GB module + 0.57 GB across the five build entries), down from 5.18 GB — a 4.9x reduction. All sizes are stored-archive bytes / 2^30, the unit the README's usage check prints. The `v3` bump is load-bearing — saves fire only on an exact-key miss, so without it the old `v2` entry (still carrying the module cache) would exact-hit forever and the smaller content would never be saved — and `gobuild-v3` drops the bare-prefix restore-key, which existed solely to borrow another flavor's copy of the module cache. Separately, `publish-dev.yml` and `release.yml` now pass `cache: false` to `actions/setup-go` (matching `goreleaser-validate.yml`), which was holding a sixth ~1 GB entry — the module tree `gomod-v1` already keeps once, plus that job's own 8-target cross-compile objects — re-saved on every cache miss. (That entry is keyed on the root `go.mod`: setup-go hashed `go.sum` through v6.2.0 and `go.mod` from v6.3.0, [actions/setup-go#705](https://github.com/actions/setup-go/pull/705).) `publish-dev.yml` re-caches only the half that pays for itself, under `gobuild-v3--go-release-` (~0.5 GB — the bundled entry minus `gomod-v1`'s share): across its last 20 runs GoReleaser takes 36–246 s with the cross-compile objects warm and 401–446 s cold (measured on `setup-go`'s bundled cache, which carried the same `~/.cache/go-build` tree), so dropping the cross-compile objects outright would have cost roughly 2.5–7 minutes on every push to main (mean delta ≈4.8 min). The `-release` suffix keeps those 8-target objects from being restored by CI's native-only flavors and vice versa. Because those timings were taken with setup-go's bundled entry (which also held `~/go/pkg/mod`), `publish-dev` additionally *restores* `gomod-v1` from `main`'s scope via `actions/cache/restore` — read-only, so it costs no budget and cannot write a partial tree to the key every `ci.yml` Go job shares. Without that restore the job would re-download ~112 MB of modules per push and land above the warm range quoted above. Both Go keys now hash `go.mod` alongside `go.sum`, for a different reason each: the GOTOOLCHAIN=auto toolchain lives in `~/go/pkg/mod` and `go.sum` records no entry for it, so a `go`-directive bump would otherwise exact-hit a toolchain-less archive and — saves firing only on an exact-key miss — re-download it every run; and the compiler's build ID keys every build object, so the same bump invalidates `~/.cache/go-build` too, where the failure mode is a permanent cold recompile rather than a re-download. Two guards come with the shared entry: `setup-env` now fails a `go: true` job that passes no `go-cache-suffix` (an empty one yields a restore-key prefix-matching every other flavor), and `make cov` gains the `go-mod-download` prerequisite its siblings already had — CI's coverage job shares the unsuffixed `gomod-v1` and races to save it, but ran only `go run ./scripts/cov report`, so winning that race would have stored a partial `~/go/pkg/mod` that then exact-hit for every other job until the next rotation. The workflows README gains a sizing policy — the 10 GB cap, the two-generations rule, how to check the current footprint, and the rule that lockfile-derived content is keyed once and shared — plus the narrowing-rotation exception to the key-versioning policy. + - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. - **SSE streams emit a periodic keepalive comment so quiet connections survive proxy and tunnel idle timeouts** (`internal/stream/` (new package: `subscriber.go`, `bucket.go`, `heartbeat.go` + tests), `internal/api/{stream,stream_test}.go`, `internal/config/{config,config_test}.go`, `cmd/wavehouse/main.go`, `config.yaml`, `docs/src/content/docs/{reverse-proxy.mdx,api.md,configuration.mdx,architecture.md,deployment.md}`): closes #226. The stream handler (`internal/stream/metrics.go`, `internal/api/{hub,hub_test}.go` also touched) wrote a single `: connected` comment on open and then sent nothing until an event arrived, so on a quiet table an intermediary's idle timeout reset the connection — Cloudflare's edge (and a Cloudflare Tunnel) dropped quiet streams about every two minutes in dogfooding, and every `curl`/server-side reconnect re-ran NATS gap-fill (browser `EventSource` masked it by auto-reconnecting). A single shared `Heartbeater` goroutine now drives keepalives for every live connection: connections are spread across a ring of buckets and one bucket is pushed a minimal `:` SSE keepalive comment per tick — the writes don't all fire at the same instant, and the per-connection period comes from one timer instead of a `time.Ticker` per connection. The user-facing knob is **`stream.keepalive_interval`** (`WH_STREAM_KEEPALIVE_INTERVAL`, default **30s**) — the longest a quiet stream goes without a write — chosen to clear the common 55–60s idle windows (nginx/ingress-nginx `proxy_read_timeout`, AWS ALB, Heroku) with ~2× margin while still clearing Cloudflare's ~120s edge; `stream.keepalive_buckets` (`WH_STREAM_KEEPALIVE_BUCKETS`, default 3) is an advanced load-spreading knob and the wheel ticks every `keepalive_interval ÷ keepalive_buckets`, so one rotation always spans exactly the interval regardless of bucket count. The wheel lives in a new `internal/stream` package (groundwork for #294, which will move the broadcast hub in alongside it) and exposes a `Bucket` interface (push a byte slice to a set of connections) so the #294 delivery-path throughput work can reuse it to serialize once per (role, table) rather than per subscriber. The keepalive is a standard SSE comment (ignored by `EventSource` and spec-compliant parsers; `curl` just prints it), and a failed keepalive write doubles as a liveness check that ends the handler once a connection has gone away. The reverse-proxy guide gains a per-provider/per-software idle-timeout reference table (with source links and how-to-change notes, plus the absolute-cap exceptions a keepalive can't fix, e.g. Envoy's 15s route timeout), and `-race` tests cover the concurrent connect/disconnect teardown path (the wheel pushing to a connection that is mid-teardown); raising the proxy idle timeout for `/v1/stream` is now optional rather than required. Streams are observed through metrics rather than per-event traces: the handler records `wavehouse_sse_active_streams`, `wavehouse_sse_stream_duration_seconds`, and `wavehouse_sse_frames_sent_total` / `wavehouse_sse_bytes_sent_total` (labeled by `kind`: `keepalive`, `event`, `replay`), and the per-event `SSE.PushEvent` span was dropped — the router already excludes `/v1/stream` from HTTP tracing, and a span per delivered event per subscriber is high-volume, low-value, and existed only to read the hub's `trace_headers` envelope, now collapsed so the hub broadcasts the raw event bytes instead of base64-wrapping them. diff --git a/Makefile b/Makefile index de1ebea2..e49fa3b2 100644 --- a/Makefile +++ b/Makefile @@ -734,7 +734,16 @@ test-all: ## Run all suites sequentially + one consolidated Go + TS coverage rep # (collect-only). Standalone `make cov` is "show me the numbers without # re-running tests." Fails if NO suite has data (a stray `make cov`). .PHONY: cov -cov: ## Consolidated coverage report (Go + TS) + gate against thresholds (auto-runs after test-all / ci) +# go-mod-download is not optional here even though `go run ./scripts/cov` +# would fetch what it needs on its own. CI's coverage job shares the +# unsuffixed gomod-v1 cache with every other ci.yml Go job (via +# .github/actions/setup-env), and all of them race to save it on a key +# rotation. `go run` populates only the modules that one program imports, +# so if this job won that race it would store a PARTIAL ~/go/pkg/mod under +# the shared key — which then exact-hits for every other job, forever, +# until the next rotation. Downloading the full graph first keeps the +# shared entry complete whoever wins. See #443. +cov: go-mod-download ## Consolidated coverage report (Go + TS) + gate against thresholds (auto-runs after test-all / ci) @go run ./scripts/cov report ##@ CI