From d629aa7e3d2bfa4dba188b83094ed241bcb48bac Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Tue, 21 Apr 2026 21:50:13 -0400 Subject: [PATCH 01/77] docs(notes): draft ADR-001 cd.yaml to buildbot migration (v0) docs(notes): revise ADR-001 v0->v1.1 (drop ntfy/set-variables, fold Phase 0, correct flake path to onPush.default, replace sidecar framing with perRepoSecretFiles pipeline, flip purity mappings, add D1-D14 resolutions) --- .../ADR-001-cd-to-buildbot-migration.md | 290 ++++++++++++++++++ 1 file changed, 290 insertions(+) create mode 100644 docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md diff --git a/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md b/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md new file mode 100644 index 000000000..c11311769 --- /dev/null +++ b/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md @@ -0,0 +1,290 @@ +# ADR-001: cd.yaml → buildbot-nix / hercules-ci-effects migration + +Status: Accepted 2026-04-21; revised 2026-04-22 to incorporate discovery resolutions and correct architectural claims. + +## Identity + +Migrate `.github/workflows/cd.yaml` and everything it transitively invokes (reusable workflows, composite actions, inline script bodies) off GitHub Actions and onto magnetite's buildbot-nix deployment, using hercules-ci-effects for impure execution and `writeShellApplication` flake apps for script bodies. Scope is strictly the CD surface — `ci.yaml` migration already landed pre-epic. + +## Context + +Three options were considered for the caching problem that surfaced in the origin artifact (`./logs/vanixiets-2026-04-21-test-cluster-cache-strategy.txt`), where `cd.yaml`'s coarse cache invalidation in `cached-ci-job/action.yaml` caused unrelated-input bumps to re-run the full job set: + +- **Option A — narrow the `flake.lock` cache key via `jq` node extraction.** Recompute the key from just the subset of lock-file nodes relevant to a given job. Rejected outright: `flake.lock`'s node graph is denormalized (follows-resolutions, transitive inputs), and correctly computing a per-job projection is its own correctness problem with no test harness behind it. Brittle in-place of a principled fix. + +- **Option B — drvPath-derived cache key in `cached-ci-job/action.yaml`.** Compute `nix eval --raw '.#checks...drvPath'` once per job and key `actions/cache` on that. Principled (the drvPath is the canonical content hash for a nix build) but is bespoke GHA machinery that becomes garbage the moment buildbot-nix takes over. Retained as **designated fallback** if Option-C discovery blocks on any single job; otherwise discarded. + +- **Option C — migrate `cd.yaml` to buildbot-nix using `writeShellApplication` flake apps + hercules-ci-effects.** Realizes "run only when pure closure changes" as the native semantic of the build system. Chosen direction for the full `cd.yaml` surface. + +The caching question is the proximate trigger; the underlying decision is driven by `nix-7v7`, which established self-sovereign build infrastructure on magnetite: niks3 binary cache on Cloudflare R2, buildbot-nix with GitHub + Gitea forge integration, and Gitea self-hosted forge. `buildbot-nix.toml` already configures magnetite's buildbot to evaluate `checks.x86_64-linux` against vanixiets. This epic realizes the CI/CD yield of that infrastructure investment; without it, magnetite evaluates vanixiets checks but does not gate releases. + +Hercules-ci-effects is available to buildbot-nix as a transitive flake-lock pin only. It is **not** a top-level flake input of vanixiets today, **not** imported as a flake-parts module, and no `herculesCI` / `onPush` / `mkEffect` attribute is defined anywhere in `./modules`. Phase 3 introduces all three. (Source: `flake.nix`, `flake.lock:655-678`.) Confidence: HIGH. + +The steady-state target on magnetite (Hetzner, a CX53 instance type) is niks3 + buildbot-nix + Gitea colocated, with hercules-ci-effects enabled and a docker-compatible container runtime provisioned for k3d-running effects. The exact CX53 shape is internally inconsistent in-repo (`modules/nixos/buildbot.nix` sets `cores = 16`; `modules/terranix/hetzner.nix:25-30` documents 16 vCPU / 32 GB / 320 GB; an inline comment in `buildbot.nix:103` says "8 vCPU / 16 GB") and requires a live `nproc` / `free -h` / `df -h` check before capacity claims become load-bearing — noted as a Phase 4 entry condition, not fabricated as a resolved value. + +`cached-ci-job/action.yaml`'s hashing algorithm has been verified (D14): it hashes `flake.lock` in its entirety via a single `git hash-object` call; the `hash-sources` input is a whitespace-separated glob list iterated with `set -f` disabling shell expansion; `**`-containing patterns are expanded by shelling to `find -type f -name ` rather than bash globstar; the action auto-includes the invoking workflow file and itself, and excludes `packages/docs/src/content/docs/notes/*`. The final key is `job-result--<12-char sha256 prefix>` over concatenated `git hash-object` outputs. Confidence: HIGH. This confirms the original cache-coarseness hypothesis and informs Option B fallback design if discovery blocks. + +## Decision — target architecture + +### Module layout + +Four domain-organized subdirectories under `modules/apps/` host migrated job logic: + +- `modules/apps/cluster/` — k3d local integration and forward-compatible Hetzner production cluster orchestration +- `modules/apps/docs/` — documentation preview/release/deploy (partially present per nix-a8g precedent) +- `modules/apps/release/` — production release-packages +- `modules/apps/bootstrap/` — bootstrap-verification + +Each app follows the nix-a8g template: `.nix` declares `pkgs.writeShellApplication` with `runtimeInputs` for the hermetic package closure; `.sh` holds the script body ingested via `readFile`. + +Template bifurcation (per nix-a8g extraction): `modules/apps/docs/deploy.nix` uses string-interpolation form `text = "${builtins.readFile ./deploy.sh}"` because it injects nix-computed variables at eval time (`SOPS_SECRETS_FILE`, `DOCS_PAYLOAD`); `release.nix` and `preview-version.nix` use pure `text = builtins.readFile ./release.sh`. Cluster apps requiring injection of nix-computed paths (e.g., `CLUSTER_CONFIG`, `SOPS_AGE_KEY_PATH`) use the interpolation form; otherwise pure readFile. Phase 1 documents this bifurcation as part of the cluster-app template guide. Confidence: HIGH. + +Dual-maintenance between justfile recipe and flake app is convention-only. Justfile recipes wrap flake apps via `nix run .#`; no enforced lint. Indirect safeguards: (a) CI hash-sources pin the coupling so drift surfaces as rebuild during Phase 5; (b) shellcheck at build time catches script-level regressions. Phase 1 documents dual-maintenance as a review responsibility. Confidence: HIGH. + +### Execution model + +Pure data jobs become package-classified derivations where the work is genuinely a nix derivation producing a consumed artifact. The GHA `set-variables` job does **not** survive as a single derivation in the target architecture; its dispatch-variable surface is distributed across three native buildbot-nix / hercules-ci-effects mechanisms rather than centralized in one emitting package. See "Trigger translation" below for the per-variable mapping: `branch` / `rev` / `shortRev` arrive as top-level arguments to each effect via buildbot-effects; `debug` is the `buildbot-effects run --debug` flag; `force-ci` has no analog (every effect run is fresh — there is no GHA-style cache-hit skipping to override); `sanitized_branch` is computed inline inside each effect that needs it; `deploy_enabled` collapses into per-effect `hci-effects.runIf` gating plus `effects_branches` configuration; the `packages` matrix becomes flake-eval-time expansion — one attribute per package under `onPush.default.outputs.effects` or under `packages.x86_64-linux.*`. No synthetic `cd-variables` package exists in the target architecture. Confidence: HIGH. + +Impure jobs become hercules-ci-effects under a **single fixed attribute path**. buildbot-nix reads `flake.outputs.herculesCI(args).onPush.default.outputs.effects` on every evaluation; the literal `default` is not a branch name — it is the one and only attribute path buildbot-nix consumes. Source of truth: `buildbot-nix/buildbot_effects/buildbot_effects/__init__.py:142-159`. Per-branch `onPush.` nodes are allowed by the hercules type system but are ignored by buildbot-nix. Branch-specific gating is expressed two ways, neither of them in the Nix attribute path: + +- **What runs:** `herculesCI.onPush.default.outputs.effects.` (always the same set per evaluation). +- **When it runs:** `effects_branches = ["main", "release/*", ...]` (glob list) and `effects_on_pull_requests = true|false` in `buildbot-nix.toml`, **always read from the default-branch copy** via `git show origin/:buildbot-nix.toml`. A PR author cannot self-authorize by modifying their PR's toml. Source: `buildbot-nix/buildbot_nix/buildbot_nix/nix_eval.py:596-632`. +- **Within the Nix expression:** `hci-effects.runIf ` gates individual effects at eval time (e.g., `runIf (args.branch == "main")`). + +Inter-effect dependencies are **not expressible** at the buildbot-nix surface. Every attribute under `onPush.default.outputs.effects` becomes one independent Triggerable build on the `/run-effect` builder; all effects are triggered in parallel with `waitForFinish=True, haltOnFailure=True, flunkOnFailure=True` (`nix_eval.py:708-729`). The only cross-effect ordering guarantee is "the prior `nix-build` matrix has succeeded"; there is no "effect A before effect B" edge. Former GHA `needs:` edges carrying no data are dropped; edges carrying data collapse to derivation-input references; edges demanding execution ordering are expressed via `runIf` gating on a prior effect's completion-signal derivation or by composing into a single larger effect. Confidence: HIGH. + +### Secret pipeline (not a sidecar) + +There is no buildbot-effects sidecar process. `buildbot-effects` is a CLI tool in the worker's Python environment. The end-to-end secret flow for an effect build is: + +1. `services.buildbot-nix.master.effects.perRepoSecretFiles.":/" = ` declared on the master's NixOS config (option at `buildbot-nix/nixosModules/master.nix:716-732`). +2. Master loads the file as a systemd `LoadCredential` entry. +3. Buildbot reads it as a `SecretInAFile` via `$CREDENTIALS_DIRECTORY`. +4. For a scheduled effect build, the master writes the JSON blob to `../secrets.json` relative to the worker's build directory and invokes `buildbot-effects run --secrets ../secrets.json `. +5. `buildbot-effects` starts a `bwrap` sandbox, bind-mounts the file at `/run/secrets.json`, and sets `HERCULES_CI_SECRETS_JSON=/run/secrets.json`. +6. The effect script reads that env var and parses the JSON to obtain secrets at runtime. + +Canonical end-to-end example: Harmonia's codecov token, wired at `~/projects/nix-workspace/mic92-clan-dotfiles/machines/eve/modules/buildbot.nix:50-66` and consumed at `~/projects/nix-workspace/harmonia/nix/herculesCI.nix:54-60` with `jq -r '.codecov.data.token // empty' "$HERCULES_CI_SECRETS_JSON"`. Confidence: HIGH. + +clan-infra web01 is the **secret-wiring and niks3/buildbot colocation reference** only. Exhaustive grep of `~/projects/nix-workspace/clan-infra/` for `hercules|effects|mkEffect|onPush|herculesCI` returns no matches beyond the flake-parts URL. web01 does not run hercules effects. It uses two parallel secret systems — `sops.secrets.*` for buildbot forge credentials and `clan.core.vars.generators.*` for niks3 S3 creds, signing key, and API token — both delivered to services via systemd-managed decrypted files on disk. The effects-secrets JSON file on magnetite will follow the same delivery mechanism (clan-vars preferred, consistent with magnetite's existing convention of no sops-nix usage), but its shape (flat JSON dict) and wiring (`perRepoSecretFiles`) are dictated by buildbot-nix upstream, not web01. Confidence: HIGH. + +### Per-job purity mapping + +The `set-variables` job is intentionally absent from the table below: as explained in the execution-model paragraph above, its responsibilities (dispatch variables, package matrix, debug/force flags, branch gating) have more natural homes in the buildbot-nix / hercules-ci-effects surface — effect arguments, `runIf` gating, `effects_branches` configuration, and flake-eval-time attribute expansion — rather than as a single synthetic package derivation. See "Trigger translation" below. + +| Job | Classification | Confidence | Rationale | +|---|---|---|---| +| `preview-release-version` | pure → `checks.x86_64-linux.preview-release-` (or `packages.`) | HIGH | `@semantic-release/github` is explicitly filtered from `--plugins`; no `git push`; trap-restored local-only `git update-ref`; `contents: write` permission is vestigial (semantic-release `verifyAuth` requires it even in dry-run); can be kept or dropped | +| `bootstrap-verification` | effect | HIGH | Mutates `~/.config/sops/age/keys.txt`; `make bootstrap` installs nix daemon via the nix-installer + creates `nixbld` users + writes `/etc/nix/nix.conf`; `make setup-user` generates a fresh age key — all outside the nix sandbox. Cannot be subsumed under buildbot's check graph because its job is to test the bootstrapping path that *makes* nix usable | +| `test-cluster` | effect (local-only) | HIGH | Docker / k3d / ephemeral filesystem mutation; no cross-network mutation but still nix-sandbox-external | +| `preview-docs-deploy` | effect | HIGH | `wrangler versions upload` creates a preview alias on Cloudflare Workers | +| `production-docs-deploy` | effect | HIGH | `wrangler versions deploy @100%` against production `infra.cameronraysmith.net` | +| `production-release-packages` | effect | HIGH | semantic-release with `--dry-run=false`, git tag push, GitHub Release creation, `npmPublish: false`; authority via `GITHUB_TOKEN` | + +### Trigger translation + +Mapping GHA triggers to buildbot-nix / hercules-ci-effects using the two-axis model (Nix attribute path + `buildbot-nix.toml` configuration): + +- **`push` on branches** — effect discovery always happens at `onPush.default.outputs.effects.*`. Execution is gated by the default-branch `buildbot-nix.toml`: default branch always runs effects; other branches run iff their name matches an `effects_branches` glob. +- **`pull_request`** — effects execute iff `effects_on_pull_requests = true` in the default-branch `buildbot-nix.toml`. `checks..*` builds run unconditionally in the Nix sandbox. See "Fork-PR posture" below. +- **`schedule`** — expressed as `herculesCI.onSchedule. = { when = { minute; hour; dayOfWeek; dayOfMonth; }; outputs.effects. = ...; }`. The schema is a structured submodule, **not** a cron string; `dayOfWeek` is a list of `"Mon".."Sun"` translated to buildbot's `0..6`. Missing fields default to deterministic-seeded values to avoid thundering herd. Schedule changes propagate on the next successful default-branch `nix-eval` and trigger a `master.reconfig()`. Source: `buildbot-nix/buildbot_nix/buildbot_nix/scheduled.py`, `buildbot-nix/checks/test-flake/flake.nix`. +- **`workflow_dispatch`** — three substitute surfaces, in priority order: + 1. **CLI over ZeroTier (primary):** `ssh magnetite.zt buildbot-effects run github:cameronraysmith/vanixiets/# [--debug] [--secrets ...]`. The CLI is verified in `buildbot-nix/buildbot_effects/buildbot_effects/cli.py`; subcommands are `list`, `run`, `list-schedules`, `run-scheduled`; flags include `--rev`, `--branch`, `--repo`, `--path`, `--debug`, `--secrets `; flakeref syntax (`github:org/repo/branch#effect`) is supported. The master runs exactly this same command on the `run-effect` builder. Packaging: `just ci-dispatch [flags]` wraps the SSH invocation. Confidence: HIGH. + 2. **Web-UI Rebuild (secondary):** The "Rebuild" button on a prior `run-effect` build gives per-effect re-run granularity at the prior rev. Available only if the effect has already run at least once at the desired rev. The web-UI "Force Build" affordance is wired only to `{project}/nix-eval` — it re-runs the whole evaluation, not a single effect; not a per-effect substitute. + 3. **Thin GHA shim (fallback only):** A `cd-dispatch.yaml` workflow with matching `inputs` that dispatches via buildbot REST or a trailer-parsed commit push. Retained as fallback for any case where operators demand a GitHub UI surface; adds a GHA layer that defeats simplification. + +Path filters (`paths-ignore: '*.md'` is the only one in `cd.yaml` and exists at workflow level, not per-job) become derivation-input scoping — restricting a derivation's `src` to the relevant subtree via `lib.fileset.*`. For content-scoped `runIf` gating, hash path content and compare in the effect declaration. + +Arguments passed to effects: buildbot-effects passes `{ name, branch, ref, tag, rev, shortRev, remoteHttpUrl, primaryRepo }` at top level, with `primaryRepo` containing the same fields. `ref` is always `null` (TODO in upstream). Fields that hercules-ci-agent natively provides (`owner`, `remoteSshUrl`, `webUrl`, `forgeType`) are **not** set by buildbot-effects; accessing them throws under the hercules flake-module unless effects are written to degrade gracefully. Effect scripts must only rely on the fields above. + +### Fork-PR posture + +buildbot-nix has no author/contributor allowlist for PR builds. The PR scheduler matches `category="pull"` unconditionally; `GitLocalPrMerge` fetches fork HEAD via the base-repo URL (`refs/pull//head`) and merges as normal. `userAllowlist`/`repoAllowlist` filter which *repositories buildbot manages*, not which PRs it accepts. Source: `buildbot-nix/buildbot_nix/buildbot_nix/project_config.py:85-99`, `common.py:116-150`. + +Under buildbot-nix defaults (`effects_on_pull_requests = false`), fork PRs receive no effect-secrets: the effects builder returns `util.SKIPPED`, and `checks..*` runs in the Nix sandbox with no wired secrets. If the flag is flipped to `true`, fork PRs receive the full `effects_per_repo_secrets` JSON with **no author allowlist, no fork-vs-same-repo differentiation, no differential privilege, and no Nix sandbox** — effects run as impure shell commands on the worker with the secrets file on disk. The upstream README (`buildbot-nix/README.md:184-190`) explicitly warns this is exploitable. Vanixiets currently has `effects.perRepoSecretFiles = {}` and the flag unset. + +**Recommended default: Posture A.** `effects_on_pull_requests = false`. Preview-* effects run only on default-branch merges. Contributors see `checks..*` feedback (safe, Nix-sandboxed) on their PRs but no contributor-triggered preview-deploy. + +**Named future option: Posture B.** Re-push contributor PR commits onto base-repo `preview/` branches; add `effects_branches = ["preview/*"]` so secrets reach only writers with base-repo push access. Mirrors GHA's `pull_request_target` trust boundary. Adoption contingent on contributor preview-feedback becoming a priority. + +## Phase structure + +**Phase 1 — `writeShellApplication` foundation across four domains.** Per-job script bodies plus their transitive just/shell recipes are converted to the nix-a8g template. Justfile recipes rewrite as thin wrappers invoking `nix run .#`. Phase-1 conversion set (from inventory research 03): `list-packages-json`, `k3d-integration-ci`, `k3d-full`, `k3d-bootstrap-secrets`, `k3d-configure-dns`, `k3d-wait-ready`, `k3d-wait-argocd-sync`, `k3d-test-coverage`, `nixidy-build`, `nixidy-bootstrap`, `nixidy-sync`, `nixidy-push`, and `scripts/k3d-test-coverage.sh`. Scope explicitly excludes composite-action and reusable-workflow disappearance work — that belongs to Phase 6. No production cutover; existing GHA still runs. + +**Phase 2 — Per-job branch-point decision.** Per `cd.yaml` job, confirm Option-C viability. Jobs may diverge: `test-cluster` may proceed to C while `release-packages` awaits secret-pipeline work. Per-job decision, not global. + +**Phase 3 — Effects wiring.** Entry conditions: (a) add `hercules-ci-effects` as a top-level flake input with `inputs.flake-parts.follows = "flake-parts"; inputs.nixpkgs.follows = "nixpkgs";`; (b) introduce a flake-level module importing `inputs.hercules-ci-effects.flakeModule` under the deferred-module composition; (c) declare at least an empty `herculesCI = { ... }: { onPush.default.outputs.effects = { }; }`. Then per job confirmed in Phase 2, populate `herculesCI.onPush.default.outputs.effects.` and gate branches via `effects_branches` in `buildbot-nix.toml`. Overlaps with Phase 5. + +**Phase 4 — Worker provisioning.** Entry conditions: + +1. **Live CX53 capacity confirmation.** `ssh magnetite.zt 'nproc && free -h && df -h /'`. Reconcile with `modules/terranix/hetzner.nix:25-30` and `modules/nixos/buildbot.nix` `cores = 16`. The stale inline comment in `buildbot.nix:103` ("CX53 (8 vCPU, 16 GB RAM)") is either corrected or confirmed; capacity claims downstream become load-bearing only after live verification. +2. **Docker runtime.** magnetite currently runs only `virtualisation.podman` (for gitea-actions-runner, storage at `zroot/root/podman`). k3d effects need docker. Enable `virtualisation.docker.enable = true;` and provision a dedicated ZFS dataset `zroot/root/docker` in `modules/machines/nixos/magnetite/disko.nix` (mirroring the podman pattern). Validate that the docker socket is reachable by the buildbot worker user. +3. **`perRepoSecretFiles` wiring.** Add a clan-vars generator emitting the effects-secret JSON blob (shape: `{ "secretName": "value", ... }` — flat dict consumable as `HERCULES_CI_SECRETS_JSON`). Wire `services.buildbot-nix.master.effects.perRepoSecretFiles."github:cameronraysmith/vanixiets" = config.clan.core.vars.generators.buildbot-effects-vanixiets.files."secrets.json".path;`. +4. **Optional cgroup isolation.** `systemd.slices.effects` with `MemoryMax` and `CPUQuota` caps, attaching buildbot-effects runs to that slice; reduce `gitea-actions-runner.numInstances` during migration window if contention surfaces. + +**Phase 5 — Per-job parity validation.** Entry condition: per-job parity-N threshold locked per the rollback rubric in "Resolutions" (D9 row). Both GHA and buildbot-nix paths run simultaneously. + +- Reversible jobs (`bootstrap-verification`, `preview-release-version`, `preview-docs-deploy`, `test-cluster`): N = 2–3. (`set-variables` is absent from the migration target per "Execution model" / "Per-job purity mapping" above.) +- Irreversible jobs (`production-docs-deploy`, `production-release-packages`): N ≥ 5 with mandatory rollback rehearsal at least once. Dual-writer mitigation is mandatory during parity: keep buildbot's semantic-release in `dry-run: true` so only GHA publishes; flip to `dry-run: false` at cutover. Symmetric approach for tag push and production deploys. + +Compared: success/failure consistency, timing, log quality, secret handling, observability. Abort parity if divergence rate exceeds 20% within the first 10 runs per job or if any single divergence occurs on an irreversible job. + +**Phase 6 — Per-job sunset + `cd.yaml` archival + composite-action/reusable-workflow disappearance + drift cleanup.** + +Disappearance cluster splits into two sub-clusters: + +- **Disappears without replacement** (exactly two composite actions per the inventory): + - `.github/actions/cached-ci-job/action.yaml` — subsumed by the content-addressed nix store + binary cache. + - `.github/actions/setup-nix/action.yml` — buildbot-nix workers have nix pre-provisioned. +- **Artifact disappears, logic migrates:** + - `.github/workflows/test-cluster.yaml` — logic migrates to `modules/apps/cluster/*.{nix,sh}` plus the `test-cluster` effect definition; the workflow file is removed. + - `.github/workflows/deploy-docs.yaml` — logic migrates to `modules/apps/docs/deploy.{nix,sh}` (already present) plus the `preview-docs-deploy` and `production-docs-deploy` effects. + - `.github/workflows/package-release.yaml` — logic migrates to `modules/apps/release/*.{nix,sh}` plus the `production-release-packages` effect. + +Per-job removal from `cd.yaml` after parity threshold met. Final archival of `cd.yaml` once all migrated jobs are confirmed — `cd.yaml` preserved in `.github/deprecated/` per existing precedent, enabling rollback by un-archiving individual jobs. + +**Drift and dead-surface cleanup (Phase 6):** + +- `scripts/preview-version.sh` (legacy root copy, 8684 bytes, not referenced by any active workflow; consumed only by `package.json:18` and by the deprecated `.github/deprecated/*.yaml` hash-sources lines) — delete. +- `package.json:18` (`"preview-version": "./scripts/preview-version.sh"`) — repoint to `nix run .#preview-version` or drop. +- `.github/deprecated/ci-nix-fast-build.yaml` and `.github/deprecated/ci-pre-nix-check.yaml` — drop `scripts/preview-version.sh` references from `hash-sources` strings, or leave as historical if the deprecated files are themselves earmarked for deletion. +- Documentation drift: update `packages/docs/src/content/docs/about/contributing/semantic-release-preview.md` to reference `nix run .#preview-version` and `just preview-version `. (Path differs from earlier ADR drafts that said `docs/content/.../` — the actual path is `packages/docs/src/content/docs/about/contributing/semantic-release-preview.md`.) +- `cd.yaml` `workflow_call` inputs `target_configs`, `cache_control`, `job_selection` — declared, never referenced anywhere in the file. Dead input surface; strip during migration. +- `test-cluster.yaml` `env.CACHIX_BINARY_CACHE: cameronraysmith` — set, never consumed by any action. Dead env var; strip. +- `inputs.job` selector value `'docs-deploy'` vs actual job name `production-docs-deploy` — normalize during migration (rename selector to `production-docs-deploy` or document the alias). +- `permissions: contents: write` on `preview-release-version` — vestigial (semantic-release `verifyAuth` requires it even in dry-run). Document if kept; drop if the dry-run plugin filter eliminates the dependency. + +### Exit criteria (mutually exclusive) + +- **Fully-migrated.** All 7 jobs migrated. `cd.yaml` archived to `.github/deprecated/`. Composite actions and reusable workflows disappeared per Phase 6 sub-clusters. Worker provisioning complete. Rollback recoverable via un-archiving individual job definitions. +- **Hybrid-stable.** Subset migrated; remainder stays on GHA indefinitely due to blocking outcomes or coordination requirements. `cd.yaml` active for the GHA residue. Revisit trigger: quarterly review of still-on-GHA jobs against the blocker that kept them there. Prevents drift into indefinite hybrid. +- **Discovery-blocks.** A hard blocker (e.g., a job whose secret model cannot safely migrate, or a k3d-on-docker incompatibility) surfaces during Phase 3 or 4. Fall back to Option B scoped to affected jobs — `writeShellApplication` conversion from Phase 1 still lands as independently valuable infrastructure; Option B cache-key extension applied to `cached-ci-job/action.yaml` completes the fallback for the residue. + +## Resolutions + +Phase 0 is folded into Phase 3/4/5 entry conditions; the table below summarizes discovery items from the prior draft, their resolution, and the research report that resolved each. Reports are under `.factory/research/adr-001-validation/`. + +| Item | Status | Resolution | Reference | +|---|---|---|---| +| **D1** — magnetite capacity / Docker / k3d | OPEN (live check) | CX53 shape internally inconsistent in-repo; requires `ssh magnetite.zt 'nproc && free -h && df -h'` before load-bearing use. Docker not currently enabled (only podman); Phase 4 adds `virtualisation.docker.enable = true` + dedicated ZFS dataset. | research/02 | +| **D2** — hercules-ci-effects + buildbot-nix integration | RESOLVED (HIGH) | Attribute path is fixed at `herculesCI.onPush.default.outputs.effects.`; per-branch paths are ignored. Branch gating via `effects_branches` and `effects_on_pull_requests` in `buildbot-nix.toml` (read from default branch). Secrets via `perRepoSecretFiles` → JSON file → `HERCULES_CI_SECRETS_JSON` inside bwrap sandbox. hercules-ci-effects is currently only a transitive flake-lock pin on vanixiets; Phase 3 entry adds it as a top-level input + flake-parts module. | research/01, 02 | +| **D5a** — per-job secret inventory | RESOLVED (HIGH) | `set-variables`: none. `preview-release-version`: declared `contents: write` but plugin filter removes `GITHUB_TOKEN` consumption. `preview-docs-deploy`: `SOPS_AGE_KEY` (decrypts Cloudflare creds from `secrets/shared.yaml`). `bootstrap-verification`: none. `test-cluster`: `SOPS_AGE_KEY` (for k3d `sops-age-key` Kubernetes secret bootstrap). `production-release-packages`: explicit `SOPS_AGE_KEY` + implicit `GITHUB_TOKEN`. `production-docs-deploy`: `SOPS_AGE_KEY`. | research/03 | +| **D5b** — fork-PR security posture | RESOLVED (HIGH) | No author allowlist, no fork-vs-same-repo differentiation, no Nix sandbox for effects. `effects_on_pull_requests = false` default keeps fork PRs safe. Posture A (keep default) chosen; Posture B (preview/* base-repo branches) named as upgrade path. | research/04 | +| **D5c** — secret pipeline design | RESOLVED (HIGH) | "Sidecar" framing was incorrect; actual model is `perRepoSecretFiles` → systemd `LoadCredential` → JSON file in bwrap. Magnetite follows clan-vars convention (no sops-nix yet) to generate the JSON blob. | research/01, 02 | +| **D7a** — trigger-surface mapping | RESOLVED (HIGH) | See "Trigger translation" section. Two-axis model: Nix attribute path (`onPush.default.outputs.effects`) + `buildbot-nix.toml` config (`effects_branches`, `effects_on_pull_requests`) + `onSchedule..when` structured submodule + CLI for manual dispatch. | research/01, 05 | +| **D7b** — path-filter audit | RESOLVED (HIGH) | cd.yaml has a single workflow-level `paths-ignore: '*.md'`; no job-level path filters. Translates to `lib.fileset.*` scoping of derivation `src` where desired, or is dropped as trivially handled by nix content-addressing. | research/03 | +| **D7c** — workflow_dispatch substitute | RESOLVED (HIGH) | `buildbot-effects run` is verified: subcommands `list`, `run`, `list-schedules`, `run-scheduled`; flags `--rev`, `--branch`, `--repo`, `--path`, `--debug`, `--secrets`; flakeref syntax supported. Web-UI "Force Build" is only wired to `nix-eval` (whole-evaluation); per-effect "Rebuild" requires prior run. CLI over ZeroTier is primary; Rebuild is secondary; thin GHA shim is fallback-only. Confidence upgraded MEDIUM → HIGH. | research/01, 05 | +| **D8a** — per-job purity confirmation | RESOLVED (HIGH) | See "Per-job purity mapping" table. | research/03 | +| **D8b** — bootstrap-verification rubric | RESOLVED (HIGH) | EFFECT. `make bootstrap` installs nix daemon + `nixbld` users + systemd/launchd units; `make setup-user` writes `~/.config/sops/age/keys.txt`. By construction cannot be a buildbot check; either remains a minimal GHA job gated on bootstrap-relevant paths, or becomes an effect that provisions and tests a fresh worker. | research/03 | +| **D8c** — preview-release-version tag-push resolution | RESOLVED (HIGH) | PURE. Both `scripts/preview-version.sh` and `modules/apps/docs/preview-version.sh` operate in a throwaway worktree with trap-restored local-only `git update-ref`; `@semantic-release/github` plugin is explicitly filtered from `--plugins`; no `git push` anywhere. Classification is check (or package), not pure-effect or split. | research/03 | +| **D8d** — composite-action + reusable-workflow inventory | RESOLVED (HIGH) | Exactly two composite actions: `.github/actions/cached-ci-job/action.yaml` and `.github/actions/setup-nix/action.yml`. Both disappear without replacement. Reusable workflows: `deploy-docs.yaml`, `test-cluster.yaml`, `package-release.yaml` — logic migrates to `modules/apps/`. | research/03 | +| **D9** — rollback posture | RESOLVED (HIGH) | Two classes. Reversible (parity N = 2–3; fast-revert by un-archiving from `.github/deprecated/`; trigger rollback at 1–2 consecutive divergences). Irreversible (parity N ≥ 5; mandatory rollback rehearsal; dual-writer mitigation with semantic-release `dry-run: true` during parity). Automated rollback triggers depend on ntfy observability (D12); without it, detection is eyeball-only. | research/05 | +| **D10** — Ironstar history comparison | DEFERRED (out of epic scope per revision) | Research did not cover. Accretion-vs-load-bearing audit of `cd.yaml` patterns is independent of the migration mechanics and can be deferred. | — | +| **D12** — observability transition | RESOLVED (HIGH) | Tier 1 (per-effect GitHub Commit Status via `FilteredGitHubStatusPush` + `nix_status_generator.py`; each effect posts its own context `effects.`) is sufficient for mission scope. Tier 2 (ntfy `HttpStatusPush` → `https://ntfy.zt/vanixiets-ci-fail` on default-branch failures) is deferred as an operational improvement post-mission. matrix-synapse further deferred behind ntfy. Gap vs GHA: implicit email-on-failure has no default replacement within mission scope; subscribers rely on GitHub Commit Status notifications until ntfy is wired. | research/05 | +| **D13** — cost posture | OPEN (live check) | Depends on D1 CX53-shape confirmation. CX53 at public Hetzner pricing ≈ €14/month; R2 storage ≈ $7.5/month at 500 GB. If live `nproc` shows 8 vCPU / 16 GB, headroom for concurrent k3d effects is tight and CX63 or CCX33 upgrade becomes a consideration. | research/02 | +| **D14** — cached-ci-job hashing | RESOLVED (HIGH) | Hashes `flake.lock` whole via single `git hash-object`; `hash-sources` is a whitespace-separated glob list iterated with `set -f`; `**` expanded via `find -type f -name `; auto-includes workflow file + the action itself; excludes `packages/docs/src/content/docs/notes/*`; key = `job-result--<12-char sha256 prefix>`. | research/03 | + +## Organizational shape + +Single parent epic with internal clustering. Rejected alternative: parent epic + child epics per domain. + +Justification: Phase 4 worker provisioning is cross-cutting across all effectful jobs; Phase 2 per-job branch-point decisions need a single coordination view; a unified "how is the migration going" view matters for duration tracking; dependency coordination via edges is cheaper than epic-metadata overhead. + +Estimated duration: 6–12 weeks. With ~90% of discovery resolved (see Resolutions table), the range is anchored on Phase-1/3/4/5 execution time, not discovery outcomes. Phase 4 live CX53 verification may revise the upper bound if capacity forces a server upgrade. + +Internal clusters: + +- Cluster-domain app conversion (`modules/apps/cluster/`) — 13 recipes/scripts +- Docs-domain app conversion (completion + drift cleanup) +- Release-domain app conversion (`modules/apps/release/`) +- Bootstrap-domain app conversion (`modules/apps/bootstrap/`) +- Flake-level effects wiring (hercules-ci-effects input + flakeModule + `herculesCI` attribute + per-effect declarations) +- Worker provisioning (magnetite NixOS module: docker + ZFS dataset + `perRepoSecretFiles` + optional cgroup isolation; ntfy reporter deferred post-mission) +- Parity validation (per confirmed job, per N-run rubric) +- Sunset + disappearance (per-job removal, composite-action and reusable-workflow deletion, `cd.yaml` archival, drift cleanup per Phase 6 touchpoints) + +## Consequences + +Positive: + +- Self-sovereign CI execution aligned with `nix-7v7` investment. +- Enables `nix-7v7` infrastructure to gate releases, not just evaluate checks. +- Per-effect granular caching via native hercules semantics; no bespoke GHA cache machinery. +- Each effect posts its own GitHub Commit Status context (`effects.`) — observability contract for PR authors is preserved and arguably sharper. +- Domain-organized app layout supports forward-compatible Hetzner production cluster migration. +- `writeShellApplication` + `.sh` sidecar decouples shellcheck hygiene from nix string-templating. + +Negative / risks: + +- Magnetite becomes CI single-point-of-failure. **Mitigation:** `cd.yaml` is archived in `.github/deprecated/` during Phase 6, not deleted; un-archiving individual jobs restores the GHA fallback path without code rewrites. Rollback acceptance criteria for irreversible jobs (D9) include a mandatory rehearsal. +- Secret-pipeline migration has security-adjacent complexity — fork-PR secret exposure is a real footgun if `effects_on_pull_requests` is ever flipped. Posture A (default off) is the explicit guardrail. +- For irreversible jobs (`production-release-packages`, `production-docs-deploy`), the revert path cannot un-publish artifacts; it can only return publish authority to GHA for subsequent runs. **Mitigation:** dual-writer rule during Phase 5 parity — buildbot's semantic-release runs with `dry-run: true` so only GHA publishes until cutover. +- User-facing observability shifts from GHA UI to `buildbot.scientistexperience.net` with per-effect GitHub Commit Status contexts as the primary feedback channel on PRs. Implicit GHA email-on-failure has **no default replacement within mission scope** (ntfy Tier 2 is deferred as a post-mission operational improvement); subscribers rely on GitHub Commit Status notifications until the ntfy reporter is wired. +- Transient developer-ergonomics cost during hybrid state — PRs show both GHA and buildbot commit-status contexts until Phase 6 completes per job. +- `workflow_dispatch` ergonomics change (CLI substitute instead of GitHub UI). ZT access is required to trigger effects manually; non-admin contributors cannot force-run an effect. +- Effect debug UX is strictly more powerful (`buildbot-effects run --debug`) but strictly less ergonomic than `action-tmate@v3` for non-ZT contributors. Permanent ergonomic cost. +- Fixed-cost posture shift: magnetite (CX53) supersedes GHA's effectively-free public-repo CI capacity. Absolute cost minor at Hetzner pricing, but pending D1/D13 live confirmation, capacity headroom alongside niks3 + buildbot + Gitea + 2 gitea-actions-runner podman instances is not yet quantitatively validated. +- Drift between `.sh` sidecars and legacy copies (`scripts/preview-version.sh`, `package.json:18`, docs reference, deprecated workflow hash-sources) requires active Phase 6 cleanup per enumerated touchpoints. + +## Explicit deferrals + +- Individual issue bodies and beads IDs — not ADR content. +- Ironstar-style accretion-vs-load-bearing audit (D10) — out of epic scope per revision. +- Per-phase duration estimates beyond the top-level 6–12 week range. +- Posture B adoption (fork-PR preview via `preview/*` base-repo branches) — deferred until contributor preview-feedback becomes a priority. +- ntfy `HttpStatusPush` reporter wiring — operational steady-state concern, deferred as post-mission improvement. Tier 1 (per-effect GitHub Commit Status via `FilteredGitHubStatusPush`) plus PR-based validation (`gh pr checks` + `buildbot-logs`) covers the mission-scope dev loop; the email-on-failure gap is acknowledged under Consequences. +- matrix-synapse reporter wiring — deferred behind ntfy Tier 2 (which is itself post-mission). +- Phase 5 per-job parity validation is a wall-clock observational activity that begins after all mission features complete; the mission does **not** gate on parity confirmation. Parity windows run on calendar time, not on the mission's feature-completion critical path. +- `cd.yaml` archival to `.github/deprecated/`, deletion of the reusable workflows (`deploy-docs.yaml`, `test-cluster.yaml`, `package-release.yaml`), and deletion of the composite actions (`cached-ci-job/action.yaml`, `setup-nix/action.yml`) — deferred until post-mission parity observation completes (Phase 6 depends on Phase 5 exit). The mission as currently scoped stops before Phases 5 and 6 run their full course, though both phases remain the ADR's target end-state. +- Binary-cache poisoning analysis for fork-PR-triggered niks3 uploads — content-addressing makes direct collision attacks infeasible, but trust-in-cache-contents is out of scope. + +## References + +Codebase: + +- GHA workflow authoritative source: `/Users/crs58/projects/nix-workspace/vanixiets/.github/workflows/cd.yaml` +- Reusable workflows: `.github/workflows/test-cluster.yaml`, `.github/workflows/deploy-docs.yaml`, `.github/workflows/package-release.yaml` +- Composite actions: `.github/actions/cached-ci-job/action.yaml`, `.github/actions/setup-nix/action.yml` +- nix-a8g precedent template: `modules/apps/docs/` +- Hermetic-deps derivation shape: `pkgs/by-name/vanixiets-docs-deps/package.nix` +- Legacy preview-version drift: `scripts/preview-version.sh`, `package.json:18` +- Docs-reference drift: `packages/docs/src/content/docs/about/contributing/semantic-release-preview.md` +- Buildbot worker NixOS module: `modules/nixos/buildbot.nix` +- niks3 NixOS module: `modules/nixos/niks3.nix` +- Buildbot-nix project config: `buildbot-nix.toml` +- Magnetite machine config: `modules/machines/nixos/magnetite/default.nix`, `modules/machines/nixos/magnetite/disko.nix` +- Terranix shape declaration: `modules/terranix/hetzner.nix:25-30` +- cinnabar ntfy deployment: `modules/machines/nixos/cinnabar/ntfy.nix` + +Research reports (this revision's evidence base): + +- `.factory/research/adr-001-validation/01-hercules-effects-buildbot-nix-mechanics.md` +- `.factory/research/adr-001-validation/02-magnetite-state-and-web01-pattern.md` +- `.factory/research/adr-001-validation/03-cd-yaml-inventory.md` +- `.factory/research/adr-001-validation/04-fork-pr-security.md` +- `.factory/research/adr-001-validation/05-ops-triggers-observability-rollback.md` + +External sources: + +- buildbot-nix upstream: `~/projects/nix-workspace/buildbot-nix/` + - Effects CLI: `buildbot_effects/buildbot_effects/cli.py` + - Effects flake-attr reader: `buildbot_effects/buildbot_effects/__init__.py:142-159` + - Effects dispatch + gating: `buildbot_nix/buildbot_nix/nix_eval.py:596-632`, `708-729` + - Master module effects options: `nixosModules/master.nix:716-732`, `1010-1040` + - Scheduled effects: `buildbot_nix/buildbot_nix/scheduled.py`, `models.py:ScheduleWhen` + - Commit-status generator: `buildbot_nix/buildbot_nix/nix_status_generator.py` + - Security warning (fork-PR): `README.md:184-190` +- hercules-ci-effects upstream: `~/projects/nix-workspace/hercules-ci-effects/` + - flakeModule: `flake-modules/herculesCI-attribute.nix` + - `runIf`: `effects/default.nix:47-63` +- Reference implementation (secret-wiring + niks3/buildbot colocation only — **not** an effects reference): `~/projects/nix-workspace/clan-infra/` +- End-to-end effects-secrets example (codecov token): `~/projects/nix-workspace/mic92-clan-dotfiles/machines/eve/modules/buildbot.nix:50-66`, `~/projects/nix-workspace/harmonia/nix/herculesCI.nix:54-60` +- Origin artifact (caching question that triggered the epic): `./logs/vanixiets-2026-04-21-test-cluster-cache-strategy.txt` + +Skills: + +- `~/.claude/skills/preferences-nix-ci-cd-integration/SKILL.md` +- `~/.claude/skills/preferences-nix-checks-architecture/SKILL.md` +- `~/.claude/skills/preferences-secrets/SKILL.md` +- `~/.claude/skills/preferences-adaptive-planning/SKILL.md` +- `~/.claude/skills/stigmergic-convention/SKILL.md` From 563c991207eecbed3f016286352df6ee4caa1165 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 03:30:03 -0400 Subject: [PATCH 02/77] feat(apps/cluster): convert k3d + nixidy + list-packages-json recipes to writeShellApplication flake apps Land 12 cluster-domain flake apps under modules/apps/cluster/ per M1 of the cd.yaml -> buildbot-nix migration (ADR-001 Phase 1): k3d-integration-ci, k3d-full, k3d-bootstrap-secrets, k3d-configure-dns, k3d-wait-ready, k3d-wait-argocd-sync, k3d-test-coverage, nixidy-build, nixidy-bootstrap, nixidy-sync, nixidy-push, list-packages-json Each app uses the nix-a8g pure-readFile writeShellApplication template (no cluster app required nix-computed variable injection). runtimeInputs are declared meticulously per app; writeShellApplication's shellcheck hook runs at build time. Every .sh carries a --help handler that exits 0 with usage, and (where applicable) argument-validation errors exit nonzero with a usage hint on stderr. Apps nixidy-sync and k3d-integration-ci compose sibling apps (nixidy-build, nixidy-push, nixidy-bootstrap, k3d-full, k3d-wait-*, k3d-test-coverage) either via their bin names on PATH (nixidy-sync) or via just-wrapper delegation (k3d-integration-ci, k3d-full). Justfile recipes rewrite as thin 'nix run .# -- {{ARGS}}' wrappers with comments pointing back at the modules/apps/cluster/ source pair. Per the mission, GHA workflows still invoke the legacy recipe paths; no cutover in M1. Argument pass-through is preserved for k3d-test-coverage, which previously accepted chainsaw flags via *ARGS. scripts/k3d-test-coverage.sh is reduced to a thin backward-compat shim that exec's 'nix run .#k3d-test-coverage'. The legacy path is retained because .github/workflows/test-cluster.yaml:57 references it in hash-sources; removal is deferred to M5 drift cleanup. Verified locally via (aarch64-darwin): - nix eval .#apps.aarch64-darwin..program --raw (12/12 OK) - nix run .# -- --help (12/12 exit 0) - shellcheck modules/apps/cluster/*.sh scripts/k3d-test-coverage.sh (clean) - just list-packages-json | jq . (regression) - nix run .#nixidy-push (no result/) -> exit 1 with expected error Fulfills (M1 cluster sub-area of validation-contract.md): VAL-WRITESHELL-CLUSTER-001..013, CLUSTER-014/015 (shellcheck), CLUSTER-020 (nixidy-push usage contract), CLUSTER-022 (justfile delegation), CLUSTER-023 (arg pass-through). PR-based assertions for k3d-bootstrap-secrets idempotence and nixidy-sync composition (017-019, 021) require a live k3d cluster and are deferred to M4 validation. --- justfile | 266 ++------- .../apps/cluster/k3d-bootstrap-secrets.nix | 32 ++ modules/apps/cluster/k3d-bootstrap-secrets.sh | 57 ++ modules/apps/cluster/k3d-configure-dns.nix | 32 ++ modules/apps/cluster/k3d-configure-dns.sh | 51 ++ modules/apps/cluster/k3d-full.nix | 34 ++ modules/apps/cluster/k3d-full.sh | 39 ++ modules/apps/cluster/k3d-integration-ci.nix | 37 ++ modules/apps/cluster/k3d-integration-ci.sh | 76 +++ modules/apps/cluster/k3d-test-coverage.nix | 41 ++ modules/apps/cluster/k3d-test-coverage.sh | 535 ++++++++++++++++++ modules/apps/cluster/k3d-wait-argocd-sync.nix | 30 + modules/apps/cluster/k3d-wait-argocd-sync.sh | 88 +++ modules/apps/cluster/k3d-wait-ready.nix | 29 + modules/apps/cluster/k3d-wait-ready.sh | 53 ++ modules/apps/cluster/list-packages-json.nix | 30 + modules/apps/cluster/list-packages-json.sh | 44 ++ modules/apps/cluster/nixidy-bootstrap.nix | 34 ++ modules/apps/cluster/nixidy-bootstrap.sh | 28 + modules/apps/cluster/nixidy-build.nix | 35 ++ modules/apps/cluster/nixidy-build.sh | 27 + modules/apps/cluster/nixidy-push.nix | 30 + modules/apps/cluster/nixidy-push.sh | 63 +++ modules/apps/cluster/nixidy-sync.nix | 60 ++ modules/apps/cluster/nixidy-sync.sh | 31 + scripts/k3d-test-coverage.sh | 520 +---------------- 26 files changed, 1567 insertions(+), 735 deletions(-) create mode 100644 modules/apps/cluster/k3d-bootstrap-secrets.nix create mode 100644 modules/apps/cluster/k3d-bootstrap-secrets.sh create mode 100644 modules/apps/cluster/k3d-configure-dns.nix create mode 100644 modules/apps/cluster/k3d-configure-dns.sh create mode 100644 modules/apps/cluster/k3d-full.nix create mode 100644 modules/apps/cluster/k3d-full.sh create mode 100644 modules/apps/cluster/k3d-integration-ci.nix create mode 100644 modules/apps/cluster/k3d-integration-ci.sh create mode 100644 modules/apps/cluster/k3d-test-coverage.nix create mode 100644 modules/apps/cluster/k3d-test-coverage.sh create mode 100644 modules/apps/cluster/k3d-wait-argocd-sync.nix create mode 100644 modules/apps/cluster/k3d-wait-argocd-sync.sh create mode 100644 modules/apps/cluster/k3d-wait-ready.nix create mode 100644 modules/apps/cluster/k3d-wait-ready.sh create mode 100644 modules/apps/cluster/list-packages-json.nix create mode 100644 modules/apps/cluster/list-packages-json.sh create mode 100644 modules/apps/cluster/nixidy-bootstrap.nix create mode 100644 modules/apps/cluster/nixidy-bootstrap.sh create mode 100644 modules/apps/cluster/nixidy-build.nix create mode 100644 modules/apps/cluster/nixidy-build.sh create mode 100644 modules/apps/cluster/nixidy-push.nix create mode 100644 modules/apps/cluster/nixidy-push.sh create mode 100644 modules/apps/cluster/nixidy-sync.nix create mode 100644 modules/apps/cluster/nixidy-sync.sh diff --git a/justfile b/justfile index 3b974e864..edc5b72a5 100644 --- a/justfile +++ b/justfile @@ -898,49 +898,17 @@ k3d-up: # Bootstrap secrets required before first deployment (idempotent) # Supports both CI (SOPS_AGE_KEY env var) and local dev (file-based) workflows +# Body lives in modules/apps/cluster/k3d-bootstrap-secrets.{nix,sh}. [group('k3d')] -k3d-bootstrap-secrets: - #!/usr/bin/env bash - set -euo pipefail - kubectl create namespace sops-secrets-operator --dry-run=client -o yaml | kubectl apply -f - - # Determine age key file: env var (CI) or local file (dev) - if [ -n "${SOPS_AGE_KEY:-}" ]; then - echo "Using SOPS_AGE_KEY from environment variable" - KEYFILE=$(mktemp) - echo "${SOPS_AGE_KEY}" > "$KEYFILE" - trap "rm -f '$KEYFILE'" EXIT - else - echo "Using SOPS age key from file: ${HOME}/.config/sops/age/keys.txt" - KEYFILE="${HOME}/.config/sops/age/keys.txt" - fi - kubectl create secret generic sops-age-key \ - --namespace=sops-secrets-operator \ - --from-file=age.key="$KEYFILE" \ - --dry-run=client -o yaml | kubectl apply -f - +k3d-bootstrap-secrets *ARGS: + {{nix_cmd}} run --no-warn-dirty .#k3d-bootstrap-secrets -- {{ARGS}} # Configure CoreDNS to forward sslip.io queries to public DNS resolvers # Required because OrbStack's DNS (192.168.107.1) cannot resolve sslip.io wildcards +# Body lives in modules/apps/cluster/k3d-configure-dns.{nix,sh}. [group('k3d')] -k3d-configure-dns: - #!/usr/bin/env bash - set -euo pipefail - echo "Waiting for CoreDNS to be running..." - kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=120s - echo "Patching CoreDNS ConfigMap to forward sslip.io to public DNS..." - CURRENT=$(kubectl get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}') - if echo "$CURRENT" | grep -q "sslip.io"; then - echo "CoreDNS already configured for sslip.io forwarding" - exit 0 - fi - SSLIP_BLOCK=$'sslip.io:53 {\n forward . 1.1.1.1 8.8.8.8\n cache 30\n}\n' - PATCHED="${SSLIP_BLOCK}${CURRENT}" - PATCH_JSON=$(jq -n --arg corefile "$PATCHED" '{"data": {"Corefile": $corefile}}') - kubectl patch configmap coredns -n kube-system --type=merge -p "$PATCH_JSON" - echo "Restarting CoreDNS deployment..." - kubectl rollout restart deployment coredns -n kube-system - echo "Waiting for CoreDNS to be ready..." - kubectl rollout status deployment coredns -n kube-system --timeout=120s - echo "CoreDNS configured for sslip.io forwarding" +k3d-configure-dns *ARGS: + {{nix_cmd}} run --no-warn-dirty .#k3d-configure-dns -- {{ARGS}} # Delete local k3d cluster [group('k3d')] @@ -1009,11 +977,12 @@ k3d-deploy-infrastructure: {{nix_cmd}} run .#k8s-deploy-local-k3d-infrastructure -- --yes # Full k3d workflow: create cluster, bootstrap secrets, deploy all layers +# Body lives in modules/apps/cluster/k3d-full.{nix,sh}; delegates back to +# the k3d-down, k3d-up, and k3d-deploy recipes above (none of which are +# flake-app converted in M1). [group('k3d')] -k3d-full: - just k3d-down || true - just k3d-up - just k3d-deploy +k3d-full *ARGS: + {{nix_cmd}} run --no-warn-dirty .#k3d-full -- {{ARGS}} # Run all kubernetes tests (foundation + infrastructure) [group('k3d')] @@ -1032,106 +1001,23 @@ k3d-test-infrastructure: # Run tests with coverage report showing tested vs deployed resources # Respects NO_COLOR env var and auto-detects CI environments +# Body lives in modules/apps/cluster/k3d-test-coverage.{nix,sh}; +# scripts/k3d-test-coverage.sh retained as a thin backward-compat shim. [group('k3d')] k3d-test-coverage *ARGS: - ./scripts/k3d-test-coverage.sh {{ARGS}} + {{nix_cmd}} run --no-warn-dirty .#k3d-test-coverage -- {{ARGS}} # Wait for kluctl-deployed foundation and infrastructure pods to be ready +# Body lives in modules/apps/cluster/k3d-wait-ready.{nix,sh}. [group('k3d')] -k3d-wait-ready: - #!/usr/bin/env bash - set -euo pipefail - - echo "=== Waiting for Foundation (CNI) ===" - echo "Waiting for Cilium Agent..." - kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=cilium-agent -n kube-system --timeout=300s - - echo "Waiting for Cilium Operator..." - kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=cilium-operator -n kube-system --timeout=300s - - echo "" - echo "=== Waiting for Infrastructure ===" - echo "Waiting for ArgoCD deployments..." - kubectl wait --for=condition=Available deployment --all -n argocd --timeout=300s - - echo "Waiting for ArgoCD Application Controller..." - kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=argocd-application-controller -n argocd --timeout=300s - - echo "Waiting for step-ca..." - # Use StatefulSet pod label to exclude Helm test-connection pod (which always fails) - kubectl wait --for=condition=Ready pod -l statefulset.kubernetes.io/pod-name=step-ca-step-certificates-0 -n step-ca --timeout=300s - - echo "Waiting for sops-secrets-operator..." - kubectl wait --for=condition=Available deployment --all -n sops-secrets-operator --timeout=300s - - echo "" - echo "=== All foundation and infrastructure pods ready ===" +k3d-wait-ready *ARGS: + {{nix_cmd}} run --no-warn-dirty .#k3d-wait-ready -- {{ARGS}} # Wait for all ArgoCD Applications to reach Synced + Healthy status +# Body lives in modules/apps/cluster/k3d-wait-argocd-sync.{nix,sh}. [group('k3d')] -k3d-wait-argocd-sync: - #!/usr/bin/env bash - set -euo pipefail - - echo "=== Waiting for ArgoCD Applications ===" - echo "Applications managed by nixidy sync waves:" - echo " Wave -1 (adoption): cilium, argocd, sops-secrets-operator, step-ca" - echo " Wave 0: cert-manager" - echo " Wave 1-2: cluster-issuer, gateway, gateway-api" - echo " Wave 3: argocd-route" - echo "" - - # All expected applications (app-of-apps creates these asynchronously) - EXPECTED_APPS=( - apps - argocd - argocd-route - cert-manager - cilium - cluster-issuer - gateway - gateway-api - sops-secrets-operator - step-ca - ) - - echo "Waiting for all ${#EXPECTED_APPS[@]} applications to exist..." - for app in "${EXPECTED_APPS[@]}"; do - echo -n " Waiting for $app... " - # kubectl wait fails immediately if resource doesn't exist, so poll instead - timeout 300 bash -c "until kubectl get application/$app -n argocd &>/dev/null; do sleep 2; done" - echo "exists" - done - - echo "" - echo "Listing applications..." - kubectl get applications -n argocd -o wide || true - echo "" - - echo "Waiting for all applications to be Healthy..." - for app in "${EXPECTED_APPS[@]}"; do - echo -n " Waiting for $app to be Healthy... " - kubectl wait --for=jsonpath='{.status.health.status}'=Healthy application/"$app" -n argocd --timeout=600s >/dev/null - echo "done" - done - - echo "" - echo "Waiting for all applications to be Synced..." - for app in "${EXPECTED_APPS[@]}"; do - echo -n " Waiting for $app to be Synced... " - kubectl wait --for=jsonpath='{.status.sync.status}'=Synced application/"$app" -n argocd --timeout=300s >/dev/null - echo "done" - done - - echo "" - echo "=== Waiting for Gateway to be programmed ===" - # ArgoCD reports Healthy before Cilium fully programs the Gateway - # Wait for the actual Gateway condition, not just ArgoCD's view - kubectl wait --for=condition=Programmed gateway/main-gateway -n gateway-system --timeout=300s - - echo "" - echo "=== All ArgoCD applications synced and healthy ===" - kubectl get applications -n argocd -o wide +k3d-wait-argocd-sync *ARGS: + {{nix_cmd}} run --no-warn-dirty .#k3d-wait-argocd-sync -- {{ARGS}} # Full integration test: cluster creation, deployment, GitOps sync, and validation [group('k3d')] @@ -1168,52 +1054,10 @@ k3d-integration: # Full CI integration test: local manifests, cluster, GitOps sync, tests # Uses file:///manifests instead of remote repo - no GitHub credentials needed # The /tmp/k3d-manifests directory is volume-mounted into the cluster at /manifests +# Body lives in modules/apps/cluster/k3d-integration-ci.{nix,sh}. [group('k3d')] -k3d-integration-ci: - #!/usr/bin/env bash - set -euo pipefail - - echo "=== Phase 1: Build manifests with local repo URL ===" - export ARGOCD_REPO_URL="file:///manifests" - just nixidy-build - - echo "" - echo "=== Phase 2: Prepare local git repo (before cluster for volume mount) ===" - # Ensure writable before cleanup (Nix store copies may be read-only) - chmod -R +w /tmp/k3d-manifests 2>/dev/null || true - rm -rf /tmp/k3d-manifests - mkdir -p /tmp/k3d-manifests - rsync -aL --delete --chmod=Du+w,Fu+w result/ /tmp/k3d-manifests/ - cd /tmp/k3d-manifests - git init -b main - git config user.email "ci@localhost" - git config user.name "CI" - git add . - git commit -m "CI manifests" - cd - - - echo "" - echo "=== Phase 3: Create cluster and deploy via kluctl ===" - just k3d-full - - echo "" - echo "=== Phase 4: Wait for infrastructure ready ===" - just k3d-wait-ready - - echo "" - echo "=== Phase 5: Bootstrap ArgoCD (syncs from file:///manifests) ===" - just nixidy-bootstrap - - echo "" - echo "=== Phase 6: Wait for ArgoCD sync ===" - just k3d-wait-argocd-sync - - echo "" - echo "=== Phase 7: Run integration tests ===" - just k3d-test-coverage - - echo "" - echo "=== CI integration complete ===" +k3d-integration-ci *ARGS: + {{nix_cmd}} run --no-warn-dirty .#k3d-integration-ci -- {{ARGS}} ## nixidy (Phase 4 GitOps) # Per ADR-006: Rendered manifests are pushed to separate private repos per cluster. @@ -1223,9 +1067,10 @@ k3d-integration-ci: local_k3d_repo := env("LOCAL_K3D_REPO", home_directory() / "projects/nix-workspace/local-k3d") # Build nixidy manifests for local-k3d environment (renders to ./result) +# Body lives in modules/apps/cluster/nixidy-build.{nix,sh}. [group('nixidy')] -nixidy-build: - {{nix_cmd}} run .#nixidy -- build .#local-k3d +nixidy-build *ARGS: + {{nix_cmd}} run --no-warn-dirty .#nixidy-build -- {{ARGS}} # Show nixidy environment info [group('nixidy')] @@ -1234,49 +1079,26 @@ nixidy-info: # Push rendered manifests to local-k3d private repository # Prerequisites: nixidy-build must be run first, local-k3d repo must exist +# Body lives in modules/apps/cluster/nixidy-push.{nix,sh}. +# LOCAL_K3D_REPO env var overrides the default target path; justfile- +# level local_k3d_repo is preserved as a convenience for scripted callers. [group('nixidy')] -nixidy-push: - #!/usr/bin/env bash - set -euo pipefail - - if [[ ! -d "result" ]]; then - echo "Error: result/ directory not found. Run 'just nixidy-build' first." - exit 1 - fi - - if [[ ! -d "{{ local_k3d_repo }}" ]]; then - echo "Error: local-k3d repo not found at {{ local_k3d_repo }}" - echo "Clone it with: git clone git@github.com:cameronraysmith/local-k3d.git {{ local_k3d_repo }}" - exit 1 - fi - - echo "Syncing rendered manifests to {{ local_k3d_repo }}..." - # -L dereferences symlinks (nix store paths) to copy actual content - # --checksum compares by content hash (Nix store files have epoch timestamps) - # --chmod fixes read-only permissions from nix store - rsync -aL --delete --checksum --chmod=Du+w,Fu+w --exclude='.git' result/ "{{ local_k3d_repo }}/" - - echo "Committing and pushing to local-k3d repo..." - cd "{{ local_k3d_repo }}" - git add -A - if git diff --cached --quiet; then - echo "No changes to push." - else - git commit -m "chore: update rendered manifests from vanixiets" - git push - echo "Manifests pushed to local-k3d repo." - fi +nixidy-push *ARGS: + LOCAL_K3D_REPO="{{ local_k3d_repo }}" {{nix_cmd}} run --no-warn-dirty .#nixidy-push -- {{ARGS}} # Build and push manifests in one step +# Body lives in modules/apps/cluster/nixidy-sync.{nix,sh}. [group('nixidy')] -nixidy-sync: nixidy-build nixidy-push +nixidy-sync *ARGS: + LOCAL_K3D_REPO="{{ local_k3d_repo }}" {{nix_cmd}} run --no-warn-dirty .#nixidy-sync -- {{ARGS}} # Bootstrap ArgoCD app-of-apps (transition from Phase 3 to Phase 4) # Prerequisites: k3d-full must complete, manifests must be pushed to local-k3d repo # Note: ArgoCD needs credentials to access private repo (configure via argocd CLI or UI) +# Body lives in modules/apps/cluster/nixidy-bootstrap.{nix,sh}. [group('nixidy')] -nixidy-bootstrap: - {{nix_cmd}} run .#nixidy -- bootstrap .#local-k3d | kubectl apply -f - +nixidy-bootstrap *ARGS: + {{nix_cmd}} run --no-warn-dirty .#nixidy-bootstrap -- {{ARGS}} # Full GitOps workflow: Phase 3 bootstrap + Phase 4 ArgoCD takeover # Note: Requires local-k3d repo to exist and ArgoCD to have access credentials @@ -1750,18 +1572,10 @@ list-packages: @ls -1 packages/ # List packages in JSON format for CI matrix +# Body lives in modules/apps/cluster/list-packages-json.{nix,sh}. [group('CI/CD')] -list-packages-json: - #!/usr/bin/env bash - cd packages - packages=() - for dir in */; do - pkg_name="${dir%/}" - if [ -f "$dir/package.json" ]; then - packages+=("{\"name\":\"$pkg_name\",\"path\":\"packages/$pkg_name\"}") - fi - done - echo "[$(IFS=,; echo "${packages[*]}")]" +list-packages-json *ARGS: + @{{nix_cmd}} run --no-warn-dirty .#list-packages-json -- {{ARGS}} # Validate package structure [group('CI/CD')] diff --git a/modules/apps/cluster/k3d-bootstrap-secrets.nix b/modules/apps/cluster/k3d-bootstrap-secrets.nix new file mode 100644 index 000000000..5f70cdff0 --- /dev/null +++ b/modules/apps/cluster/k3d-bootstrap-secrets.nix @@ -0,0 +1,32 @@ +# k3d-bootstrap-secrets.nix - Bootstrap sops-age-key into a running k3d cluster. +# +# Usage: +# nix run .#k3d-bootstrap-secrets +# +# Template form: pure readFile (no nix-computed variable injection). +# Idempotent: second invocation leaves the secret byte-identical. +# +# Supports two key-source branches: +# - SOPS_AGE_KEY env var present -> write to tmpfile, use +# - otherwise -> read $HOME/.config/sops/age/keys.txt +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.k3d-bootstrap-secrets = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "k3d-bootstrap-secrets"; + runtimeInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.kubectl + ]; + text = builtins.readFile ./k3d-bootstrap-secrets.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/k3d-bootstrap-secrets.sh b/modules/apps/cluster/k3d-bootstrap-secrets.sh new file mode 100644 index 000000000..f080727ab --- /dev/null +++ b/modules/apps/cluster/k3d-bootstrap-secrets.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Bootstrap the sops-age-key Kubernetes secret required by +# sops-secrets-operator to decrypt SopsSecret custom resources in the +# local-k3d cluster. Idempotent: reapplies cleanly and does not mutate +# the secret when the key source has not changed. +# +# Usage: +# k3d-bootstrap-secrets [--help] +# +# Key sources (first one found wins): +# SOPS_AGE_KEY env var - used directly (CI pathway) +# ~/.config/sops/age/keys.txt - file-based (local dev pathway) +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: k3d-bootstrap-secrets [--help] + +Creates the sops-secrets-operator namespace (if missing) and the +sops-age-key secret containing an age private key used to decrypt +SopsSecret CRs. Idempotent: subsequent invocations leave the secret +byte-identical. Requires kubectl context pointing at the live k3d +cluster. + +Key source (first found): + SOPS_AGE_KEY environment variable (CI) + ~/.config/sops/age/keys.txt file (local dev) +EOF + exit 0 + ;; +esac + +kubectl create namespace sops-secrets-operator \ + --dry-run=client -o yaml | kubectl apply -f - + +# Determine age key file: env var (CI) or local file (dev) +if [ -n "${SOPS_AGE_KEY:-}" ]; then + echo "Using SOPS_AGE_KEY from environment variable" + KEYFILE=$(mktemp) + echo "${SOPS_AGE_KEY}" > "$KEYFILE" + trap 'rm -f "$KEYFILE"' EXIT +else + echo "Using SOPS age key from file: ${HOME}/.config/sops/age/keys.txt" + KEYFILE="${HOME}/.config/sops/age/keys.txt" + if [ ! -f "$KEYFILE" ]; then + echo "error: age key file not found: $KEYFILE" >&2 + echo " either set SOPS_AGE_KEY or create the file" >&2 + exit 1 + fi +fi + +kubectl create secret generic sops-age-key \ + --namespace=sops-secrets-operator \ + --from-file=age.key="$KEYFILE" \ + --dry-run=client -o yaml | kubectl apply -f - diff --git a/modules/apps/cluster/k3d-configure-dns.nix b/modules/apps/cluster/k3d-configure-dns.nix new file mode 100644 index 000000000..be784057b --- /dev/null +++ b/modules/apps/cluster/k3d-configure-dns.nix @@ -0,0 +1,32 @@ +# k3d-configure-dns.nix - Patch CoreDNS to forward sslip.io queries to public DNS. +# +# Usage: +# nix run .#k3d-configure-dns +# +# Template form: pure readFile (no nix-computed variable injection). +# Required because OrbStack's default DNS (192.168.107.1) cannot resolve +# sslip.io wildcards used by the local ArgoCD application routes. +# Idempotent: second invocation exits 0 without patching. +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.k3d-configure-dns = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "k3d-configure-dns"; + runtimeInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.gnugrep + pkgs.jq + pkgs.kubectl + ]; + text = builtins.readFile ./k3d-configure-dns.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/k3d-configure-dns.sh b/modules/apps/cluster/k3d-configure-dns.sh new file mode 100644 index 000000000..1047a2b82 --- /dev/null +++ b/modules/apps/cluster/k3d-configure-dns.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Patch the k3d cluster's CoreDNS to forward sslip.io queries to public +# DNS resolvers so ArgoCD Application routes using .sslip.io domains +# resolve inside the cluster. Idempotent: re-running on an +# already-configured cluster detects the existing "sslip.io" block and +# exits 0 without mutation. +# +# Usage: +# k3d-configure-dns [--help] +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: k3d-configure-dns [--help] + +Patches the kube-system/coredns ConfigMap to add a + sslip.io:53 { forward . 1.1.1.1 8.8.8.8; cache 30 } +stanza, then rolls the coredns Deployment so the new Corefile takes +effect. Idempotent; re-running on an already-patched cluster is a no-op. + +Requires kubectl context pointing at a running k3d cluster with +Cilium (or another CNI) already Ready. +EOF + exit 0 + ;; +esac + +echo "Waiting for CoreDNS to be running..." +kubectl wait --for=condition=Ready pod -l k8s-app=kube-dns -n kube-system --timeout=120s + +echo "Patching CoreDNS ConfigMap to forward sslip.io to public DNS..." +CURRENT=$(kubectl get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}') +if echo "$CURRENT" | grep -q "sslip.io"; then + echo "CoreDNS already configured for sslip.io forwarding" + exit 0 +fi + +SSLIP_BLOCK=$'sslip.io:53 {\n forward . 1.1.1.1 8.8.8.8\n cache 30\n}\n' +PATCHED="${SSLIP_BLOCK}${CURRENT}" +PATCH_JSON=$(jq -n --arg corefile "$PATCHED" '{"data": {"Corefile": $corefile}}') +kubectl patch configmap coredns -n kube-system --type=merge -p "$PATCH_JSON" + +echo "Restarting CoreDNS deployment..." +kubectl rollout restart deployment coredns -n kube-system + +echo "Waiting for CoreDNS to be ready..." +kubectl rollout status deployment coredns -n kube-system --timeout=120s + +echo "CoreDNS configured for sslip.io forwarding" diff --git a/modules/apps/cluster/k3d-full.nix b/modules/apps/cluster/k3d-full.nix new file mode 100644 index 000000000..78f8c29d1 --- /dev/null +++ b/modules/apps/cluster/k3d-full.nix @@ -0,0 +1,34 @@ +# k3d-full.nix - Full local-k3d lifecycle: down -> up -> deploy. +# +# Usage: +# nix run .#k3d-full +# +# Template form: pure readFile (no nix-computed variable injection). +# Orchestration wrapper that delegates to the underlying justfile +# recipes for k3d-down, k3d-up, and k3d-deploy — none of which are in +# the M1 flake-app conversion scope. `just` is therefore included as a +# runtimeInput. The recipes themselves still need k3d, ctlptl, kubectl, +# etc. on PATH; those come from the user's dev environment (writeShell- +# Application prepends runtimeInputs to $PATH without stripping it). +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.k3d-full = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "k3d-full"; + runtimeInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.git + pkgs.just + ]; + text = builtins.readFile ./k3d-full.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/k3d-full.sh b/modules/apps/cluster/k3d-full.sh new file mode 100644 index 000000000..b07cb705e --- /dev/null +++ b/modules/apps/cluster/k3d-full.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Full local-k3d lifecycle: tear down any existing cluster, recreate it, +# and deploy foundation + infrastructure layers. Delegates to the +# original just recipes (k3d-down, k3d-up, k3d-deploy) which remain the +# single source of truth for the cluster wiring during the M1 transition +# window. +# +# Usage: +# k3d-full [--help] +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: k3d-full [--help] + +Runs, in order: + just k3d-down || true (idempotent teardown) + just k3d-up (ctlptl apply + bootstrap-secrets) + just k3d-deploy (foundation + infrastructure layers) + +The invocation must happen from a directory inside the vanixiets git +worktree (repo root resolution via `git rev-parse --show-toplevel`), +since the underlying just recipes reference +kubernetes/clusters/local-k3d/cluster.yaml by relative path. +EOF + exit 0 + ;; +esac + +repo_root=$(git rev-parse --show-toplevel 2>/dev/null || true) +if [[ -n "$repo_root" ]]; then + cd "$repo_root" +fi + +just k3d-down || true +just k3d-up +just k3d-deploy diff --git a/modules/apps/cluster/k3d-integration-ci.nix b/modules/apps/cluster/k3d-integration-ci.nix new file mode 100644 index 000000000..46890231a --- /dev/null +++ b/modules/apps/cluster/k3d-integration-ci.nix @@ -0,0 +1,37 @@ +# k3d-integration-ci.nix - CI-variant full integration: file:///manifests + tests. +# +# Usage: +# nix run .#k3d-integration-ci +# +# Template form: pure readFile (no nix-computed variable injection). +# Orchestrates the seven-phase CI integration flow that is currently +# invoked by `.github/workflows/test-cluster.yaml`. Delegates to the +# sibling cluster/docs flake apps (nixidy-build, nixidy-bootstrap, +# k3d-wait-*, k3d-test-coverage) via `just `; those recipes are +# thin `nix run` wrappers after M1. `just` is the single external +# dispatch mechanism, so it is the only orchestration-layer runtimeInput; +# the underlying tools (ctlptl, k3d, kubectl, …) come from the invoking +# dev shell's PATH (writeShellApplication prepends runtimeInputs to $PATH). +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.k3d-integration-ci = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "k3d-integration-ci"; + runtimeInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.git + pkgs.just + pkgs.rsync + ]; + text = builtins.readFile ./k3d-integration-ci.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/k3d-integration-ci.sh b/modules/apps/cluster/k3d-integration-ci.sh new file mode 100644 index 000000000..c35296c96 --- /dev/null +++ b/modules/apps/cluster/k3d-integration-ci.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# CI integration driver for the local-k3d cluster. Uses the local +# file:///manifests repo URL (no GitHub credentials required) and +# orchestrates the full seven-phase flow consumed by +# .github/workflows/test-cluster.yaml's `integration` job. +# +# Usage: +# k3d-integration-ci [--help] +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: k3d-integration-ci [--help] + +Phases: + 1. nixidy-build with ARGOCD_REPO_URL=file:///manifests + 2. Stage /tmp/k3d-manifests as a fresh git repo (cluster volume mount target) + 3. k3d-full (ctlptl create + kluctl deploy) + 4. k3d-wait-ready (foundation + infra Ready) + 5. nixidy-bootstrap (app-of-apps sync via file:///manifests) + 6. k3d-wait-argocd-sync (all Applications Synced + Healthy) + 7. k3d-test-coverage (chainsaw tests + coverage report) + +Must be invoked from a directory inside the vanixiets git worktree. +EOF + exit 0 + ;; +esac + +repo_root=$(git rev-parse --show-toplevel) +cd "$repo_root" + +echo "=== Phase 1: Build manifests with local repo URL ===" +export ARGOCD_REPO_URL="file:///manifests" +just nixidy-build + +echo "" +echo "=== Phase 2: Prepare local git repo (before cluster for volume mount) ===" +# Ensure writable before cleanup (Nix store copies may be read-only) +chmod -R +w /tmp/k3d-manifests 2>/dev/null || true +rm -rf /tmp/k3d-manifests +mkdir -p /tmp/k3d-manifests +rsync -aL --delete --chmod=Du+w,Fu+w result/ /tmp/k3d-manifests/ +( + cd /tmp/k3d-manifests + git init -b main + git config user.email "ci@localhost" + git config user.name "CI" + git add . + git commit -m "CI manifests" +) + +echo "" +echo "=== Phase 3: Create cluster and deploy via kluctl ===" +just k3d-full + +echo "" +echo "=== Phase 4: Wait for infrastructure ready ===" +just k3d-wait-ready + +echo "" +echo "=== Phase 5: Bootstrap ArgoCD (syncs from file:///manifests) ===" +just nixidy-bootstrap + +echo "" +echo "=== Phase 6: Wait for ArgoCD sync ===" +just k3d-wait-argocd-sync + +echo "" +echo "=== Phase 7: Run integration tests ===" +just k3d-test-coverage + +echo "" +echo "=== CI integration complete ===" diff --git a/modules/apps/cluster/k3d-test-coverage.nix b/modules/apps/cluster/k3d-test-coverage.nix new file mode 100644 index 000000000..5a6957d63 --- /dev/null +++ b/modules/apps/cluster/k3d-test-coverage.nix @@ -0,0 +1,41 @@ +# k3d-test-coverage.nix - Run chainsaw integration tests and emit coverage report. +# +# Usage: +# nix run .#k3d-test-coverage -- [--raw] [chainsaw args...] +# +# Template form: pure readFile (no nix-computed variable injection). +# Subsumes scripts/k3d-test-coverage.sh (legacy root-level copy kept as +# a thin shim in M5 for backward compatibility). The coverage-report +# logic lives in-tree at modules/apps/cluster/k3d-test-coverage.sh. +# +# Resolves kubernetes/tests/local-k3d/ relative to the invoking git +# worktree via `git rev-parse --show-toplevel`. +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.k3d-test-coverage = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "k3d-test-coverage"; + runtimeInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.findutils + pkgs.gawk + pkgs.git + pkgs.gnugrep + pkgs.gnused + pkgs.jq + pkgs.kubectl + pkgs.kyverno-chainsaw + pkgs.libxml2 # xmllint + ]; + text = builtins.readFile ./k3d-test-coverage.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/k3d-test-coverage.sh b/modules/apps/cluster/k3d-test-coverage.sh new file mode 100644 index 000000000..d0a2c166b --- /dev/null +++ b/modules/apps/cluster/k3d-test-coverage.sh @@ -0,0 +1,535 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Run chainsaw tests with coverage report showing tested vs deployed resources. +# +# Usage: k3d-test-coverage [--help] [--raw] [chainsaw args...] +# +# Options: +# --help Show this message and exit 0 +# --raw Show raw uncategorized output (original format) +# +# Environment: +# CI, GITHUB_ACTIONS, NO_COLOR - Disable colors when set +# +# Exit codes: +# 0 - All tests passed +# 1 - Tests failed or error +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: k3d-test-coverage [--help] [--raw] [chainsaw args...] + +Run chainsaw integration tests against kubernetes/tests/local-k3d/ and +emit a coverage report categorizing deployed resources as application, +foundation, or system. + +Options: + --help Show this message and exit 0 + --raw Show raw uncategorized coverage output + +Environment: + CI / GITHUB_ACTIONS / NO_COLOR Disable ANSI colors when set +EOF + exit 0 + ;; +esac + +# Global flag for raw output mode +RAW_MODE=0 + +# shellcheck disable=SC2034 # Colors are used via variable expansion +setup_colors() { + if [[ -n "${CI:-}" ]] || [[ -n "${GITHUB_ACTIONS:-}" ]] || [[ -n "${NO_COLOR:-}" ]]; then + RED="" GREEN="" YELLOW="" BOLD="" DIM="" RESET="" + else + RED=$'\e[31m' GREEN=$'\e[32m' YELLOW=$'\e[33m' + BOLD=$'\e[1m' DIM=$'\e[2m' RESET=$'\e[0m' + fi +} + +run_chainsaw_tests() { + local report_dir="$1" + local test_dir="$2" + shift 2 + + echo "${BOLD}Running chainsaw tests...${RESET}" + echo "" + + if chainsaw test "$test_dir" "$@" \ + --report-format JUNIT-OPERATION \ + --report-path "$report_dir" 2>&1; then + return 0 + else + return 1 + fi +} + +print_test_summary() { + local report_file="$1" + + echo "" + echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" + echo "${BOLD} TEST SUMMARY ${RESET}" + echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" + echo "" + + if [[ ! -f "$report_file" ]]; then + echo " ${YELLOW}Warning: No test report found${RESET}" + return + fi + + local total_ops total_time + total_ops=$(xmllint --xpath 'string(/testsuites/@tests)' "$report_file" 2>/dev/null || echo "0") + total_time=$(xmllint --xpath 'string(/testsuites/@time)' "$report_file" 2>/dev/null || echo "0") + + echo "${BOLD}Test Execution:${RESET}" + echo " Total operations: ${GREEN}${total_ops}${RESET}" + echo " Total time: ${DIM}${total_time}s${RESET}" + echo "" + + echo "${BOLD}By Test Suite:${RESET}" + local suite suite_tests suite_failures suite_time status + for suite in foundation infrastructure local-k3d; do + suite_tests=$(xmllint --xpath "string(//testsuite[@name='$suite']/@tests)" "$report_file" 2>/dev/null || echo "0") + suite_failures=$(xmllint --xpath "string(//testsuite[@name='$suite']/@failures)" "$report_file" 2>/dev/null || echo "0") + suite_time=$(xmllint --xpath "string(//testsuite[@name='$suite']/@time)" "$report_file" 2>/dev/null || echo "0") + + if [[ "$suite_tests" != "0" ]]; then + if [[ "$suite_failures" == "0" ]]; then + status="${GREEN}PASS${RESET}" + else + status="${RED}FAIL${RESET}" + fi + printf " %-20s %s %3s ops ${DIM}%ss${RESET}\n" "$suite" "$status" "$suite_tests" "$suite_time" + fi + done +} + +collect_deployed_resources() { + local -n deployed_ref=$1 + local -n type_counts_ref=$2 + + # Workloads (Deployment, StatefulSet, DaemonSet) + local line ns name kind key + while IFS= read -r line; do + [[ -z "$line" ]] && continue + ns=$(awk '{print $1}' <<< "$line") + name=$(awk '{print $2}' <<< "$line") + kind=$(awk '{print $3}' <<< "$line") + key="${kind}/${ns}/${name}" + deployed_ref["$key"]=1 + done < <(kubectl get deploy,sts,ds -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,KIND:.kind' --no-headers 2>/dev/null | grep -v '^$') + + # Gateway API resources + while IFS= read -r line; do + [[ -z "$line" ]] && continue + ns=$(awk '{print $1}' <<< "$line") + name=$(awk '{print $2}' <<< "$line") + deployed_ref["Gateway/${ns}/${name}"]=1 + done < <(kubectl get gateway -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') + + while IFS= read -r line; do + [[ -z "$line" ]] && continue + ns=$(awk '{print $1}' <<< "$line") + name=$(awk '{print $2}' <<< "$line") + deployed_ref["HTTPRoute/${ns}/${name}"]=1 + done < <(kubectl get httproute -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') + + # Certificates + while IFS= read -r line; do + [[ -z "$line" ]] && continue + ns=$(awk '{print $1}' <<< "$line") + name=$(awk '{print $2}' <<< "$line") + deployed_ref["Certificate/${ns}/${name}"]=1 + done < <(kubectl get certificate -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') + + # ClusterIssuers (cluster-scoped) + while IFS= read -r line; do + [[ -z "$line" ]] && continue + deployed_ref["ClusterIssuer/-/${line}"]=1 + done < <(kubectl get clusterissuer -o custom-columns='NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') + + # Count by type + for key in "${!deployed_ref[@]}"; do + kind="${key%%/*}" + type_counts_ref["$kind"]=$(( ${type_counts_ref["$kind"]:-0} + 1 )) + done +} + +collect_tested_resources() { + local -n tested_ref=$1 + # shellcheck disable=SC2178 # nameref to associative array + local -n type_counts_ref=$2 + local test_dir="$3" + + local file current_kind current_name current_ns line key + + while IFS= read -r file; do + current_kind="" + current_name="" + current_ns="-" + + while IFS= read -r line; do + # New document resets state + if [[ "$line" == "---" ]]; then + if [[ -n "$current_kind" && -n "$current_name" ]]; then + key="${current_kind}/${current_ns}/${current_name}" + tested_ref["$key"]=1 + fi + current_kind="" + current_name="" + current_ns="-" + continue + fi + + # Extract kind + if [[ "$line" =~ ^kind:\ *(.+)$ ]]; then + current_kind="${BASH_REMATCH[1]}" + fi + + # Extract name (first name field is metadata.name) + if [[ "$line" =~ ^[[:space:]]+name:\ *(.+)$ ]]; then + if [[ -z "$current_name" ]]; then + current_name="${BASH_REMATCH[1]}" + fi + fi + + # Extract namespace + if [[ "$line" =~ ^[[:space:]]+namespace:\ *(.+)$ ]]; then + current_ns="${BASH_REMATCH[1]}" + fi + done < "$file" + + # Last resource in file + if [[ -n "$current_kind" && -n "$current_name" ]]; then + key="${current_kind}/${current_ns}/${current_name}" + tested_ref["$key"]=1 + fi + done < <(find "$test_dir" -name "*assert*.yaml" -type f) + + # Count by type + for key in "${!tested_ref[@]}"; do + kind="${key%%/*}" + type_counts_ref["$kind"]=$(( ${type_counts_ref["$kind"]:-0} + 1 )) + done +} + +print_resource_table() { + local -n counts_ref=$1 + local total=$2 + + local kind count + for kind in Deployment StatefulSet DaemonSet Gateway HTTPRoute Certificate ClusterIssuer; do + count="${counts_ref[$kind]:-0}" + if [[ "$count" -gt 0 ]]; then + printf " %-15s %3d\n" "$kind" "$count" + fi + done + echo " ${DIM}─────────────────────${RESET}" + printf " %-15s %3d\n" "Total" "$total" +} + +# Categorize a resource as application, foundation, or system +# Returns: "application", "foundation", or "system" +categorize_resource() { + local key="$1" + local kind="${key%%/*}" + local rest="${key#*/}" + local ns="${rest%%/*}" + local name="${rest#*/}" + + # System components (k3s internals, Cilium internals, auto-generated) + # These are excluded from coverage calculation because they are: + # - Not managed by our nixidy/ArgoCD stack + # - Auto-created by k3s or other controllers + # - Internal components of our foundation layer + case "$key" in + # k3s DNS - managed by k3s, not our stack + Deployment/kube-system/coredns) echo "system"; return ;; + # k3s storage provisioner - managed by k3s + Deployment/kube-system/local-path-provisioner) echo "system"; return ;; + # k3s metrics - managed by k3s + Deployment/kube-system/metrics-server) echo "system"; return ;; + # Cilium internal envoy proxy - managed by Cilium operator + DaemonSet/kube-system/cilium-envoy) echo "system"; return ;; + # Auto-generated by cert-manager gateway-shim from HTTPRoute annotation + # Duplicates our explicit step-ca-tls Certificate + Certificate/gateway-system/test-cert-tls) echo "system"; return ;; + esac + + # k3s servicelb auto-created DaemonSets (svclb-*) + # These are auto-created by k3s for LoadBalancer services + if [[ "$kind" == "DaemonSet" && "$ns" == "kube-system" && "$name" == svclb-* ]]; then + echo "system" + return + fi + + # Foundation resources (CNI layer we deploy but is infrastructure) + case "$key" in + DaemonSet/kube-system/cilium) echo "foundation"; return ;; + Deployment/kube-system/cilium-operator) echo "foundation"; return ;; + esac + + # Application resources (our nixidy/ArgoCD managed stack) + # Includes: argocd, cert-manager, sops-secrets-operator, step-ca, + # gateway-system, plus Gateway API resources + echo "application" +} + +# Get human-readable description for system components +get_system_description() { + local key="$1" + + case "$key" in + Deployment/kube-system/coredns) echo "k3s DNS" ;; + Deployment/kube-system/local-path-provisioner) echo "k3s storage" ;; + Deployment/kube-system/metrics-server) echo "k3s metrics" ;; + DaemonSet/kube-system/cilium-envoy) echo "Cilium internal" ;; + Certificate/gateway-system/test-cert-tls) echo "gateway-shim duplicate" ;; + DaemonSet/kube-system/svclb-*) echo "k3s servicelb auto-created" ;; + *) echo "system component" ;; + esac +} + +print_coverage_report() { + local test_dir="$1" + + echo "" + echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" + echo "${BOLD} RESOURCE COVERAGE ${RESET}" + echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" + echo "" + + declare -A deployed_resources + # shellcheck disable=SC2034 # passed to function via nameref + declare -A deployed_type_counts + declare -A tested_resources + # shellcheck disable=SC2034 # passed to function via nameref + declare -A tested_type_counts + + collect_deployed_resources deployed_resources deployed_type_counts + collect_tested_resources tested_resources tested_type_counts "$test_dir" + + local deployed_count=${#deployed_resources[@]} + local tested_count=${#tested_resources[@]} + + echo "${BOLD}Deployed Resources:${RESET}" + print_resource_table deployed_type_counts "$deployed_count" + + echo "" + echo "${BOLD}Tested Resources:${RESET}" + print_resource_table tested_type_counts "$tested_count" + + echo "" + echo "${BOLD}Coverage Analysis:${RESET}" + + # Categorize resources and calculate coverage + local matched=0 + local untested=() + local key category + + # Categorized counts + local app_total=0 app_tested=0 + local foundation_total=0 foundation_tested=0 + local system_total=0 system_tested=0 + local system_resources=() + + for key in "${!deployed_resources[@]}"; do + category=$(categorize_resource "$key") + + case "$category" in + application) + (( app_total++ )) || true + if [[ -n "${tested_resources[$key]:-}" ]]; then + (( app_tested++ )) || true + (( matched++ )) || true + else + untested+=("$key") + fi + ;; + foundation) + (( foundation_total++ )) || true + if [[ -n "${tested_resources[$key]:-}" ]]; then + (( foundation_tested++ )) || true + (( matched++ )) || true + else + untested+=("$key") + fi + ;; + system) + (( system_total++ )) || true + system_resources+=("$key") + if [[ -n "${tested_resources[$key]:-}" ]]; then + (( system_tested++ )) || true + (( matched++ )) || true + fi + ;; + esac + done + + # Calculate raw coverage (all resources) + local raw_coverage=0 + if [[ $deployed_count -gt 0 ]]; then + raw_coverage=$(( matched * 100 / deployed_count )) + fi + + # Calculate managed coverage (excluding system components) + local managed_total=$(( app_total + foundation_total )) + local managed_tested=$(( app_tested + foundation_tested )) + local managed_coverage=0 + if [[ $managed_total -gt 0 ]]; then + managed_coverage=$(( managed_tested * 100 / managed_total )) + fi + + if [[ $RAW_MODE -eq 1 ]]; then + # Original raw output format + local cov_color="$RED" + if [[ $raw_coverage -ge 80 ]]; then + cov_color="$GREEN" + elif [[ $raw_coverage -ge 50 ]]; then + cov_color="$YELLOW" + fi + + echo "" + echo " Resource instance coverage: ${cov_color}${BOLD}${raw_coverage}%${RESET} (${matched}/${deployed_count})" + echo "" + + if [[ ${#untested[@]} -gt 0 ]] || [[ ${#system_resources[@]} -gt 0 ]]; then + echo "${BOLD}Untested Resources:${RESET}" + local rest ns name + # Combine untested managed resources with untested system resources + local all_untested=() + for key in "${untested[@]}"; do + all_untested+=("$key") + done + for key in "${system_resources[@]}"; do + if [[ -z "${tested_resources[$key]:-}" ]]; then + all_untested+=("$key") + fi + done + for key in "${all_untested[@]}"; do + kind="${key%%/*}" + rest="${key#*/}" + ns="${rest%%/*}" + name="${rest#*/}" + if [[ "$ns" == "-" ]]; then + printf " ${YELLOW}○${RESET} ${DIM}%-15s${RESET} %s\n" "$kind" "$name" + else + printf " ${YELLOW}○${RESET} ${DIM}%-15s${RESET} %s ${DIM}(%s)${RESET}\n" "$kind" "$name" "$ns" + fi + done | sort + else + echo " ${GREEN}All deployed resources have test coverage${RESET}" + fi + else + # Categorized output format + echo "" + echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" + echo "${BOLD} COVERAGE BY CATEGORY ${RESET}" + echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" + echo "" + + # Application coverage + local app_cov_pct=0 + if [[ $app_total -gt 0 ]]; then + app_cov_pct=$(( app_tested * 100 / app_total )) + fi + local app_color="$RED" + [[ $app_cov_pct -ge 80 ]] && app_color="$GREEN" + [[ $app_cov_pct -ge 50 && $app_cov_pct -lt 80 ]] && app_color="$YELLOW" + printf " Application Resources: ${app_color}%2d/%2d (%3d%%)${RESET}\n" "$app_tested" "$app_total" "$app_cov_pct" + + # Foundation coverage + local fnd_cov_pct=0 + if [[ $foundation_total -gt 0 ]]; then + fnd_cov_pct=$(( foundation_tested * 100 / foundation_total )) + fi + local fnd_color="$RED" + [[ $fnd_cov_pct -ge 80 ]] && fnd_color="$GREEN" + [[ $fnd_cov_pct -ge 50 && $fnd_cov_pct -lt 80 ]] && fnd_color="$YELLOW" + printf " Foundation Resources: ${fnd_color}%2d/%2d (%3d%%)${RESET}\n" "$foundation_tested" "$foundation_total" "$fnd_cov_pct" + + echo " ${DIM}─────────────────────────────────────────────────────────────────${RESET}" + + # Managed total + local mgd_color="$RED" + [[ $managed_coverage -ge 80 ]] && mgd_color="$GREEN" + [[ $managed_coverage -ge 50 && $managed_coverage -lt 80 ]] && mgd_color="$YELLOW" + printf " ${BOLD}Managed Resources Total: ${mgd_color}%2d/%2d (%3d%%)${RESET}\n" "$managed_tested" "$managed_total" "$managed_coverage" + + # Untested managed resources + if [[ ${#untested[@]} -gt 0 ]]; then + echo "" + echo "${BOLD}Untested Managed Resources:${RESET}" + local rest ns name + for key in "${untested[@]}"; do + kind="${key%%/*}" + rest="${key#*/}" + ns="${rest%%/*}" + name="${rest#*/}" + if [[ "$ns" == "-" ]]; then + printf " ${YELLOW}○${RESET} ${DIM}%-15s${RESET} %s\n" "$kind" "$name" + else + printf " ${YELLOW}○${RESET} ${DIM}%-15s${RESET} %s ${DIM}(%s)${RESET}\n" "$kind" "$name" "$ns" + fi + done | sort + fi + + # System components section + echo "" + echo "${BOLD}System Components (excluded from coverage):${RESET}" + local rest ns name desc + for key in "${system_resources[@]}"; do + kind="${key%%/*}" + rest="${key#*/}" + ns="${rest%%/*}" + name="${rest#*/}" + desc=$(get_system_description "$key") + printf " ${DIM}○${RESET} ${DIM}%-15s${RESET} %s ${DIM}(%s)${RESET} - ${DIM}%s${RESET}\n" "$kind" "$name" "$ns" "$desc" + done | sort + + echo "" + printf " ${DIM}Raw Resource Count: %2d/%2d (%3d%%)${RESET}\n" "$matched" "$deployed_count" "$raw_coverage" + fi + + echo "" + echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" +} + +main() { + setup_colors + + # Parse --raw flag + local args=() + for arg in "$@"; do + if [[ "$arg" == "--raw" ]]; then + RAW_MODE=1 + else + args+=("$arg") + fi + done + + # Resolve test directory relative to the invoking worktree so the + # flake app is location-independent. + local repo_root + repo_root=$(git rev-parse --show-toplevel) + local test_dir="${repo_root}/kubernetes/tests/local-k3d" + + local report_dir + report_dir=$(mktemp -d) + trap 'rm -rf "$report_dir"' EXIT + + local test_failed=0 + if ! run_chainsaw_tests "$report_dir" "$test_dir" "${args[@]}"; then + test_failed=1 + fi + + print_test_summary "$report_dir/chainsaw-report.xml" + print_coverage_report "$test_dir" + + exit $test_failed +} + +main "$@" diff --git a/modules/apps/cluster/k3d-wait-argocd-sync.nix b/modules/apps/cluster/k3d-wait-argocd-sync.nix new file mode 100644 index 000000000..226907f2b --- /dev/null +++ b/modules/apps/cluster/k3d-wait-argocd-sync.nix @@ -0,0 +1,30 @@ +# k3d-wait-argocd-sync.nix - Wait for all ArgoCD Applications to reach Synced + Healthy. +# +# Usage: +# nix run .#k3d-wait-argocd-sync +# +# Template form: pure readFile (no nix-computed variable injection). +# Matches the Phase-4 post-bootstrap gating from the justfile +# `k3d-wait-argocd-sync` recipe. The expected-apps list mirrors the +# nixidy sync-wave declarations and is the source of truth at this layer. +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.k3d-wait-argocd-sync = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "k3d-wait-argocd-sync"; + runtimeInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.kubectl + ]; + text = builtins.readFile ./k3d-wait-argocd-sync.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/k3d-wait-argocd-sync.sh b/modules/apps/cluster/k3d-wait-argocd-sync.sh new file mode 100644 index 000000000..2bd1b255c --- /dev/null +++ b/modules/apps/cluster/k3d-wait-argocd-sync.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Block until every ArgoCD Application in the local-k3d cluster is both +# Healthy and Synced, then verify the root Gateway is Programmed by +# Cilium's Gateway API implementation. +# +# Usage: +# k3d-wait-argocd-sync [--help] +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: k3d-wait-argocd-sync [--help] + +Waits for the app-of-apps-managed ArgoCD Applications to come online in +the cluster, then gates on each becoming Healthy and Synced, and finally +waits for the main-gateway Gateway to be Programmed by Cilium. + +Sync waves (from nixidy local-k3d env): + Wave -1 (adoption): cilium, argocd, sops-secrets-operator, step-ca + Wave 0: cert-manager + Wave 1-2: cluster-issuer, gateway, gateway-api + Wave 3: argocd-route +EOF + exit 0 + ;; +esac + +echo "=== Waiting for ArgoCD Applications ===" +echo "Applications managed by nixidy sync waves:" +echo " Wave -1 (adoption): cilium, argocd, sops-secrets-operator, step-ca" +echo " Wave 0: cert-manager" +echo " Wave 1-2: cluster-issuer, gateway, gateway-api" +echo " Wave 3: argocd-route" +echo "" + +# All expected applications (app-of-apps creates these asynchronously) +EXPECTED_APPS=( + apps + argocd + argocd-route + cert-manager + cilium + cluster-issuer + gateway + gateway-api + sops-secrets-operator + step-ca +) + +echo "Waiting for all ${#EXPECTED_APPS[@]} applications to exist..." +for app in "${EXPECTED_APPS[@]}"; do + echo -n " Waiting for $app... " + # kubectl wait fails immediately if resource doesn't exist, so poll instead + timeout 300 bash -c "until kubectl get application/$app -n argocd &>/dev/null; do sleep 2; done" + echo "exists" +done + +echo "" +echo "Listing applications..." +kubectl get applications -n argocd -o wide || true +echo "" + +echo "Waiting for all applications to be Healthy..." +for app in "${EXPECTED_APPS[@]}"; do + echo -n " Waiting for $app to be Healthy... " + kubectl wait --for=jsonpath='{.status.health.status}'=Healthy application/"$app" -n argocd --timeout=600s >/dev/null + echo "done" +done + +echo "" +echo "Waiting for all applications to be Synced..." +for app in "${EXPECTED_APPS[@]}"; do + echo -n " Waiting for $app to be Synced... " + kubectl wait --for=jsonpath='{.status.sync.status}'=Synced application/"$app" -n argocd --timeout=300s >/dev/null + echo "done" +done + +echo "" +echo "=== Waiting for Gateway to be programmed ===" +# ArgoCD reports Healthy before Cilium fully programs the Gateway +# Wait for the actual Gateway condition, not just ArgoCD's view +kubectl wait --for=condition=Programmed gateway/main-gateway -n gateway-system --timeout=300s + +echo "" +echo "=== All ArgoCD applications synced and healthy ===" +kubectl get applications -n argocd -o wide diff --git a/modules/apps/cluster/k3d-wait-ready.nix b/modules/apps/cluster/k3d-wait-ready.nix new file mode 100644 index 000000000..4abb92056 --- /dev/null +++ b/modules/apps/cluster/k3d-wait-ready.nix @@ -0,0 +1,29 @@ +# k3d-wait-ready.nix - Block until kluctl-deployed foundation + infra pods are Ready. +# +# Usage: +# nix run .#k3d-wait-ready +# +# Template form: pure readFile (no nix-computed variable injection). +# Mirrors the Phase 3 post-deploy gating that sat in the justfile +# `k3d-wait-ready` recipe. All kubectl waits have deterministic timeouts. +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.k3d-wait-ready = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "k3d-wait-ready"; + runtimeInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.kubectl + ]; + text = builtins.readFile ./k3d-wait-ready.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/k3d-wait-ready.sh b/modules/apps/cluster/k3d-wait-ready.sh new file mode 100644 index 000000000..d29146d90 --- /dev/null +++ b/modules/apps/cluster/k3d-wait-ready.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Wait for the kluctl-deployed foundation (Cilium) and infrastructure +# (ArgoCD, sops-secrets-operator, step-ca) pods to reach Ready. +# +# Usage: +# k3d-wait-ready [--help] +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: k3d-wait-ready [--help] + +Blocks until all Phase-3 (foundation + infrastructure) pods are Ready in +the local-k3d cluster, in the order: + + Foundation: cilium-agent, cilium-operator (kube-system) + Infrastructure: argocd deployments, argocd-app-ctrl (argocd) + step-ca statefulset pod (step-ca) + sops-secrets-operator deployments (sops-secrets-operator) + +Each kubectl-wait carries a 300s timeout. Requires kubectl context +pointing at a live k3d cluster with all manifests already applied. +EOF + exit 0 + ;; +esac + +echo "=== Waiting for Foundation (CNI) ===" +echo "Waiting for Cilium Agent..." +kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=cilium-agent -n kube-system --timeout=300s + +echo "Waiting for Cilium Operator..." +kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=cilium-operator -n kube-system --timeout=300s + +echo "" +echo "=== Waiting for Infrastructure ===" +echo "Waiting for ArgoCD deployments..." +kubectl wait --for=condition=Available deployment --all -n argocd --timeout=300s + +echo "Waiting for ArgoCD Application Controller..." +kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=argocd-application-controller -n argocd --timeout=300s + +echo "Waiting for step-ca..." +# Use StatefulSet pod label to exclude Helm test-connection pod (which always fails) +kubectl wait --for=condition=Ready pod -l statefulset.kubernetes.io/pod-name=step-ca-step-certificates-0 -n step-ca --timeout=300s + +echo "Waiting for sops-secrets-operator..." +kubectl wait --for=condition=Available deployment --all -n sops-secrets-operator --timeout=300s + +echo "" +echo "=== All foundation and infrastructure pods ready ===" diff --git a/modules/apps/cluster/list-packages-json.nix b/modules/apps/cluster/list-packages-json.nix new file mode 100644 index 000000000..5ae137ee0 --- /dev/null +++ b/modules/apps/cluster/list-packages-json.nix @@ -0,0 +1,30 @@ +# list-packages-json.nix - Emit a JSON matrix of workspace packages. +# +# Usage: +# nix run .#list-packages-json +# +# Template form: pure readFile (no nix-computed variable injection). +# +# Enumerates packages// directories containing a package.json +# and emits a JSON array of {name, path} entries consumed by the +# preview-release-version matrix step in cd.yaml's set-variables job. +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.list-packages-json = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "list-packages-json"; + runtimeInputs = [ + pkgs.coreutils + pkgs.git + ]; + text = builtins.readFile ./list-packages-json.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/list-packages-json.sh b/modules/apps/cluster/list-packages-json.sh new file mode 100644 index 000000000..a005cd2e1 --- /dev/null +++ b/modules/apps/cluster/list-packages-json.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Emit a JSON matrix entry per packages// with a package.json. +# +# Usage: +# list-packages-json [--help] +# +# Output: a single JSON array line of {name, path} objects on stdout. +# Resolves the repo root via `git rev-parse --show-toplevel`, so callers +# may invoke from any subdirectory of the vanixiets worktree. +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: list-packages-json [--help] + +Emit a JSON array of {"name": "", "path": "packages/"} for every +packages// directory containing a package.json. Consumed by the +preview-release-version CI matrix in cd.yaml (set-variables job). + +No positional arguments; must run inside a git worktree rooted at the +vanixiets repo (or subdirectory thereof). +EOF + exit 0 + ;; +esac + +repo_root=$(git rev-parse --show-toplevel) +cd "$repo_root/packages" + +packages=() +for dir in */; do + pkg_name="${dir%/}" + if [ -f "${dir}package.json" ]; then + packages+=("{\"name\":\"$pkg_name\",\"path\":\"packages/$pkg_name\"}") + fi +done + +# Emit a JSON array; empty case still produces a valid "[]". +( + IFS=, + echo "[${packages[*]}]" +) diff --git a/modules/apps/cluster/nixidy-bootstrap.nix b/modules/apps/cluster/nixidy-bootstrap.nix new file mode 100644 index 000000000..6042ff007 --- /dev/null +++ b/modules/apps/cluster/nixidy-bootstrap.nix @@ -0,0 +1,34 @@ +# nixidy-bootstrap.nix - Apply the local-k3d app-of-apps bootstrap Application CR. +# +# Usage: +# nix run .#nixidy-bootstrap +# +# Template form: pure readFile (no nix-computed variable injection). +# Emits the bootstrap Application CR to stdout and pipes it into +# `kubectl apply -f -` against the live k3d cluster context. +{ ... }: +{ + perSystem = + { + pkgs, + lib, + config, + ... + }: + { + apps.nixidy-bootstrap = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "nixidy-bootstrap"; + runtimeInputs = [ + pkgs.coreutils + pkgs.kubectl + config.packages.nixidy + ]; + text = builtins.readFile ./nixidy-bootstrap.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/nixidy-bootstrap.sh b/modules/apps/cluster/nixidy-bootstrap.sh new file mode 100644 index 000000000..a5cf31667 --- /dev/null +++ b/modules/apps/cluster/nixidy-bootstrap.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Apply the app-of-apps bootstrap Application CR for the local-k3d +# environment: renders the manifest with `nixidy bootstrap` and pipes it +# into kubectl apply -f - against the cluster context in use. +# +# Usage: +# nixidy-bootstrap [--help] +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: nixidy-bootstrap [--help] + +Equivalent to: + nixidy bootstrap .#local-k3d | kubectl apply -f - + +Transitions Phase 3 (kluctl-driven) infrastructure to Phase 4 (ArgoCD +app-of-apps). ArgoCD must already be Available before invoking, and it +must have credentials to access the local-k3d manifest repo referenced +by the rendered Application CR. +EOF + exit 0 + ;; +esac + +nixidy bootstrap .#local-k3d | kubectl apply -f - diff --git a/modules/apps/cluster/nixidy-build.nix b/modules/apps/cluster/nixidy-build.nix new file mode 100644 index 000000000..0ab6f0670 --- /dev/null +++ b/modules/apps/cluster/nixidy-build.nix @@ -0,0 +1,35 @@ +# nixidy-build.nix - Render nixidy manifests for local-k3d to ./result. +# +# Usage: +# nix run .#nixidy-build +# +# Template form: pure readFile (no nix-computed variable injection). +# The nixidy CLI is exposed via config.packages.nixidy (set in +# modules/nixidy.nix) and added to runtimeInputs; the flake-app +# invocation resolves the env at `.#local-k3d` using the current +# system's nixidyEnvs..local-k3d output. +{ ... }: +{ + perSystem = + { + pkgs, + lib, + config, + ... + }: + { + apps.nixidy-build = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "nixidy-build"; + runtimeInputs = [ + pkgs.coreutils + config.packages.nixidy + ]; + text = builtins.readFile ./nixidy-build.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/nixidy-build.sh b/modules/apps/cluster/nixidy-build.sh new file mode 100644 index 000000000..cdce60749 --- /dev/null +++ b/modules/apps/cluster/nixidy-build.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Build nixidy-rendered Kubernetes manifests for the local-k3d env into +# ./result. Equivalent to `nixidy build .#local-k3d`; preserved here so +# the invocation is packaged as a first-class flake app for CI effects +# and justfile wrappers. +# +# Usage: +# nixidy-build [--help] +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: nixidy-build [--help] + +Invokes `nixidy build .#local-k3d`, producing a ./result/ symlink at the +working directory that materializes the rendered manifest tree. +ARGOCD_REPO_URL may be set in the environment to override the default +remote repo URL baked into the rendered Application resources (see +modules/nixidy.nix and the ARGOCD_REPO_URL env hook in kubernetes/nixidy). +EOF + exit 0 + ;; +esac + +exec nixidy build .#local-k3d diff --git a/modules/apps/cluster/nixidy-push.nix b/modules/apps/cluster/nixidy-push.nix new file mode 100644 index 000000000..492d6e341 --- /dev/null +++ b/modules/apps/cluster/nixidy-push.nix @@ -0,0 +1,30 @@ +# nixidy-push.nix - Rsync rendered manifests to the local-k3d private repo. +# +# Usage: +# nix run .#nixidy-push +# +# Template form: pure readFile (no nix-computed variable injection). +# The target repo path is resolved at runtime from the LOCAL_K3D_REPO +# env var (fallback: $HOME/projects/nix-workspace/local-k3d), mirroring +# the justfile `local_k3d_repo` convention. +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.nixidy-push = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "nixidy-push"; + runtimeInputs = [ + pkgs.coreutils + pkgs.git + pkgs.rsync + ]; + text = builtins.readFile ./nixidy-push.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/nixidy-push.sh b/modules/apps/cluster/nixidy-push.sh new file mode 100644 index 000000000..625ba8d8c --- /dev/null +++ b/modules/apps/cluster/nixidy-push.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Sync the ./result/ tree produced by nixidy-build into the private +# local-k3d manifest repo, then commit and push any diff. +# +# Usage: +# nixidy-push [--help] +# +# Environment: +# LOCAL_K3D_REPO path to the local-k3d manifest repo +# (default: $HOME/projects/nix-workspace/local-k3d) +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: nixidy-push [--help] + +Prerequisites: + - `nixidy-build` has run in the current directory, producing ./result + - The LOCAL_K3D_REPO directory exists and has a configured git remote + +rsync copies result/ → $LOCAL_K3D_REPO/ with --delete, dereferencing +nix-store symlinks and normalizing permissions. Exits 0 cleanly when +there is nothing to push (no changes detected). + +Environment: + LOCAL_K3D_REPO target repo path + (default: $HOME/projects/nix-workspace/local-k3d) +EOF + exit 0 + ;; +esac + +LOCAL_K3D_REPO="${LOCAL_K3D_REPO:-$HOME/projects/nix-workspace/local-k3d}" + +if [[ ! -d "result" ]]; then + echo "Error: result/ directory not found. Run 'just nixidy-build' first." >&2 + exit 1 +fi + +if [[ ! -d "$LOCAL_K3D_REPO" ]]; then + echo "Error: local-k3d repo not found at $LOCAL_K3D_REPO" >&2 + echo "Clone it with: git clone git@github.com:cameronraysmith/local-k3d.git $LOCAL_K3D_REPO" >&2 + exit 1 +fi + +echo "Syncing rendered manifests to $LOCAL_K3D_REPO..." +# -L dereferences symlinks (nix store paths) to copy actual content +# --checksum compares by content hash (Nix store files have epoch timestamps) +# --chmod fixes read-only permissions from nix store +rsync -aL --delete --checksum --chmod=Du+w,Fu+w --exclude='.git' result/ "$LOCAL_K3D_REPO/" + +echo "Committing and pushing to local-k3d repo..." +cd "$LOCAL_K3D_REPO" +git add -A +if git diff --cached --quiet; then + echo "No changes to push." +else + git commit -m "chore: update rendered manifests from vanixiets" + git push + echo "Manifests pushed to local-k3d repo." +fi diff --git a/modules/apps/cluster/nixidy-sync.nix b/modules/apps/cluster/nixidy-sync.nix new file mode 100644 index 000000000..cd2407a31 --- /dev/null +++ b/modules/apps/cluster/nixidy-sync.nix @@ -0,0 +1,60 @@ +# nixidy-sync.nix - Compose nixidy-build then nixidy-push. +# +# Usage: +# nix run .#nixidy-sync +# +# Template form: pure readFile (no nix-computed variable injection). +# Composes nixidy-build and nixidy-push by invoking them sequentially +# via their published bin names (both exposed as runtimeInputs). Running +# the sidecars directly—rather than going through `nix run .#...`—means +# the sync app is self-contained and does not need `nix` on PATH. +{ ... }: +{ + perSystem = + { + pkgs, + lib, + config, + ... + }: + let + # writeShellApplication closures for the two composed apps. These are + # already referenced by apps.nixidy-build / apps.nixidy-push via + # lib.getExe; re-declaring the derivations here lets nixidy-sync + # place both on its own PATH through runtimeInputs, avoiding a + # dependency on `nix` or `just` being present at runtime. + nixidyBuild = pkgs.writeShellApplication { + name = "nixidy-build"; + runtimeInputs = [ + pkgs.coreutils + config.packages.nixidy + ]; + text = builtins.readFile ./nixidy-build.sh; + }; + nixidyPush = pkgs.writeShellApplication { + name = "nixidy-push"; + runtimeInputs = [ + pkgs.coreutils + pkgs.git + pkgs.rsync + ]; + text = builtins.readFile ./nixidy-push.sh; + }; + in + { + apps.nixidy-sync = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "nixidy-sync"; + runtimeInputs = [ + pkgs.coreutils + nixidyBuild + nixidyPush + ]; + text = builtins.readFile ./nixidy-sync.sh; + } + ); + }; + }; +} diff --git a/modules/apps/cluster/nixidy-sync.sh b/modules/apps/cluster/nixidy-sync.sh new file mode 100644 index 000000000..7bb8d48b4 --- /dev/null +++ b/modules/apps/cluster/nixidy-sync.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Build nixidy manifests then push them to the local-k3d private repo. +# Composition of `nixidy-build` followed by `nixidy-push`; both live on +# PATH as writeShellApplication-wrapped commands supplied via +# runtimeInputs in nixidy-sync.nix. +# +# Usage: +# nixidy-sync [--help] +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'EOF' +Usage: nixidy-sync [--help] + +Runs `nixidy-build` then `nixidy-push` in-process (no just/nix run +indirection). Requires the same preconditions as the two sub-apps: + + - A configured LOCAL_K3D_REPO directory with a git remote + - A current directory writable to receive ./result (nixidy-build) + - An ARGOCD_REPO_URL env override when building for file:/// manifests + +See `nixidy-build --help` and `nixidy-push --help` for details. +EOF + exit 0 + ;; +esac + +nixidy-build +nixidy-push diff --git a/scripts/k3d-test-coverage.sh b/scripts/k3d-test-coverage.sh index 999dbceae..83e1bfb37 100755 --- a/scripts/k3d-test-coverage.sh +++ b/scripts/k3d-test-coverage.sh @@ -1,511 +1,13 @@ #!/usr/bin/env bash -# Run chainsaw tests with coverage report showing tested vs deployed resources -# -# Usage: ./scripts/k3d-test-coverage.sh [--raw] [chainsaw args...] -# -# Options: -# --raw Show raw uncategorized output (original format) -# -# Environment: -# CI, GITHUB_ACTIONS, NO_COLOR - Disable colors when set -# -# Exit codes: -# 0 - All tests passed -# 1 - Tests failed or error - +# shellcheck shell=bash +# Backward-compat shim — the authoritative implementation now lives at +# modules/apps/cluster/k3d-test-coverage.{nix,sh} (flake app +# `k3d-test-coverage`). This shim is preserved so that out-of-tree +# consumers pinning the legacy `scripts/k3d-test-coverage.sh` path (e.g., +# the `hash-sources` entry in `.github/workflows/test-cluster.yaml`) +# continue to work during the M1→M5 transition. Once M5 drops the legacy +# path from workflow hash sources, this file may be deleted outright. set -euo pipefail - -# Global flag for raw output mode -RAW_MODE=0 - -# shellcheck disable=SC2034 # Colors are used via variable expansion -setup_colors() { - if [[ -n "${CI:-}" ]] || [[ -n "${GITHUB_ACTIONS:-}" ]] || [[ -n "${NO_COLOR:-}" ]]; then - RED="" GREEN="" YELLOW="" BOLD="" DIM="" RESET="" - else - RED=$'\e[31m' GREEN=$'\e[32m' YELLOW=$'\e[33m' - BOLD=$'\e[1m' DIM=$'\e[2m' RESET=$'\e[0m' - fi -} - -run_chainsaw_tests() { - local report_dir="$1" - shift - - echo "${BOLD}Running chainsaw tests...${RESET}" - echo "" - - if chainsaw test kubernetes/tests/local-k3d/ "$@" \ - --report-format JUNIT-OPERATION \ - --report-path "$report_dir" 2>&1; then - return 0 - else - return 1 - fi -} - -print_test_summary() { - local report_file="$1" - - echo "" - echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" - echo "${BOLD} TEST SUMMARY ${RESET}" - echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" - echo "" - - if [[ ! -f "$report_file" ]]; then - echo " ${YELLOW}Warning: No test report found${RESET}" - return - fi - - local total_ops total_time - total_ops=$(xmllint --xpath 'string(/testsuites/@tests)' "$report_file" 2>/dev/null || echo "0") - total_time=$(xmllint --xpath 'string(/testsuites/@time)' "$report_file" 2>/dev/null || echo "0") - - echo "${BOLD}Test Execution:${RESET}" - echo " Total operations: ${GREEN}${total_ops}${RESET}" - echo " Total time: ${DIM}${total_time}s${RESET}" - echo "" - - echo "${BOLD}By Test Suite:${RESET}" - local suite suite_tests suite_failures suite_time status - for suite in foundation infrastructure local-k3d; do - suite_tests=$(xmllint --xpath "string(//testsuite[@name='$suite']/@tests)" "$report_file" 2>/dev/null || echo "0") - suite_failures=$(xmllint --xpath "string(//testsuite[@name='$suite']/@failures)" "$report_file" 2>/dev/null || echo "0") - suite_time=$(xmllint --xpath "string(//testsuite[@name='$suite']/@time)" "$report_file" 2>/dev/null || echo "0") - - if [[ "$suite_tests" != "0" ]]; then - if [[ "$suite_failures" == "0" ]]; then - status="${GREEN}PASS${RESET}" - else - status="${RED}FAIL${RESET}" - fi - printf " %-20s %s %3s ops ${DIM}%ss${RESET}\n" "$suite" "$status" "$suite_tests" "$suite_time" - fi - done -} - -collect_deployed_resources() { - local -n deployed_ref=$1 - local -n type_counts_ref=$2 - - # Workloads (Deployment, StatefulSet, DaemonSet) - local line ns name kind key - while IFS= read -r line; do - [[ -z "$line" ]] && continue - ns=$(awk '{print $1}' <<< "$line") - name=$(awk '{print $2}' <<< "$line") - kind=$(awk '{print $3}' <<< "$line") - key="${kind}/${ns}/${name}" - deployed_ref["$key"]=1 - done < <(kubectl get deploy,sts,ds -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,KIND:.kind' --no-headers 2>/dev/null | grep -v '^$') - - # Gateway API resources - while IFS= read -r line; do - [[ -z "$line" ]] && continue - ns=$(awk '{print $1}' <<< "$line") - name=$(awk '{print $2}' <<< "$line") - deployed_ref["Gateway/${ns}/${name}"]=1 - done < <(kubectl get gateway -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') - - while IFS= read -r line; do - [[ -z "$line" ]] && continue - ns=$(awk '{print $1}' <<< "$line") - name=$(awk '{print $2}' <<< "$line") - deployed_ref["HTTPRoute/${ns}/${name}"]=1 - done < <(kubectl get httproute -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') - - # Certificates - while IFS= read -r line; do - [[ -z "$line" ]] && continue - ns=$(awk '{print $1}' <<< "$line") - name=$(awk '{print $2}' <<< "$line") - deployed_ref["Certificate/${ns}/${name}"]=1 - done < <(kubectl get certificate -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') - - # ClusterIssuers (cluster-scoped) - while IFS= read -r line; do - [[ -z "$line" ]] && continue - deployed_ref["ClusterIssuer/-/${line}"]=1 - done < <(kubectl get clusterissuer -o custom-columns='NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') - - # Count by type - for key in "${!deployed_ref[@]}"; do - kind="${key%%/*}" - type_counts_ref["$kind"]=$(( ${type_counts_ref["$kind"]:-0} + 1 )) - done -} - -collect_tested_resources() { - local -n tested_ref=$1 - # shellcheck disable=SC2178 # nameref to associative array - local -n type_counts_ref=$2 - local test_dir="$3" - - local file current_kind current_name current_ns line key - - while IFS= read -r file; do - current_kind="" - current_name="" - current_ns="-" - - while IFS= read -r line; do - # New document resets state - if [[ "$line" == "---" ]]; then - if [[ -n "$current_kind" && -n "$current_name" ]]; then - key="${current_kind}/${current_ns}/${current_name}" - tested_ref["$key"]=1 - fi - current_kind="" - current_name="" - current_ns="-" - continue - fi - - # Extract kind - if [[ "$line" =~ ^kind:\ *(.+)$ ]]; then - current_kind="${BASH_REMATCH[1]}" - fi - - # Extract name (first name field is metadata.name) - if [[ "$line" =~ ^[[:space:]]+name:\ *(.+)$ ]]; then - if [[ -z "$current_name" ]]; then - current_name="${BASH_REMATCH[1]}" - fi - fi - - # Extract namespace - if [[ "$line" =~ ^[[:space:]]+namespace:\ *(.+)$ ]]; then - current_ns="${BASH_REMATCH[1]}" - fi - done < "$file" - - # Last resource in file - if [[ -n "$current_kind" && -n "$current_name" ]]; then - key="${current_kind}/${current_ns}/${current_name}" - tested_ref["$key"]=1 - fi - done < <(find "$test_dir" -name "*assert*.yaml" -type f) - - # Count by type - for key in "${!tested_ref[@]}"; do - kind="${key%%/*}" - type_counts_ref["$kind"]=$(( ${type_counts_ref["$kind"]:-0} + 1 )) - done -} - -print_resource_table() { - local -n counts_ref=$1 - local total=$2 - - local kind count - for kind in Deployment StatefulSet DaemonSet Gateway HTTPRoute Certificate ClusterIssuer; do - count="${counts_ref[$kind]:-0}" - if [[ "$count" -gt 0 ]]; then - printf " %-15s %3d\n" "$kind" "$count" - fi - done - echo " ${DIM}─────────────────────${RESET}" - printf " %-15s %3d\n" "Total" "$total" -} - -# Categorize a resource as application, foundation, or system -# Returns: "application", "foundation", or "system" -categorize_resource() { - local key="$1" - local kind="${key%%/*}" - local rest="${key#*/}" - local ns="${rest%%/*}" - local name="${rest#*/}" - - # System components (k3s internals, Cilium internals, auto-generated) - # These are excluded from coverage calculation because they are: - # - Not managed by our nixidy/ArgoCD stack - # - Auto-created by k3s or other controllers - # - Internal components of our foundation layer - case "$key" in - # k3s DNS - managed by k3s, not our stack - Deployment/kube-system/coredns) echo "system"; return ;; - # k3s storage provisioner - managed by k3s - Deployment/kube-system/local-path-provisioner) echo "system"; return ;; - # k3s metrics - managed by k3s - Deployment/kube-system/metrics-server) echo "system"; return ;; - # Cilium internal envoy proxy - managed by Cilium operator - DaemonSet/kube-system/cilium-envoy) echo "system"; return ;; - # Auto-generated by cert-manager gateway-shim from HTTPRoute annotation - # Duplicates our explicit step-ca-tls Certificate - Certificate/gateway-system/test-cert-tls) echo "system"; return ;; - esac - - # k3s servicelb auto-created DaemonSets (svclb-*) - # These are auto-created by k3s for LoadBalancer services - if [[ "$kind" == "DaemonSet" && "$ns" == "kube-system" && "$name" == svclb-* ]]; then - echo "system" - return - fi - - # Foundation resources (CNI layer we deploy but is infrastructure) - case "$key" in - DaemonSet/kube-system/cilium) echo "foundation"; return ;; - Deployment/kube-system/cilium-operator) echo "foundation"; return ;; - esac - - # Application resources (our nixidy/ArgoCD managed stack) - # Includes: argocd, cert-manager, sops-secrets-operator, step-ca, - # gateway-system, plus Gateway API resources - echo "application" -} - -# Get human-readable description for system components -get_system_description() { - local key="$1" - local kind="${key%%/*}" - local rest="${key#*/}" - local ns="${rest%%/*}" - local name="${rest#*/}" - - case "$key" in - Deployment/kube-system/coredns) echo "k3s DNS" ;; - Deployment/kube-system/local-path-provisioner) echo "k3s storage" ;; - Deployment/kube-system/metrics-server) echo "k3s metrics" ;; - DaemonSet/kube-system/cilium-envoy) echo "Cilium internal" ;; - Certificate/gateway-system/test-cert-tls) echo "gateway-shim duplicate" ;; - DaemonSet/kube-system/svclb-*) echo "k3s servicelb auto-created" ;; - *) echo "system component" ;; - esac -} - -print_coverage_report() { - local test_dir="$1" - - echo "" - echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" - echo "${BOLD} RESOURCE COVERAGE ${RESET}" - echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" - echo "" - - declare -A deployed_resources - # shellcheck disable=SC2034 # passed to function via nameref - declare -A deployed_type_counts - declare -A tested_resources - # shellcheck disable=SC2034 # passed to function via nameref - declare -A tested_type_counts - - collect_deployed_resources deployed_resources deployed_type_counts - collect_tested_resources tested_resources tested_type_counts "$test_dir" - - local deployed_count=${#deployed_resources[@]} - local tested_count=${#tested_resources[@]} - - echo "${BOLD}Deployed Resources:${RESET}" - print_resource_table deployed_type_counts "$deployed_count" - - echo "" - echo "${BOLD}Tested Resources:${RESET}" - print_resource_table tested_type_counts "$tested_count" - - echo "" - echo "${BOLD}Coverage Analysis:${RESET}" - - # Categorize resources and calculate coverage - local matched=0 - local untested=() - local key category - - # Categorized counts - local app_total=0 app_tested=0 - local foundation_total=0 foundation_tested=0 - local system_total=0 system_tested=0 - local system_resources=() - - for key in "${!deployed_resources[@]}"; do - category=$(categorize_resource "$key") - - case "$category" in - application) - (( app_total++ )) || true - if [[ -n "${tested_resources[$key]:-}" ]]; then - (( app_tested++ )) || true - (( matched++ )) || true - else - untested+=("$key") - fi - ;; - foundation) - (( foundation_total++ )) || true - if [[ -n "${tested_resources[$key]:-}" ]]; then - (( foundation_tested++ )) || true - (( matched++ )) || true - else - untested+=("$key") - fi - ;; - system) - (( system_total++ )) || true - system_resources+=("$key") - if [[ -n "${tested_resources[$key]:-}" ]]; then - (( system_tested++ )) || true - (( matched++ )) || true - fi - ;; - esac - done - - # Calculate raw coverage (all resources) - local raw_coverage=0 - if [[ $deployed_count -gt 0 ]]; then - raw_coverage=$(( matched * 100 / deployed_count )) - fi - - # Calculate managed coverage (excluding system components) - local managed_total=$(( app_total + foundation_total )) - local managed_tested=$(( app_tested + foundation_tested )) - local managed_coverage=0 - if [[ $managed_total -gt 0 ]]; then - managed_coverage=$(( managed_tested * 100 / managed_total )) - fi - - if [[ $RAW_MODE -eq 1 ]]; then - # Original raw output format - local cov_color="$RED" - if [[ $raw_coverage -ge 80 ]]; then - cov_color="$GREEN" - elif [[ $raw_coverage -ge 50 ]]; then - cov_color="$YELLOW" - fi - - echo "" - echo " Resource instance coverage: ${cov_color}${BOLD}${raw_coverage}%${RESET} (${matched}/${deployed_count})" - echo "" - - if [[ ${#untested[@]} -gt 0 ]] || [[ ${#system_resources[@]} -gt 0 ]]; then - echo "${BOLD}Untested Resources:${RESET}" - local rest ns name - # Combine untested managed resources with untested system resources - local all_untested=() - for key in "${untested[@]}"; do - all_untested+=("$key") - done - for key in "${system_resources[@]}"; do - if [[ -z "${tested_resources[$key]:-}" ]]; then - all_untested+=("$key") - fi - done - for key in "${all_untested[@]}"; do - kind="${key%%/*}" - rest="${key#*/}" - ns="${rest%%/*}" - name="${rest#*/}" - if [[ "$ns" == "-" ]]; then - printf " ${YELLOW}○${RESET} ${DIM}%-15s${RESET} %s\n" "$kind" "$name" - else - printf " ${YELLOW}○${RESET} ${DIM}%-15s${RESET} %s ${DIM}(%s)${RESET}\n" "$kind" "$name" "$ns" - fi - done | sort - else - echo " ${GREEN}All deployed resources have test coverage${RESET}" - fi - else - # Categorized output format - echo "" - echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" - echo "${BOLD} COVERAGE BY CATEGORY ${RESET}" - echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" - echo "" - - # Application coverage - local app_cov_pct=0 - if [[ $app_total -gt 0 ]]; then - app_cov_pct=$(( app_tested * 100 / app_total )) - fi - local app_color="$RED" - [[ $app_cov_pct -ge 80 ]] && app_color="$GREEN" - [[ $app_cov_pct -ge 50 && $app_cov_pct -lt 80 ]] && app_color="$YELLOW" - printf " Application Resources: ${app_color}%2d/%2d (%3d%%)${RESET}\n" "$app_tested" "$app_total" "$app_cov_pct" - - # Foundation coverage - local fnd_cov_pct=0 - if [[ $foundation_total -gt 0 ]]; then - fnd_cov_pct=$(( foundation_tested * 100 / foundation_total )) - fi - local fnd_color="$RED" - [[ $fnd_cov_pct -ge 80 ]] && fnd_color="$GREEN" - [[ $fnd_cov_pct -ge 50 && $fnd_cov_pct -lt 80 ]] && fnd_color="$YELLOW" - printf " Foundation Resources: ${fnd_color}%2d/%2d (%3d%%)${RESET}\n" "$foundation_tested" "$foundation_total" "$fnd_cov_pct" - - echo " ${DIM}─────────────────────────────────────────────────────────────────${RESET}" - - # Managed total - local mgd_color="$RED" - [[ $managed_coverage -ge 80 ]] && mgd_color="$GREEN" - [[ $managed_coverage -ge 50 && $managed_coverage -lt 80 ]] && mgd_color="$YELLOW" - printf " ${BOLD}Managed Resources Total: ${mgd_color}%2d/%2d (%3d%%)${RESET}\n" "$managed_tested" "$managed_total" "$managed_coverage" - - # Untested managed resources - if [[ ${#untested[@]} -gt 0 ]]; then - echo "" - echo "${BOLD}Untested Managed Resources:${RESET}" - local rest ns name - for key in "${untested[@]}"; do - kind="${key%%/*}" - rest="${key#*/}" - ns="${rest%%/*}" - name="${rest#*/}" - if [[ "$ns" == "-" ]]; then - printf " ${YELLOW}○${RESET} ${DIM}%-15s${RESET} %s\n" "$kind" "$name" - else - printf " ${YELLOW}○${RESET} ${DIM}%-15s${RESET} %s ${DIM}(%s)${RESET}\n" "$kind" "$name" "$ns" - fi - done | sort - fi - - # System components section - echo "" - echo "${BOLD}System Components (excluded from coverage):${RESET}" - local rest ns name desc - for key in "${system_resources[@]}"; do - kind="${key%%/*}" - rest="${key#*/}" - ns="${rest%%/*}" - name="${rest#*/}" - desc=$(get_system_description "$key") - printf " ${DIM}○${RESET} ${DIM}%-15s${RESET} %s ${DIM}(%s)${RESET} - ${DIM}%s${RESET}\n" "$kind" "$name" "$ns" "$desc" - done | sort - - echo "" - printf " ${DIM}Raw Resource Count: %2d/%2d (%3d%%)${RESET}\n" "$matched" "$deployed_count" "$raw_coverage" - fi - - echo "" - echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" -} - -main() { - setup_colors - - # Parse --raw flag - local args=() - for arg in "$@"; do - if [[ "$arg" == "--raw" ]]; then - RAW_MODE=1 - else - args+=("$arg") - fi - done - - local report_dir - report_dir=$(mktemp -d) - trap 'rm -rf "$report_dir"' EXIT - - local test_failed=0 - if ! run_chainsaw_tests "$report_dir" "${args[@]}"; then - test_failed=1 - fi - - print_test_summary "$report_dir/chainsaw-report.xml" - print_coverage_report "kubernetes/tests/local-k3d" - - exit $test_failed -} - -main "$@" +exec nix run --accept-flake-config --no-warn-dirty \ + "$(git rev-parse --show-toplevel 2>/dev/null || echo .)#k3d-test-coverage" \ + -- "$@" From 983ecaaff4a57869ebf2126372641a32f28c7784 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 04:00:02 -0400 Subject: [PATCH 03/77] refactor(apps/docs): audit docs flake apps and document writeShellApplication template bifurcation Audit pass for m1-docs-apps: verifies existing deploy-docs, release, and preview-version flake apps match the nix-a8g template precedent, adds inline comments to each .nix file documenting the writeShellApplication template bifurcation (interpolation vs pure readFile form), and brings release.sh and preview-version.sh into conformance with the cluster sidecar convention (#!/usr/bin/env bash + shellcheck shell=bash directive + set -euo pipefail) so direct shellcheck runs pass. - deploy.nix uses the interpolation form because it injects DOCS_PAYLOAD (config.packages.vanixiets-docs outPath) and SOPS_SECRETS_FILE (inputs.self path) into the script preamble at nix eval time. - release.nix and preview-version.nix use the pure readFile form; their only nix-injected value is DOCS_NODE_MODULES, exposed via runtimeEnv at invocation time (not text interpolation). runtimeInputs closure is unchanged and matches VAL-WRITESHELL-DOCS-007 (deploy: nodejs_24, sops, age, jq, coreutils, git; release: nodejs-slim, git; preview-version: nodejs-slim, git, jq, gnugrep, coreutils). Fulfills VAL-WRITESHELL-DOCS-001..008. --- modules/apps/docs/deploy.nix | 8 ++++++++ modules/apps/docs/preview-version.nix | 8 ++++++++ modules/apps/docs/preview-version.sh | 4 ++++ modules/apps/docs/release.nix | 8 ++++++++ modules/apps/docs/release.sh | 4 ++++ 5 files changed, 32 insertions(+) diff --git a/modules/apps/docs/deploy.nix b/modules/apps/docs/deploy.nix index d4071e7f2..5f72c1aaf 100644 --- a/modules/apps/docs/deploy.nix +++ b/modules/apps/docs/deploy.nix @@ -6,6 +6,14 @@ # Consumes the nix-built CF Worker payload from config.packages.vanixiets-docs # ($out/{dist/,.wrangler/,wrangler.jsonc}) and dispatches to wrangler via # sops exec-env for declarative Cloudflare credential access. +# +# Template bifurcation (writeShellApplication): INTERPOLATION FORM. +# `text` is a nix string that injects two eval-time-computed paths +# (DOCS_PAYLOAD via config.packages.vanixiets-docs and SOPS_SECRETS_FILE via +# inputs.self) into the script preamble before the readFile'd sidecar body. +# Contrast with `release.nix` and `preview-version.nix`, which use the pure +# `text = builtins.readFile ./.sh` form because they have no +# nix-eval-time path injection requirement (they rely on runtimeEnv only). { inputs, ... }: { perSystem = diff --git a/modules/apps/docs/preview-version.nix b/modules/apps/docs/preview-version.nix index c6529b9a2..67a7322b1 100644 --- a/modules/apps/docs/preview-version.nix +++ b/modules/apps/docs/preview-version.nix @@ -9,6 +9,14 @@ # vanixiets-docs-deps derivation (linked into the worktree at runtime); the app # is self-contained and does not depend on a prior `bun install` or on # pkgs.semantic-release. +# +# Template bifurcation (writeShellApplication): PURE READFILE FORM. +# `text = builtins.readFile ./preview-version.sh` — the sidecar is consumed +# verbatim, no nix-eval-time string interpolation. The only nix-injected +# value is DOCS_NODE_MODULES, exposed via `runtimeEnv` at invocation time. +# Contrast with `deploy.nix`, which uses the interpolation form because it +# must inject DOCS_PAYLOAD and SOPS_SECRETS_FILE store paths into the +# script preamble. { ... }: { perSystem = diff --git a/modules/apps/docs/preview-version.sh b/modules/apps/docs/preview-version.sh index 2ed6047cc..b7cad0d54 100644 --- a/modules/apps/docs/preview-version.sh +++ b/modules/apps/docs/preview-version.sh @@ -1,3 +1,5 @@ +#!/usr/bin/env bash +# shellcheck shell=bash # preview-version.sh - Preview semantic-release version after merging to target branch # # Usage: @@ -17,6 +19,8 @@ # directly via node_modules/.bin, bypassing any need for bun or a prior # `bun install`. +set -euo pipefail + # Configuration TARGET_BRANCH="${1:-main}" PACKAGE_PATH="${2:-}" diff --git a/modules/apps/docs/release.nix b/modules/apps/docs/release.nix index d38bf9592..c5c769665 100644 --- a/modules/apps/docs/release.nix +++ b/modules/apps/docs/release.nix @@ -10,6 +10,14 @@ # # Expected caller environment (not loaded from sops; CI-only): # GITHUB_TOKEN - GitHub authentication for the @semantic-release/github plugin +# +# Template bifurcation (writeShellApplication): PURE READFILE FORM. +# `text = builtins.readFile ./release.sh` — the sidecar is consumed verbatim, +# no nix-eval-time string interpolation. The only nix-injected value is +# DOCS_NODE_MODULES, which is provided via `runtimeEnv` (an env var set by +# the writeShellApplication wrapper at invocation time), not via `text` +# interpolation. Contrast with `deploy.nix`, which uses the interpolation +# form because it must inject derivation outPaths into the script preamble. { ... }: { perSystem = diff --git a/modules/apps/docs/release.sh b/modules/apps/docs/release.sh index 8523a38a4..de3f1ad8a 100644 --- a/modules/apps/docs/release.sh +++ b/modules/apps/docs/release.sh @@ -1,3 +1,5 @@ +#!/usr/bin/env bash +# shellcheck shell=bash # release.sh - Production semantic-release runner for a monorepo package. # # Usage: @@ -15,6 +17,8 @@ # Required environment (pass through from caller; CI-only): # GITHUB_TOKEN - required by @semantic-release/github to publish tags/releases. +set -euo pipefail + package_path="${1:?usage: release [extra semantic-release args...]}" shift From ded040f1432478747522310900dcbd65b0ef4bca Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 04:15:35 -0400 Subject: [PATCH 04/77] feat(apps/release): convert package-release.yaml body into flake app Relocate the release flake app from modules/apps/docs/release.{nix,sh} to modules/apps/release/release.{nix,sh} and expand it to absorb the production-release-packages workflow body from .github/workflows/package-release.yaml: - Add a 'release info' subcommand that emits release-info JSON (version, tag, released) from the latest git tag matching the package (semantic-release-monorepo -vX.Y.Z convention). - Add a '--dry-run' flag that passes --dry-run --no-ci to semantic-release and filters @semantic-release/github out of the plugin list so GITHUB_TOKEN is not required for previews. - Configure a default git identity (overridable via GIT_USER_NAME / GIT_USER_EMAIL) so semantic-release can create tags/commits without a separate setup step. - Preserve the hermetic node_modules symlink guard and trap-based cleanup from the prior app. Retire the docs-domain release app (duplicate apps.release attribute) and update package-release.yaml cache hash-sources to reference the new paths. Add a 'just release *args' recipe that delegates to 'nix run .#release --'. Fulfills VAL-WRITESHELL-RELEASE-{001..005,007}. --- .github/workflows/package-release.yaml | 2 +- justfile | 9 + modules/apps/docs/release.sh | 49 ----- modules/apps/{docs => release}/release.nix | 28 ++- modules/apps/release/release.sh | 204 +++++++++++++++++++++ 5 files changed, 236 insertions(+), 56 deletions(-) delete mode 100644 modules/apps/docs/release.sh rename modules/apps/{docs => release}/release.nix (53%) create mode 100644 modules/apps/release/release.sh diff --git a/.github/workflows/package-release.yaml b/.github/workflows/package-release.yaml index 1c4c3b1da..e0320f26c 100644 --- a/.github/workflows/package-release.yaml +++ b/.github/workflows/package-release.yaml @@ -110,7 +110,7 @@ jobs: uses: ./.github/actions/cached-ci-job with: check-name: ${{ inputs.package-name }}-release - hash-sources: 'packages/${{ inputs.package-name }}/**/* packages/docs/package.json bun.lock flake.lock flake.nix modules/apps/docs/release.nix modules/apps/docs/release.sh justfile .github/actions/setup-nix/action.yml .github/workflows/package-release.yaml' + hash-sources: 'packages/${{ inputs.package-name }}/**/* packages/docs/package.json bun.lock flake.lock flake.nix modules/apps/release/release.nix modules/apps/release/release.sh justfile .github/actions/setup-nix/action.yml .github/workflows/package-release.yaml' force-run: ${{ inputs.force-run }} - name: Setup Nix diff --git a/justfile b/justfile index edc5b72a5..4a5078b41 100644 --- a/justfile +++ b/justfile @@ -1595,6 +1595,15 @@ test-package package: preview-version target="main" package="": nix run --accept-flake-config .#preview-version -- "{{target}}" "{{package}}" +# Run the release flake app with passthrough args (see modules/apps/release/release.{nix,sh}) +# Examples: +# just release --help +# just release info packages/docs +# just release packages/docs --dry-run +[group('CI/CD')] +release *args: + {{nix_cmd}} run --no-warn-dirty .#release -- {{args}} + # Release a package using semantic-release [group('CI/CD')] release-package package dry_run="false": diff --git a/modules/apps/docs/release.sh b/modules/apps/docs/release.sh deleted file mode 100644 index de3f1ad8a..000000000 --- a/modules/apps/docs/release.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/usr/bin/env bash -# shellcheck shell=bash -# release.sh - Production semantic-release runner for a monorepo package. -# -# Usage: -# nix run .#release -- [extra semantic-release args...] -# -# Examples: -# nix run .#release -- packages/docs -# -# Hermetic: DOCS_NODE_MODULES (set by release.nix) points to a read-only -# node_modules tree produced by the vanixiets-docs-deps derivation. This script -# links it into the target package directory and invokes semantic-release -# directly via node_modules/.bin, bypassing any need for bun or a prior -# `bun install`. -# -# Required environment (pass through from caller; CI-only): -# GITHUB_TOKEN - required by @semantic-release/github to publish tags/releases. - -set -euo pipefail - -package_path="${1:?usage: release [extra semantic-release args...]}" -shift - -repo_root="$(git rev-parse --show-toplevel)" -cd "$repo_root" - -if [ ! -d "$package_path" ]; then - printf 'error: package path %q does not exist relative to %s\n' \ - "$package_path" "$repo_root" >&2 - exit 1 -fi - -cd "$package_path" - -# Guard against clobbering a real local node_modules from a developer's -# bun install; only proceed if the slot is empty or already our symlink. -if [[ -e node_modules && ! -L node_modules ]]; then - echo "error: $package_path/node_modules exists and is not a symlink; refusing to overwrite a local bun install" >&2 - exit 1 -fi -trap 'rm -f "$PWD/node_modules"' EXIT -ln -snf "$DOCS_NODE_MODULES" node_modules - -# This is a real release path: semantic-release will create a tag and publish -# a GitHub release when invoked. Use `preview-version` (dry-run) or -# `test-release` for previewing. -echo "running production semantic-release in ${package_path}..." -exec node ./node_modules/.bin/semantic-release "$@" diff --git a/modules/apps/docs/release.nix b/modules/apps/release/release.nix similarity index 53% rename from modules/apps/docs/release.nix rename to modules/apps/release/release.nix index c5c769665..223e4e8d9 100644 --- a/modules/apps/docs/release.nix +++ b/modules/apps/release/release.nix @@ -2,22 +2,35 @@ # # Usage: # nix run .#release -- -# nix run .#release -- packages/docs +# nix run .#release -- --dry-run +# nix run .#release -- info +# nix run .#release -- --help +# +# Absorbs the `production-release-packages` job body from +# .github/workflows/package-release.yaml: configures git, invokes +# semantic-release against the target monorepo package, filters +# `@semantic-release/github` out of the plugin list when `--dry-run` +# is set (so GITHUB_TOKEN is not required for previews), and provides +# a `info` subcommand that emits release info (version, tag, released) +# as JSON. # # Hermetic: semantic-release and all plugins are provided by the -# vanixiets-docs-deps derivation and linked into the package directory at runtime. -# Callers do not need to run `bun install`. +# vanixiets-docs-deps derivation and linked into the package directory at +# runtime. Callers do not need to run `bun install`. # # Expected caller environment (not loaded from sops; CI-only): -# GITHUB_TOKEN - GitHub authentication for the @semantic-release/github plugin +# GITHUB_TOKEN - required by @semantic-release/github for production releases +# SOPS_AGE_KEY - passthrough for semantic-release hooks that may decrypt +# secrets via sops (not consumed by this script directly) # # Template bifurcation (writeShellApplication): PURE READFILE FORM. # `text = builtins.readFile ./release.sh` — the sidecar is consumed verbatim, # no nix-eval-time string interpolation. The only nix-injected value is # DOCS_NODE_MODULES, which is provided via `runtimeEnv` (an env var set by # the writeShellApplication wrapper at invocation time), not via `text` -# interpolation. Contrast with `deploy.nix`, which uses the interpolation -# form because it must inject derivation outPaths into the script preamble. +# interpolation. Contrast with `deploy.nix` (modules/apps/docs/), which uses +# the interpolation form because it must inject derivation outPaths into +# the script preamble. { ... }: { perSystem = @@ -36,6 +49,9 @@ runtimeInputs = [ pkgs.nodejs-slim pkgs.git + pkgs.jq + pkgs.gnugrep + pkgs.coreutils ]; runtimeEnv = { DOCS_NODE_MODULES = "${config.packages.vanixiets-docs-deps}/packages/docs/node_modules"; diff --git a/modules/apps/release/release.sh b/modules/apps/release/release.sh new file mode 100644 index 000000000..3d4bc5c91 --- /dev/null +++ b/modules/apps/release/release.sh @@ -0,0 +1,204 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# release.sh - Production semantic-release runner for a monorepo package. +# +# Usage: +# release [--dry-run] [-- extra semantic-release args] +# release info [] +# release --help +# +# Subcommands: +# (default) Run semantic-release against . Tag/publish on +# success; when --dry-run is passed, runs a preview without +# the @semantic-release/github plugin so GITHUB_TOKEN is not +# required. +# info Emit a JSON object describing the most recent release for +# (or the repo root when omitted). Fields: +# { "version": "X.Y.Z", "tag": "pkg-vX.Y.Z", "released": true } +# On no prior release: { "version": "unknown", "tag": "", +# "released": false }. +# +# Flags: +# --dry-run Pass --dry-run and --no-ci to semantic-release and strip the +# @semantic-release/github plugin from the invocation plugin +# list. Mirrors the preview-version trick (no GITHUB_TOKEN +# needed). +# --help Print this usage and exit 0. +# +# Environment: +# GITHUB_TOKEN Required by @semantic-release/github for production +# releases. Not consulted for --dry-run. +# SOPS_AGE_KEY Passthrough for semantic-release hooks that may +# decrypt secrets via sops. Not used directly. +# DOCS_NODE_MODULES Hermetic node_modules tree injected by release.nix. +# Must point at a directory containing a resolved +# node_modules/.bin/semantic-release. +# GIT_USER_NAME, Optional overrides for the git identity used by +# GIT_USER_EMAIL semantic-release commit/tag operations; defaults +# to `semantic-release` / `semantic-release@vanixiets.local`. + +set -euo pipefail + +usage() { + cat <<'EOF' +usage: release [--dry-run] [-- extra semantic-release args] + release info [] + release --help + +Run semantic-release against a monorepo package, or extract release info. + +Subcommands: + (default) Run semantic-release for . + info Emit release info JSON (version, tag, released) from latest + git tag matching the package. + +Flags: + --dry-run Dry-run (skips @semantic-release/github; no GITHUB_TOKEN needed). + --help Print this usage and exit. + +Environment: + GITHUB_TOKEN, SOPS_AGE_KEY, DOCS_NODE_MODULES, GIT_USER_NAME, + GIT_USER_EMAIL (see release.sh header for details). +EOF +} + +emit_release_info() { + local package_path="${1:-}" + local latest_tag="" + local version="" + + if [ -n "$package_path" ]; then + # Monorepo tag convention (semantic-release-monorepo): -vX.Y.Z + local package_name + package_name=$(basename "$package_path") + latest_tag=$(git tag --list "${package_name}-v*" --sort=-v:refname 2>/dev/null | head -1 || true) + else + latest_tag=$(git describe --tags --abbrev=0 2>/dev/null || true) + fi + + if [ -n "$latest_tag" ]; then + version=$(printf '%s\n' "$latest_tag" \ + | grep -oE '[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z]+\.[0-9]+)?' \ + | head -1 || true) + if [ -z "$version" ]; then + version="unknown" + fi + jq -cn \ + --arg v "$version" \ + --arg t "$latest_tag" \ + '{version: $v, tag: $t, released: true}' + else + jq -cn '{version: "unknown", tag: "", released: false}' + fi +} + +# Handle top-level dispatch: --help, info subcommand, or fall through +# to the default "run semantic-release" mode. +if [ $# -eq 0 ]; then + usage >&2 + exit 2 +fi + +case "$1" in + -h|--help) + usage + exit 0 + ;; + info) + shift + emit_release_info "${1:-}" + exit 0 + ;; +esac + +# Default mode: semantic-release runner. +# Parse positional arg + --dry-run flag; forward remaining args through to +# node ./node_modules/.bin/semantic-release. +dry_run=0 +package_path="" +extra_args=() + +while [ $# -gt 0 ]; do + case "$1" in + --dry-run) + dry_run=1 + shift + ;; + -h|--help) + usage + exit 0 + ;; + --) + shift + extra_args+=("$@") + break + ;; + -*) + extra_args+=("$1") + shift + ;; + *) + if [ -z "$package_path" ]; then + package_path="$1" + else + extra_args+=("$1") + fi + shift + ;; + esac +done + +if [ -z "$package_path" ]; then + echo "error: missing required " >&2 + usage >&2 + exit 2 +fi + +repo_root="$(git rev-parse --show-toplevel)" +cd "$repo_root" + +if [ ! -d "$package_path" ]; then + printf 'error: package path %q does not exist relative to %s\n' \ + "$package_path" "$repo_root" >&2 + exit 1 +fi + +# Configure a git identity if none is present so semantic-release can +# create tags/commits without a separate setup step. A pre-configured +# identity (e.g. from the caller's ~/.gitconfig) is preserved. +if [ -z "$(git config user.email 2>/dev/null || true)" ]; then + git config user.email "${GIT_USER_EMAIL:-semantic-release@vanixiets.local}" +fi +if [ -z "$(git config user.name 2>/dev/null || true)" ]; then + git config user.name "${GIT_USER_NAME:-semantic-release}" +fi + +cd "$package_path" + +# Guard node_modules slot against clobbering a developer's real install. +if [[ -e node_modules && ! -L node_modules ]]; then + echo "error: $package_path/node_modules exists and is not a symlink; refusing to overwrite a local bun install" >&2 + exit 1 +fi +trap 'rm -f "$PWD/node_modules"' EXIT +ln -snf "$DOCS_NODE_MODULES" node_modules + +if [ "$dry_run" -eq 1 ]; then + # Filter @semantic-release/github so GITHUB_TOKEN is not required for + # a preview. Mirrors the plugin list used by preview-version.sh plus + # the changelog + major-tag plugins that the package.json "release" + # block declares (still safe under --dry-run: prepare/publish steps + # are no-ops in dry-run mode). + plugins="@semantic-release/commit-analyzer,@semantic-release/release-notes-generator,@semantic-release/changelog,semantic-release-major-tag" + echo "running semantic-release (dry-run, no GitHub plugin) in ${package_path}..." + node ./node_modules/.bin/semantic-release \ + --dry-run \ + --no-ci \ + --plugins "$plugins" \ + "${extra_args[@]}" +else + # Production release path: semantic-release will create a tag and + # publish a GitHub release when invoked. GITHUB_TOKEN is required. + echo "running production semantic-release in ${package_path}..." + node ./node_modules/.bin/semantic-release "${extra_args[@]}" +fi From 25550421e75256f0e946d38e1a9f6169c5a9bff0 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 04:15:35 -0400 Subject: [PATCH 05/77] chore(scripts): replace preview-version.sh with a thin shim over the flake app Reduce scripts/preview-version.sh to a thin shim that delegates to 'nix run .#preview-version --', and repoint the package.json 'preview-version' script to invoke the flake app directly. Removes dual-maintenance drift between scripts/preview-version.sh and the authoritative modules/apps/docs/preview-version.{nix,sh} implementation. Fulfills VAL-WRITESHELL-RELEASE-006. --- package.json | 2 +- scripts/preview-version.sh | 208 ++----------------------------------- 2 files changed, 10 insertions(+), 200 deletions(-) diff --git a/package.json b/package.json index 16ec893a2..a8cb86fd6 100644 --- a/package.json +++ b/package.json @@ -15,7 +15,7 @@ "scripts": { "reinstall": "rm -rf node_modules packages/docs/node_modules && bun install", "test-release": "semantic-release --dry-run --no-ci", - "preview-version": "./scripts/preview-version.sh" + "preview-version": "nix run --accept-flake-config --no-warn-dirty .#preview-version --" }, "devDependencies": { "@semantic-release/changelog": "^6.0.3", diff --git a/scripts/preview-version.sh b/scripts/preview-version.sh index d8111b181..842ed874a 100755 --- a/scripts/preview-version.sh +++ b/scripts/preview-version.sh @@ -1,207 +1,17 @@ #!/usr/bin/env bash -# preview-version.sh - Preview semantic-release version after merging to target branch +# preview-version.sh - Thin shim over the preview-version flake app. +# +# The authoritative implementation lives at modules/apps/preview-version/ (via +# modules/apps/docs/preview-version.{nix,sh}), invoked through the flake app +# `.#preview-version`. This shim is retained so out-of-tree callers that still +# reference `./scripts/preview-version.sh` (notably `package.json:18`) keep +# working. # # Usage: # ./scripts/preview-version.sh [target-branch] [package-path] # -# Examples: -# ./scripts/preview-version.sh # Preview root version on main -# ./scripts/preview-version.sh main packages/docs # Preview docs package version on main -# ./scripts/preview-version.sh beta packages/docs # Preview docs version on beta -# -# This script simulates merging the current branch into the target branch and -# runs semantic-release in dry-run mode to preview what version would be released. +# Forwards all arguments to `nix run .#preview-version --`. set -euo pipefail -NIX_CMD="nix --accept-flake-config" - -# Configuration -TARGET_BRANCH="${1:-main}" -PACKAGE_PATH="${2:-}" -CURRENT_BRANCH=$(git branch --show-current) -REPO_ROOT=$(git rev-parse --show-toplevel) -WORKTREE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/semantic-release-preview.XXXXXX") - -# Save original target branch HEAD for restoration -ORIGINAL_TARGET_HEAD="" -ORIGINAL_REMOTE_HEAD="" - -# Colors for output -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -NC='\033[0m' # No Color - -# Cleanup function -cleanup() { - local exit_code=$? - - # Always restore target branch to original state if we modified it - if [ -n "$ORIGINAL_TARGET_HEAD" ]; then - echo -e "\n${BLUE}restoring ${TARGET_BRANCH} to original state...${NC}" - git update-ref "refs/heads/$TARGET_BRANCH" "$ORIGINAL_TARGET_HEAD" 2>/dev/null || true - fi - - # Always restore remote-tracking branch to original state if we modified it - if [ -n "$ORIGINAL_REMOTE_HEAD" ]; then - git update-ref "refs/remotes/origin/$TARGET_BRANCH" "$ORIGINAL_REMOTE_HEAD" 2>/dev/null || true - fi - - # Clean up worktree - if [ -d "$WORKTREE_DIR" ]; then - echo -e "${BLUE}cleaning up worktree...${NC}" - git worktree remove --force "$WORKTREE_DIR" 2>/dev/null || true - # Prune any stale worktree references - git worktree prune 2>/dev/null || true - fi - - exit $exit_code -} - -trap cleanup EXIT INT TERM - -# Validation -if [ "$CURRENT_BRANCH" == "$TARGET_BRANCH" ]; then - echo -e "${YELLOW}already on target branch ${TARGET_BRANCH}${NC}" - echo -e "${YELLOW}running test-release instead of preview${NC}\n" - if [ -n "$PACKAGE_PATH" ]; then - cd "$REPO_ROOT/$PACKAGE_PATH" - fi - exec $NIX_CMD develop -c bun run test-release -fi - -# Display what we're doing -echo -e "${BLUE}═══════════════════════════════════════════════════════════════${NC}" -echo -e "${BLUE}semantic-release version preview${NC}" -echo -e "${BLUE}═══════════════════════════════════════════════════════════════${NC}" -echo -e "current branch: ${GREEN}${CURRENT_BRANCH}${NC}" -echo -e "target branch: ${GREEN}${TARGET_BRANCH}${NC}" -if [ -n "$PACKAGE_PATH" ]; then - echo -e "package: ${GREEN}${PACKAGE_PATH}${NC}" -else - echo -e "package: ${GREEN}(root)${NC}" -fi -echo -e "${BLUE}───────────────────────────────────────────────────────────────${NC}\n" - -# Verify target branch exists -if ! git show-ref --verify --quiet "refs/heads/$TARGET_BRANCH"; then - echo -e "${RED}error: target branch '${TARGET_BRANCH}' does not exist${NC}" >&2 - exit 1 -fi - -# Save original target branch HEAD before any modifications -ORIGINAL_TARGET_HEAD=$(git rev-parse "$TARGET_BRANCH") - -# Save original remote-tracking branch HEAD before any modifications -ORIGINAL_REMOTE_HEAD=$(git rev-parse "origin/$TARGET_BRANCH" 2>/dev/null || echo "") - -# Create merge tree to test if merge is possible -echo -e "${BLUE}simulating merge of ${CURRENT_BRANCH} → ${TARGET_BRANCH}...${NC}" - -# Perform merge-tree operation to test if merge is possible -MERGE_OUTPUT=$(git merge-tree --write-tree "$TARGET_BRANCH" "$CURRENT_BRANCH" 2>&1) -MERGE_EXIT=$? - -if [ $MERGE_EXIT -ne 0 ]; then - echo -e "${RED}error: merge conflicts detected${NC}" >&2 - echo -e "${YELLOW}please resolve conflicts in your branch before previewing${NC}" >&2 - echo -e "\n${YELLOW}conflict details:${NC}" >&2 - echo "$MERGE_OUTPUT" >&2 - exit 1 -fi - -# Extract tree hash from merge-tree output (first line) -MERGE_TREE=$(echo "$MERGE_OUTPUT" | head -1) - -if [ -z "$MERGE_TREE" ]; then - echo -e "${RED}error: failed to create merge tree${NC}" >&2 - exit 1 -fi - -# Create temporary merge commit -echo -e "${BLUE}creating temporary merge commit...${NC}" -TEMP_COMMIT=$(git commit-tree -p "$TARGET_BRANCH" -p "$CURRENT_BRANCH" \ - -m "Temporary merge for semantic-release preview" "$MERGE_TREE") - -if [ -z "$TEMP_COMMIT" ]; then - echo -e "${RED}error: failed to create temporary merge commit${NC}" >&2 - exit 1 -fi - -# Temporarily update target branch to point to merge commit -# This allows semantic-release to analyze the correct commit history -# The cleanup function will ALWAYS restore the original branch HEAD -echo -e "${BLUE}temporarily updating ${TARGET_BRANCH} ref for analysis...${NC}" -git update-ref "refs/heads/$TARGET_BRANCH" "$TEMP_COMMIT" - -# Also update remote-tracking branch to match (so semantic-release sees them as synchronized) -git update-ref "refs/remotes/origin/$TARGET_BRANCH" "$TEMP_COMMIT" - -# Create worktree at target branch (now pointing to merge commit) -echo -e "${BLUE}creating temporary worktree at ${TARGET_BRANCH}...${NC}" -git worktree add --quiet "$WORKTREE_DIR" "$TARGET_BRANCH" - -# Navigate to worktree -cd "$WORKTREE_DIR" - -# Install dependencies in worktree (bun uses global cache, so this is fast) -echo -e "${BLUE}installing dependencies in worktree...${NC}" -$NIX_CMD develop -c bun install --silent - -# Navigate to package if specified -if [ -n "$PACKAGE_PATH" ]; then - if [ ! -d "$PACKAGE_PATH" ]; then - echo -e "${RED}error: package path '${PACKAGE_PATH}' does not exist${NC}" >&2 - exit 1 - fi - cd "$PACKAGE_PATH" -fi - -# Run semantic-release in dry-run mode -echo -e "\n${BLUE}running semantic-release analysis...${NC}\n" - -# Capture output and parse version -# Exclude @semantic-release/github to avoid GitHub token requirement for preview -# This is safe because dry-run skips publish/success/fail steps anyway -PLUGINS="@semantic-release/commit-analyzer,@semantic-release/release-notes-generator" - -if [ -n "$PACKAGE_PATH" ]; then - # For monorepo packages, check if package.json has specific plugins configured - OUTPUT=$(GITHUB_REF="refs/heads/$TARGET_BRANCH" $NIX_CMD develop -c bun run semantic-release --dry-run --no-ci --branches "$TARGET_BRANCH" --plugins "$PLUGINS" 2>&1 || true) -else - # For root package - OUTPUT=$(GITHUB_REF="refs/heads/$TARGET_BRANCH" $NIX_CMD develop -c bun run semantic-release --dry-run --no-ci --branches "$TARGET_BRANCH" --plugins "$PLUGINS" 2>&1 || true) -fi - -# Display semantic-release summary (filter out verbose plugin repetition) -echo "$OUTPUT" | grep -v "^$" | grep -vE "(No more plugins|does not provide step)" | \ - grep -E "(semantic-release|Running|analyzing|Found.*commits|release version|Release note|Features|Bug Fixes|Breaking Changes|Published|\*\s)" || true - -echo -e "\n${BLUE}═══════════════════════════════════════════════════════════════${NC}" - -# Extract and display the next version -if echo "$OUTPUT" | grep -q "There are no relevant changes"; then - echo -e "${YELLOW}no version bump required${NC}" - echo -e "no semantic commits found since last release" -elif echo "$OUTPUT" | grep -q "is not configured to publish from"; then - echo -e "${YELLOW}cannot determine version${NC}" - echo -e "branch ${TARGET_BRANCH} is not in release configuration" -elif VERSION=$(echo "$OUTPUT" | grep -oP 'next release version is \K[0-9]+\.[0-9]+\.[0-9]+(-[a-z]+\.[0-9]+)?' | head -1); then - echo -e "${GREEN}next version: ${VERSION}${NC}" - - # Extract release type if available - if TYPE=$(echo "$OUTPUT" | grep -oP 'Release type: \K[a-z]+' | head -1); then - echo -e "release type: ${TYPE}" - fi -else - echo -e "${YELLOW}could not parse version from output${NC}" - echo -e "check the semantic-release output above for details" -fi - -echo -e "${BLUE}═══════════════════════════════════════════════════════════════${NC}\n" - -# Preview completed successfully - exit 0 regardless of whether a version bump is pending. -# "No version bump required" is a valid outcome, not an error. -exit 0 +exec nix run --accept-flake-config --no-warn-dirty .#preview-version -- "$@" From 8bffe4e574d6e838b1e0e427d22c57372a0533e4 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 04:22:39 -0400 Subject: [PATCH 06/77] feat(apps/bootstrap): add bootstrap, verify, setup-user flake apps wrapping Makefile targets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Create modules/apps/bootstrap/{bootstrap,verify,setup-user}.{nix,sh} as writeShellApplication flake apps mirroring the Makefile bootstrap flow for hosts that already have nix installed: - bootstrap: idempotent post-nix bootstrap — installs direnv via `nix profile install nixpkgs#direnv` only when missing - verify: audits nix + flakes + direnv + flake-metadata, exits nonzero on missing nix/flakes (mirrors `make verify`, no make dependency) - setup-user: generates age key at ~/.config/sops/age/keys.txt; no-op reprint of the public key when the file already exists Each .nix uses the pure readFile template form (no nix-eval-time path injection) and documents the chicken-and-egg scope: the flake apps require nix to already be present (`nix run` is their entry point), so `make bootstrap` remains the clean-host first-contact installer. Shellcheck clean; all three apps verified idempotent on a nix-ready host. Adds just bootstrap / just bootstrap-verify / just bootstrap-setup-user wrappers (new `bootstrap` group) that delegate to the flake apps. Leaves the existing `just verify` recipe (system-configuration rebuild via scripts/verify-system.sh) untouched to avoid semantic collision. --- justfile | 20 +++++ modules/apps/bootstrap/bootstrap.nix | 45 +++++++++++ modules/apps/bootstrap/bootstrap.sh | 75 +++++++++++++++++++ modules/apps/bootstrap/setup-user.nix | 41 ++++++++++ modules/apps/bootstrap/setup-user.sh | 65 ++++++++++++++++ modules/apps/bootstrap/verify.nix | 39 ++++++++++ modules/apps/bootstrap/verify.sh | 104 ++++++++++++++++++++++++++ 7 files changed, 389 insertions(+) create mode 100644 modules/apps/bootstrap/bootstrap.nix create mode 100644 modules/apps/bootstrap/bootstrap.sh create mode 100644 modules/apps/bootstrap/setup-user.nix create mode 100644 modules/apps/bootstrap/setup-user.sh create mode 100644 modules/apps/bootstrap/verify.nix create mode 100644 modules/apps/bootstrap/verify.sh diff --git a/justfile b/justfile index 4a5078b41..82b078770 100644 --- a/justfile +++ b/justfile @@ -331,6 +331,26 @@ bootstrap-shell: "nixpkgs#git" \ "nixpkgs#just" +# Idempotent post-nix bootstrap: install direnv if missing, report status +# Body lives in modules/apps/bootstrap/bootstrap.{nix,sh}. +# Chicken-and-egg: for first-contact nix install, use `make bootstrap`. +[group('bootstrap')] +bootstrap *ARGS: + {{nix_cmd}} run --no-warn-dirty .#bootstrap -- {{ARGS}} + +# Verify host nix/flakes/direnv/flake-metadata (mirror of `make verify`) +# Body lives in modules/apps/bootstrap/verify.{nix,sh}. +[group('bootstrap')] +bootstrap-verify *ARGS: + {{nix_cmd}} run --no-warn-dirty .#verify -- {{ARGS}} + +# Generate ~/.config/sops/age/keys.txt (mirror of `make setup-user`) +# Body lives in modules/apps/bootstrap/setup-user.{nix,sh}. +# Idempotent: re-print public key and exit 0 if the key already exists. +[group('bootstrap')] +bootstrap-setup-user *ARGS: + {{nix_cmd}} run --no-warn-dirty .#setup-user -- {{ARGS}} + # nix run home-manager -- build --flake ".#{{ profile }}" # Bootstrap build home-manager with flake [group('nix-home-manager')] diff --git a/modules/apps/bootstrap/bootstrap.nix b/modules/apps/bootstrap/bootstrap.nix new file mode 100644 index 000000000..e785cf476 --- /dev/null +++ b/modules/apps/bootstrap/bootstrap.nix @@ -0,0 +1,45 @@ +# Flake app: re-run the bootstrap flow from an already-nix-ready host. +# +# Usage: +# nix run .#bootstrap # install direnv if missing, confirm nix +# nix run .#bootstrap -- --help +# +# Chicken-and-egg note: The repo's primary bootstrap entry point is the +# Makefile (`make bootstrap`), which installs nix itself via the NixOS +# community installer and only then installs direnv. This flake app, by +# contrast, can only run once nix is already present (since `nix run` +# requires nix). It exists for reproducibility / scripting on hosts that +# have nix but want to (idempotently) finish the direnv half of bootstrap +# or re-verify that bootstrap has been completed. +# +# Idempotent: detects existing nix and direnv via `command -v`; only +# attempts `nix profile install nixpkgs#direnv` when direnv is missing. +# Does not mutate /nix or /etc/nix; only touches the user's nix profile. +# +# Template bifurcation (writeShellApplication): PURE READFILE FORM. +# The sidecar needs no nix-eval-time path injection. +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.bootstrap = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "bootstrap"; + runtimeInputs = [ + pkgs.coreutils + pkgs.gnugrep + # `nix` is in runtimeInputs so `nix profile install` works + # from within the hermetic PATH. The host must already have + # a running nix daemon; this app is explicitly not a + # first-contact installer (the Makefile is). + pkgs.nix + ]; + text = builtins.readFile ./bootstrap.sh; + } + ); + }; + }; +} diff --git a/modules/apps/bootstrap/bootstrap.sh b/modules/apps/bootstrap/bootstrap.sh new file mode 100644 index 000000000..2a0fe097b --- /dev/null +++ b/modules/apps/bootstrap/bootstrap.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Re-run the bootstrap flow (direnv install + status report) on a host +# that already has nix installed. +# +# Chicken-and-egg: `make bootstrap` is the real first-contact installer; +# it installs nix itself. This flake app runs UNDER nix, so by definition +# nix is already present. Treat this app as the post-nix half of +# bootstrap: it ensures direnv is installed and reports status. +# +# Idempotent: re-runs produce no new state when direnv is already present. +set -euo pipefail + +usage() { + cat <<'EOF' +Usage: bootstrap [--help] + +Idempotent post-nix bootstrap: installs direnv via `nix profile install` +if it is missing, then reports the tool versions. Assumes nix is already +installed (that is the precondition of `nix run`). For a clean-host +first-contact install, use `make bootstrap` instead (installs nix first, +then direnv). + +Mirrors the `make bootstrap` target in the repo-root Makefile for the +direnv half of the flow; the nix-installer half is skipped because it +cannot run from inside a nix sandbox. +EOF +} + +case "${1:-}" in + -h|--help) + usage + exit 0 + ;; +esac + +printf '=== Bootstrap (nix-present host) ===\n\n' + +# Step 1: confirm nix (cannot be missing since we're running under nix, +# but surface the version for parity with `make verify`). +if command -v nix >/dev/null 2>&1; then + printf '● nix found at %s\n' "$(command -v nix)" + nix --version +else + # Unreachable under `nix run`, but keep the guard for defence in depth. + printf '⊘ nix not found on PATH (unexpected inside a nix sandbox)\n' >&2 + # shellcheck disable=SC2016 # backticks in string are literal output, not command substitution + printf 'Run `make bootstrap` from a nix-free shell to install nix first.\n' >&2 + exit 1 +fi +printf '\n' + +# Step 2: install direnv if missing. `nix profile install` is idempotent +# against the same attribute path; we guard with `command -v` for a +# cleaner no-op output when direnv is already on PATH. +if command -v direnv >/dev/null 2>&1; then + printf '● direnv already installed at %s\n' "$(command -v direnv)" +else + # shellcheck disable=SC2016 # backticks in string are literal output, not command substitution + printf 'Installing direnv via `nix profile install nixpkgs#direnv`...\n' + nix --accept-flake-config profile install nixpkgs#direnv + printf '● direnv installed\n' +fi +printf '\n' + +printf '=== ● Bootstrap complete ===\n\n' +printf 'Next steps:\n' +# shellcheck disable=SC2016 # backticks in strings are literal output, not command substitution +printf ' 1. Run `nix run .#verify` to audit your installation.\n' +# shellcheck disable=SC2016 +printf ' 2. Run `nix run .#setup-user` once to generate your age key.\n' +# shellcheck disable=SC2016 +printf ' 3. Run `nix develop` to enter the development environment.\n' +printf '\n' +printf 'See https://direnv.net/docs/hook.html to add direnv to your shell.\n' diff --git a/modules/apps/bootstrap/setup-user.nix b/modules/apps/bootstrap/setup-user.nix new file mode 100644 index 000000000..3c6557f1c --- /dev/null +++ b/modules/apps/bootstrap/setup-user.nix @@ -0,0 +1,41 @@ +# Flake app: generate the user's age key for sops-nix secrets (first-time +# user setup only; idempotent on re-run). +# +# Usage: +# nix run .#setup-user # generate key if absent; print public key +# nix run .#setup-user -- --help +# +# Chicken-and-egg note: Mirrors `make setup-user` from the repo-root +# Makefile. Requires nix to be already installed (this flake app cannot +# run before nix). For a clean-host first-contact, use `make setup-user` +# instead; both targets share the same idempotence guarantee. +# +# Idempotent: if ~/.config/sops/age/keys.txt already exists, the script +# re-prints the public key and exits 0 WITHOUT regenerating. Only the +# first invocation writes keys.txt (mode 0600). +# +# Template bifurcation (writeShellApplication): PURE READFILE FORM. +# The sidecar needs no nix-eval-time path injection. +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.setup-user = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "setup-user"; + runtimeInputs = [ + pkgs.coreutils + # `age` provides age-keygen directly, avoiding a nested + # `nix shell nixpkgs#age` invocation (cleaner dependency + # closure than the Makefile's approach). + pkgs.age + ]; + text = builtins.readFile ./setup-user.sh; + } + ); + }; + }; +} diff --git a/modules/apps/bootstrap/setup-user.sh b/modules/apps/bootstrap/setup-user.sh new file mode 100644 index 000000000..13deea88c --- /dev/null +++ b/modules/apps/bootstrap/setup-user.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Generate the user's age key at ~/.config/sops/age/keys.txt. If the key +# already exists, print the public key and exit 0 without regenerating. +# +# Idempotent: re-running on a host with an existing key file is a no-op +# aside from stdout (no mutation of the key file or its parent dir). +set -euo pipefail + +usage() { + cat <<'EOF' +Usage: setup-user [--help] + +Generates an age keypair for sops-nix secrets on the current user at +~/.config/sops/age/keys.txt and prints the public key. If the file +already exists, re-prints the public key and exits 0 WITHOUT +regenerating. Mode 0600 on the private key. + +First-time setup: after running, back up the contents of keys.txt to +Bitwarden as a secure note `age-key-`, and send the public +key to the admin for addition to `.sops.yaml`. +EOF +} + +case "${1:-}" in + -h|--help) + usage + exit 0 + ;; +esac + +key_dir="${HOME}/.config/sops/age" +key_file="${key_dir}/keys.txt" + +printf '\n=== Age key setup ===\n\n' + +if [ -f "$key_file" ]; then + printf '⚠ Age key already exists at %s\n' "$key_file" + printf 'To regenerate, manually delete the file first.\n' + printf '\nYour public key is:\n' + if ! age-keygen -y "$key_file" 2>/dev/null; then + printf 'Error reading existing key (is the file corrupted?)\n' >&2 + exit 1 + fi + exit 0 +fi + +# First run: create the key file with a locked-down mode. +mkdir -p "$key_dir" +age-keygen -o "$key_file" +chmod 600 "$key_file" + +printf '\n● Age key generated successfully!\n\n' +printf 'Your public key is:\n' +age-keygen -y "$key_file" + +cat <<'EOF' + +⚠ IMPORTANT: Back up your private key to Bitwarden! + 1. Copy the content of ~/.config/sops/age/keys.txt + 2. Store in Bitwarden as a secure note: `age-key-` + 3. Send your PUBLIC key (shown above) to the admin + +See docs/new-user-host.md for complete setup instructions. +EOF diff --git a/modules/apps/bootstrap/verify.nix b/modules/apps/bootstrap/verify.nix new file mode 100644 index 000000000..724cce46c --- /dev/null +++ b/modules/apps/bootstrap/verify.nix @@ -0,0 +1,39 @@ +# Flake app: verify the host's nix + flakes + direnv + devShell setup. +# +# Usage: +# nix run .#verify # full status report; exit nonzero on missing nix/flakes +# nix run .#verify -- --help +# +# Chicken-and-egg note: Mirrors `make verify` from the repo-root Makefile. +# The Makefile version is callable from a nix-free shell (it's plain +# make + shell). This flake-app version assumes nix is already installed +# (since `nix run` requires nix); it exists so scripted contexts (CI, +# post-bootstrap sanity checks, buildbot effects) can invoke the audit +# without depending on GNU make being on PATH. +# +# Idempotent / pure: only reads state. Writes nothing; touches no files. +# +# Template bifurcation (writeShellApplication): PURE READFILE FORM. +# The sidecar needs no nix-eval-time path injection. +{ ... }: +{ + perSystem = + { pkgs, lib, ... }: + { + apps.verify = { + type = "app"; + program = lib.getExe ( + pkgs.writeShellApplication { + name = "verify"; + runtimeInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.gnugrep + pkgs.nix + ]; + text = builtins.readFile ./verify.sh; + } + ); + }; + }; +} diff --git a/modules/apps/bootstrap/verify.sh b/modules/apps/bootstrap/verify.sh new file mode 100644 index 000000000..f5a15dd59 --- /dev/null +++ b/modules/apps/bootstrap/verify.sh @@ -0,0 +1,104 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Verify the host's nix installation, flakes support, direnv presence, +# and flake metadata. Mirrors `make verify` from the repo-root Makefile, +# minus the devShell build (which is expensive and duplicates what +# `nix flake check` already covers). +# +# Idempotent / pure: does not mutate state. +set -euo pipefail + +usage() { + cat <<'EOF' +Usage: verify [--help] + +Audits the current host for a working nix + flakes + direnv setup and +validates that the invoking flake parses. Exits 0 on success, 1 if nix +or flakes are missing or the flake fails to parse. Prints a status line +per check. Read-only; does not mutate any system state. + +Equivalent to `make verify` but invokable from a nix-only shell (no +dependency on GNU make). +EOF +} + +case "${1:-}" in + -h|--help) + usage + exit 0 + ;; +esac + +failed=0 + +printf '\n=== Verifying installation ===\n\n' + +# Check 1: nix binary +printf 'Checking nix installation: ' +if command -v nix >/dev/null 2>&1; then + printf '● nix found at %s\n' "$(command -v nix)" + nix --version +else + printf '⊘ nix not found\n' + # shellcheck disable=SC2016 # backticks in string are literal output, not command substitution + printf ' Run `make install-nix` from a nix-free shell to install nix.\n' + failed=1 +fi +printf '\n' + +# Check 2: flakes support +printf 'Checking nix flakes support: ' +if nix flake --help >/dev/null 2>&1; then + printf '● flakes enabled\n' +else + printf '⊘ flakes not enabled\n' + failed=1 +fi +printf '\n' + +# Check 3: direnv (optional) +printf 'Checking direnv installation: ' +if command -v direnv >/dev/null 2>&1; then + printf '● direnv found at %s\n' "$(command -v direnv)" +else + printf '⚠ direnv not found (optional but recommended)\n' + # shellcheck disable=SC2016 # backticks in string are literal output, not command substitution + printf ' Run `nix run .#bootstrap` to install.\n' +fi +printf '\n' + +# Check 4: flake metadata parseable +printf 'Checking flake validity: ' +if nix --accept-flake-config flake metadata . >/dev/null 2>&1; then + printf '● flake is valid\n' +else + printf '⊘ flake has errors\n' + failed=1 +fi +printf '\n' + +# Check 5: surface /etc/nix/nix.conf for auditability (match make verify) +printf '/etc/nix/nix.conf:\n' +printf '==================\n' +if [ -f /etc/nix/nix.conf ]; then + cat /etc/nix/nix.conf +else + printf '(file not found)\n' +fi +printf '==================\n' + +if [ -f /etc/nix/nix.custom.conf ]; then + printf '\n/etc/nix/nix.custom.conf:\n' + printf '==================\n' + cat /etc/nix/nix.custom.conf + printf '==================\n' +fi +printf '\n' + +if [ "$failed" -eq 0 ]; then + printf '● All verification checks passed!\n\n' + exit 0 +else + printf '⊘ One or more verification checks failed.\n\n' >&2 + exit 1 +fi From 8537036e2a20e77cf3f3b3c1f4de89cb80ca28d3 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 04:35:44 -0400 Subject: [PATCH 07/77] docs(apps/cluster): cross-reference template bifurcation form in k3d-integration-ci header --- modules/apps/cluster/k3d-integration-ci.nix | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/modules/apps/cluster/k3d-integration-ci.nix b/modules/apps/cluster/k3d-integration-ci.nix index 46890231a..a296b44f4 100644 --- a/modules/apps/cluster/k3d-integration-ci.nix +++ b/modules/apps/cluster/k3d-integration-ci.nix @@ -3,7 +3,22 @@ # Usage: # nix run .#k3d-integration-ci # -# Template form: pure readFile (no nix-computed variable injection). +# Template bifurcation (writeShellApplication): PURE READFILE FORM. +# `text = builtins.readFile ./k3d-integration-ci.sh` — the sidecar is +# consumed verbatim, no nix-eval-time string interpolation. This is the +# cluster-domain representative of the pure form and the canonical +# starting point when converting a cluster script. +# +# Choose PURE form when the sidecar needs no nix-eval-time path injection +# (all inputs come from env vars, CLI args, or runtimeInputs). Choose +# INTERPOLATION form (a nix-string text attribute that concatenates an +# eval-time preamble with builtins.readFile of the sidecar) only when +# you must inject a nix-computed store path or derivation outPath into +# the script preamble — for the canonical example see +# `modules/apps/docs/deploy.nix`, which injects DOCS_PAYLOAD +# (config.packages.vanixiets-docs) and SOPS_SECRETS_FILE (inputs.self) +# at eval time. +# # Orchestrates the seven-phase CI integration flow that is currently # invoked by `.github/workflows/test-cluster.yaml`. Delegates to the # sibling cluster/docs flake apps (nixidy-build, nixidy-bootstrap, From 2fc608997014f652a829c344a30d6686737846bb Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 10:57:29 -0400 Subject: [PATCH 08/77] fix(apps/docs): add --help handlers and harden preview-version merge-tree error handling --- modules/apps/docs/deploy.sh | 51 ++++++++++++++++++++++++ modules/apps/docs/preview-version.sh | 58 +++++++++++++++++++++++++--- 2 files changed, 104 insertions(+), 5 deletions(-) diff --git a/modules/apps/docs/deploy.sh b/modules/apps/docs/deploy.sh index b035210d8..49d0239fa 100644 --- a/modules/apps/docs/deploy.sh +++ b/modules/apps/docs/deploy.sh @@ -10,12 +10,62 @@ # Usage: # deploy-docs preview # deploy-docs production +# deploy-docs --help set -euo pipefail +usage() { + cat <<'EOF' +usage: deploy-docs preview + deploy-docs production + deploy-docs --help + +Deploy the nix-built vanixiets-docs payload to Cloudflare Workers. + +Subcommands: + preview Upload a Cloudflare Workers preview version tagged with + the current HEAD short SHA, aliased at b-. + defaults to `git branch --show-current`; explicit + value required when HEAD is detached. + production Promote the existing preview version matching the current + HEAD short SHA to 100% production traffic, or fall back + to a direct deploy of the nix-built payload when no + matching preview exists. + +Flags: + --help, -h Print this usage and exit 0. + +Environment contract (populated by deploy.nix; required at runtime): + DOCS_PAYLOAD Absolute path to the vanixiets-docs derivation output + ($out/{dist/, .wrangler/, wrangler.jsonc}). + SOPS_SECRETS_FILE Absolute path to secrets/shared.yaml under $inputs.self; + source of Cloudflare credentials via `sops exec-env`. + DOCS_NODE_MODULES Absolute path to the vanixiets-docs-deps node_modules + tree (hosts the hermetic wrangler binary). + +Optional environment: + GITHUB_ACTIONS / GITHUB_ACTOR / GITHUB_WORKFLOW + When set, the production deploy message is prefixed with + the GitHub Actions context; otherwise whoami and hostname + are used. + +Examples: + nix run .#deploy-docs -- preview my-feature-branch + nix run .#deploy-docs -- production +EOF +} + mode="${1:-}" +case "$mode" in + -h | --help) + usage + exit 0 + ;; +esac + if [[ -z "$mode" ]]; then echo "error: missing subcommand" >&2 echo "usage: deploy-docs preview | deploy-docs production" >&2 + echo "(run with --help for full usage and env-var contract)" >&2 exit 2 fi shift @@ -207,6 +257,7 @@ case "$mode" in *) echo "error: unknown subcommand '$mode'" >&2 echo "usage: deploy-docs preview | deploy-docs production" >&2 + echo "(run with --help for full usage and env-var contract)" >&2 exit 2 ;; esac diff --git a/modules/apps/docs/preview-version.sh b/modules/apps/docs/preview-version.sh index b7cad0d54..11949a35c 100644 --- a/modules/apps/docs/preview-version.sh +++ b/modules/apps/docs/preview-version.sh @@ -21,6 +21,49 @@ set -euo pipefail +usage() { + cat <<'EOF' +usage: preview-version [target-branch] [package-path] + preview-version --help + +Preview the semantic-release version that would be published after merging +the current branch into . Simulates the merge via +`git merge-tree --write-tree`, runs semantic-release in --dry-run / --no-ci +mode against a temporary worktree, and prints the next version (or a +no-bump / unsupported-branch notice). + +Positional arguments: + target-branch Release branch to simulate merging into (default: main). + package-path Monorepo package directory relative to the repo root + (e.g., packages/docs). Defaults to the root package. + +Flags: + --help, -h Print this usage and exit 0. + +Environment: + DOCS_NODE_MODULES (required) Absolute path to the vanixiets-docs-deps + node_modules tree, provided by preview-version.nix. + Symlinked into the temporary worktree so + semantic-release and its plugins are resolvable. + CURRENT_BRANCH (optional) Bookmark/branch name to attach HEAD to when + invoked from a jj-colocated detached-HEAD setup. When + set while HEAD is detached, the script checks out the + branch for the run and restores detached state on exit. + +Examples: + nix run .#preview-version # root package on main + nix run .#preview-version -- main packages/docs # docs package on main + nix run .#preview-version -- beta packages/docs # docs package on beta +EOF +} + +case "${1:-}" in + -h | --help) + usage + exit 0 + ;; +esac + # Configuration TARGET_BRANCH="${1:-main}" PACKAGE_PATH="${2:-}" @@ -184,11 +227,16 @@ ORIGINAL_REMOTE_HEAD=$(git rev-parse "origin/$TARGET_BRANCH" 2>/dev/null || echo # Create merge tree to test if merge is possible echo -e "${BLUE}simulating merge of ${CURRENT_BRANCH} → ${TARGET_BRANCH}...${NC}" -# Perform merge-tree operation to test if merge is possible -MERGE_OUTPUT=$(git merge-tree --write-tree "$TARGET_BRANCH" "$CURRENT_BRANCH" 2>&1) -MERGE_EXIT=$? - -if [ $MERGE_EXIT -ne 0 ]; then +# Perform merge-tree operation to test if merge is possible. +# +# The `if ! MERGE_OUTPUT=$(...)` form is deliberate: under `set -e`, a +# failing command substitution in a bare assignment (`VAR=$(cmd)`) does +# NOT cause the script to exit in every bash version/mode and does not +# propagate `$?` reliably when combined with `inherit_errexit` — the prior +# pattern of `VAR=$(cmd); RC=$?` was fragile. Guard the assignment with +# `if !` so merge-conflict detection is explicit and independent of +# errexit semantics. +if ! MERGE_OUTPUT=$(git merge-tree --write-tree "$TARGET_BRANCH" "$CURRENT_BRANCH" 2>&1); then echo -e "${RED}error: merge conflicts detected${NC}" >&2 echo -e "${YELLOW}please resolve conflicts in your branch before previewing${NC}" >&2 echo -e "\n${YELLOW}conflict details:${NC}" >&2 From 2d314e77c2e4a6f0ad378578139f21a4f001b81b Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 11:09:19 -0400 Subject: [PATCH 09/77] fix(apps/release): allow --dry-run to proceed with existing non-symlink node_modules --- modules/apps/release/release.sh | 35 +++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/modules/apps/release/release.sh b/modules/apps/release/release.sh index 3d4bc5c91..78a327184 100644 --- a/modules/apps/release/release.sh +++ b/modules/apps/release/release.sh @@ -176,12 +176,43 @@ fi cd "$package_path" # Guard node_modules slot against clobbering a developer's real install. +# Production (non-dry-run): strict — refuse to overwrite a real node_modules +# directory. Only an empty slot or a pre-existing symlink is safe to clobber. +# Dry-run: proceed safely via two strategies that NEVER mutate the +# developer's real install in place: +# (b) reuse the existing node_modules directly if it already contains a +# usable semantic-release binary (common when the dev ran `bun install` +# to completion), or +# (a) move the existing node_modules aside to a tempdir, symlink +# DOCS_NODE_MODULES in its place for the duration of the run, and +# atomically restore the original on EXIT (including on error/SIGINT). +nm_exists_real=0 if [[ -e node_modules && ! -L node_modules ]]; then + nm_exists_real=1 +fi + +if [ "$nm_exists_real" -eq 1 ] && [ "$dry_run" -ne 1 ]; then echo "error: $package_path/node_modules exists and is not a symlink; refusing to overwrite a local bun install" >&2 exit 1 fi -trap 'rm -f "$PWD/node_modules"' EXIT -ln -snf "$DOCS_NODE_MODULES" node_modules + +if [ "$nm_exists_real" -eq 1 ] && [ -x node_modules/.bin/semantic-release ]; then + # Dry-run strategy (b): reuse existing node_modules in place. + echo "dry-run: reusing existing node_modules (.bin/semantic-release present)" >&2 +elif [ "$nm_exists_real" -eq 1 ]; then + # Dry-run strategy (a): move existing node_modules aside, symlink for the + # duration of the run, restore atomically on exit. + backup_dir="$(mktemp -d)" + echo "dry-run: moving existing node_modules to ${backup_dir} (restored on exit)" >&2 + mv node_modules "${backup_dir}/node_modules" + # shellcheck disable=SC2064 + trap "rm -f '${PWD}/node_modules'; mv '${backup_dir}/node_modules' '${PWD}/node_modules' 2>/dev/null || true; rmdir '${backup_dir}' 2>/dev/null || true" EXIT + ln -snf "$DOCS_NODE_MODULES" node_modules +else + # Slot is empty or already a symlink — safe to (re)link. + trap 'rm -f "$PWD/node_modules"' EXIT + ln -snf "$DOCS_NODE_MODULES" node_modules +fi if [ "$dry_run" -eq 1 ]; then # Filter @semantic-release/github so GITHUB_TOKEN is not required for From 7be4499c4a0376372c8b5dc0e988ad69936d4a2c Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 12:11:09 -0400 Subject: [PATCH 10/77] fix(cd): install nix for setup-variables job --- .github/workflows/cd.yaml | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/.github/workflows/cd.yaml b/.github/workflows/cd.yaml index 85c73be73..8fba35052 100644 --- a/.github/workflows/cd.yaml +++ b/.github/workflows/cd.yaml @@ -133,10 +133,17 @@ jobs: - name: Checkout for package discovery uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: - sparse-checkout: | - packages - justfile - sparse-checkout-cone-mode: false + fetch-depth: 0 + # sparse-checkout: | + # packages + # justfile + # sparse-checkout-cone-mode: false + + - name: Setup Nix + uses: ./.github/actions/setup-nix + with: + installer: quick + system: x86_64-linux # alternative: extractions/setup-just@v3 (blocked on node20->node24 upgrade via extractions/setup-crate#9) - name: Install just @@ -171,18 +178,18 @@ jobs: - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: - ref: ${{ github.head_ref }} # Checkout actual PR branch, not merge commit - fetch-depth: 0 # Full history needed for semantic-release analysis - fetch-tags: true # Explicitly fetch all tags for version detection + ref: ${{ github.head_ref }} # Checkout actual PR branch, not merge commit + fetch-depth: 0 # Full history needed for semantic-release analysis + fetch-tags: true # Explicitly fetch all tags for version detection - name: Check execution cache id: cache uses: ./.github/actions/cached-ci-job with: check-name: ${{ matrix.package.name }}-preview-release - hash-sources: 'packages/${{ matrix.package.name }}/**/* .github/actions/setup-nix/action.yml .github/workflows/cd.yaml flake.nix flake.lock bun.lock modules/apps/docs/**/* pkgs/by-name/vanixiets-docs/**/*' + hash-sources: "packages/${{ matrix.package.name }}/**/* .github/actions/setup-nix/action.yml .github/workflows/cd.yaml flake.nix flake.lock bun.lock modules/apps/docs/**/* pkgs/by-name/vanixiets-docs/**/*" # Always run: semantic-release analyzes commit history which changes constantly - force-run: 'true' + force-run: "true" - name: Fetch target branch for preview if: steps.cache.outputs.should-run == 'true' @@ -283,14 +290,14 @@ jobs: - name: checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: - fetch-depth: 0 # for git diff in composite action + fetch-depth: 0 # for git diff in composite action - name: Check execution cache id: cache uses: ./.github/actions/cached-ci-job with: check-name: ${{ github.job }} - hash-sources: 'Makefile .envrc .github/actions/setup-nix/action.yml' + hash-sources: "Makefile .envrc .github/actions/setup-nix/action.yml" force-run: ${{ needs.set-variables.outputs.force-ci }} - name: run make bootstrap From 54f5dfd08d003743ee1cd3c71c9b9b17e45a86ef Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 13:00:33 -0400 Subject: [PATCH 11/77] feat(apps/docs): add wrangler post-condition assertions with 4.84.x-compatible extraction Establishes a VAL-WRITESHELL-DOCS-010-style post-condition harness for wrangler versions upload / versions deploy / deploy / deployments list so silent-success regressions fail loudly. Previous reliance on wrangler's exit code masked upload failures that wrangler reports as exit 0 with no Worker Version ID produced. Uses WRANGLER_OUTPUT_FILE_PATH NDJSON event capture (supported since wrangler 3.x; --json is rejected by 4.84.x on versions upload/deploy subcommands) with stdout grep fallback. Gates success on: - (a) non-empty version_id / deployment_id extracted from either NDJSON type:"version-upload" event or stdout Worker Version ID - (b) wrangler versions list --json contains an entry whose annotations["workers/tag"] matches the current HEAD short SHA, confirming the upload persisted server-side - (c) authoritative success echo gated on (a) and (b) --- modules/apps/docs/deploy.sh | 347 +++++++++++++++++++++++++++++++++--- 1 file changed, 319 insertions(+), 28 deletions(-) diff --git a/modules/apps/docs/deploy.sh b/modules/apps/docs/deploy.sh index 49d0239fa..e149c50da 100644 --- a/modules/apps/docs/deploy.sh +++ b/modules/apps/docs/deploy.sh @@ -93,7 +93,10 @@ fi # Hermetic wrangler via bun-managed node_modules (vanixiets-docs-deps derivation). # Must be exported so sops exec-env subshells inherit it for single-quoted command strings. -export WRANGLER="$DOCS_NODE_MODULES/.bin/wrangler" +# The `${WRANGLER:-...}` fallback allows test harnesses (e.g. the no-op wrangler stub +# used to exercise the post-condition error paths for VAL-WRITESHELL-DOCS-010) to +# override the hermetic binary without rewriting this script. +export WRANGLER="${WRANGLER:-$DOCS_NODE_MODULES/.bin/wrangler}" # Resolve repo root so git metadata commands work independently of callsite. repo_root=$(git rev-parse --show-toplevel) @@ -104,7 +107,12 @@ cd "$repo_root" # resolves against the config file's location, and wrangler may write state to # .wrangler/ during deploy — both require a writable tree outside /nix/store. tmpdir=$(mktemp -d -t deploy-docs.XXXXXX) -trap 'rm -rf "$tmpdir"' EXIT +if [[ -n "${DEPLOY_DOCS_DEBUG:-}" ]]; then + echo "[deploy-docs] DEBUG: preserving tmpdir at $tmpdir" >&2 + trap 'echo "[deploy-docs] DEBUG: tmpdir preserved at '\''$tmpdir'\''" >&2' EXIT +else + trap 'rm -rf "$tmpdir"' EXIT +fi cp -R "$DOCS_PAYLOAD"/. "$tmpdir/" chmod -R u+w "$tmpdir" @@ -167,16 +175,126 @@ case "$mode" in export SAFE_BRANCH="$safe_branch" export WRANGLER_CONFIG="$wrangler_config" + # Capture wrangler's machine-readable NDJSON event log via + # WRANGLER_OUTPUT_FILE_PATH (supported by wrangler >= 3.x; confirmed on + # 4.84.1 by grepping `WRANGLER_OUTPUT_FILE_PATH` + `type: "version-upload"` + # in packages/docs/node_modules/wrangler/wrangler-dist/cli.js). The + # previous revision of this script used `--json` on `wrangler versions + # upload`, but wrangler 4.84.x does NOT accept `--json` on that subcommand + # (GHA re-run against cd-via-effects @ 6ce9fca2 exited 1 with "Unknown + # argument: json"); `--json` is only supported on the `versions list` and + # `deployments list` subcommands. The NDJSON stream is emitted to the file + # named by WRANGLER_OUTPUT_FILE_PATH; each line is a JSON object with a + # `type` discriminator. For `versions upload` we look for the + # `version-upload` event, which carries `version_id`, `worker_tag`, + # `preview_url`, and `preview_alias_url`. + # + # Three post-conditions enforce the no-silent-success invariant + # (VAL-WRITESHELL-DOCS-010): + # (a) the NDJSON event log contains a `type == "version-upload"` entry + # with a non-empty `version_id` (primary authoritative source) + # (b) `wrangler versions list --json` contains an entry whose + # annotations["workers/tag"] matches $commit_tag (server-side + # persistence cross-check) + # (c) only then is the user-visible success block echoed, including the + # authoritative Worker Version ID parsed from (a). + wrangler_upload_ndjson="$tmpdir/wrangler-versions-upload.ndjson" + wrangler_upload_stdout="$tmpdir/wrangler-versions-upload.stdout" + : > "$wrangler_upload_ndjson" + : > "$wrangler_upload_stdout" + export WRANGLER_OUTPUT_FILE_PATH="$wrangler_upload_ndjson" + + # Tee stdout so we can both display wrangler output live AND parse it as a + # fallback version_id source (Option B). Observed 2026-04-22: wrangler + # 4.84.1 occasionally completes the `/versions` upload (server-side + # version is persisted, `Worker Version ID: ...` is logged to stdout) but + # then hangs/terminates on the subsequent `/workers/subdomain` GET + # request, preventing the `writeOutput({type: "version-upload", ...})` + # call from firing. Capturing stdout in parallel lets us recover the + # authoritative version_id even in that partial-completion case. + # # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell sops exec-env "$SOPS_SECRETS_FILE" ' "$WRANGLER" --config "$WRANGLER_CONFIG" versions upload \ --preview-alias "b-${SAFE_BRANCH}" \ --tag "$VERSION_TAG" \ --message "$VERSION_MESSAGE" - ' + ' | tee "$wrangler_upload_stdout" + + unset WRANGLER_OUTPUT_FILE_PATH + + # Post-condition (a): extract a non-empty Worker Version ID. + # Primary source: NDJSON `version-upload` event (Option A) + # Fallback source: stdout line `Worker Version ID: ` (Option B) + # The cross-check in post-condition (b) below guarantees the version + # actually persisted server-side regardless of which source produced it. + version_id="" + if [[ -s "$wrangler_upload_ndjson" ]]; then + version_id=$( + jq -rs ' + map(select(type == "object" and (.type // "") == "version-upload")) + | .[0].version_id // empty + ' "$wrangler_upload_ndjson" 2>/dev/null || true + ) + fi + if [[ -z "$version_id" ]]; then + version_id=$( + grep -oE 'Worker Version ID: [a-f0-9-]+' "$wrangler_upload_stdout" 2>/dev/null \ + | awk '{print $NF}' \ + | head -1 || true + ) + fi + if [[ -z "$version_id" ]]; then + echo "" >&2 + echo "error: wrangler exited 0 but produced no Worker Version ID" >&2 + echo " post-condition (a) failed: neither WRANGLER_OUTPUT_FILE_PATH NDJSON" >&2 + echo " event log nor wrangler stdout contained" >&2 + echo " a recognizable Worker Version ID" >&2 + echo " raw wrangler event log: $wrangler_upload_ndjson" >&2 + echo " raw wrangler stdout: $wrangler_upload_stdout" >&2 + echo " hints:" >&2 + echo " - confirm CF_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 + echo " $SOPS_SECRETS_FILE" >&2 + echo " - if running in GHA, wrangler's CI-detection branch may have" >&2 + echo " silently exited; rerun with WRANGLER_LOG=debug for stderr trace" >&2 + echo " - inspect the raw NDJSON and stdout paths above for any output" >&2 + exit 1 + fi + # Post-condition (b): cross-check via versions list that the upload landed + # server-side with the expected commit tag annotation. The `| cat` pipe + # ensures wrangler's stdout is delivered through a pipe-shaped fd before + # being redirected to disk (observed empirically: `wrangler ... --json > + # file` intermittently produces zero bytes whereas `wrangler ... --json | + # cat > file` reliably produces the full JSON output, which suggests + # wrangler inspects stdout before emitting when the fd points directly at + # a file). + wrangler_list_json="$tmpdir/wrangler-versions-list.json" + + # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell + sops exec-env "$SOPS_SECRETS_FILE" ' + "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json + ' | cat > "$wrangler_list_json" + + matched_count=$(jq --arg tag "$commit_tag" \ + '[.[] | select(.annotations["workers/tag"] == $tag)] | length' \ + "$wrangler_list_json" 2>/dev/null || echo 0) + if [[ "$matched_count" -lt 1 ]]; then + echo "" >&2 + echo "error: uploaded version with tag ${commit_tag} not found in versions list" >&2 + echo " post-condition (b) failed: wrangler versions list returned no entries" >&2 + echo " with annotations[\"workers/tag\"] == ${commit_tag}" >&2 + echo " raw versions list output: $wrangler_list_json" >&2 + echo " hint: wrangler reported a version_id locally but the Cloudflare API did" >&2 + echo " not persist it; retry with WRANGLER_LOG=debug or inspect the raw" >&2 + echo " versions list for surrounding entries" >&2 + exit 1 + fi + + # Post-condition (c): authoritative success echo with parsed Worker Version ID. echo "" echo "Version uploaded successfully" + echo " Worker Version ID: ${version_id}" echo " Tag: ${commit_tag}" echo " Full SHA: ${commit_sha}" echo " Message: ${version_message}" @@ -194,11 +312,22 @@ case "$mode" in export WRANGLER_CONFIG="$wrangler_config" # Query for an existing version uploaded from this commit (via preview). + # Capture versions list to a tempfile so post-condition verification below + # can reuse it (avoids a second API call purely for the existing-version + # lookup) and any diagnostic error messages can reference the raw JSON. + # `| cat >` is used instead of `>` to route wrangler's stdout through a + # pipe-shaped fd; see the preview subcommand's equivalent comment for the + # empirical rationale. + wrangler_list_json="$tmpdir/wrangler-versions-list.json" + # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - existing_version=$(sops exec-env "$SOPS_SECRETS_FILE" ' + sops exec-env "$SOPS_SECRETS_FILE" ' "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json - ' | jq -r --arg tag "$commit_tag" \ - '.[] | select(.annotations["workers/tag"] == $tag) | .id' | head -1) + ' | cat > "$wrangler_list_json" + + existing_version=$(jq -r --arg tag "$commit_tag" \ + '.[] | select(.annotations["workers/tag"] == $tag) | .id' \ + "$wrangler_list_json" 2>/dev/null | head -1 || true) if [[ -n "$existing_version" ]]; then echo "found existing version: ${existing_version}" @@ -207,26 +336,107 @@ case "$mode" in echo "" export DEPLOYMENT_MESSAGE="$deploy_msg" + export EXISTING_VERSION="$existing_version" + + # Post-condition verification mirrors the preview path: capture + # wrangler's NDJSON event log via WRANGLER_OUTPUT_FILE_PATH, assert a + # non-empty deployment_id on the `version-deploy` event, then cross-check + # via `wrangler deployments list --json` before declaring success. Like + # `versions upload`, `versions deploy` does NOT accept `--json` on + # wrangler 4.84.x — the event log is the authoritative machine-readable + # output channel. Detects wrangler's silent-exit failure mode + # (VAL-WRITESHELL-DOCS-010 + diagnostic session 45961bc9) when the + # CI-detection branch exits 0 without actually performing the promotion. + deploy_ndjson="$tmpdir/wrangler-versions-deploy.ndjson" + deploy_stdout="$tmpdir/wrangler-versions-deploy.stdout" + : > "$deploy_ndjson" + : > "$deploy_stdout" + export WRANGLER_OUTPUT_FILE_PATH="$deploy_ndjson" # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - if sops exec-env "$SOPS_SECRETS_FILE" ' + sops exec-env "$SOPS_SECRETS_FILE" ' "$WRANGLER" --config "$WRANGLER_CONFIG" versions deploy \ - "'"$existing_version"'@100%" \ + "${EXISTING_VERSION}@100%" \ --yes \ --message "$DEPLOYMENT_MESSAGE" - '; then - echo "" - echo "successfully promoted version ${existing_version} to production" - echo " tag: ${commit_tag}" - echo " full SHA: ${commit_sha}" - echo " deployed by: ${deploy_msg}" - echo " production URL: https://infra.cameronraysmith.net" - else - echo "" - echo "error: failed to promote version ${existing_version}" >&2 - echo " deployment was cancelled or failed" >&2 + ' | tee "$deploy_stdout" + + unset WRANGLER_OUTPUT_FILE_PATH + + # Post-condition (a): extract a non-empty Deployment ID. Primary source + # is the NDJSON `version-deploy` event (Option A); fallback is stdout + # parsing for a recognizable deployment identifier (Option B). + deployment_id="" + if [[ -s "$deploy_ndjson" ]]; then + deployment_id=$( + jq -rs ' + map(select(type == "object" and (.type // "") == "version-deploy")) + | .[0].deployment_id // empty + ' "$deploy_ndjson" 2>/dev/null || true + ) + fi + if [[ -z "$deployment_id" ]]; then + # stdout fallback: match patterns like "Deployment ID: " or + # "deployment_id: " that wrangler prints on the console. + deployment_id=$( + grep -oiE '(Deployment ID|deployment_id)[[:space:]]*:[[:space:]]*[a-f0-9-]+' \ + "$deploy_stdout" 2>/dev/null \ + | awk '{print $NF}' \ + | head -1 || true + ) + fi + if [[ -z "$deployment_id" ]]; then + echo "" >&2 + echo "error: wrangler exited 0 but produced no Deployment ID" >&2 + echo " post-condition (a) failed: neither WRANGLER_OUTPUT_FILE_PATH NDJSON" >&2 + echo " event log nor wrangler stdout contained" >&2 + echo " a recognizable Deployment ID" >&2 + echo " raw wrangler event log: $deploy_ndjson" >&2 + echo " raw wrangler stdout: $deploy_stdout" >&2 + echo " hints:" >&2 + echo " - confirm CF_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 + echo " $SOPS_SECRETS_FILE" >&2 + echo " - if running in GHA, wrangler's CI-detection branch may have" >&2 + echo " silently exited; rerun with WRANGLER_LOG=debug for stderr trace" >&2 + exit 1 + fi + + # Post-condition (b): cross-check via deployments list that the deploy + # landed server-side. `| cat >` empirically required — see preview path + # comment for the rationale. + deployments_list_json="$tmpdir/wrangler-deployments-list.json" + + # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell + sops exec-env "$SOPS_SECRETS_FILE" ' + "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json + ' | cat > "$deployments_list_json" + + found_count=$(jq --arg did "$deployment_id" --arg vid "$existing_version" \ + '[.[] | select( + .id == $did + or .deployment_id == $did + or ((.versions // []) | map(.version_id // .id // "") | index($vid) != null) + )] | length' \ + "$deployments_list_json" 2>/dev/null || echo 0) + if [[ "$found_count" -lt 1 ]]; then + echo "" >&2 + echo "error: deployment ${deployment_id} (version ${existing_version}) not found in deployments list" >&2 + echo " post-condition (b) failed: wrangler deployments list returned no" >&2 + echo " entries matching the just-deployed id/version" >&2 + echo " raw deployments list output: $deployments_list_json" >&2 + echo " hint: wrangler reported a deployment locally but the Cloudflare API did" >&2 + echo " not persist it; retry with WRANGLER_LOG=debug" >&2 exit 1 fi + + # Post-condition (c): authoritative success echo with parsed Deployment ID. + echo "" + echo "successfully promoted version ${existing_version} to production" + echo " Deployment ID: ${deployment_id}" + echo " tag: ${commit_tag}" + echo " full SHA: ${commit_sha}" + echo " deployed by: ${deploy_msg}" + echo " production URL: https://infra.cameronraysmith.net" else echo "warning: no existing version found with tag: ${commit_tag}" echo " this should only happen if:" @@ -239,18 +449,99 @@ case "$mode" in export DEPLOYMENT_MESSAGE="$deploy_msg" + # Fallback direct-deploy: same post-condition pattern, but the relevant + # NDJSON event is `type == "deploy"` which carries `version_id` (no + # deployment_id field on this event type — see wrangler cli.js + # writeOutput block for `deploy`). `wrangler deploy` does NOT accept + # `--json` on wrangler 4.84.x; WRANGLER_OUTPUT_FILE_PATH is the + # authoritative machine-readable channel. + deploy_ndjson="$tmpdir/wrangler-deploy.ndjson" + deploy_stdout="$tmpdir/wrangler-deploy.stdout" + : > "$deploy_ndjson" + : > "$deploy_stdout" + export WRANGLER_OUTPUT_FILE_PATH="$deploy_ndjson" + # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - if sops exec-env "$SOPS_SECRETS_FILE" ' - "$WRANGLER" --config "$WRANGLER_CONFIG" deploy --message "$DEPLOYMENT_MESSAGE" - '; then - echo "" - echo "deployed nix-built payload directly to production" - echo " warning: this version was not tested in preview first" - else - echo "" - echo "error: failed to deploy" >&2 + sops exec-env "$SOPS_SECRETS_FILE" ' + "$WRANGLER" --config "$WRANGLER_CONFIG" deploy \ + --message "$DEPLOYMENT_MESSAGE" + ' | tee "$deploy_stdout" + + unset WRANGLER_OUTPUT_FILE_PATH + + # Post-condition (a): extract the just-deployed version_id. Primary + # source: NDJSON `deploy` event (Option A). Fallback: stdout grep + # (Option B) for the "Current Version ID: " or similar line + # wrangler prints on direct deploy. + deploy_version_id="" + if [[ -s "$deploy_ndjson" ]]; then + deploy_version_id=$( + jq -rs ' + map(select(type == "object" and (.type // "") == "deploy")) + | .[0].version_id // empty + ' "$deploy_ndjson" 2>/dev/null || true + ) + fi + if [[ -z "$deploy_version_id" ]]; then + deploy_version_id=$( + grep -oiE '(Current Version ID|Worker Version ID|version_id)[[:space:]]*:[[:space:]]*[a-f0-9-]+' \ + "$deploy_stdout" 2>/dev/null \ + | awk '{print $NF}' \ + | head -1 || true + ) + fi + if [[ -z "$deploy_version_id" ]]; then + echo "" >&2 + echo "error: wrangler exited 0 but produced no Deployment Version ID (fallback direct deploy)" >&2 + echo " post-condition (a) failed: neither WRANGLER_OUTPUT_FILE_PATH NDJSON" >&2 + echo " event log nor wrangler stdout contained" >&2 + echo " a recognizable version_id" >&2 + echo " raw wrangler event log: $deploy_ndjson" >&2 + echo " raw wrangler stdout: $deploy_stdout" >&2 + echo " hints:" >&2 + echo " - confirm CF_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 + echo " $SOPS_SECRETS_FILE" >&2 + echo " - if running in GHA, wrangler's CI-detection branch may have" >&2 + echo " silently exited; rerun with WRANGLER_LOG=debug for stderr trace" >&2 + exit 1 + fi + # Reuse deployment_id slot below (it now holds the just-deployed version_id + # since `wrangler deploy` emits no server-assigned deployment id directly). + deployment_id="$deploy_version_id" + + deployments_list_json="$tmpdir/wrangler-deployments-list.json" + + # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell + sops exec-env "$SOPS_SECRETS_FILE" ' + "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json + ' | cat > "$deployments_list_json" + + found_count=$(jq --arg vid "$deploy_version_id" \ + '[.[] | select( + .id == $vid + or .deployment_id == $vid + or ((.versions // []) | map(.version_id // .id // "") | index($vid) != null) + )] | length' \ + "$deployments_list_json" 2>/dev/null || echo 0) + if [[ "$found_count" -lt 1 ]]; then + echo "" >&2 + echo "error: deployment for version ${deploy_version_id} not found in deployments list (fallback direct deploy)" >&2 + echo " post-condition (b) failed: wrangler deployments list returned no" >&2 + echo " entries matching the just-deployed version_id" >&2 + echo " raw deployments list output: $deployments_list_json" >&2 + echo " hint: wrangler reported a deployment locally but the Cloudflare API did" >&2 + echo " not persist it; retry with WRANGLER_LOG=debug" >&2 exit 1 fi + + echo "" + echo "deployed nix-built payload directly to production" + echo " Deployment Version ID: ${deployment_id}" + echo " tag: ${commit_tag}" + echo " full SHA: ${commit_sha}" + echo " deployed by: ${deploy_msg}" + echo " production URL: https://infra.cameronraysmith.net" + echo " warning: this version was not tested in preview first" fi ;; From 2e54884a2d5ffa35ec6e23352e272f8146b8dd89 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 14:29:31 -0400 Subject: [PATCH 12/77] fix(apps/docs): invoke wrangler via real node to bypass bun-fake-node silent fetch hang MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wrangler's .bin/wrangler wrapper shebang resolves to bun-with-fake-node/bin/node (bun in node-compat mode) because bun installed the node_modules tree. On linux-x64, bun's fetch() silently hangs on api.cloudflare.com keep-alive connection reuse — wrangler versions upload / versions deploy / deploy exit 0 with no Worker Version ID produced, no asset upload, no error output. Root cause is in bun's native (Zig) fetch implementation, not wrangler or undici. Prefix wrangler invocations with \`node\` to force real-node runtime (undici fetch), matching the established repo pattern at pkgs/by-name/vanixiets-docs/package.nix:141 (astro build: "bun's incomplete ws shim") and :248 (playwright test: "bun's child_process.fork() IPC"). The \`node\` binary is already available via modules/apps/docs/deploy.nix runtimeInputs (pkgs.nodejs_24). Diagnosis confirmed empirically on magnetite linux-x64 (2026-04-22) and validated in GHA. Preserves the diagnostic plumbing (stderr split-capture, wrangler exit-code capture, wrangler internal log cat on failure via glob-based path lookup, NDJSON/stdout/stderr dumps in the post-condition failure path) as load-bearing regression observability for future wrangler upgrades. --- modules/apps/docs/deploy.sh | 139 ++++++++++++++++++++++++++++-------- 1 file changed, 108 insertions(+), 31 deletions(-) diff --git a/modules/apps/docs/deploy.sh b/modules/apps/docs/deploy.sh index e149c50da..5658c2d22 100644 --- a/modules/apps/docs/deploy.sh +++ b/modules/apps/docs/deploy.sh @@ -98,6 +98,17 @@ fi # override the hermetic binary without rewriting this script. export WRANGLER="${WRANGLER:-$DOCS_NODE_MODULES/.bin/wrangler}" +# Invoke wrangler via real node, not the .bin/wrangler shebang: +# bun's .bin wrappers point at bun-with-fake-node/bin/node (bun in node- +# compat mode), but bun's fetch() on linux-x64 silently hangs on keep- +# alive connection reuse to api.cloudflare.com — wrangler `versions +# upload` / `versions deploy` exit 0 with no Worker Version ID produced +# and no error. Prefixing `node` forces real-node (undici) runtime. +# Matches pkgs/by-name/vanixiets-docs/package.nix:141 (astro) and :248 +# (playwright) precedent for tools with known bun incompatibilities. +# Empirical: diagnosed 2026-04-22 via magnetite linux-x64 reproducer; +# same machine + wrangler runs fine under real node, hangs under bun. + # Resolve repo root so git metadata commands work independently of callsite. repo_root=$(git rev-parse --show-toplevel) cd "$repo_root" @@ -200,26 +211,44 @@ case "$mode" in # authoritative Worker Version ID parsed from (a). wrangler_upload_ndjson="$tmpdir/wrangler-versions-upload.ndjson" wrangler_upload_stdout="$tmpdir/wrangler-versions-upload.stdout" + wrangler_upload_stderr="$tmpdir/wrangler-versions-upload.stderr" : > "$wrangler_upload_ndjson" : > "$wrangler_upload_stdout" + : > "$wrangler_upload_stderr" export WRANGLER_OUTPUT_FILE_PATH="$wrangler_upload_ndjson" - - # Tee stdout so we can both display wrangler output live AND parse it as a - # fallback version_id source (Option B). Observed 2026-04-22: wrangler - # 4.84.1 occasionally completes the `/versions` upload (server-side - # version is persisted, `Worker Version ID: ...` is logged to stdout) but - # then hangs/terminates on the subsequent `/workers/subdomain` GET - # request, preventing the `writeOutput({type: "version-upload", ...})` - # call from firing. Capturing stdout in parallel lets us recover the - # authoritative version_id even in that partial-completion case. + # Note: WRANGLER_LOG=debug was observed to deterministically terminate + # wrangler 4.84.1 mid-fetch (process exits 0 after POST + # /assets-upload-session request, before response; on GHA similar early + # termination at GET /workers/services/). Upload then never + # completes. Do NOT re-enable without gating it to a retry-only code + # path. Wrangler's internal log file at ~/.wrangler/logs/wrangler-*.log + # is written at default level regardless and is captured on failure. + + # Tee stdout so we both display wrangler output live AND parse it as a + # fallback version_id source when the NDJSON event stream from + # WRANGLER_OUTPUT_FILE_PATH doesn't produce the expected + # `type:"version-upload"` event. Retained as defense-in-depth against + # future wrangler silent-success regressions. See top-of-file rationale + # (lines ~101-110) for the diagnostic history. # + # Diagnostic: echo the exact upload command line to stderr so the GHA log + # shows what the shell is about to invoke (minus secret env vars which are + # expanded by sops-wrapped subshell). + printf '>> wrangler upload command: node %s --config %s versions upload --preview-alias %s --tag %s --message %q\n' \ + "$WRANGLER" "$WRANGLER_CONFIG" "b-${SAFE_BRANCH}" "$VERSION_TAG" "$VERSION_MESSAGE" >&2 + + set +e # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell sops exec-env "$SOPS_SECRETS_FILE" ' - "$WRANGLER" --config "$WRANGLER_CONFIG" versions upload \ + node "$WRANGLER" --config "$WRANGLER_CONFIG" versions upload \ --preview-alias "b-${SAFE_BRANCH}" \ --tag "$VERSION_TAG" \ --message "$VERSION_MESSAGE" - ' | tee "$wrangler_upload_stdout" + ' \ + > >(tee "$wrangler_upload_stdout") \ + 2> >(tee "$wrangler_upload_stderr" >&2) + wrangler_upload_rc=$? + set -e unset WRANGLER_OUTPUT_FILE_PATH @@ -245,19 +274,66 @@ case "$mode" in ) fi if [[ -z "$version_id" ]]; then + # Relax errexit for the entire diagnostic dump block. grep/sed/cat/head + # failures here (missing stdout match, empty NDJSON, nonexistent log + # file) must not abort before every dump section fires — the script's + # fail contract is satisfied by the explicit `exit 1` at the end of + # this block, not by intermediate pipeline exit codes. + set +e echo "" >&2 echo "error: wrangler exited 0 but produced no Worker Version ID" >&2 echo " post-condition (a) failed: neither WRANGLER_OUTPUT_FILE_PATH NDJSON" >&2 echo " event log nor wrangler stdout contained" >&2 echo " a recognizable Worker Version ID" >&2 + echo " wrangler exit code: $wrangler_upload_rc" >&2 echo " raw wrangler event log: $wrangler_upload_ndjson" >&2 echo " raw wrangler stdout: $wrangler_upload_stdout" >&2 + echo " raw wrangler stderr: $wrangler_upload_stderr" >&2 echo " hints:" >&2 - echo " - confirm CF_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 + echo " - confirm CLOUDFLARE_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 echo " $SOPS_SECRETS_FILE" >&2 - echo " - if running in GHA, wrangler's CI-detection branch may have" >&2 - echo " silently exited; rerun with WRANGLER_LOG=debug for stderr trace" >&2 - echo " - inspect the raw NDJSON and stdout paths above for any output" >&2 + echo " - if linux-x64 regression, confirm wrangler invoked under real node and" >&2 + echo " not bun-fake-node (see top-of-file rationale at lines ~101-110)" >&2 + echo " - inspect the wrangler internal log dumped below / raw NDJSON and stdout paths above for any output" >&2 + echo "" >&2 + # Locate wrangler's internal log file by glob + newest mtime across + # platform-specific candidate locations. Avoids depending on wrangler's + # stdout `Writing logs to "..."` announcement (only printed under + # WRANGLER_LOG=debug, which we no longer set). The log file contains + # full HTTP request/response bodies and any internal stack traces that + # are otherwise destroyed with the GHA runner — dump it first as the + # most informative diagnostic source when NDJSON/stdout/stderr are + # empty or truncated. + wrangler_log_path="" + for candidate_dir in "$HOME/.wrangler/logs" "$HOME/.config/.wrangler/logs"; do + if [[ -d "$candidate_dir" ]]; then + # Filename format is `wrangler-YYYY-MM-DD_HH-MM-SS_mmm.log` — the + # embedded timestamp is zero-padded and lexicographically sortable, + # so `sort | tail -1` selects the newest without needing ls -t. + newest=$(find "$candidate_dir" -maxdepth 1 -type f -name 'wrangler-*.log' 2>/dev/null | sort | tail -1 || true) + if [[ -n "$newest" ]]; then + wrangler_log_path="$newest" + break + fi + fi + done + if [[ -n "$wrangler_log_path" && -f "$wrangler_log_path" ]]; then + echo "--- begin wrangler internal log ($wrangler_log_path) ---" >&2 + cat "$wrangler_log_path" >&2 || true + echo "--- end wrangler internal log ---" >&2 + else + echo "wrangler internal log: no file found under \$HOME/.wrangler/logs or \$HOME/.config/.wrangler/logs" >&2 + fi + echo "--- begin raw wrangler NDJSON ($wrangler_upload_ndjson) ---" >&2 + cat "$wrangler_upload_ndjson" >&2 || true + echo "--- end raw wrangler NDJSON ---" >&2 + echo "--- begin raw wrangler stdout ($wrangler_upload_stdout) ---" >&2 + cat "$wrangler_upload_stdout" >&2 || true + echo "--- end raw wrangler stdout ---" >&2 + echo "--- begin raw wrangler stderr ($wrangler_upload_stderr) ---" >&2 + cat "$wrangler_upload_stderr" >&2 || true + echo "--- end raw wrangler stderr ---" >&2 + set -e exit 1 fi @@ -273,7 +349,7 @@ case "$mode" in # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell sops exec-env "$SOPS_SECRETS_FILE" ' - "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json + node "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json ' | cat > "$wrangler_list_json" matched_count=$(jq --arg tag "$commit_tag" \ @@ -286,8 +362,7 @@ case "$mode" in echo " with annotations[\"workers/tag\"] == ${commit_tag}" >&2 echo " raw versions list output: $wrangler_list_json" >&2 echo " hint: wrangler reported a version_id locally but the Cloudflare API did" >&2 - echo " not persist it; retry with WRANGLER_LOG=debug or inspect the raw" >&2 - echo " versions list for surrounding entries" >&2 + echo " not persist it; inspect the raw versions list for surrounding entries" >&2 exit 1 fi @@ -322,7 +397,7 @@ case "$mode" in # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell sops exec-env "$SOPS_SECRETS_FILE" ' - "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json + node "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json ' | cat > "$wrangler_list_json" existing_version=$(jq -r --arg tag "$commit_tag" \ @@ -355,7 +430,7 @@ case "$mode" in # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell sops exec-env "$SOPS_SECRETS_FILE" ' - "$WRANGLER" --config "$WRANGLER_CONFIG" versions deploy \ + node "$WRANGLER" --config "$WRANGLER_CONFIG" versions deploy \ "${EXISTING_VERSION}@100%" \ --yes \ --message "$DEPLOYMENT_MESSAGE" @@ -394,10 +469,11 @@ case "$mode" in echo " raw wrangler event log: $deploy_ndjson" >&2 echo " raw wrangler stdout: $deploy_stdout" >&2 echo " hints:" >&2 - echo " - confirm CF_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 + echo " - confirm CLOUDFLARE_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 echo " $SOPS_SECRETS_FILE" >&2 - echo " - if running in GHA, wrangler's CI-detection branch may have" >&2 - echo " silently exited; rerun with WRANGLER_LOG=debug for stderr trace" >&2 + echo " - if linux-x64 regression, confirm wrangler invoked under real node and" >&2 + echo " not bun-fake-node (see top-of-file rationale at lines ~101-110)" >&2 + echo " - inspect the wrangler internal log dumped below / raw capture paths above for any output" >&2 exit 1 fi @@ -408,7 +484,7 @@ case "$mode" in # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell sops exec-env "$SOPS_SECRETS_FILE" ' - "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json + node "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json ' | cat > "$deployments_list_json" found_count=$(jq --arg did "$deployment_id" --arg vid "$existing_version" \ @@ -425,7 +501,7 @@ case "$mode" in echo " entries matching the just-deployed id/version" >&2 echo " raw deployments list output: $deployments_list_json" >&2 echo " hint: wrangler reported a deployment locally but the Cloudflare API did" >&2 - echo " not persist it; retry with WRANGLER_LOG=debug" >&2 + echo " not persist it; inspect the raw deployments list for surrounding entries" >&2 exit 1 fi @@ -463,7 +539,7 @@ case "$mode" in # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell sops exec-env "$SOPS_SECRETS_FILE" ' - "$WRANGLER" --config "$WRANGLER_CONFIG" deploy \ + node "$WRANGLER" --config "$WRANGLER_CONFIG" deploy \ --message "$DEPLOYMENT_MESSAGE" ' | tee "$deploy_stdout" @@ -499,10 +575,11 @@ case "$mode" in echo " raw wrangler event log: $deploy_ndjson" >&2 echo " raw wrangler stdout: $deploy_stdout" >&2 echo " hints:" >&2 - echo " - confirm CF_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 + echo " - confirm CLOUDFLARE_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 echo " $SOPS_SECRETS_FILE" >&2 - echo " - if running in GHA, wrangler's CI-detection branch may have" >&2 - echo " silently exited; rerun with WRANGLER_LOG=debug for stderr trace" >&2 + echo " - if linux-x64 regression, confirm wrangler invoked under real node and" >&2 + echo " not bun-fake-node (see top-of-file rationale at lines ~101-110)" >&2 + echo " - inspect the wrangler internal log dumped below / raw capture paths above for any output" >&2 exit 1 fi # Reuse deployment_id slot below (it now holds the just-deployed version_id @@ -513,7 +590,7 @@ case "$mode" in # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell sops exec-env "$SOPS_SECRETS_FILE" ' - "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json + node "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json ' | cat > "$deployments_list_json" found_count=$(jq --arg vid "$deploy_version_id" \ @@ -530,7 +607,7 @@ case "$mode" in echo " entries matching the just-deployed version_id" >&2 echo " raw deployments list output: $deployments_list_json" >&2 echo " hint: wrangler reported a deployment locally but the Cloudflare API did" >&2 - echo " not persist it; retry with WRANGLER_LOG=debug" >&2 + echo " not persist it; inspect the raw deployments list for surrounding entries" >&2 exit 1 fi From 30bcedca7c961018110f85d0e4f2956b5c4fb96b Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 15:39:53 -0400 Subject: [PATCH 13/77] fix(cd): enable magic-nix-cache on CD paths to reduce bun-drv rebuilds Every GHA run of cd.yaml and deploy-docs.yaml rebuilds hundreds of bun-pkg-*, wrangler-*.tgz, vanixiets-docs-deps, and deploy-docs drvs from scratch because all CD-path callers of .github/actions/setup-nix use installer: quick, which skips both magic-nix-cache-action and cachix-action. With no writable cache configured, every run pays full build cost. Commit b816a1355 (2026-04-16) removed cachix with the rationale that nix checks are delegated to buildbot-nix on magnetite, but the cd.yaml leg was never migrated to consume buildbot-nix cache pushes. Flipping installer: quick -> installer: full enables magic-nix-cache (GHA-cache-backed, token-less) without re-introducing cachix coupling; the first run after the change still builds but populates the cache, and subsequent runs should hit it. Covers three call sites: cd.yaml set-variables job (line 145), cd.yaml preview-release-version job (line 210), deploy-docs.yaml deploy-docs job (line 95). Orthogonal to the docs-deploy diagnostic hardening on the same chain. --- .github/workflows/cd.yaml | 4 ++-- .github/workflows/deploy-docs.yaml | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/cd.yaml b/.github/workflows/cd.yaml index 8fba35052..6c45b46bf 100644 --- a/.github/workflows/cd.yaml +++ b/.github/workflows/cd.yaml @@ -142,7 +142,7 @@ jobs: - name: Setup Nix uses: ./.github/actions/setup-nix with: - installer: quick + installer: full system: x86_64-linux # alternative: extractions/setup-just@v3 (blocked on node20->node24 upgrade via extractions/setup-crate#9) @@ -207,7 +207,7 @@ jobs: if: steps.cache.outputs.should-run == 'true' uses: ./.github/actions/setup-nix with: - installer: quick + installer: full system: x86_64-linux - name: Setup tmate debug session diff --git a/.github/workflows/deploy-docs.yaml b/.github/workflows/deploy-docs.yaml index 6293bfd88..8a06a0c64 100644 --- a/.github/workflows/deploy-docs.yaml +++ b/.github/workflows/deploy-docs.yaml @@ -92,7 +92,7 @@ jobs: env: SOPS_AGE_KEY: ${{ secrets.SOPS_AGE_KEY }} with: - installer: quick + installer: full system: x86_64-linux - name: Setup tmate debug session From 556ce42d5bd79b779d767cc5b057f600b4a78373 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 19:21:11 -0400 Subject: [PATCH 14/77] feat(flake): scaffold hercules-ci-effects with empty herculesCI attribute MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds hercules-ci-effects as a top-level flake input with follows for flake-parts and nixpkgs (no duplicate lock nodes). Introduces modules/hercules-ci.nix importing the flakeModule and declaring an empty herculesCI function that materializes the fixed buildbot-nix attribute path herculesCI.onPush.default.outputs.effects = {} as required by M2 of the cd.yaml → buildbot-nix migration (ADR-001). No effects are defined yet; per-job effects land in M4. Verified locally with: nix eval --apply 'f: builtins.attrNames (f {...}).onPush.default.outputs.effects' .#herculesCI -> [ ] nix flake show -> herculesCI: unknown nix flake check --system x86_64-darwin -> all checks passed fix(flake): apply treefmt formatting for herculesCI module --- flake.lock | 24 ++++++++++++++++++++++++ flake.nix | 4 ++++ modules/hercules-ci.nix | 29 +++++++++++++++++++++++++++++ 3 files changed, 57 insertions(+) create mode 100644 modules/hercules-ci.nix diff --git a/flake.lock b/flake.lock index 82fecd6f4..756c563c8 100644 --- a/flake.lock +++ b/flake.lock @@ -677,6 +677,29 @@ "type": "github" } }, + "hercules-ci-effects_2": { + "inputs": { + "flake-parts": [ + "flake-parts" + ], + "nixpkgs": [ + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1776603440, + "narHash": "sha256-wA+ONiwbvQIy7ERJx/ruhV7y5xku6XKstXCII5bIbdI=", + "owner": "hercules-ci", + "repo": "hercules-ci-effects", + "rev": "e2456ee419f9d75f8382e3d6c5af4690b316a5a8", + "type": "github" + }, + "original": { + "owner": "hercules-ci", + "repo": "hercules-ci-effects", + "type": "github" + } + }, "home-manager": { "inputs": { "nixpkgs": [ @@ -1365,6 +1388,7 @@ "flake-parts": "flake-parts", "gateway-api-src": "gateway-api-src", "git-hooks": "git-hooks", + "hercules-ci-effects": "hercules-ci-effects_2", "home-manager": "home-manager", "import-tree": "import-tree", "lazyvim-nix": "lazyvim-nix", diff --git a/flake.nix b/flake.nix index 0457c36ca..077b0a639 100644 --- a/flake.nix +++ b/flake.nix @@ -148,6 +148,10 @@ buildbot-nix.inputs.nixpkgs.follows = "nixpkgs"; buildbot-nix.inputs.flake-parts.follows = "flake-parts"; buildbot-nix.inputs.treefmt-nix.follows = "treefmt-nix"; + + hercules-ci-effects.url = "github:hercules-ci/hercules-ci-effects"; + hercules-ci-effects.inputs.flake-parts.follows = "flake-parts"; + hercules-ci-effects.inputs.nixpkgs.follows = "nixpkgs"; }; # sync with lib/caches.nix for machine modules diff --git a/modules/hercules-ci.nix b/modules/hercules-ci.nix new file mode 100644 index 000000000..6cd803ec2 --- /dev/null +++ b/modules/hercules-ci.nix @@ -0,0 +1,29 @@ +# Flake-level effects framework scaffolding (M2 — mission ADR-001). +# +# Imports the `hercules-ci-effects` flake-parts module so that the +# top-level flake output `herculesCI` is wired to the schema consumed +# by buildbot-nix (`flake.outputs.herculesCI(args).onPush.default.outputs.effects`, +# per `buildbot-nix/buildbot_effects/buildbot_effects/__init__.py:142-159`). +# +# At this milestone no effects are declared yet — the attribute is +# explicitly set to an empty attrset so the eval surface is well-formed +# for downstream buildbot-nix consumption. Per-job effects land in M4 +# under this same `onPush.default.outputs.effects.` path. +# +# See `docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md` +# for the full rationale and the fixed-attribute-path contract. +{ inputs, ... }: +{ + imports = [ + inputs.hercules-ci-effects.flakeModule + ]; + + # Empty but well-formed `herculesCI` function. The `onPush.default.outputs.effects` + # key is materialized as an empty attrset so `buildbot-effects list` returns `[]` + # rather than an error when the master evaluates this flake. + herculesCI = + { ... }: + { + onPush.default.outputs.effects = { }; + }; +} From a58bc6bc4e2ef138c318a0409c18b1c0a2f2f3bd Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 21:45:17 -0400 Subject: [PATCH 15/77] refactor(buildbot): update CX53 eval sizing comment with live-verified 16 vCPU, 32 GB RAM capacity --- modules/nixos/buildbot.nix | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/nixos/buildbot.nix b/modules/nixos/buildbot.nix index d7edc9e70..143f48d1a 100644 --- a/modules/nixos/buildbot.nix +++ b/modules/nixos/buildbot.nix @@ -150,7 +150,7 @@ topic = "build-with-buildbot"; }; - # Conservative eval sizing for CX53 (8 vCPU, 16 GB RAM) + # Conservative eval sizing for CX53 (16 vCPU, 32 GB RAM) — current evalWorkerCount=4 × evalMaxMemorySize=2048MB = 8 GB peak, leaving ample headroom on 32 GB host; sizing not increased at this time # 4 workers * 2048 MB = 8 GB max, leaving headroom for niks3 + PostgreSQL + nginx evalWorkerCount = 4; evalMaxMemorySize = 2048; From 46cdb00c9219aa3be911e02f21ce804a0e335d45 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 21:57:19 -0400 Subject: [PATCH 16/77] feat(magnetite): provision docker runtime on dedicated zroot/root/docker dataset Adds a second container runtime alongside the existing podman stack used by gitea-actions-runner. Required because the M4 test-cluster effect drives k3d via ctlptl which invokes the docker binary directly. Uses docker's native ZFS storage driver (overlay2 does not layer cleanly on ZFS) and dedicates zroot/root/docker mounted at /var/lib/docker so the graphroot is on its own dataset. Adds buildbot-worker to the docker group so effects running as that user can drive the daemon. Podman stack and zroot/root/podman dataset are unchanged (additive only). Fulfills m3-docker-runtime Phase 0/1/2 (docker viability GO verdict: kernel modules already exercised by podman, cgroup v2 supported since docker 20.10, /dev/kvm not required by k3d-in-container, kernel 26.05 well above docker minimum; ZFS driver is the canonical mitigation for the overlay2-on-ZFS incompatibility). Awaiting user `clan machines update magnetite` to deploy. --- modules/machines/nixos/magnetite/default.nix | 1 + modules/machines/nixos/magnetite/disko.nix | 9 +++++ modules/nixos/docker.nix | 37 ++++++++++++++++++++ 3 files changed, 47 insertions(+) create mode 100644 modules/nixos/docker.nix diff --git a/modules/machines/nixos/magnetite/default.nix b/modules/machines/nixos/magnetite/default.nix index 05d3d52ed..719771ab2 100644 --- a/modules/machines/nixos/magnetite/default.nix +++ b/modules/machines/nixos/magnetite/default.nix @@ -35,6 +35,7 @@ in buildbot gitea gitea-actions-runner + docker ]); # Make flake available to all modules (required by ssh-known-hosts) diff --git a/modules/machines/nixos/magnetite/disko.nix b/modules/machines/nixos/magnetite/disko.nix index 04229fdbf..e93af59f4 100644 --- a/modules/machines/nixos/magnetite/disko.nix +++ b/modules/machines/nixos/magnetite/disko.nix @@ -66,6 +66,15 @@ options.mountpoint = "/var/lib/containers"; mountpoint = "/var/lib/containers"; }; + # Dedicated dataset for docker graphroot to use the native ZFS + # storage driver (docker's overlay2 does not layer cleanly on ZFS, + # and /var/lib/docker must be its own dataset for the zfs driver). + # Coexists with zroot/root/podman; disjoint mountpoints. + "root/docker" = { + type = "zfs_fs"; + options.mountpoint = "/var/lib/docker"; + mountpoint = "/var/lib/docker"; + }; }; }; }; diff --git a/modules/nixos/docker.nix b/modules/nixos/docker.nix new file mode 100644 index 000000000..e6e5e8699 --- /dev/null +++ b/modules/nixos/docker.nix @@ -0,0 +1,37 @@ +# Docker runtime for magnetite +# +# Provisions real docker as a second container runtime alongside the existing +# podman stack used by gitea-actions-runner. Required because the +# test-cluster effect drives k3d via ctlptl (~/projects/sciops-workspace/ctlptl) +# which invokes the `docker` binary directly and has no production-quality +# podman support. +# +# Storage: docker's native ZFS storage driver is used (overlay2 does not +# layer cleanly on ZFS). This requires /var/lib/docker to be its own ZFS +# dataset; see modules/machines/nixos/magnetite/disko.nix for the +# zroot/root/docker dataset declaration. +# +# The buildbot-worker user is added to the docker group so effects running +# as that user can talk to /var/run/docker.sock (e.g. the forthcoming +# test-cluster effect invoking k3d via ctlptl). +{ + ... +}: +{ + flake.modules.nixos.docker = + { ... }: + { + # Real docker daemon, additive to the existing podman stack. + virtualisation.docker = { + enable = true; + # Native ZFS storage driver; requires /var/lib/docker to be its own + # ZFS dataset (declared in disko.nix as zroot/root/docker). + storageDriver = "zfs"; + }; + + # Grant the buildbot-worker runtime user access to the docker socket + # so effects executed by the worker (e.g. test-cluster via k3d+ctlptl) + # can drive the docker daemon without sudo. + users.users.buildbot-worker.extraGroups = [ "docker" ]; + }; +} From db78183e5dc5bf0e2038889ed62e2ba55b65131d Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 22:09:59 -0400 Subject: [PATCH 17/77] feat(buildbot): wire perRepoSecretFiles for vanixiets effects secrets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a clan-vars generator `buildbot-effects-vanixiets` emitting a flat-dict `secrets.json` (CLOUDFLARE_API_TOKEN, SOPS_AGE_KEY, GITHUB_TOKEN) with placeholder values so the `perRepoSecretFiles → LoadCredential → bwrap HERCULES_CI_SECRETS_JSON` pipeline can be smoke-tested before any real tokens exist. Wires `services.buildbot-nix.master.effects.perRepoSecretFiles` with the single key `github:cameronraysmith/vanixiets` pointing at the generated file path. File inherits clan-core defaults (mode 0400, secret=true) with owner=buildbot so systemd `LoadCredential` on buildbot-master can read it. Posture A (`effects_on_pull_requests` absent → default false) is preserved. Rotate placeholders to real tokens in place via: clan vars set magnetite buildbot-effects-vanixiets/secrets.json < real-secrets.json No deploy performed; awaiting user `clan machines update magnetite`. --- modules/nixos/buildbot.nix | 42 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/modules/nixos/buildbot.nix b/modules/nixos/buildbot.nix index 143f48d1a..f2b82284e 100644 --- a/modules/nixos/buildbot.nix +++ b/modules/nixos/buildbot.nix @@ -10,6 +10,9 @@ # - buildbot-worker: auto-generated (worker password + workers.json) # - buildbot-oauth2-cookie-secret: auto-generated (oauth2-proxy cookie encryption) # - buildbot-http-basic-auth-password: auto-generated (oauth2-proxy to buildbot internal auth) +# - buildbot-effects-vanixiets: prompts-populated `secrets.json` flat JSON dict for +# HERCULES_CI_SECRETS_JSON consumption by hercules-ci-effects inside the master's +# bwrap sandbox, wired via `services.buildbot-nix.master.effects.perRepoSecretFiles`. # Gitea-specific credentials are declared in gitea.nix: # - buildbot-gitea-token: manual `clan vars set` (API token with write:repository, write:user) # - buildbot-gitea-webhook-secret: auto-generated @@ -82,6 +85,36 @@ ''; }; + # Effects secrets for github:cameronraysmith/vanixiets + # (flat-dict JSON consumed as HERCULES_CI_SECRETS_JSON inside bwrap). + # The auto-generated body seeds placeholder values so the full + # `perRepoSecretFiles → LoadCredential → bwrap` pipeline can be + # smoke-tested end-to-end before any real tokens exist. Rotate to + # real tokens in place via: + # clan vars set magnetite buildbot-effects-vanixiets/secrets.json < real-secrets.json + # Keys MUST match the names effect scripts `jq -r` out of + # $HERCULES_CI_SECRETS_JSON: + # - CLOUDFLARE_API_TOKEN: wrangler-driven docs-deploy effects + # - SOPS_AGE_KEY: sops-secrets-operator bootstrap inside test-cluster + # - GITHUB_TOKEN: semantic-release + gh api consumers + # Consumed below by + # `services.buildbot-nix.master.effects.perRepoSecretFiles`. + # Default file mode is 0400 (clan-core invariant) and owner is the + # buildbot master user so systemd `LoadCredential` can read it. + clan.core.vars.generators.buildbot-effects-vanixiets = { + files."secrets.json" = { + owner = "buildbot"; + }; + runtimeInputs = [ pkgs.jq ]; + script = '' + jq -n '{ + CLOUDFLARE_API_TOKEN: "placeholder-cloudflare-api-token-rotate-via-clan-vars-set", + SOPS_AGE_KEY: "placeholder-sops-age-key-rotate-via-clan-vars-set", + GITHUB_TOKEN: "placeholder-github-token-rotate-via-clan-vars-set" + }' > "$out/secrets.json" + ''; + }; + # Worker credentials (auto-generated password + workers.json) # CX53: 16 logical CPUs (nproc) — cores must match for correct worker count clan.core.vars.generators.buildbot-worker = { @@ -155,6 +188,15 @@ evalWorkerCount = 4; evalMaxMemorySize = 2048; + # Per-repo effects secrets (flat-dict JSON → HERCULES_CI_SECRETS_JSON inside bwrap). + # Fork-PR posture A (`effects_on_pull_requests = false` in buildbot-nix.toml) + # keeps this file off fork-PR effect runs; in-repo pushes on matched + # `effects_branches` receive it. Key must be the forge-prefixed + # `:/` literal that buildbot-effects uses to look up + # the matching secrets file at dispatch time. + effects.perRepoSecretFiles."github:cameronraysmith/vanixiets" = + config.clan.core.vars.generators.buildbot-effects-vanixiets.files."secrets.json".path; + # niks3 binary cache integration (push built paths after successful builds) # Uses public URL to support future remote workers (e.g. cinnabar) niks3 = { From 6ed956a4bd0ab9933555ae47d9ecddd15cb2fab3 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 22:13:17 -0400 Subject: [PATCH 18/77] vars: update via generator buildbot-effects-vanixiets (machine: magnetite) --- .../secrets.json/machines/magnetite | 1 + .../secrets.json/secret | 18 ++++++++++++++++++ .../secrets.json/users/cameron | 1 + 3 files changed, 20 insertions(+) create mode 120000 vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/machines/magnetite create mode 100644 vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/secret create mode 120000 vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/users/cameron diff --git a/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/machines/magnetite b/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/machines/magnetite new file mode 120000 index 000000000..41bd9646c --- /dev/null +++ b/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/machines/magnetite @@ -0,0 +1 @@ +../../../../../../sops/machines/magnetite \ No newline at end of file diff --git a/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/secret b/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/secret new file mode 100644 index 000000000..e77f86a78 --- /dev/null +++ b/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/secret @@ -0,0 +1,18 @@ +{ + "data": "ENC[AES256_GCM,data:/ZNFhCcBJtRuDIqVGQAmJAKnM69Wg4mNMSrUoWn7D30Z5QaoFFpak2qUD10S7+TDiCVaHgnmmj4dlHB3pe4k6hFfqqBGO7b5EI+Us5VWiSqSO5nZjzK8v7/tW2M4jinqlQGvKqJ/Idw9Uaf2vsXtivvDxLTVDZEeP4R5lXXtlf4LgYURbGfz/VJPLk7TMIqJ72uqrvQU/RYXftPndDDQeaeWFN3rY5VSN2mKSnxtwgejiCqyXP8ak3HOKdWuMzwt32JMrqVia5A3EgJLrnZKLnqNmRnUdob8pbCAzo0i0qsYHQjJ05D/kg==,iv:sd3gvsUEaJ6C57LE4lTnDGOemHVxrQDLen81pj3YEKg=,tag:0IkJkx0rCnl2osvhITN6Cw==,type:str]", + "sops": { + "age": [ + { + "recipient": "age1a7a70qcpjemlvk6q4uaf4k77p9eq7lj7wcal5jdj3xuetznyqdrs3mfnsf", + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxbWVMaWo1R0g5MVc4L1lO\nM2lvUi83VzZ4ZHZYVU05SlBZaUMyTnpybUZrClhsc0sybWpROWd1V011MjE2V09w\nLzY0bWZNMEl0cHo2bmx1VTRIZmZGL0EKLS0tIFhzR25JWDZ2ZkR3UXRDSHMyZDVH\na200RlZvUXBFdG1NbXFpTXgrNTJ5NWMKbaVkB1OVKiU7No+CdKZGNDzXURbhutVU\nlLApeDcsV9T3FH6pq9uHlXERNHt8KUXkMYsDIHXJue07ahfLTmQoyQ==\n-----END AGE ENCRYPTED FILE-----\n" + }, + { + "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuenZOVkR5OWgrVTV4a3F2\nQVR2OXc0dFQ5SndlcVBDaGFzZll5L1J1aTJZCjNLckM5MnpqSkxKc1lIWCt2d1VH\nL0RJQmlINFR5YmFkai9VSGJCTSs3YWsKLS0tIG5yM3R1THJXMWk1UXk3em5ESmwx\nNDJtTjJaVXk1R2F1YWozMWppOXNxcmsKatUi/oGyzlfdemgwetd6bfhzVt+d4wUM\nofx+Rsy/RUP39AxHiv95lIGNgjdTSApeq+KoUZqjkmDp5Qqr8+/D3Q==\n-----END AGE ENCRYPTED FILE-----\n" + } + ], + "lastmodified": "2026-04-23T02:13:17Z", + "mac": "ENC[AES256_GCM,data:XbVJVvsIaZSM/3O4UehddlPFDTvH5akoINA55R4l3JOP+YUL+7Fy0igkOxThOZFqoQqFBkgA7QUXfXd9LOtzI7UcYodnfak9UkkhU/A/US9UVUrBUoHduzMA5cAIDDuLOHTP3K3xnrsFqNxOOvse9L3x+gYnYES8sFDI+UkSyd0=,iv:vaBAyNalKFXXHWIkrjzv3E6NH6EvIM8AcGGW5zlE8Js=,tag:l5x+9vLy0I5PxzBneSNNrQ==,type:str]", + "version": "3.12.2" + } +} diff --git a/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/users/cameron b/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/users/cameron new file mode 120000 index 000000000..015130152 --- /dev/null +++ b/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/users/cameron @@ -0,0 +1 @@ +../../../../../../sops/users/cameron \ No newline at end of file From ee5ad79a2d4b29961600dc720f746188e4702931 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 22:47:34 -0400 Subject: [PATCH 19/77] feat(hercules-ci): add effects.smoke for M3 end-to-end pipeline diagnostic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Declares herculesCI.onPush.default.outputs.effects.smoke using hercules-ci-effects.lib.withPkgs + mkEffect. The effect prints the buildbot-effects-passed repo args (branch/ref/rev/shortRev/tag) and enumerates HERCULES_CI_SECRETS_JSON keys only (values intentionally masked via 'jq -r to_entries | map(.key) | @csv'), then exits 0. Establishes a reusable diagnostic baseline for M4 per-job effects and fulfills VAL-PROVISIONING-SMOKE-001 / -005 / -006 / -007 for feature m3-deploy-smoke. Branch gating lives in buildbot-nix.toml (Posture A) rather than in a Nix-level runIf — see AGENTS.md mission-wide gating decisions. --- modules/hercules-ci.nix | 94 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 84 insertions(+), 10 deletions(-) diff --git a/modules/hercules-ci.nix b/modules/hercules-ci.nix index 6cd803ec2..ff63d807f 100644 --- a/modules/hercules-ci.nix +++ b/modules/hercules-ci.nix @@ -5,25 +5,99 @@ # by buildbot-nix (`flake.outputs.herculesCI(args).onPush.default.outputs.effects`, # per `buildbot-nix/buildbot_effects/buildbot_effects/__init__.py:142-159`). # -# At this milestone no effects are declared yet — the attribute is -# explicitly set to an empty attrset so the eval surface is well-formed -# for downstream buildbot-nix consumption. Per-job effects land in M4 -# under this same `onPush.default.outputs.effects.` path. +# Per-job effects land under this same `onPush.default.outputs.effects.` +# path (M3 smoke, M4 per-job cutover). Branch gating is expressed in +# `buildbot-nix.toml` (`effects_branches`, `effects_on_pull_requests`), +# not in the Nix attribute path. # # See `docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md` # for the full rationale and the fixed-attribute-path contract. -{ inputs, ... }: +{ inputs, lib, ... }: +let + # Effects execute on x86_64-linux (magnetite's buildbot-worker arch). + pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux; + + # `lib.withPkgs` returns the hercules-ci-effects helper set + # (mkEffect, runIf, modularEffect, ...). See + # hercules-ci-effects/flake-public-outputs.nix `lib.withPkgs`. + hci-effects = inputs.hercules-ci-effects.lib.withPkgs pkgs; +in { imports = [ inputs.hercules-ci-effects.flakeModule ]; - # Empty but well-formed `herculesCI` function. The `onPush.default.outputs.effects` - # key is materialized as an empty attrset so `buildbot-effects list` returns `[]` - # rather than an error when the master evaluates this flake. herculesCI = - { ... }: + { config, ... }: { - onPush.default.outputs.effects = { }; + onPush.default.outputs.effects = { + # effects.smoke — minimal diagnostic effect (M3 feature `m3-deploy-smoke`). + # + # Purpose: exercise the full buildbot-nix + hercules-ci-effects + # pipeline end-to-end (flake eval → nix-eval builder discovery → + # `run-effect` builder scheduling → bwrap execution → + # `HERCULES_CI_SECRETS_JSON` read → masked key enumeration → exit 0). + # Establishes a reusable diagnostic baseline for M4 per-job effects. + # + # Security invariants: + # - Prints only secret KEY NAMES, never VALUES (uses + # `jq -r 'to_entries | map(.key) | @csv'`). + # - Default-branch-only via `effects_branches = ["main"]` + + # `effects_on_pull_requests = false` (Posture A) in + # `buildbot-nix.toml`. No fork-PR or feature-branch exposure. + # + # Verification: see VAL-PROVISIONING-SMOKE-00{1..9} in + # `.factory/mission/validation-contract.md`. + smoke = hci-effects.mkEffect { + name = "smoke"; + + # buildbot-effects populates these from the push metadata. + # `toString` coerces `null` (unset tag/branch) to the empty + # string so Nix string interpolation does not throw during + # eval of the effect derivation. + effectScript = + let + branch = toString (config.repo.branch or ""); + ref = toString (config.repo.ref or ""); + rev = toString (config.repo.rev or ""); + shortRev = toString (config.repo.shortRev or ""); + tag = toString (config.repo.tag or ""); + in + '' + set -euo pipefail + + echo "=== effects.smoke: buildbot-nix + hercules-ci-effects pipeline smoke test ===" + + # buildbot-effects-passed args (captured at Nix eval time via config.repo). + echo "branch: ${lib.escapeShellArg branch}" + echo "ref: ${lib.escapeShellArg ref}" + echo "rev: ${lib.escapeShellArg rev}" + echo "shortRev: ${lib.escapeShellArg shortRev}" + echo "tag: ${lib.escapeShellArg tag}" + + # HERCULES_CI_SECRETS_JSON is set by buildbot-nix inside the + # bwrap sandbox to the path of the JSON secrets blob produced + # by the `perRepoSecretFiles` pipeline. See + # buildbot-nix/buildbot_effects/buildbot_effects/__init__.py:250-290. + echo "HERCULES_CI_SECRETS_JSON=''${HERCULES_CI_SECRETS_JSON:-}" + + if [ -n "''${HERCULES_CI_SECRETS_JSON:-}" ] \ + && [ -f "''${HERCULES_CI_SECRETS_JSON}" ]; then + echo "secrets file exists: true" + # Key-only enumeration. VALUES ARE INTENTIONALLY OMITTED. + # Do not change this to `jq -r 'to_entries[] | .value'` + # or equivalent — that would leak secret payloads to the + # buildbot log (VAL-PROVISIONING-SMOKE-006). + echo -n "secret keys: " + jq -r 'to_entries | map(.key) | @csv' \ + "''${HERCULES_CI_SECRETS_JSON}" + else + echo "secrets file exists: false" + fi + + echo "=== smoke effect complete (exit 0) ===" + ''; + }; + }; }; } From b44adcd08f653bd771b062c92feb914b4e70c749 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Wed, 22 Apr 2026 23:38:58 -0400 Subject: [PATCH 20/77] fix(checks): declare hercules-ci-effects nix-unit.inputs --- modules/checks/nix-unit.nix | 1 + 1 file changed, 1 insertion(+) diff --git a/modules/checks/nix-unit.nix b/modules/checks/nix-unit.nix index da93abf7c..afe696ce0 100644 --- a/modules/checks/nix-unit.nix +++ b/modules/checks/nix-unit.nix @@ -28,6 +28,7 @@ nuenv llm-agents catppuccin + hercules-ci-effects ; inherit self; }; From c61fb2030445853e927b07ddff83bc39534dbf5a Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Thu, 23 Apr 2026 00:34:37 -0400 Subject: [PATCH 21/77] fix(hercules-ci): drop config.repo.ref reference to unblock buildbot-effects eval buildbot-effects hard-codes `"ref": None` in its primaryRepo JSON payload (buildbot_effects/__init__.py:108, `# TODO: support ref`) while hercules-ci-effects declares `repo.ref` as non-nullable `types.str` (herculesCI-attribute.nix:18), causing module type-checking to reject the null at eval time. Upstream's own mkEffect example reads only branch/tag/rev for the same reason. --- modules/hercules-ci.nix | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/modules/hercules-ci.nix b/modules/hercules-ci.nix index ff63d807f..32628da9f 100644 --- a/modules/hercules-ci.nix +++ b/modules/hercules-ci.nix @@ -52,13 +52,20 @@ in name = "smoke"; # buildbot-effects populates these from the push metadata. - # `toString` coerces `null` (unset tag/branch) to the empty - # string so Nix string interpolation does not throw during - # eval of the effect derivation. + # `toString` coerces a null tag to the empty string so Nix string + # interpolation does not throw during eval of the effect derivation. + # + # `config.repo.ref` is intentionally not referenced: buildbot-effects + # hard-codes `"ref": None` in its JSON payload (see + # buildbot_effects/__init__.py:108, `# TODO: support ref`), and + # hercules-ci-effects declares `repo.ref` as non-nullable + # `types.str` (herculesCI-attribute.nix:18), so reading it would + # fail module type-checking at eval time. Upstream's own mkEffect + # example in buildbot-nix/nix/herculesCI/flake-module.nix follows + # the same pattern (branch/tag/rev only). effectScript = let branch = toString (config.repo.branch or ""); - ref = toString (config.repo.ref or ""); rev = toString (config.repo.rev or ""); shortRev = toString (config.repo.shortRev or ""); tag = toString (config.repo.tag or ""); @@ -70,7 +77,6 @@ in # buildbot-effects-passed args (captured at Nix eval time via config.repo). echo "branch: ${lib.escapeShellArg branch}" - echo "ref: ${lib.escapeShellArg ref}" echo "rev: ${lib.escapeShellArg rev}" echo "shortRev: ${lib.escapeShellArg shortRev}" echo "tag: ${lib.escapeShellArg tag}" From d0033a3985f1f8b512ffdea8539bd5682c40cb9f Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Thu, 23 Apr 2026 00:45:45 -0400 Subject: [PATCH 22/77] feat(devshell): add buildbot-effects CLI for local effect dispatch Exposes the buildbot-effects CLI (see buildbot-nix/docs/EFFECTS.md) in the default devshell so effects can be dispatched and iterated locally on a Linux host before pushing to buildbot. Gated on pkgs.stdenv.isLinux because the upstream package depends on bwrap (see buildbot-nix/packages/flake-module.nix:29) and has no darwin build. --- modules/devshells/default.nix | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/modules/devshells/default.nix b/modules/devshells/default.nix index e96b48811..d4c516730 100644 --- a/modules/devshells/default.nix +++ b/modules/devshells/default.nix @@ -1,6 +1,7 @@ { perSystem = { + lib, pkgs, inputs', config, @@ -66,6 +67,12 @@ # Document typesetting pkgs.typstWithPackages pkgs.svgo + ] + # buildbot-effects CLI for local dispatch of hercules-ci-effects + # (see buildbot-nix/docs/EFFECTS.md). Linux-only: the package + # depends on bwrap and is gated at buildbot-nix/packages/flake-module.nix:29. + ++ lib.optionals pkgs.stdenv.isLinux [ + inputs'.buildbot-nix.packages.buildbot-effects ]; passthru.meta.description = "Development environment with clan CLI and build tools"; From b29343ed802932340f280b56691bd01c02e7e690 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Thu, 23 Apr 2026 20:08:05 -0400 Subject: [PATCH 23/77] refactor(apps): declare explicit env-var contracts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refactor the five writeShellApplication scripts to declare explicit env-var contracts and replace implicit secret provisioning with caller-side injection, per docs/notes/operations/nix-7v7/m4/ env-var-contract-design.md §4. Changed scripts (headers + `:?` guards + no in-script `sops exec-env`): - modules/apps/docs/deploy.sh: - Added contract header documenting required secrets (CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID) and config (DOCS_PAYLOAD, DOCS_NODE_MODULES), plus optional overrides. - Added :? guards for the 4 required env vars. - Removed all 7 `sops exec-env \"\$SOPS_SECRETS_FILE\"` wrappers around wrangler invocations; wrangler now authenticates purely via inherited env vars. - Re-pointed error hints at the caller-mechanism menu. - modules/apps/release/release.sh: - Added contract header (GITHUB_TOKEN conditionally required when not --dry-run, SOPS_AGE_KEY passthrough, DOCS_NODE_MODULES required config). - Added DOCS_NODE_MODULES :? guard and conditional GITHUB_TOKEN :? guard gated on dry_run -ne 1, ordered before node_modules setup so missing-token fails fast. - modules/apps/cluster/k3d-bootstrap-secrets.sh: - Added narrow-exception header documenting the single permitted SOPS_AGE_KEY consumer (env-first, file-fallback dual-branch). - Reordered: env-or-file key check now precedes the kubectl invocation (fail-fast on missing key). - modules/apps/docs/preview-version.sh: - Added contract header (DOCS_NODE_MODULES required, no secret env). - Added DOCS_NODE_MODULES :? guard. - modules/apps/cluster/k3d-integration-ci.sh: - Added transitive-contract header (SOPS_AGE_KEY transitively required via k3d-bootstrap-secrets, ARGOCD_REPO_URL optional). - Added ARGOCD_REPO_URL default via := expansion. Sibling .nix cleanups: - modules/apps/docs/deploy.nix: dropped SOPS_SECRETS_FILE export, pkgs.sops / pkgs.age from runtimeInputs, and the now-unused `inputs` parameter. Updated header comment. - modules/apps/docs/preview-version.nix: removed stale SOPS_SECRETS_FILE reference from comment. - modules/apps/cluster/k3d-integration-ci.nix: removed stale SOPS_SECRETS_FILE reference; noted secret env vars are never injected via the nix preamble for this app. Caller-site updates: - justfile: wrapped docs-deploy-preview / docs-deploy-production recipes with `sops exec-env secrets/shared.yaml` on the caller side so CLOUDFLARE_API_TOKEN / CLOUDFLARE_ACCOUNT_ID reach the nested `nix run .#deploy-docs` invocation. - .github/workflows/deploy-docs.yaml: wrapped the `nix run .#deploy-docs` invocation with `nix develop -c sops exec-env secrets/shared.yaml \"\"`; the age key for the sops decrypt is supplied via the step env: block from the repo secrets. Verification: - nix flake check --system x86_64-darwin: all checks passed. - shellcheck on all 5 scripts: clean. - nix eval .#apps.x86_64-linux..program: resolved for deploy-docs, release, preview-version, k3d-bootstrap-secrets, k3d-integration-ci. - Guard smoke tests: - env -u CLOUDFLARE_API_TOKEN -u CLOUDFLARE_ACCOUNT_ID nix run .#deploy-docs -- production -> exit 1 with contract guard message referring to deploy.sh header. - env -u GITHUB_TOKEN nix run .#release -- packages/docs -> exit 1 with contract guard message (not --dry-run). - env -u SOPS_AGE_KEY HOME=/nonexistent-dir nix run .#k3d-bootstrap-secrets -> exit 1 with the \"set SOPS_AGE_KEY or create the file\" guard message. - just --dry-run docs-deploy-preview: emits the caller-side sops-wrapped command as expected. - Cross-cutting assertions (VAL-ENVCONTRACT-CROSS-01..10) all satisfied: no SOPS_SECRETS_FILE anywhere, no `sops exec-env \"\$SOPS_SECRETS_FILE\"` in modules/apps/, no secrets on argv, SOPS_AGE_KEY usage restricted to the two permitted k3d files, all 5 scripts carry \"Env-var contract\" headers and :? guards. --- .github/workflows/deploy-docs.yaml | 17 +- justfile | 19 ++- modules/apps/cluster/k3d-bootstrap-secrets.sh | 38 ++++- modules/apps/cluster/k3d-integration-ci.nix | 7 +- modules/apps/cluster/k3d-integration-ci.sh | 33 +++- modules/apps/docs/deploy.nix | 23 +-- modules/apps/docs/deploy.sh | 155 +++++++++--------- modules/apps/docs/preview-version.nix | 3 +- modules/apps/docs/preview-version.sh | 17 ++ modules/apps/release/release.sh | 56 +++++-- 10 files changed, 254 insertions(+), 114 deletions(-) diff --git a/.github/workflows/deploy-docs.yaml b/.github/workflows/deploy-docs.yaml index 8a06a0c64..9e24355e5 100644 --- a/.github/workflows/deploy-docs.yaml +++ b/.github/workflows/deploy-docs.yaml @@ -103,6 +103,9 @@ jobs: if: steps.cache.outputs.should-run == 'true' id: deployment env: + # SOPS_AGE_KEY decrypts secrets/shared.yaml at step time. + # Per ADR-002 env-var contract, deploy.sh no longer calls + # `sops exec-env` internally — the wrap moves to the caller. SOPS_AGE_KEY: ${{ secrets.SOPS_AGE_KEY }} GITHUB_ACTIONS: "true" GITHUB_ACTOR: ${{ github.actor }} @@ -110,10 +113,20 @@ jobs: DEPLOY_ENVIRONMENT: ${{ inputs.environment }} DEPLOY_BRANCH: ${{ inputs.branch }} run: | + # Use `nix develop -c` so sops is guaranteed on PATH + # (setup-nix composite action provides nix, but not sops on + # the bare runner); sops exec-env populates CLOUDFLARE_API_TOKEN + # and CLOUDFLARE_ACCOUNT_ID before the nix-run invocation. + # `sops exec-env` takes (file, single-shell-command-string) so + # the nested nix-run invocation is a single quoted string. if [ "$DEPLOY_ENVIRONMENT" = "preview" ]; then - nix run .#deploy-docs -- preview "$DEPLOY_BRANCH" + nix develop --accept-flake-config -c \ + sops exec-env secrets/shared.yaml \ + "nix run --accept-flake-config .#deploy-docs -- preview \"$DEPLOY_BRANCH\"" else - nix run .#deploy-docs -- production + nix develop --accept-flake-config -c \ + sops exec-env secrets/shared.yaml \ + "nix run --accept-flake-config .#deploy-docs -- production" fi - name: Create job result marker diff --git a/justfile b/justfile index 82b078770..1f43c9855 100644 --- a/justfile +++ b/justfile @@ -769,15 +769,26 @@ docs-test-e2e-report: docs-test-coverage: cd packages/docs && bun run test:coverage -# Deploy documentation to Cloudflare Workers (preview) +# Deploy documentation to Cloudflare Workers (preview). +# Wraps with `sops exec-env secrets/shared.yaml ''` so +# CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID are exported per the +# deploy-docs env-var contract (ADR-002 / env-var-contract-design.md +# §2.1.3 Call site A). Devs with a local `.env` already exporting the +# vars can skip the wrap; the sops prefix is idempotent and keeps fresh +# clones without `.env` working. `sops exec-env` requires exactly two +# positional args (file + single shell-command string), so the nix-run +# invocation is quoted as one arg. [group('docs')] docs-deploy-preview branch=`git branch --show-current`: - nix run --accept-flake-config .#deploy-docs -- preview "{{branch}}" + sops exec-env secrets/shared.yaml \ + 'nix run --accept-flake-config .#deploy-docs -- preview "{{branch}}"' -# Deploy documentation to Cloudflare Workers (production) +# Deploy documentation to Cloudflare Workers (production). +# See docs-deploy-preview header for the sops wrap rationale. [group('docs')] docs-deploy-production: - nix run --accept-flake-config .#deploy-docs -- production + sops exec-env secrets/shared.yaml \ + 'nix run --accept-flake-config .#deploy-docs -- production' # List recent Cloudflare deployments [group('docs')] diff --git a/modules/apps/cluster/k3d-bootstrap-secrets.sh b/modules/apps/cluster/k3d-bootstrap-secrets.sh index f080727ab..d8a9395ed 100644 --- a/modules/apps/cluster/k3d-bootstrap-secrets.sh +++ b/modules/apps/cluster/k3d-bootstrap-secrets.sh @@ -8,9 +8,29 @@ # Usage: # k3d-bootstrap-secrets [--help] # -# Key sources (first one found wins): -# SOPS_AGE_KEY env var - used directly (CI pathway) -# ~/.config/sops/age/keys.txt - file-based (local dev pathway) +# Env-var contract (per ADR-002 / env-var-contract-design.md §2.4): +# One of the following MUST be satisfied (narrow exception; env-first): +# SOPS_AGE_KEY (env) single-line AGE-SECRET-KEY-… +# body (CI / M4 effect preamble) +# $HOME/.config/sops/age/keys.txt (file) local dev pathway +# +# This is the ONLY flake app in modules/apps/ that intentionally consumes +# SOPS_AGE_KEY directly. Per ADR-002 ("SOPS_AGE_KEY exposure as a general +# pattern is REJECTED"), no other M4 effect or app is permitted to expose +# it. Rationale: the k3d bootstrap flow needs an age key INSIDE the +# ephemeral cluster for sops-secrets-operator to decrypt SopsSecret CRs +# at runtime — this is a load-bearing narrow exception. +# +# Caller mechanisms: +# - Local dev: file-branch via $HOME/.config/sops/age/keys.txt +# - GHA env: GHA `env:` block with SOPS_AGE_KEY from repo secrets +# - M4 effect: effect preamble extracts SOPS_AGE_KEY from +# HERCULES_CI_SECRETS_JSON and exports before invoking +# the transitive caller (k3d-integration-ci) +# +# NB: intentionally uses if-else ladder rather than `: "${VAR:?…}"` because +# the "try env, fall back to file" behaviour is the contract shape; a single +# `:?` guard cannot express the env-OR-file dual branch. set -euo pipefail case "${1:-}" in @@ -32,10 +52,11 @@ EOF ;; esac -kubectl create namespace sops-secrets-operator \ - --dry-run=client -o yaml | kubectl apply -f - - -# Determine age key file: env var (CI) or local file (dev) +# Env-var contract: validate key source BEFORE any kubectl invocation so +# the failure surface points at the contract (SOPS_AGE_KEY env OR the +# keys.txt file) rather than an opaque kubectl/api error. This is the +# ordering that satisfies VAL-ENVCONTRACT-K3DBOOT-04's "fails fast" intent. +# Determine age key file: env var (CI / effect preamble) or local file (dev). if [ -n "${SOPS_AGE_KEY:-}" ]; then echo "Using SOPS_AGE_KEY from environment variable" KEYFILE=$(mktemp) @@ -51,6 +72,9 @@ else fi fi +kubectl create namespace sops-secrets-operator \ + --dry-run=client -o yaml | kubectl apply -f - + kubectl create secret generic sops-age-key \ --namespace=sops-secrets-operator \ --from-file=age.key="$KEYFILE" \ diff --git a/modules/apps/cluster/k3d-integration-ci.nix b/modules/apps/cluster/k3d-integration-ci.nix index a296b44f4..ad56f3a0f 100644 --- a/modules/apps/cluster/k3d-integration-ci.nix +++ b/modules/apps/cluster/k3d-integration-ci.nix @@ -16,8 +16,11 @@ # you must inject a nix-computed store path or derivation outPath into # the script preamble — for the canonical example see # `modules/apps/docs/deploy.nix`, which injects DOCS_PAYLOAD -# (config.packages.vanixiets-docs) and SOPS_SECRETS_FILE (inputs.self) -# at eval time. +# (config.packages.vanixiets-docs) at eval time. Secret env vars are +# never injected via the nix preamble (per ADR-002 env-var contract); +# the caller provides them through sops exec-env, direnv dotenv, GHA +# env:, or the M4 effect preamble that extracts from +# HERCULES_CI_SECRETS_JSON. # # Orchestrates the seven-phase CI integration flow that is currently # invoked by `.github/workflows/test-cluster.yaml`. Delegates to the diff --git a/modules/apps/cluster/k3d-integration-ci.sh b/modules/apps/cluster/k3d-integration-ci.sh index c35296c96..026f2e949 100644 --- a/modules/apps/cluster/k3d-integration-ci.sh +++ b/modules/apps/cluster/k3d-integration-ci.sh @@ -7,6 +7,33 @@ # # Usage: # k3d-integration-ci [--help] +# +# Env-var contract (per ADR-002 / env-var-contract-design.md §2.5): +# Transitively required (consumed by k3d-bootstrap-secrets, the leaf): +# SOPS_AGE_KEY age key body for sops-secrets-operator inside the +# ephemeral k3d cluster. Enforcement is deferred to +# the leaf script (k3d-bootstrap-secrets.sh) which +# accepts the env-or-file dual-branch; this wrapper +# does NOT add a top-level `${SOPS_AGE_KEY:?…}` guard +# so that local-dev runs using the file-branch +# ($HOME/.config/sops/age/keys.txt) remain usable. +# Optional (config, defaulted inside this script): +# ARGOCD_REPO_URL defaults to file:///manifests; callers may override +# for remote-repo testing. +# +# Caller mechanisms: +# - Local dev: .envrc dotenv or file-branch ($HOME/.config/sops/...) +# - GHA env: job-level `env:` block populates SOPS_AGE_KEY from +# repo secrets (.github/workflows/test-cluster.yaml) +# - M4 effect: test-cluster effect preamble extracts SOPS_AGE_KEY +# from HERCULES_CI_SECRETS_JSON and exports before +# invoking ${config.apps.k3d-integration-ci.program} +# +# NB: required-env guard documented here via the `:?` idiom lives in the +# leaf k3d-bootstrap-secrets.sh; this file intentionally has no top-level +# `${SOPS_AGE_KEY:?…}` enforcement (the transitive contract is surfaced +# via k3d-bootstrap-secrets.sh's fail-fast behaviour when neither env nor +# file is present). set -euo pipefail case "${1:-}" in @@ -33,7 +60,11 @@ repo_root=$(git rev-parse --show-toplevel) cd "$repo_root" echo "=== Phase 1: Build manifests with local repo URL ===" -export ARGOCD_REPO_URL="file:///manifests" +# ARGOCD_REPO_URL default applied via `:=` bash parameter expansion so +# callers can override via env for remote-repo testing without editing +# this script. See env-var contract header for the caller mechanisms. +: "${ARGOCD_REPO_URL:=file:///manifests}" +export ARGOCD_REPO_URL just nixidy-build echo "" diff --git a/modules/apps/docs/deploy.nix b/modules/apps/docs/deploy.nix index 5f72c1aaf..332898152 100644 --- a/modules/apps/docs/deploy.nix +++ b/modules/apps/docs/deploy.nix @@ -4,17 +4,20 @@ # nix run .#deploy-docs -- production # # Consumes the nix-built CF Worker payload from config.packages.vanixiets-docs -# ($out/{dist/,.wrangler/,wrangler.jsonc}) and dispatches to wrangler via -# sops exec-env for declarative Cloudflare credential access. +# ($out/{dist/,.wrangler/,wrangler.jsonc}) and dispatches to wrangler against +# the inherited environment per the ADR-002 env-var contract; the caller +# supplies CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID via one of +# {sops exec-env, direnv dotenv, GHA step env, M4 effect preamble reading +# HERCULES_CI_SECRETS_JSON}. See deploy.sh header for the full contract. # # Template bifurcation (writeShellApplication): INTERPOLATION FORM. -# `text` is a nix string that injects two eval-time-computed paths -# (DOCS_PAYLOAD via config.packages.vanixiets-docs and SOPS_SECRETS_FILE via -# inputs.self) into the script preamble before the readFile'd sidecar body. -# Contrast with `release.nix` and `preview-version.nix`, which use the pure +# `text` is a nix string that injects one eval-time-computed path +# (DOCS_PAYLOAD via config.packages.vanixiets-docs) into the script preamble +# before the readFile'd sidecar body. Contrast with `release.nix` and +# `preview-version.nix`, which use the pure # `text = builtins.readFile ./.sh` form because they have no # nix-eval-time path injection requirement (they rely on runtimeEnv only). -{ inputs, ... }: +{ ... }: { perSystem = { @@ -29,10 +32,11 @@ program = lib.getExe ( pkgs.writeShellApplication { name = "deploy-docs"; + # Per ADR-002 env-var contract: secrets flow via inherited env + # (never via `sops exec-env` inside the script), so pkgs.sops / + # pkgs.age are no longer required runtime inputs. runtimeInputs = [ pkgs.nodejs_24 - pkgs.sops - pkgs.age pkgs.jq pkgs.coreutils pkgs.git @@ -42,7 +46,6 @@ }; text = '' export DOCS_PAYLOAD=${lib.escapeShellArg config.packages.vanixiets-docs} - export SOPS_SECRETS_FILE=${lib.escapeShellArg "${inputs.self}/secrets/shared.yaml"} ${builtins.readFile ./deploy.sh} ''; } diff --git a/modules/apps/docs/deploy.sh b/modules/apps/docs/deploy.sh index 5658c2d22..d63ca2e21 100644 --- a/modules/apps/docs/deploy.sh +++ b/modules/apps/docs/deploy.sh @@ -2,10 +2,45 @@ # shellcheck shell=bash # Docs deployment dispatcher invoked via `nix run .#deploy-docs`. # -# Environment inputs (set by deploy.nix): -# DOCS_PAYLOAD absolute path to the vanixiets-docs derivation output -# ({dist/,.wrangler/,wrangler.jsonc} layout) -# SOPS_SECRETS_FILE absolute path to secrets/shared.yaml under $inputs.self +# Env-var contract (per ADR-002 / env-var-contract-design.md §2.1): +# Required (secret, provided by caller): +# CLOUDFLARE_API_TOKEN wrangler auth token +# CLOUDFLARE_ACCOUNT_ID Cloudflare account id (wrangler requires this +# for account-scoped operations such as +# `versions upload` on a Worker attached to an +# account-level resource) +# Required (config, injected by deploy.nix): +# DOCS_PAYLOAD store path of the vanixiets-docs derivation +# ($out/{dist/, .wrangler/, wrangler.jsonc}) +# DOCS_NODE_MODULES store path of vanixiets-docs-deps node_modules +# tree (runtimeEnv of deploy.nix) +# Optional: +# WRANGLER override binary path for test harnesses; default +# $DOCS_NODE_MODULES/.bin/wrangler +# DEPLOY_DOCS_DEBUG preserve tmpdir on exit when set +# GITHUB_ACTIONS / GITHUB_ACTOR / GITHUB_WORKFLOW +# prefix the deploy message with GHA context +# +# Caller mechanisms (satisfy the secret-env contract via one of): +# - Local dev: caller-side sops wrapper (justfile `docs-deploy-*` +# recipes wrap with `sops` to decrypt secrets/shared.yaml +# and export the Cloudflare env before the nested nix run) +# OR direnv dotenv (.envrc loads .env with the Cloudflare +# env vars already exported). +# - GHA env: deploy-docs.yaml step wraps the nix run with the same +# caller-side sops decrypt inside a nix-develop wrapper; +# the age key is provided via the step `env:` block from +# the repo secrets (see deploy-docs.yaml). +# - M4 effect: preview-docs-deploy / production-docs-deploy effect +# preamble extracts CLOUDFLARE_API_TOKEN and +# CLOUDFLARE_ACCOUNT_ID from $HERCULES_CI_SECRETS_JSON +# and exports before invoking the embedded store path +# ${config.apps.deploy-docs.program}. +# +# Secret passing rule (per ADR-002): wrangler authentication flows ONLY +# through inherited env vars; no authentication CLI flags are used. +# No caller-side sops wrappers inside this script (the caller wraps if +# their mechanism is sops-based). # # Usage: # deploy-docs preview @@ -34,16 +69,16 @@ Subcommands: Flags: --help, -h Print this usage and exit 0. -Environment contract (populated by deploy.nix; required at runtime): - DOCS_PAYLOAD Absolute path to the vanixiets-docs derivation output - ($out/{dist/, .wrangler/, wrangler.jsonc}). - SOPS_SECRETS_FILE Absolute path to secrets/shared.yaml under $inputs.self; - source of Cloudflare credentials via `sops exec-env`. - DOCS_NODE_MODULES Absolute path to the vanixiets-docs-deps node_modules - tree (hosts the hermetic wrangler binary). - -Optional environment: - GITHUB_ACTIONS / GITHUB_ACTOR / GITHUB_WORKFLOW +Environment contract (see top-of-file header for full details): + Required (secret, caller-provided): + CLOUDFLARE_API_TOKEN wrangler auth token + CLOUDFLARE_ACCOUNT_ID Cloudflare account id (account-scoped ops) + Required (config, injected by deploy.nix): + DOCS_PAYLOAD path to the vanixiets-docs derivation output + DOCS_NODE_MODULES path to vanixiets-docs-deps node_modules tree + Optional: + WRANGLER, DEPLOY_DOCS_DEBUG + GITHUB_ACTIONS / GITHUB_ACTOR / GITHUB_WORKFLOW When set, the production deploy message is prefixed with the GitHub Actions context; otherwise whoami and hostname are used. @@ -70,29 +105,15 @@ if [[ -z "$mode" ]]; then fi shift -if [[ -z "${DOCS_PAYLOAD:-}" ]]; then - echo "error: DOCS_PAYLOAD not set; deploy.nix must pass the nix-built payload" >&2 - exit 1 -fi -if [[ ! -d "$DOCS_PAYLOAD" ]]; then - echo "error: DOCS_PAYLOAD=$DOCS_PAYLOAD is not a directory" >&2 - exit 1 -fi -if [[ -z "${SOPS_SECRETS_FILE:-}" ]]; then - echo "error: SOPS_SECRETS_FILE not set; deploy.nix must interpolate secrets path" >&2 - exit 1 -fi -if [[ ! -f "$SOPS_SECRETS_FILE" ]]; then - echo "error: SOPS_SECRETS_FILE=$SOPS_SECRETS_FILE does not exist" >&2 - exit 1 -fi -if [[ -z "${DOCS_NODE_MODULES:-}" ]]; then - echo "error: DOCS_NODE_MODULES not set; deploy.nix must expose vanixiets-docs-deps" >&2 - exit 1 -fi +# Env-var contract guards (per ADR-002 / env-var-contract-design.md §2.1.2). +# Fail fast before any wrangler / filesystem work if the contract is unmet. +: "${DOCS_PAYLOAD:?DOCS_PAYLOAD not set; deploy.nix must pass the nix-built payload}" +[[ -d "$DOCS_PAYLOAD" ]] || { echo "error: DOCS_PAYLOAD=$DOCS_PAYLOAD is not a directory" >&2; exit 1; } +: "${DOCS_NODE_MODULES:?DOCS_NODE_MODULES not set; deploy.nix must expose vanixiets-docs-deps via runtimeEnv}" +: "${CLOUDFLARE_API_TOKEN:?CLOUDFLARE_API_TOKEN is required (see deploy.sh header for caller mechanisms: effect preamble, direnv, caller-side sops wrapper, or GHA env)}" +: "${CLOUDFLARE_ACCOUNT_ID:?CLOUDFLARE_ACCOUNT_ID is required (see deploy.sh header for caller mechanisms: effect preamble, direnv, caller-side sops wrapper, or GHA env)}" # Hermetic wrangler via bun-managed node_modules (vanixiets-docs-deps derivation). -# Must be exported so sops exec-env subshells inherit it for single-quoted command strings. # The `${WRANGLER:-...}` fallback allows test harnesses (e.g. the no-op wrangler stub # used to exercise the post-condition error paths for VAL-WRITESHELL-DOCS-010) to # override the hermetic binary without rewriting this script. @@ -232,19 +253,17 @@ case "$mode" in # (lines ~101-110) for the diagnostic history. # # Diagnostic: echo the exact upload command line to stderr so the GHA log - # shows what the shell is about to invoke (minus secret env vars which are - # expanded by sops-wrapped subshell). + # shows what the shell is about to invoke (CLOUDFLARE_API_TOKEN and + # CLOUDFLARE_ACCOUNT_ID are expected in the inherited env per the + # env-var contract; never printed on argv). printf '>> wrangler upload command: node %s --config %s versions upload --preview-alias %s --tag %s --message %q\n' \ "$WRANGLER" "$WRANGLER_CONFIG" "b-${SAFE_BRANCH}" "$VERSION_TAG" "$VERSION_MESSAGE" >&2 set +e - # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - sops exec-env "$SOPS_SECRETS_FILE" ' - node "$WRANGLER" --config "$WRANGLER_CONFIG" versions upload \ + node "$WRANGLER" --config "$WRANGLER_CONFIG" versions upload \ --preview-alias "b-${SAFE_BRANCH}" \ --tag "$VERSION_TAG" \ - --message "$VERSION_MESSAGE" - ' \ + --message "$VERSION_MESSAGE" \ > >(tee "$wrangler_upload_stdout") \ 2> >(tee "$wrangler_upload_stderr" >&2) wrangler_upload_rc=$? @@ -290,8 +309,8 @@ case "$mode" in echo " raw wrangler stdout: $wrangler_upload_stdout" >&2 echo " raw wrangler stderr: $wrangler_upload_stderr" >&2 echo " hints:" >&2 - echo " - confirm CLOUDFLARE_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 - echo " $SOPS_SECRETS_FILE" >&2 + echo " - confirm CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID are exported by" >&2 + echo " the caller (see deploy.sh env-var contract header for caller mechanisms)" >&2 echo " - if linux-x64 regression, confirm wrangler invoked under real node and" >&2 echo " not bun-fake-node (see top-of-file rationale at lines ~101-110)" >&2 echo " - inspect the wrangler internal log dumped below / raw NDJSON and stdout paths above for any output" >&2 @@ -347,10 +366,8 @@ case "$mode" in # a file). wrangler_list_json="$tmpdir/wrangler-versions-list.json" - # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - sops exec-env "$SOPS_SECRETS_FILE" ' - node "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json - ' | cat > "$wrangler_list_json" + node "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json \ + | cat > "$wrangler_list_json" matched_count=$(jq --arg tag "$commit_tag" \ '[.[] | select(.annotations["workers/tag"] == $tag)] | length' \ @@ -395,10 +412,8 @@ case "$mode" in # empirical rationale. wrangler_list_json="$tmpdir/wrangler-versions-list.json" - # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - sops exec-env "$SOPS_SECRETS_FILE" ' - node "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json - ' | cat > "$wrangler_list_json" + node "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json \ + | cat > "$wrangler_list_json" existing_version=$(jq -r --arg tag "$commit_tag" \ '.[] | select(.annotations["workers/tag"] == $tag) | .id' \ @@ -428,13 +443,11 @@ case "$mode" in : > "$deploy_stdout" export WRANGLER_OUTPUT_FILE_PATH="$deploy_ndjson" - # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - sops exec-env "$SOPS_SECRETS_FILE" ' - node "$WRANGLER" --config "$WRANGLER_CONFIG" versions deploy \ + node "$WRANGLER" --config "$WRANGLER_CONFIG" versions deploy \ "${EXISTING_VERSION}@100%" \ --yes \ - --message "$DEPLOYMENT_MESSAGE" - ' | tee "$deploy_stdout" + --message "$DEPLOYMENT_MESSAGE" \ + | tee "$deploy_stdout" unset WRANGLER_OUTPUT_FILE_PATH @@ -469,8 +482,8 @@ case "$mode" in echo " raw wrangler event log: $deploy_ndjson" >&2 echo " raw wrangler stdout: $deploy_stdout" >&2 echo " hints:" >&2 - echo " - confirm CLOUDFLARE_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 - echo " $SOPS_SECRETS_FILE" >&2 + echo " - confirm CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID are exported by" >&2 + echo " the caller (see deploy.sh env-var contract header for caller mechanisms)" >&2 echo " - if linux-x64 regression, confirm wrangler invoked under real node and" >&2 echo " not bun-fake-node (see top-of-file rationale at lines ~101-110)" >&2 echo " - inspect the wrangler internal log dumped below / raw capture paths above for any output" >&2 @@ -482,10 +495,8 @@ case "$mode" in # comment for the rationale. deployments_list_json="$tmpdir/wrangler-deployments-list.json" - # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - sops exec-env "$SOPS_SECRETS_FILE" ' - node "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json - ' | cat > "$deployments_list_json" + node "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json \ + | cat > "$deployments_list_json" found_count=$(jq --arg did "$deployment_id" --arg vid "$existing_version" \ '[.[] | select( @@ -537,11 +548,9 @@ case "$mode" in : > "$deploy_stdout" export WRANGLER_OUTPUT_FILE_PATH="$deploy_ndjson" - # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - sops exec-env "$SOPS_SECRETS_FILE" ' - node "$WRANGLER" --config "$WRANGLER_CONFIG" deploy \ - --message "$DEPLOYMENT_MESSAGE" - ' | tee "$deploy_stdout" + node "$WRANGLER" --config "$WRANGLER_CONFIG" deploy \ + --message "$DEPLOYMENT_MESSAGE" \ + | tee "$deploy_stdout" unset WRANGLER_OUTPUT_FILE_PATH @@ -575,8 +584,8 @@ case "$mode" in echo " raw wrangler event log: $deploy_ndjson" >&2 echo " raw wrangler stdout: $deploy_stdout" >&2 echo " hints:" >&2 - echo " - confirm CLOUDFLARE_API_TOKEN (and CLOUDFLARE_ACCOUNT_ID) are present in" >&2 - echo " $SOPS_SECRETS_FILE" >&2 + echo " - confirm CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID are exported by" >&2 + echo " the caller (see deploy.sh env-var contract header for caller mechanisms)" >&2 echo " - if linux-x64 regression, confirm wrangler invoked under real node and" >&2 echo " not bun-fake-node (see top-of-file rationale at lines ~101-110)" >&2 echo " - inspect the wrangler internal log dumped below / raw capture paths above for any output" >&2 @@ -588,10 +597,8 @@ case "$mode" in deployments_list_json="$tmpdir/wrangler-deployments-list.json" - # shellcheck disable=SC2016 # single-quoted $VARs are intentional; expanded by sops-wrapped subshell - sops exec-env "$SOPS_SECRETS_FILE" ' - node "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json - ' | cat > "$deployments_list_json" + node "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json \ + | cat > "$deployments_list_json" found_count=$(jq --arg vid "$deploy_version_id" \ '[.[] | select( diff --git a/modules/apps/docs/preview-version.nix b/modules/apps/docs/preview-version.nix index 67a7322b1..c395add08 100644 --- a/modules/apps/docs/preview-version.nix +++ b/modules/apps/docs/preview-version.nix @@ -15,8 +15,7 @@ # verbatim, no nix-eval-time string interpolation. The only nix-injected # value is DOCS_NODE_MODULES, exposed via `runtimeEnv` at invocation time. # Contrast with `deploy.nix`, which uses the interpolation form because it -# must inject DOCS_PAYLOAD and SOPS_SECRETS_FILE store paths into the -# script preamble. +# must inject the DOCS_PAYLOAD store path into the script preamble. { ... }: { perSystem = diff --git a/modules/apps/docs/preview-version.sh b/modules/apps/docs/preview-version.sh index 11949a35c..5a0691319 100644 --- a/modules/apps/docs/preview-version.sh +++ b/modules/apps/docs/preview-version.sh @@ -18,9 +18,26 @@ # links it into the worktree's package directory and invokes semantic-release # directly via node_modules/.bin, bypassing any need for bun or a prior # `bun install`. +# +# Env-var contract (per ADR-002 / env-var-contract-design.md §2.2): +# Required (config, injected by preview-version.nix runtimeEnv): +# DOCS_NODE_MODULES store path of the vanixiets-docs-deps node_modules +# tree (hosts node_modules/.bin/semantic-release). +# Optional (caller-provided): +# CURRENT_BRANCH bookmark/branch name to attach HEAD to when invoked +# from a jj-colocated detached-HEAD setup. +# +# This script does NOT require secret env vars (CLOUDFLARE_API_TOKEN, +# CLOUDFLARE_ACCOUNT_ID, GITHUB_TOKEN, SOPS_AGE_KEY). semantic-release is +# invoked in --dry-run with @semantic-release/github filtered out of the +# plugin list, so no secret env required for any caller (direnv dotenv, +# `sops exec-env` wrapper, GHA `env:` block, or M4 effect preamble +# reading HERCULES_CI_SECRETS_JSON). set -euo pipefail +: "${DOCS_NODE_MODULES:?DOCS_NODE_MODULES not set; preview-version.nix must expose vanixiets-docs-deps via runtimeEnv}" + usage() { cat <<'EOF' usage: preview-version [target-branch] [package-path] diff --git a/modules/apps/release/release.sh b/modules/apps/release/release.sh index 78a327184..734b2cf02 100644 --- a/modules/apps/release/release.sh +++ b/modules/apps/release/release.sh @@ -25,20 +25,41 @@ # needed). # --help Print this usage and exit 0. # -# Environment: -# GITHUB_TOKEN Required by @semantic-release/github for production -# releases. Not consulted for --dry-run. -# SOPS_AGE_KEY Passthrough for semantic-release hooks that may -# decrypt secrets via sops. Not used directly. -# DOCS_NODE_MODULES Hermetic node_modules tree injected by release.nix. -# Must point at a directory containing a resolved -# node_modules/.bin/semantic-release. -# GIT_USER_NAME, Optional overrides for the git identity used by -# GIT_USER_EMAIL semantic-release commit/tag operations; defaults -# to `semantic-release` / `semantic-release@vanixiets.local`. +# Env-var contract (per ADR-002 / env-var-contract-design.md §2.3): +# Required (secret, production path only — not --dry-run): +# GITHUB_TOKEN @semantic-release/github auth for tag push and +# release publish. Filtered-out plugin list under +# --dry-run means no token is consulted in that mode. +# Required (config, injected by release.nix runtimeEnv): +# DOCS_NODE_MODULES vanixiets-docs-deps node_modules tree hosting +# node_modules/.bin/semantic-release. +# Optional (all modes): +# SOPS_AGE_KEY reserved passthrough for sops-decrypt hooks +# (no consumer in the current tree; declared but +# NOT enforced via :? guard — see ADR-002, which +# REJECTS SOPS_AGE_KEY as a general pattern). +# GIT_USER_NAME git identity; default: semantic-release +# GIT_USER_EMAIL git identity; default: semantic-release@vanixiets.local +# +# Caller mechanisms: +# - Local dev dry-run: `nix run .#release -- packages/ --dry-run` +# needs no secret env (plugin filter strips github) +# - Local dev prod: caller-side sops wrapper (decrypt +# secrets/shared.yaml before the nix run) OR +# direnv dotenv (.envrc `dotenv` + .env) +# - GHA env: step `env:` block populates GITHUB_TOKEN from +# the repo secrets (package-release.yaml) +# - M4 effect: production-release-packages effect preamble +# extracts GITHUB_TOKEN from HERCULES_CI_SECRETS_JSON +# and exports before invoking the app program path +# +# Secret passing rule (per ADR-002): NO secrets are passed as CLI flags. +# Authentication flows exclusively through the inherited environment. set -euo pipefail +: "${DOCS_NODE_MODULES:?DOCS_NODE_MODULES not set; release.nix must expose vanixiets-docs-deps via runtimeEnv}" + usage() { cat <<'EOF' usage: release [--dry-run] [-- extra semantic-release args] @@ -175,6 +196,15 @@ fi cd "$package_path" +# Production-path env-var contract guard: fail fast on missing GITHUB_TOKEN +# BEFORE any node_modules / workspace mutation so the error points at the +# contract rather than at an opaque state-mutation side effect. Gated on +# dry_run so the dry-run path (with @semantic-release/github filtered out) +# continues to work without any secret env. +if [ "$dry_run" -ne 1 ]; then + : "${GITHUB_TOKEN:?GITHUB_TOKEN is required for production semantic-release (see release.sh header for caller mechanisms; not needed for --dry-run)}" +fi + # Guard node_modules slot against clobbering a developer's real install. # Production (non-dry-run): strict — refuse to overwrite a real node_modules # directory. Only an empty slot or a pre-existing symlink is safe to clobber. @@ -229,7 +259,9 @@ if [ "$dry_run" -eq 1 ]; then "${extra_args[@]}" else # Production release path: semantic-release will create a tag and - # publish a GitHub release when invoked. GITHUB_TOKEN is required. + # publish a GitHub release when invoked. GITHUB_TOKEN is enforced via + # the early :? guard above (placed before node_modules setup so failure + # modes are contract-first). echo "running production semantic-release in ${package_path}..." node ./node_modules/.bin/semantic-release "${extra_args[@]}" fi From 8c5f4541d037bb440cf64894a3acb6a0a53abd67 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Thu, 23 Apr 2026 20:15:58 -0400 Subject: [PATCH 24/77] feat(effects): add vanixiets effects-secrets generator module Materialize the per-repo clan-vars effects-secrets generator for github:cameronraysmith/vanixiets as a flake-parts deferred NixOS module named effects-vanixiets-secrets, per ADR-002 Pattern C'-refined (mic92 idiom). Declares four prompts (cloudflare-api-token hidden, cloudflare-account-id line, github-token hidden, sops-age-key hidden) composing into the canonical hercules-ci-effects nested JSON shape { : { data: { value: ... } } }. Magnetite opts in via its flake-modules import list. The legacy inline perRepoSecretFiles wire in modules/nixos/buildbot.nix (pointing at the older buildbot-effects-vanixiets generator) remains in place during the three-step secrets-tree cutover (m4-01a/b/c, ADR-002); the new module's wire is declared with lib.mkDefault so the legacy normal-priority definition wins until m4-01c retires it. Once the user completes m4-01b (clan vars generate --regenerate for the new generator) and m4-01c removes the legacy wire, the mkDefault wrapper can be dropped to return to the skeleton's verbatim assignment form. Local verification: - nix flake check --system x86_64-darwin: exit 0 - nixos-rebuild dry-build --flake .#magnetite: exit 0, no errors - nix eval ...vanixiets-effects-secrets.files.secrets.secret: true - prompts attrNames: 4 prompts match expected set --- modules/effects/vanixiets/secrets.nix | 218 +++++++++++++++++++ modules/machines/nixos/magnetite/default.nix | 1 + 2 files changed, 219 insertions(+) create mode 100644 modules/effects/vanixiets/secrets.nix diff --git a/modules/effects/vanixiets/secrets.nix b/modules/effects/vanixiets/secrets.nix new file mode 100644 index 000000000..9690d0400 --- /dev/null +++ b/modules/effects/vanixiets/secrets.nix @@ -0,0 +1,218 @@ +# Pattern C'-refined (mic92 idiom) — per-repo effects-secrets generator. +# +# This file owns every clan-vars and buildbot-nix wire for the +# github:cameronraysmith/vanixiets repo's effects-secrets bundle. +# Consumer-repo effect declarations (preview-docs-deploy, +# production-docs-deploy, etc.) live in each consumer's own flake under +# `herculesCI.effects.*`; vanixiets owns only the secret-material side +# of the contract. +# +# When integrated into the live tree at modules/effects/vanixiets/secrets.nix: +# - The one-line wire +# services.buildbot-nix.master.effects.perRepoSecretFiles."github:cameronraysmith/vanixiets" = …; +# currently in modules/nixos/buildbot.nix MUST be removed; it is +# authoritative here. NixOS module-system semantics allow additive +# extension of `services.buildbot-nix.master.effects.perRepoSecretFiles` +# from this module without duplicating `services.buildbot-nix.master.enable`. +# - magnetite's host module (modules/machines/nixos/magnetite/default.nix) +# MUST add `effects-vanixiets-secrets` to its `with flakeModules; [ … ]` +# list so the flake-parts deferred module is included in magnetite's +# NixOS configuration. +# +# Operator runbook (routine): +# clan vars generate --regenerate \ +# --generator vanixiets-effects-secrets magnetite +# # Walks the four prompts (cloudflare-api-token, cloudflare-account-id, +# # github-token, sops-age-key); "Enter to keep, Backspace for new" per field. +# # Composed `secrets` file is re-encrypted and git-committed automatically. +# +# Operator runbook (escape hatch — single-token non-interactive rotation): +# printf '%s' "$TOK" | clan vars set magnetite \ +# vanixiets-effects-secrets/github-token +# clan vars generate --regenerate \ +# --generator vanixiets-effects-secrets magnetite +# +# Reference: ADR-002 (Pattern C'-refined) and its amendment-A; reference +# implementation is +# ~/projects/nix-workspace/mic92-clan-dotfiles/machines/eve/modules/buildbot.nix +# (`harmonia-effects-secrets`). +# +# This file contributes a flake-parts deferred NixOS module named +# `effects-vanixiets-secrets`, following the outer-lambda shape of +# modules/nixos/buildbot.nix so that import-tree can auto-discover it and +# magnetite can `imports = with flakeModules; [ … effects-vanixiets-secrets ];`. +{ + config, + inputs, + ... +}: +{ + flake.modules.nixos.effects-vanixiets-secrets = + { + config, + pkgs, + lib, + ... + }: + { + # Per-repo effects-secrets generator for github:cameronraysmith/vanixiets. + # + # Composes the four operator-sourced tokens + # (CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, GITHUB_TOKEN, + # SOPS_AGE_KEY) into a single `secrets` file shaped as + # hercules-ci-effects-nested JSON: + # + # { + # "": { "data": { "value": "" } }, + # … + # } + # + # consumed by buildbot-nix at dispatch time as + # HERCULES_CI_SECRETS_JSON inside the bwrap sandbox. The keys below + # match the secret identifiers referenced by effect scripts in + # modules/effects/vanixiets/herculesCI/*.nix. + clan.core.vars.generators.vanixiets-effects-secrets = { + # The composed JSON file deployed to magnetite and pointed at by + # services.buildbot-nix.master.effects.perRepoSecretFiles below. + # `secret = true` is the default (see clan-core/modules/clan/vars/settings-opts.nix:30-37); + # stated explicitly to mirror mic92's harmonia-effects-secrets idiom. + files.secrets = { + secret = true; + owner = "buildbot"; + }; + + # --- Prompts (interactive capture, persisted for rotation UX) --- + + prompts.cloudflare-api-token = { + description = '' + Cloudflare API token (scope: Workers/Pages:Edit + relevant zone/R2 scopes). + Single token shared across preview and production effects. + ''; + type = "hidden"; + persist = true; + display = { + group = "vanixiets effects"; + label = "CLOUDFLARE_API_TOKEN"; + helperText = '' + Pasted once at first generate; Enter to keep existing on subsequent + `clan vars generate --regenerate` invocations. + ''; + }; + }; + + prompts.cloudflare-account-id = { + description = '' + Cloudflare account ID paired with CLOUDFLARE_API_TOKEN above. + Required by wrangler for Pages/Workers deploys. Not secret in + the cryptographic sense, but captured via the same generator + to keep the 4-env-var contract homogeneous and avoid a + parallel non-secret distribution channel. + ''; + type = "line"; + persist = true; + display = { + group = "vanixiets effects"; + label = "CLOUDFLARE_ACCOUNT_ID"; + helperText = '' + Single-line account id (32 hex chars). Enter to keep existing + on subsequent `clan vars generate --regenerate` invocations. + ''; + }; + }; + + prompts.github-token = { + description = '' + GitHub fine-grained Personal Access Token for effect scripts that + interact with the forge API (release creation, label edits, etc.). + Scope to the minimum repositories required by the effect bundle. + ''; + type = "hidden"; + persist = true; + display = { + group = "vanixiets effects"; + label = "GITHUB_TOKEN"; + helperText = '' + Fine-grained PAT, not a classic PAT. Expires per your GitHub + account default (rotate before expiry). + ''; + }; + }; + + prompts.sops-age-key = { + description = '' + Age private key used by sops-secrets-operator bootstrap effects + running inside the ephemeral test-cluster spawned during CI. + Corresponds to the age recipient recorded in .sops.yaml. + ''; + type = "hidden"; + persist = true; + display = { + group = "vanixiets effects"; + label = "SOPS_AGE_KEY"; + helperText = '' + AGE-SECRET-KEY-… literal (single line). Not the path to a key + file; paste the key body itself. + ''; + }; + }; + + # Raw prompt files are auto-materialized by `persist = true` so + # their encrypted-at-rest copy lives in the repo, enabling + # per-token rotation via the "Enter to keep" UX. They must NOT be + # deployed to magnetite: only the composed `secrets` file needs to + # reach the machine's secret store at activation time. + # (Reference: clanServices/admin/root-password.nix:17-20 uses the + # same idiom for `files.password.deploy = false`.) + files.cloudflare-api-token.deploy = false; + files.cloudflare-account-id.deploy = false; + files.github-token.deploy = false; + files.sops-age-key.deploy = false; + + # --- Composition script (mic92 harmonia-effects-secrets style) --- + + runtimeInputs = [ pkgs.jq ]; + + script = '' + jq -n \ + --arg cloudflare_api_token "$(cat "$prompts/cloudflare-api-token")" \ + --arg cloudflare_account_id "$(cat "$prompts/cloudflare-account-id")" \ + --arg github_token "$(cat "$prompts/github-token")" \ + --arg sops_age_key "$(cat "$prompts/sops-age-key")" \ + '{ + CLOUDFLARE_API_TOKEN: { data: { value: $cloudflare_api_token } }, + CLOUDFLARE_ACCOUNT_ID: { data: { value: $cloudflare_account_id } }, + GITHUB_TOKEN: { data: { value: $github_token } }, + SOPS_AGE_KEY: { data: { value: $sops_age_key } } + }' > "$out/secrets" + ''; + }; + + # Wire the composed `secrets` file to buildbot-nix's per-repo + # effects-secret map. The attribute `services.buildbot-nix.master` + # is declared as an option by + # inputs.buildbot-nix.nixosModules.buildbot-master (imported by + # modules/machines/nixos/magnetite/default.nix); the NixOS module + # system merges this additive attribute with the rest of the master + # config in modules/nixos/buildbot.nix. In particular we do NOT + # redeclare master.enable, workersFile, github.*, accessMode.*, or + # any other authoritative option here — only the perRepoSecretFiles + # attribute keyed on this repo's forge identifier. + # + # `lib.mkDefault` is a transitional priority marker for the + # three-step secrets-tree cutover (ADR-002 / AGENTS.md §"Three-step + # secrets-tree cutover (M4-01 pattern)"): during the implement + # phase (m4-01a) the legacy inline wire at + # `modules/nixos/buildbot.nix` coexists with this declaration and + # wins via its normal-priority definition, so the production + # perRepoSecretFiles entry continues to resolve to the legacy + # `buildbot-effects-vanixiets` generator path until the user-driven + # retire phase (m4-01c) removes the legacy wire. At that point the + # `lib.mkDefault` below becomes the sole definition and resolves to + # the new `vanixiets-effects-secrets` generator's composed secrets + # file. Once m4-01c lands, the mkDefault wrapper should be dropped + # (returning to the skeleton's verbatim assignment form). + services.buildbot-nix.master.effects.perRepoSecretFiles."github:cameronraysmith/vanixiets" = + lib.mkDefault + config.clan.core.vars.generators.vanixiets-effects-secrets.files.secrets.path; + }; +} diff --git a/modules/machines/nixos/magnetite/default.nix b/modules/machines/nixos/magnetite/default.nix index 719771ab2..e3e41e442 100644 --- a/modules/machines/nixos/magnetite/default.nix +++ b/modules/machines/nixos/magnetite/default.nix @@ -36,6 +36,7 @@ in gitea gitea-actions-runner docker + effects-vanixiets-secrets ]); # Make flake available to all modules (required by ssh-known-hosts) From fbb15aac88fa1b1f6eaa3d6cc5a45c29599e2cb3 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Fri, 24 Apr 2026 02:07:28 -0400 Subject: [PATCH 25/77] vars: update via generator vanixiets-effects-secrets (machine: magnetite) --- .../cloudflare-account-id/secret | 14 ++++++++++++++ .../cloudflare-account-id/users/cameron | 1 + .../cloudflare-api-token/secret | 14 ++++++++++++++ .../cloudflare-api-token/users/cameron | 1 + .../github-token/secret | 14 ++++++++++++++ .../github-token/users/cameron | 1 + .../secrets/machines/magnetite | 1 + .../vanixiets-effects-secrets/secrets/secret | 18 ++++++++++++++++++ .../secrets/users/cameron | 1 + .../sops-age-key/secret | 14 ++++++++++++++ .../sops-age-key/users/cameron | 1 + 11 files changed, 80 insertions(+) create mode 100644 vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/secret create mode 120000 vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/users/cameron create mode 100644 vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/secret create mode 120000 vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/users/cameron create mode 100644 vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/secret create mode 120000 vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/users/cameron create mode 120000 vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/machines/magnetite create mode 100644 vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/secret create mode 120000 vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/users/cameron create mode 100644 vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/secret create mode 120000 vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/users/cameron diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/secret new file mode 100644 index 000000000..a7b5ec04a --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/secret @@ -0,0 +1,14 @@ +{ + "data": "ENC[AES256_GCM,data:AEgZtL+XHM1oGrCSUlbzKfyBtIsYwBRfgeQ03NOrvJc=,iv:bH5kvogxXCC0DiuLfPhc5ahu0lsoVqX9SY+tYstqRa4=,tag:YxUqn5pjhyuZC5MX6iDeEg==,type:str]", + "sops": { + "age": [ + { + "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxbE45bHZ6UE5Uc2NISnZM\nRlRIMDFZVktxL1JkMTBDbC84YUU4RHdRM0JzCjdhV3N5Snd3VHZLWnNKdG5JRmc0\nY1FkeHBFM3I5cGJ2VGFwaE9aR00zWXcKLS0tIHhFSkZHeE1SMnZNNzR5S214OVpi\nZVJrcXpzeENCRmdRRVJSNXl1cDhOZmcKjTMBlJ36OfqyATLYMLry6cicewCDmJC3\nBmCbf9s1cq3os7UlmWYFrfvn5cR5mc9u6u/2Sa6djDPWXXn12m0tUA==\n-----END AGE ENCRYPTED FILE-----\n" + } + ], + "lastmodified": "2026-04-24T06:07:27Z", + "mac": "ENC[AES256_GCM,data:nF0LZlmrNffvbk7aTAzqsnZVwo7uNzPYqxd49l2xFg0w7h6SxJ5RnPGydHo49KwZ7PmBN+jef9OGf+et9MNYlo7M0aoupXsqT4EJFoCZitUMDP3KdmUy6dxc9AaitGBNohbpmh5UDnzL9AFtoGTwCJcJeHgsjgErWBY0TLrD2l0=,iv:jF8KUIwGFUgic5dss1EG/zdsLPvmXvMB10hbApezAjA=,tag:96is3O7SLbcjrS9pPq65fA==,type:str]", + "version": "3.12.2" + } +} diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/users/cameron b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/users/cameron new file mode 120000 index 000000000..015130152 --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/users/cameron @@ -0,0 +1 @@ +../../../../../../sops/users/cameron \ No newline at end of file diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/secret new file mode 100644 index 000000000..75ba61bbf --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/secret @@ -0,0 +1,14 @@ +{ + "data": "ENC[AES256_GCM,data:2fa05e29Y06qALSPVR4GdkpUbCbPgJF1yFtYof6nrti0Cmq7HBSprQ==,iv:fROlnLnp+n1Goh6bt1gCHlSviWxRyhX5cb1my4DKwu0=,tag:EfSl5ASs6D5/r0uMUpwvBQ==,type:str]", + "sops": { + "age": [ + { + "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZTWpubXVoNUFsRDdualpz\nb3ArYVNTeURuTXNGSmplcnIvM0h2OUlwd0FrCkYwUVJNcURBM2ZvZ1NIaWRZb2JY\nMFgyejFiS25Ucjl0QWNVR3ZKVnVpdDQKLS0tIGg0WElseU1DT1hwNG9VTU9rRE0w\nb21Tc2EvRGhaVW43SEttcWFjREdPNVkK1zlIYVD04enRhykmSvFOzsrKX9HZ4W2C\nBOSERkQwh6N52VuOCfHl7lUU5Xl5emUyTcYesWc4udGbBM/u5yzdhQ==\n-----END AGE ENCRYPTED FILE-----\n" + } + ], + "lastmodified": "2026-04-24T06:07:27Z", + "mac": "ENC[AES256_GCM,data:F0fPnnhgEiFgsq72R5m64pzQuerjrlwrK/3UiJPvhcrP/IYY+4GObr60ymPZ5EyuMiPV67DEZCUgvXZI4RHOVr0JqvXSrg+w/O5l8q4naYKrIFoI3hs6M/N3jdQotapYeLQ/iDkVEjKULrqTTjoLXgw/waTe2hBw/ME6s23Bj8A=,iv:mTiO2SKYvlIi07l/cZSNJrh/HcErDwtiYLwa8KET8KY=,tag:Crd90W2grb10Ce+CBf+tdg==,type:str]", + "version": "3.12.2" + } +} diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/users/cameron b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/users/cameron new file mode 120000 index 000000000..015130152 --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/users/cameron @@ -0,0 +1 @@ +../../../../../../sops/users/cameron \ No newline at end of file diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/secret new file mode 100644 index 000000000..7be71495e --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/secret @@ -0,0 +1,14 @@ +{ + "data": "ENC[AES256_GCM,data:fjjkFqnu6DEY7ip6HJNSaI0FcDKTnUt+8BkbqeuPCkrMlTD0ostxQg==,iv:gKHO4qp6u3R90DFNxhgT2WQWKdlV2iLJKyolZFSd14I=,tag:loWGQn/fAAVXb8hr8ylqWg==,type:str]", + "sops": { + "age": [ + { + "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmcEFDdnplL3pUTlNNY2RI\ncjQvSCtkS0RZY1JEUm80Z09aakVhRjJpTUVVCjB3eDR2Z2FtUGRmS3hVK3hQTWRD\nWmhISi8vRko4dkZieWhMeC85U2d2RFEKLS0tIFprTjZvcGlHeGNxYk00TndqSDlU\nOHNsc2RFZWRWeDcxUHEvVS8wQ3Q3eVkKb6PLRVBxGWYkY3SUaNtpUMs8v0YQ0Lwc\nmsm9+ipNzERg5jGnQZ0ip8pfv4nPbOZUA/cpsAFse30HKYnF2g+cbw==\n-----END AGE ENCRYPTED FILE-----\n" + } + ], + "lastmodified": "2026-04-24T06:07:27Z", + "mac": "ENC[AES256_GCM,data:o3smM5LQObtBpwubQF9GVJJEmWWUauXo4zI2BTm5qG+LJ4bXFfOuEYvjJvvhSHcz1d4wFlbCOmY6i/F/NTh7hS4wINvztIuC7Dsac3pmnJKxfRMwwMbSsWQ9ZQS8Fq3S/DYiET9grShO61g2kPQu7frZrGPQXIbi9l6kDKQ56Ag=,iv:LC5fs6mciQEJc/odi3V+wY+irYEROMQcmcZFxAqj2KE=,tag:5oOLLDcuzTKDhWkuFN6IOQ==,type:str]", + "version": "3.12.2" + } +} diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/users/cameron b/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/users/cameron new file mode 120000 index 000000000..015130152 --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/users/cameron @@ -0,0 +1 @@ +../../../../../../sops/users/cameron \ No newline at end of file diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/machines/magnetite b/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/machines/magnetite new file mode 120000 index 000000000..41bd9646c --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/machines/magnetite @@ -0,0 +1 @@ +../../../../../../sops/machines/magnetite \ No newline at end of file diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/secret new file mode 100644 index 000000000..4c477f732 --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/secret @@ -0,0 +1,18 @@ +{ + "data": "ENC[AES256_GCM,data:qo8Ka1sKx9BoqPy5DUYJsgyEpEahSkbM60jB6nmPKQYeNSlONXITExWGp2k75Iu+2gTYqRF2D78P2xGgSr+uPncdJb2zJdBepgAtZXSuLk0tLiPj3PbvEYPSlbXzCN5MZsi721nk5P/eaSh5Cwpcz+vF77g4T4gE9Na775xMbALjosMo1CDU0+n8bqJGiwBmWf9iW5EbxqtsPSolbdg70FIG59pk8/TCAktXnGQ+6B3vJ+dKJsGYN5kMx9WxYsZtMTlNBs4dpJu3Vt3Dm2tkwPw8/bGUHlHWmi/UzIgSv2AAcMAOSbGB2KogCo3fGZvyIj0WL+qsO9Wz5pl5ZmbzjhuPdy/Ri3zBNxyHUgjcmwttXHnQT8BMAlM/zfwLgF3oRuSyUQisKBTXcW6Y549RypoXDEoSx2egc0iPj+ZVjylxpRAScx9SbquVrc7Ua3UBCeT49/qVA3AaPXwhzhfG3U6DfkHh+LyLrRCz/drLmWo57FsbZqUWsltMngNOvqqmrrQ8yTSxJstN5kS6Paay6EOIwlvPNHAD4e6ETggQivwTzAeO3hfIdLBJEjTVcKIN1kLKL+IO9skyh/vqBTHYy44+WKq0bldnaXo=,iv:cu7qkM2xwzsZ2z7OZyouP9Pk637yGT6zpGMU1vK8Kng=,tag:73gDM12N6eUPhnZvL4CbAQ==,type:str]", + "sops": { + "age": [ + { + "recipient": "age1a7a70qcpjemlvk6q4uaf4k77p9eq7lj7wcal5jdj3xuetznyqdrs3mfnsf", + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4dzA0TDloUXZsUG1jMEhr\ncG1RbTJDMXRma1M4SXYyK2M4blRqWWNocVhrClM4VW02VHgrOWloQjdKNkJRODZZ\nRVB2MXp4cmR5Ym41THA5S0dqWEFQWkEKLS0tIEN6VEE3a2ZUdmN6L1ZoNTdkRHdX\nVCtkeU1FWDJZZjM5OW81b0ZHVnU0encKSl6/rKzZz816Vo7kdmo2Ivy4007DTAgy\nKTgygkhbEQ1UnfgO94YZhmeLItjJq8f56+zope+dtjgcAnlzQ205Ow==\n-----END AGE ENCRYPTED FILE-----\n" + }, + { + "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBHdENVQW1RcEJsSWYxR3BY\nNGdGNzJxSlRPRkpHelVqR0tEdUxQZEpwWUhNCjJGSDQ0TXl5TVpOYitObG0yUE5w\nN3J6UzZoSUpxUzJscnJ3bk5TVkxjcGMKLS0tIHFxSDd3bEYzdHlmVTl5YWpvMnJ5\nMUgvenN0NElwTVFvbWdrbG04alZSVDAKwGB1h5ahtW27pfIsspJVaCetzr/p1zTR\n9Kh6ZAoqRPxh6dr2iHP0kmiUP5aWDDd5f0yj07jq+y2XfO5ICxt4FA==\n-----END AGE ENCRYPTED FILE-----\n" + } + ], + "lastmodified": "2026-04-24T06:07:27Z", + "mac": "ENC[AES256_GCM,data:raOcQI23iIX0yTroEDhMy/wY9apTjX2lMFfnGkh6gySyF9d1OK3Dz9dbjMi1sPLViXfzQUYypfScyywZpSYTNLaKUDiOCPotpP37tecPuJPTm8YFTXxvEKql6rUrs2dO19b7K1TtZSgyPdr1MlBbIKYN9FwvHXCiZI01iYVj4y4=,iv:OsFByARwKr2I/vYP/irtYQ+BAE0H1U8UbcQ93KSL0vE=,tag:ayxNf1yukt9QhyJIvwE5aw==,type:str]", + "version": "3.12.2" + } +} diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/users/cameron b/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/users/cameron new file mode 120000 index 000000000..015130152 --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/users/cameron @@ -0,0 +1 @@ +../../../../../../sops/users/cameron \ No newline at end of file diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/secret new file mode 100644 index 000000000..bd1e00e97 --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/secret @@ -0,0 +1,14 @@ +{ + "data": "ENC[AES256_GCM,data:vqhuD6dBU/LSTYaQzKV6gyRVvVKPTN/Z56vq5J1zCRHilYRapEFHLc+tIGdaVK9M2owioshL+ntconal9Upg09npJ91genXQXHU=,iv:NHfF1GuC1FX+5tEYdEzkp3HuP3Da2K8byZpF1hFAy2M=,tag:1HtqkhSFu0+XLnWnwsewcQ==,type:str]", + "sops": { + "age": [ + { + "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrSHBKbUNDdkJlbWd5VThu\nTXhtQ2VkVm9NdkJyd3NnM0hTNEN6N2YxcjNRCmZuNU01S2UrQUtaVTQ1N0hBWXYr\nc3FkV3RVQkpOTGFueGtGVE5JT1BhYjQKLS0tIGN0eTlSbjBvM0ZjckhzenVZWmxx\nVklkdXMyVzErY2k3Z2ZiMHNvUzZNZFkK8Vb9w6D3CqOwLKRw+YRWe1QtJzkWc3xY\nxTmPWXi6ubPSnVZbUmZ4vvLGSvBwv6AidojFJRsmh3C2JBkR9sLLEA==\n-----END AGE ENCRYPTED FILE-----\n" + } + ], + "lastmodified": "2026-04-24T06:07:27Z", + "mac": "ENC[AES256_GCM,data:NaUGtQVXgnAYEfZpE7D6y4nygxB9QLeVY99wYJvM1xrMCV/B/XHhsF5GGH7vqISnwxwFVQRCZDwQKc/zowpufB0aifUWzp5ceWdBYTlLsGzR51/G3yiaCtCNS32G0gjZE6qV+l1SBOyDCcrcgPULTMdH/7u+deEUE56RtZplSho=,iv:TBuoSlst0YN746JD49rf8SXmCiuUGt1pBVRk8wTdSk4=,tag:R4dmtQryBuHJFI2U0BoTBA==,type:str]", + "version": "3.12.2" + } +} diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/users/cameron b/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/users/cameron new file mode 120000 index 000000000..015130152 --- /dev/null +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/users/cameron @@ -0,0 +1 @@ +../../../../../../sops/users/cameron \ No newline at end of file From 6d7f98bbf992d949f65b76709da1bf265b5a36ef Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Fri, 24 Apr 2026 02:37:42 -0400 Subject: [PATCH 26/77] refactor(buildbot): retire legacy buildbot-effects-vanixiets wiring The new vanixiets-effects-secrets generator in modules/effects/vanixiets/secrets.nix is now the sole authoritative source for the github:cameronraysmith/vanixiets perRepoSecretFiles entry. This commit: 1. Removes the inline perRepoSecretFiles wire from modules/nixos/buildbot.nix. 2. Removes the legacy buildbot-effects-vanixiets clan-vars generator declaration (and its header doc line) from modules/nixos/buildbot.nix. 3. Drops the transitional lib.mkDefault wrapper from the new wire in modules/effects/vanixiets/secrets.nix now that the legacy normal-priority definition is gone. The vars/per-machine/magnetite/buildbot-effects-vanixiets/ on-disk sops-encrypted entries are intentionally left in place as inert historical artifacts; their removal (clan vars remove) is deferred to M5 feature m5-retire-legacy-buildbot-effects-vanixiets-vars once the new wire has accumulated confidence across multiple effect runs. Verified locally on darwin: - rg -n buildbot-effects-vanixiets modules/ -> no matches - nix eval .#nixosConfigurations.magnetite.config.services.buildbot-nix.master.effects.perRepoSecretFiles -> {"github:cameronraysmith/vanixiets":"/run/secrets/vars/vanixiets-effects-secrets/secrets"} - nix build --dry-run .#nixosConfigurations.magnetite.config.system.build.toplevel -> succeeded Ready for user to run clan machines update magnetite. --- modules/effects/vanixiets/secrets.nix | 20 +++-------- modules/nixos/buildbot.nix | 48 ++++----------------------- 2 files changed, 12 insertions(+), 56 deletions(-) diff --git a/modules/effects/vanixiets/secrets.nix b/modules/effects/vanixiets/secrets.nix index 9690d0400..8726f815f 100644 --- a/modules/effects/vanixiets/secrets.nix +++ b/modules/effects/vanixiets/secrets.nix @@ -198,21 +198,11 @@ # any other authoritative option here — only the perRepoSecretFiles # attribute keyed on this repo's forge identifier. # - # `lib.mkDefault` is a transitional priority marker for the - # three-step secrets-tree cutover (ADR-002 / AGENTS.md §"Three-step - # secrets-tree cutover (M4-01 pattern)"): during the implement - # phase (m4-01a) the legacy inline wire at - # `modules/nixos/buildbot.nix` coexists with this declaration and - # wins via its normal-priority definition, so the production - # perRepoSecretFiles entry continues to resolve to the legacy - # `buildbot-effects-vanixiets` generator path until the user-driven - # retire phase (m4-01c) removes the legacy wire. At that point the - # `lib.mkDefault` below becomes the sole definition and resolves to - # the new `vanixiets-effects-secrets` generator's composed secrets - # file. Once m4-01c lands, the mkDefault wrapper should be dropped - # (returning to the skeleton's verbatim assignment form). + # This module is the sole authoritative definition of the + # `github:cameronraysmith/vanixiets` perRepoSecretFiles entry; + # m4-01c retired the legacy inline wire in modules/nixos/buildbot.nix, + # so no transitional `lib.mkDefault` priority marker is required. services.buildbot-nix.master.effects.perRepoSecretFiles."github:cameronraysmith/vanixiets" = - lib.mkDefault - config.clan.core.vars.generators.vanixiets-effects-secrets.files.secrets.path; + config.clan.core.vars.generators.vanixiets-effects-secrets.files.secrets.path; }; } diff --git a/modules/nixos/buildbot.nix b/modules/nixos/buildbot.nix index f2b82284e..43b14cc63 100644 --- a/modules/nixos/buildbot.nix +++ b/modules/nixos/buildbot.nix @@ -10,9 +10,10 @@ # - buildbot-worker: auto-generated (worker password + workers.json) # - buildbot-oauth2-cookie-secret: auto-generated (oauth2-proxy cookie encryption) # - buildbot-http-basic-auth-password: auto-generated (oauth2-proxy to buildbot internal auth) -# - buildbot-effects-vanixiets: prompts-populated `secrets.json` flat JSON dict for -# HERCULES_CI_SECRETS_JSON consumption by hercules-ci-effects inside the master's -# bwrap sandbox, wired via `services.buildbot-nix.master.effects.perRepoSecretFiles`. +# The effects-secrets generator and its `perRepoSecretFiles` wire for +# github:cameronraysmith/vanixiets are authoritatively declared in +# modules/effects/vanixiets/secrets.nix (flake module +# `effects-vanixiets-secrets`, opted-in by magnetite's host module). # Gitea-specific credentials are declared in gitea.nix: # - buildbot-gitea-token: manual `clan vars set` (API token with write:repository, write:user) # - buildbot-gitea-webhook-secret: auto-generated @@ -85,36 +86,6 @@ ''; }; - # Effects secrets for github:cameronraysmith/vanixiets - # (flat-dict JSON consumed as HERCULES_CI_SECRETS_JSON inside bwrap). - # The auto-generated body seeds placeholder values so the full - # `perRepoSecretFiles → LoadCredential → bwrap` pipeline can be - # smoke-tested end-to-end before any real tokens exist. Rotate to - # real tokens in place via: - # clan vars set magnetite buildbot-effects-vanixiets/secrets.json < real-secrets.json - # Keys MUST match the names effect scripts `jq -r` out of - # $HERCULES_CI_SECRETS_JSON: - # - CLOUDFLARE_API_TOKEN: wrangler-driven docs-deploy effects - # - SOPS_AGE_KEY: sops-secrets-operator bootstrap inside test-cluster - # - GITHUB_TOKEN: semantic-release + gh api consumers - # Consumed below by - # `services.buildbot-nix.master.effects.perRepoSecretFiles`. - # Default file mode is 0400 (clan-core invariant) and owner is the - # buildbot master user so systemd `LoadCredential` can read it. - clan.core.vars.generators.buildbot-effects-vanixiets = { - files."secrets.json" = { - owner = "buildbot"; - }; - runtimeInputs = [ pkgs.jq ]; - script = '' - jq -n '{ - CLOUDFLARE_API_TOKEN: "placeholder-cloudflare-api-token-rotate-via-clan-vars-set", - SOPS_AGE_KEY: "placeholder-sops-age-key-rotate-via-clan-vars-set", - GITHUB_TOKEN: "placeholder-github-token-rotate-via-clan-vars-set" - }' > "$out/secrets.json" - ''; - }; - # Worker credentials (auto-generated password + workers.json) # CX53: 16 logical CPUs (nproc) — cores must match for correct worker count clan.core.vars.generators.buildbot-worker = { @@ -188,14 +159,9 @@ evalWorkerCount = 4; evalMaxMemorySize = 2048; - # Per-repo effects secrets (flat-dict JSON → HERCULES_CI_SECRETS_JSON inside bwrap). - # Fork-PR posture A (`effects_on_pull_requests = false` in buildbot-nix.toml) - # keeps this file off fork-PR effect runs; in-repo pushes on matched - # `effects_branches` receive it. Key must be the forge-prefixed - # `:/` literal that buildbot-effects uses to look up - # the matching secrets file at dispatch time. - effects.perRepoSecretFiles."github:cameronraysmith/vanixiets" = - config.clan.core.vars.generators.buildbot-effects-vanixiets.files."secrets.json".path; + # Per-repo effects secrets for github:cameronraysmith/vanixiets are + # wired authoritatively in modules/effects/vanixiets/secrets.nix + # (flake module `effects-vanixiets-secrets`). # niks3 binary cache integration (push built paths after successful builds) # Uses public URL to support future remote workers (e.g. cinnabar) From 7da174d7eb3bb1262af4d7c56113e289fbabffdd Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Fri, 24 Apr 2026 12:53:13 -0400 Subject: [PATCH 27/77] feat(effects): add deploy-docs branch-dispatcher effect MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Declares herculesCI.onPush.default.outputs.effects.deploy-docs as a single branch-dispatcher effect replacing the three legacy per-job features (preview-docs-deploy, production-docs-deploy-dryrun, production-docs-deploy-cutover). Design: - Option Gamma store-path embedding: effectScript interpolates the deploy-docs flake app's resolved /nix/store path at nix-eval time (config.apps.deploy-docs.program via withSystem "x86_64-linux"), never a nix-run shell-out (bwrap does not bind the working tree). - Branch dispatch via exact string equality on primaryRepo.branch (surfaced as herculesCI.config.repo.branch): main → production promote path; any other branch or null → preview upload path. - Pattern C'-refined secrets preamble: extracts CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID from $HERCULES_CI_SECRETS_JSON at the ..data.value envelope; no SOPS_AGE_KEY / NPM_TOKEN reads. - Structured banner emission (DD-16 log-grep anchors): * DEPLOY-DOCS-ACTION: preview-upload|promote|fresh-deploy-and-promote (preview-upload / promote are emitted upfront; the post-hoc fresh-deploy-and-promote banner fires on the main path when deploy.sh logs its "falling back to direct deploy" fallback). * DEPLOY-DOCS-PREVIEW-URL: on the preview path, parsed from deploy.sh stdout so downstream log-grep can 200-probe the alias. Posture A outer gate (effects_on_pull_requests=false, effects_branches in buildbot-nix.toml) precedes this inner branch-dispatch; fork PRs never reach the dispatcher body. fulfills: VAL-EFFECT-DEPLOYDOCS-{01..26}, VAL-EFFECT-DISPATCHER-{01..04} mission: cd.yaml → buildbot-nix (m4-effects feature m4-deploy-docs) --- .../vanixiets/herculesCI/deploy-docs.nix | 189 ++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 modules/effects/vanixiets/herculesCI/deploy-docs.nix diff --git a/modules/effects/vanixiets/herculesCI/deploy-docs.nix b/modules/effects/vanixiets/herculesCI/deploy-docs.nix new file mode 100644 index 000000000..a87c99ebb --- /dev/null +++ b/modules/effects/vanixiets/herculesCI/deploy-docs.nix @@ -0,0 +1,189 @@ +# effects.deploy-docs — docs deployment branch-dispatcher (M4 feature +# `m4-deploy-docs`). Consolidates three pre-consolidation jobs +# (preview-docs-deploy, production-docs-deploy-dryrun, +# production-docs-deploy-cutover) into a single herculesCI effect that +# dispatches on `primaryRepo.branch`. +# +# Design contract (see mission AGENTS.md "ADR-002 locked decisions" and +# `.factory/validation-contract.md` VAL-EFFECT-DEPLOYDOCS-*): +# +# Option Gamma store-path embedding: +# The effect body invokes the `deploy-docs` flake app via the +# nix-eval-time resolved store path +# `${config.apps.x86_64-linux.deploy-docs.program}`. The effect +# never dispatches via a flake-app shell-out (bwrap does not bind +# the working tree, so the .# syntax cannot resolve). +# +# Branch dispatch (exact string equality): +# Selection is driven by `primaryRepo.branch == "main"`, surfaced +# through `herculesCI.config.repo.branch` which hercules-ci-effects +# populates from the primaryRepo record at flake.herculesCI entry. +# * primaryRepo.branch == "main" → production promote path +# * any other branch (or null) → preview upload path +# +# Pattern C'-refined secrets preamble: +# Extracts CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID from +# $HERCULES_CI_SECRETS_JSON at the `..data.value` envelope +# (see modules/effects/vanixiets/secrets.nix for the generator that +# emits this shape). Never extracts SOPS_AGE_KEY (ADR-002 +# exclusivity) and never references NPM_TOKEN. +# +# Posture A outer gate: +# Fork-PR exposure is blocked by `effects_on_pull_requests = false` +# + `effects_branches` in `buildbot-nix.toml`, which precedes this +# inner branch-dispatch. The dispatcher logic runs only after the +# outer gate has passed. +# +# Structured banners (DD-16 log-grep anchors): +# * `DEPLOY-DOCS-ACTION: preview-upload|promote|fresh-deploy-and-promote` +# emitted exactly once per run, identifying the dispatcher path +# taken. `fresh-deploy-and-promote` is emitted post-hoc on the +# main path only when deploy.sh reports the fallback branch. +# * `DEPLOY-DOCS-PREVIEW-URL: ` emitted on the preview path +# once the wrangler-produced preview URL is parsed from deploy.sh +# stdout, enabling downstream `curl` verification of 200 OK. +{ + config, + inputs, + lib, + withSystem, + ... +}: +{ + herculesCI = + herculesCI: + let + # primaryRepo.branch (exposed as config.repo.branch by + # hercules-ci-effects' paramModule, populated from primaryRepo.branch + # at flake.herculesCI entry). Type: nullable string — null on tag + # pushes where branch is not populated. + branch = herculesCI.config.repo.branch; + shortRev = herculesCI.config.repo.shortRev; + rev = herculesCI.config.repo.rev; + + # Branch dispatch: exact string equality. null == "main" is false + # in Nix, so tag pushes naturally fall through to the preview path. + isMain = branch == "main"; + in + { + onPush.default.outputs.effects.deploy-docs = withSystem "x86_64-linux" ( + { config, pkgs, ... }: + let + hci-effects = inputs.hercules-ci-effects.lib.withPkgs pkgs; + + # Option Gamma: resolved at nix eval time to a /nix/store path + # that the bwrap sandbox can execute without a working-tree or + # nix-daemon lookup. + deployDocsProgram = config.apps.deploy-docs.program; + + # Initial action banner — refined post-hoc on the main path if + # deploy.sh's fresh-deploy-and-promote fallback triggers. + actionBanner = if isMain then "promote" else "preview-upload"; + + # Preview branch argument: prefer the live branch name; fall + # back to the shortRev on detached / null-branch pushes so + # deploy.sh preview has a non-empty argument. + previewBranchArg = if branch != null && branch != "" then branch else shortRev; + in + hci-effects.mkEffect { + name = "deploy-docs"; + + effectScript = '' + set -euo pipefail + + echo "=== effects.deploy-docs (docs deployment dispatcher) ===" + echo "branch: ${lib.escapeShellArg (toString branch)}" + echo "rev: ${lib.escapeShellArg (toString rev)}" + echo "shortRev: ${lib.escapeShellArg (toString shortRev)}" + echo "isMain: ${if isMain then "true" else "false"}" + + # Structured banner (DD-16): emitted once per run so log-grep + # can distinguish preview-upload | promote | fresh-deploy-and-promote. + echo "DEPLOY-DOCS-ACTION: ${actionBanner}" + + # Secrets preamble — Pattern C'-refined (ADR-002): + # extract CLOUDFLARE_API_TOKEN from $HERCULES_CI_SECRETS_JSON at .data.value envelope. + export CLOUDFLARE_API_TOKEN="$(jq -r '.CLOUDFLARE_API_TOKEN.data.value' "$HERCULES_CI_SECRETS_JSON")" + # Secrets preamble — Pattern C'-refined (ADR-002): + # extract CLOUDFLARE_ACCOUNT_ID from $HERCULES_CI_SECRETS_JSON at .data.value envelope. + export CLOUDFLARE_ACCOUNT_ID="$(jq -r '.CLOUDFLARE_ACCOUNT_ID.data.value' "$HERCULES_CI_SECRETS_JSON")" + + # Env-var-contract guard: fail fast if the secrets bundle is + # missing either Cloudflare key. Message excludes the value; + # only key name is echoed (VAL-EFFECT-DEPLOYDOCS-21). + if [ -z "''${CLOUDFLARE_API_TOKEN:-}" ] || [ "$CLOUDFLARE_API_TOKEN" = "null" ]; then + echo "error: CLOUDFLARE_API_TOKEN missing from \$HERCULES_CI_SECRETS_JSON" >&2 + exit 1 + fi + if [ -z "''${CLOUDFLARE_ACCOUNT_ID:-}" ] || [ "$CLOUDFLARE_ACCOUNT_ID" = "null" ]; then + echo "error: CLOUDFLARE_ACCOUNT_ID missing from \$HERCULES_CI_SECRETS_JSON" >&2 + exit 1 + fi + + # Option Gamma store-path dispatch — the `deploy-docs` flake + # app's /nix/store path is embedded at eval time via + # the perSystem config.apps.deploy-docs.program attribute. + # No flake-app shell-out (bwrap would not resolve .#). + DEPLOY_DOCS=${deployDocsProgram} + + ${ + if isMain then + '' + # Main branch → promote-by-SHA path. + # deploy.sh's `production` subcommand looks up a Worker + # version whose workers/tag annotation matches the + # current commit short-SHA-12 (uploaded earlier on the + # pre-merge branch push). If found, it promotes via + # `wrangler versions deploy @100%` (no re-upload → + # VAL-EFFECT-DEPLOYDOCS-16). If absent, it falls back + # to a fresh deploy + promote, logged with the literal + # substring "falling back to direct deploy". The + # dispatcher re-emits a DEPLOY-DOCS-ACTION banner to + # disambiguate the two execution paths for log-grep. + deploy_log="$(mktemp -t deploy-docs-prod.XXXXXX.log)" + set +e + "$DEPLOY_DOCS" production 2>&1 | tee "$deploy_log" + deploy_rc=''${PIPESTATUS[0]} + set -e + if grep -q "falling back to direct deploy" "$deploy_log"; then + echo "DEPLOY-DOCS-ACTION: fresh-deploy-and-promote" + fi + if [ "$deploy_rc" -ne 0 ]; then + echo "error: deploy-docs production exited $deploy_rc" >&2 + exit "$deploy_rc" + fi + '' + else + '' + # Non-main → preview upload path. + # deploy.sh's `preview ` subcommand uploads a + # new Cloudflare Workers version tagged with the + # commit short-SHA-12, aliased at + # b--infra-docs.sciexp.workers.dev. + # The script emits a `Preview URL:` line on success + # which we parse + re-emit as a structured banner + # (DEPLOY-DOCS-PREVIEW-URL) for downstream 200-probes. + preview_log="$(mktemp -t deploy-docs-preview.XXXXXX.log)" + set +e + "$DEPLOY_DOCS" preview ${lib.escapeShellArg previewBranchArg} 2>&1 | tee "$preview_log" + upload_rc=''${PIPESTATUS[0]} + set -e + preview_url="$(grep -oE 'Preview URL: https://[^[:space:]]+' "$preview_log" | head -1 | awk '{print $3}' || true)" + if [ -n "$preview_url" ]; then + echo "DEPLOY-DOCS-PREVIEW-URL: $preview_url" + else + echo "warning: could not parse preview URL from deploy.sh output" >&2 + fi + if [ "$upload_rc" -ne 0 ]; then + echo "error: deploy-docs preview exited $upload_rc" >&2 + exit "$upload_rc" + fi + '' + } + + echo "=== deploy-docs effect complete (exit 0) ===" + ''; + } + ); + }; +} From a74466625a94c16acb5e7dc94d473293435932d7 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Fri, 24 Apr 2026 18:29:56 -0400 Subject: [PATCH 28/77] fix(apps/docs,effects): env-first GIT_* env-var contract to bypass bwrap-sandbox git access MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extends the ADR-002 env-var-contract pattern from secrets+config to git metadata, resolving the runtime regression where deploy.sh failed inside the buildbot-effects bwrap sandbox with 'fatal: not a git repository' (upstream-by-design: hercules-ci-effects does not bind-mount the working tree). modules/apps/docs/deploy.sh - Declare six new GIT_* env vars (GIT_REV, GIT_REV_SHORT, GIT_REV_SHORT12, GIT_BRANCH, GIT_COMMIT_MSG, GIT_WORKTREE_STATUS) in the top-of-file env-var contract block alongside the existing CLOUDFLARE_* / DOCS_* declarations. - Replace unconditional git calls with env-first / git-fallback parameter expansions (${GIT_X:-$(git ... 2>/dev/null || true)}) on the six commit-metadata derivations; error-tolerant so a missing .git in the sandbox no longer causes non-zero exit. - Drop the 'repo_root=$(git rev-parse --show-toplevel); cd "$repo_root"' preamble entirely — wrangler is invoked with absolute '--config' so CWD is immaterial, and 'git rev-parse --show-toplevel' fails in the bwrap sandbox. modules/effects/vanixiets/herculesCI/deploy-docs.nix - Extend the effect preamble to export the six GIT_* variables, interpolated at eval time from herculesCI.config.repo.{rev, shortRev,branch} via lib.escapeShellArg and builtins.substring (GIT_REV_SHORT12 computed at eval time, no runtime git needed). - GIT_WORKTREE_STATUS is hard-coded 'clean' because effect runs always dispatch from a committed revision (pristine checkout). - Document the symmetric GIT_* env-var contract in the module header comment, mirroring the Pattern C'-refined secrets preamble. Verified: - nix eval ... effects.deploy-docs.effectScript | rg 'export GIT_' emits all six GIT_* exports with the expected interpolated values - /tmp sandbox simulation with env-i + GIT_* env pre-populated bypasses the git-resolution path and reaches wrangler without any 'fatal: not a git repository' output - Local shell from the worktree (GIT_* unset, .git reachable) falls through to the git fallbacks and still succeeds - nix run .#deploy-docs -- --help works, writeShellApplication shellcheck pass at build time Fulfills VAL-EFFECT-DEPLOYDOCS-27 (regression guard for the 'fatal: not a git repository' failure in the bwrap sandbox). --- modules/apps/docs/deploy.sh | 74 +++++++++++++++---- .../vanixiets/herculesCI/deploy-docs.nix | 28 +++++++ 2 files changed, 87 insertions(+), 15 deletions(-) diff --git a/modules/apps/docs/deploy.sh b/modules/apps/docs/deploy.sh index d63ca2e21..d5e2f21e1 100644 --- a/modules/apps/docs/deploy.sh +++ b/modules/apps/docs/deploy.sh @@ -14,6 +14,25 @@ # ($out/{dist/, .wrangler/, wrangler.jsonc}) # DOCS_NODE_MODULES store path of vanixiets-docs-deps node_modules # tree (runtimeEnv of deploy.nix) +# Optional (git metadata; env-first with git-fallback) — symmetric to the +# secrets contract above. Extended in the m4-deploy-docs-git-env-contract +# feature to let the script run inside the buildbot-effects bwrap sandbox +# which does NOT bind-mount the working tree (upstream-by-design). Every +# GIT_* consumer below is expressed as `${GIT_X:-$(git … 2>/dev/null || true)}` +# so the three supported caller contexts all work: effect preamble (env +# pre-populated, no .git reachable), local shell from a git worktree (env +# unset, git fallback), and GHA after checkout (env unset, git fallback). +# GIT_REV 40-char commit SHA (fallback: git rev-parse HEAD) +# GIT_REV_SHORT 7-ish-char short SHA (fallback: git rev-parse --short HEAD) +# GIT_REV_SHORT12 12-char short SHA used for wrangler --tag / +# workers/tag cross-check (fallback: +# git rev-parse --short=12 HEAD). VAL-WRITESHELL-DOCS-010 +# commit_tag invariant sources from here. +# GIT_BRANCH current branch name (fallback: git branch --show-current) +# GIT_COMMIT_MSG HEAD subject line used in the version-message +# annotation (fallback: git log -1 --pretty=format:'%s') +# GIT_WORKTREE_STATUS literal "clean" or "dirty" (fallback: +# git diff-index --quiet HEAD -- && echo clean || echo dirty) # Optional: # WRANGLER override binary path for test harnesses; default # $DOCS_NODE_MODULES/.bin/wrangler @@ -31,11 +50,16 @@ # caller-side sops decrypt inside a nix-develop wrapper; # the age key is provided via the step `env:` block from # the repo secrets (see deploy-docs.yaml). -# - M4 effect: preview-docs-deploy / production-docs-deploy effect -# preamble extracts CLOUDFLARE_API_TOKEN and -# CLOUDFLARE_ACCOUNT_ID from $HERCULES_CI_SECRETS_JSON -# and exports before invoking the embedded store path -# ${config.apps.deploy-docs.program}. +# - M4 effect: the deploy-docs dispatcher effect preamble extracts +# CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID from +# $HERCULES_CI_SECRETS_JSON and ALSO exports the six GIT_* +# variables interpolated from herculesCI.config.repo.* +# (via lib.escapeShellArg + builtins.substring at eval time) +# before invoking the embedded store path +# ${config.apps.deploy-docs.program}. The bwrap sandbox +# does not bind-mount the working tree, so every git +# command in this script is env-first with error-tolerant +# fallback (`git … 2>/dev/null || true`). # # Secret passing rule (per ADR-002): wrangler authentication flows ONLY # through inherited env vars; no authentication CLI flags are used. @@ -130,9 +154,11 @@ export WRANGLER="${WRANGLER:-$DOCS_NODE_MODULES/.bin/wrangler}" # Empirical: diagnosed 2026-04-22 via magnetite linux-x64 reproducer; # same machine + wrangler runs fine under real node, hangs under bun. -# Resolve repo root so git metadata commands work independently of callsite. -repo_root=$(git rev-parse --show-toplevel) -cd "$repo_root" +# Git metadata is resolved below via env-first / git-fallback (see top-of- +# file env-var contract for GIT_*). Wrangler is invoked with absolute +# `--config "$WRANGLER_CONFIG"`, so CWD is immaterial — no `cd` into the +# worktree is required (and would fail inside the buildbot-effects bwrap +# sandbox, which does not bind-mount the working tree). # Materialise a writable copy of the nix payload. wrangler reads # .wrangler/deploy/config.json whose configPath ("../../dist/server/wrangler.json") @@ -156,11 +182,16 @@ chmod -R u+w "$tmpdir" # present in the source wrangler.jsonc. wrangler_config="$tmpdir/dist/server/wrangler.json" -# Commit metadata shared by preview and production subcommands. -commit_sha=$(git rev-parse HEAD) -commit_tag=$(git rev-parse --short=12 HEAD) -commit_short=$(git rev-parse --short HEAD) -current_branch=$(git branch --show-current || true) +# Commit metadata shared by preview and production subcommands. Env-first / +# git-fallback per the GIT_* env-var contract (see top-of-file header). +# All `git` invocations are guarded by `2>/dev/null || true` so that a +# missing .git (e.g. buildbot-effects bwrap sandbox, no bind-mounted worktree) +# surfaces as empty strings rather than a non-zero exit; the env-first path +# supplies the authoritative values in that context. +commit_sha="${GIT_REV:-$(git rev-parse HEAD 2>/dev/null || true)}" +commit_tag="${GIT_REV_SHORT12:-$(git rev-parse --short=12 HEAD 2>/dev/null || true)}" +commit_short="${GIT_REV_SHORT:-$(git rev-parse --short HEAD 2>/dev/null || true)}" +current_branch="${GIT_BRANCH:-$(git branch --show-current 2>/dev/null || true)}" # Compose deploy message (prefer GitHub Actions context, fall back to local). if [[ -n "${GITHUB_ACTIONS:-}" ]]; then @@ -190,8 +221,21 @@ case "$mode" in | sed 's/--*/-/g; s/^-//; s/-$//' \ | cut -c1-40) - commit_msg=$(git log -1 --pretty=format:'%s') - git_status=$(git diff-index --quiet HEAD -- && echo "clean" || echo "dirty") + # Env-first / git-fallback — see top-of-file GIT_* env-var contract. + # `git log` / `git diff-index` are error-tolerant so a missing .git + # (buildbot-effects bwrap) leaves commit_msg empty; the effect preamble + # supplies GIT_COMMIT_MSG and GIT_WORKTREE_STATUS=clean in that case. + commit_msg="${GIT_COMMIT_MSG:-$(git log -1 --pretty=format:'%s' 2>/dev/null || true)}" + if [[ -n "${GIT_WORKTREE_STATUS:-}" ]]; then + git_status="$GIT_WORKTREE_STATUS" + elif git diff-index --quiet HEAD -- 2>/dev/null; then + git_status="clean" + else + # Non-zero from `git diff-index` covers both "dirty worktree" and + # "not a git repository" — collapse both to "dirty" so downstream + # version_message is always well-formed. + git_status="dirty" + fi version_message="[${branch}] ${commit_msg} (${commit_tag}, ${git_status})" echo "Deploying preview for branch: ${branch}" diff --git a/modules/effects/vanixiets/herculesCI/deploy-docs.nix b/modules/effects/vanixiets/herculesCI/deploy-docs.nix index a87c99ebb..de4f6e7b3 100644 --- a/modules/effects/vanixiets/herculesCI/deploy-docs.nix +++ b/modules/effects/vanixiets/herculesCI/deploy-docs.nix @@ -28,6 +28,18 @@ # emits this shape). Never extracts SOPS_AGE_KEY (ADR-002 # exclusivity) and never references NPM_TOKEN. # +# Symmetric GIT_* env-var contract (m4-deploy-docs-git-env-contract): +# Exports GIT_REV, GIT_REV_SHORT, GIT_REV_SHORT12, GIT_BRANCH, +# GIT_COMMIT_MSG, GIT_WORKTREE_STATUS from herculesCI.config.repo.* +# at eval time via lib.escapeShellArg + builtins.substring. Required +# because the buildbot-effects bwrap sandbox does not bind-mount the +# working tree (upstream-by-design across all 3 reference effect +# implementations); deploy.sh's git invocations would fail with +# `fatal: not a git repository` without these exports. Symmetric to +# the secrets env-var contract: the script declares env-first / +# git-fallback on every GIT_* consumer, and this preamble supplies +# the authoritative values for the sandboxed path. +# # Posture A outer gate: # Fork-PR exposure is blocked by `effects_on_pull_requests = false` # + `effects_branches` in `buildbot-nix.toml`, which precedes this @@ -108,6 +120,22 @@ # extract CLOUDFLARE_ACCOUNT_ID from $HERCULES_CI_SECRETS_JSON at .data.value envelope. export CLOUDFLARE_ACCOUNT_ID="$(jq -r '.CLOUDFLARE_ACCOUNT_ID.data.value' "$HERCULES_CI_SECRETS_JSON")" + # Git-metadata env-var contract (m4-deploy-docs-git-env-contract): + # interpolate the six GIT_* values from herculesCI.config.repo.* at + # eval time so deploy.sh's env-first / git-fallback consumers never + # need to shell out to `git` inside the bwrap sandbox. GIT_REV_SHORT12 + # is computed from the full rev via builtins.substring (not a runtime + # `git rev-parse --short=12`, which would fail on a missing .git). + # GIT_WORKTREE_STATUS is hard-coded "clean" because an effect run + # always dispatches from a committed revision (hercules-ci-effects + # fetches a pristine checkout). + export GIT_REV=${lib.escapeShellArg (toString rev)} + export GIT_REV_SHORT=${lib.escapeShellArg (toString shortRev)} + export GIT_REV_SHORT12=${lib.escapeShellArg (builtins.substring 0 12 (toString rev))} + export GIT_BRANCH=${lib.escapeShellArg (if branch == null then "" else toString branch)} + export GIT_COMMIT_MSG=${lib.escapeShellArg "effect deploy from rev ${toString shortRev}"} + export GIT_WORKTREE_STATUS=clean + # Env-var-contract guard: fail fast if the secrets bundle is # missing either Cloudflare key. Message excludes the value; # only key name is echoed (VAL-EFFECT-DEPLOYDOCS-21). From 20e772066c5de0ef3638bf3a0d3a9ab59d33ef32 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Fri, 24 Apr 2026 19:25:56 -0400 Subject: [PATCH 29/77] fix(apps/docs,effects): extend env-var contract to DEPLOY_HOST/DEPLOY_DEPLOYER + harden runtimeInputs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Generalise the env-first / shelled-fallback pattern (introduced in a74466625 for GIT_*) to cover the second class of host-PATH binary regressions surfaced by the buildbot-effects bwrap sandbox: missing `hostname` and `whoami` binaries (exit 127 on unconditional invocation). modules/apps/docs/deploy.sh: - Replace `hostname -s` with `${DEPLOY_HOST:-${HOSTNAME%%.*}}`. The bash builtin $HOSTNAME is populated from gethostname(2) at shell startup and requires no external binary in any context; ${HOSTNAME%%.*} mimics `hostname -s` via parameter expansion. - Replace `whoami` with ${DEPLOY_DEPLOYER:-${GITHUB_ACTOR:-$(whoami 2>/dev/null || echo unknown)}}. - Update the top-of-file env-var contract block to declare all 12 caller-overridable variables (4 tokens + 6 GIT_* + 2 DEPLOY_*) with semantics, defaults, and provider mechanism for each caller context. modules/apps/docs/deploy.nix: - Add pkgs.gnugrep, pkgs.gnused, pkgs.gawk, pkgs.findutils to runtimeInputs. sed/awk/grep/find are invoked by deploy.sh and were previously supplied implicitly by hercules-ci-effects' default sandbox PATH; making them explicit closes the latent coupling and makes the writeShellApplication self-sufficient under any caller context. modules/effects/vanixiets/herculesCI/deploy-docs.nix: - Export DEPLOY_DEPLOYER=hercules-ci-effects and DEPLOY_HOST=magnetite in the effect preamble after the GIT_* block. - Generalise the header comment from 'Symmetric GIT_* env-var contract' to 'Symmetric env-var contract (GIT_* + DEPLOY_*)'. Verified via local sandbox simulation with PATH=/nonexistent — script runs without 'fatal: not a git repository', 'hostname: command not found', or 'whoami: ...' errors, all env vars resolved through the contract chain. --- modules/apps/docs/deploy.nix | 12 ++ modules/apps/docs/deploy.sh | 140 +++++++++++++----- .../vanixiets/herculesCI/deploy-docs.nix | 35 +++-- 3 files changed, 141 insertions(+), 46 deletions(-) diff --git a/modules/apps/docs/deploy.nix b/modules/apps/docs/deploy.nix index 332898152..51683b6e6 100644 --- a/modules/apps/docs/deploy.nix +++ b/modules/apps/docs/deploy.nix @@ -35,11 +35,23 @@ # Per ADR-002 env-var contract: secrets flow via inherited env # (never via `sops exec-env` inside the script), so pkgs.sops / # pkgs.age are no longer required runtime inputs. + # + # Hermeticity (m4-deploy-docs-git-env-contract): sed/awk/grep/find + # are explicitly declared even though hercules-ci-effects' default + # sandbox PATH supplies them implicitly. Closing this latent + # coupling makes the writeShellApplication self-sufficient under + # ANY caller context (not just the bwrap sandbox), satisfying the + # writeShellApplication invariant that PATH is exactly equal to + # runtimeInputs at runtime. runtimeInputs = [ pkgs.nodejs_24 pkgs.jq pkgs.coreutils pkgs.git + pkgs.gnugrep + pkgs.gnused + pkgs.gawk + pkgs.findutils ]; runtimeEnv = { DOCS_NODE_MODULES = "${config.packages.vanixiets-docs-deps}/packages/docs/node_modules"; diff --git a/modules/apps/docs/deploy.sh b/modules/apps/docs/deploy.sh index d5e2f21e1..88fc2c4d3 100644 --- a/modules/apps/docs/deploy.sh +++ b/modules/apps/docs/deploy.sh @@ -2,26 +2,43 @@ # shellcheck shell=bash # Docs deployment dispatcher invoked via `nix run .#deploy-docs`. # -# Env-var contract (per ADR-002 / env-var-contract-design.md §2.1): -# Required (secret, provided by caller): -# CLOUDFLARE_API_TOKEN wrangler auth token -# CLOUDFLARE_ACCOUNT_ID Cloudflare account id (wrangler requires this -# for account-scoped operations such as -# `versions upload` on a Worker attached to an -# account-level resource) +# Env-var contract (per ADR-002 / env-var-contract-design.md §2.1, extended +# by the m4-deploy-docs-git-env-contract feature to ALL host-PATH binary +# dependencies). 12 caller-overridable variables: 4 secret tokens (the +# closed effects bundle), 6 GIT_*, 2 DEPLOY_*. Symmetric env-first / +# shelled-fallback shape across all four caller contexts (effect preamble, +# sops exec-env, direnv, GHA env, local shell). +# +# Required (secret, provided by caller from the closed 4-key effects +# bundle — see modules/effects/vanixiets/secrets.nix): +# CLOUDFLARE_API_TOKEN wrangler auth token (CONSUMED by this +# script; required at runtime). +# CLOUDFLARE_ACCOUNT_ID Cloudflare account id (CONSUMED; wrangler +# requires this for account-scoped ops such +# as `versions upload` on a Worker attached +# to an account-level resource). +# GITHUB_TOKEN not consumed by deploy.sh; documented as +# part of the canonical effects bundle for +# homogeneity (consumed by release.sh). +# SOPS_AGE_KEY not consumed by deploy.sh; documented as +# part of the canonical effects bundle for +# homogeneity (consumed by +# k3d-bootstrap-secrets.sh). Per ADR-002, +# this script does NOT shell out to sops. # Required (config, injected by deploy.nix): # DOCS_PAYLOAD store path of the vanixiets-docs derivation # ($out/{dist/, .wrangler/, wrangler.jsonc}) # DOCS_NODE_MODULES store path of vanixiets-docs-deps node_modules # tree (runtimeEnv of deploy.nix) -# Optional (git metadata; env-first with git-fallback) — symmetric to the -# secrets contract above. Extended in the m4-deploy-docs-git-env-contract -# feature to let the script run inside the buildbot-effects bwrap sandbox -# which does NOT bind-mount the working tree (upstream-by-design). Every -# GIT_* consumer below is expressed as `${GIT_X:-$(git … 2>/dev/null || true)}` -# so the three supported caller contexts all work: effect preamble (env -# pre-populated, no .git reachable), local shell from a git worktree (env -# unset, git fallback), and GHA after checkout (env unset, git fallback). +# +# Optional (git metadata; env-first with git-fallback). Extended in the +# m4-deploy-docs-git-env-contract feature to let the script run inside +# the buildbot-effects bwrap sandbox which does NOT bind-mount the +# working tree (upstream-by-design). Every GIT_* consumer below is +# expressed as `${GIT_X:-$(git … 2>/dev/null || true)}` so the three +# supported caller contexts all work: effect preamble (env pre-populated, +# no .git reachable), local shell from a git worktree (env unset, git +# fallback), and GHA after checkout (env unset, git fallback). # GIT_REV 40-char commit SHA (fallback: git rev-parse HEAD) # GIT_REV_SHORT 7-ish-char short SHA (fallback: git rev-parse --short HEAD) # GIT_REV_SHORT12 12-char short SHA used for wrangler --tag / @@ -33,33 +50,60 @@ # annotation (fallback: git log -1 --pretty=format:'%s') # GIT_WORKTREE_STATUS literal "clean" or "dirty" (fallback: # git diff-index --quiet HEAD -- && echo clean || echo dirty) -# Optional: +# +# Optional (deploy-context metadata; env-first with bash-builtin / +# shelled-fallback). Generalisation of the same pattern to ALL host-PATH +# binary dependencies, fixing the second-bug-class regression where the +# bwrap sandbox lacks `hostname`/`whoami` on PATH (only /nix/store +# ro-bind + writeShellApplication runtimeInputs are available). The +# bash builtin `$HOSTNAME` is populated from gethostname(2) at shell +# startup — no external binary required in any context. +# DEPLOY_HOST short hostname for the production deploy +# message. Fallback: ${HOSTNAME%%.*} +# (bash builtin parameter expansion; trims +# the first dot-suffix to mimic `hostname -s` +# without shelling out). +# DEPLOY_DEPLOYER actor identity for deploy/version messages. +# Fallback chain: GITHUB_ACTOR (GHA context) +# → `whoami 2>/dev/null` (local shell with +# /etc/passwd available) → "unknown". +# +# Optional (caller debugging / overrides): # WRANGLER override binary path for test harnesses; default # $DOCS_NODE_MODULES/.bin/wrangler # DEPLOY_DOCS_DEBUG preserve tmpdir on exit when set # GITHUB_ACTIONS / GITHUB_ACTOR / GITHUB_WORKFLOW -# prefix the deploy message with GHA context +# when GITHUB_ACTIONS is set, the production +# deploy message uses GITHUB_WORKFLOW (default +# "CI") as the deploy context instead of +# DEPLOY_HOST. GITHUB_ACTOR participates in +# the DEPLOY_DEPLOYER fallback chain. # -# Caller mechanisms (satisfy the secret-env contract via one of): +# Caller mechanisms (satisfy each contract slot via one of): # - Local dev: caller-side sops wrapper (justfile `docs-deploy-*` # recipes wrap with `sops` to decrypt secrets/shared.yaml # and export the Cloudflare env before the nested nix run) # OR direnv dotenv (.envrc loads .env with the Cloudflare -# env vars already exported). +# env vars already exported). DEPLOY_HOST / DEPLOY_DEPLOYER +# left unset → bash-builtin / whoami fallback. # - GHA env: deploy-docs.yaml step wraps the nix run with the same # caller-side sops decrypt inside a nix-develop wrapper; # the age key is provided via the step `env:` block from -# the repo secrets (see deploy-docs.yaml). +# the repo secrets (see deploy-docs.yaml). GIT_* / DEPLOY_* +# left unset → git/bash-builtin/GITHUB_ACTOR fallback. # - M4 effect: the deploy-docs dispatcher effect preamble extracts # CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID from -# $HERCULES_CI_SECRETS_JSON and ALSO exports the six GIT_* +# $HERCULES_CI_SECRETS_JSON, exports the six GIT_* # variables interpolated from herculesCI.config.repo.* -# (via lib.escapeShellArg + builtins.substring at eval time) -# before invoking the embedded store path -# ${config.apps.deploy-docs.program}. The bwrap sandbox -# does not bind-mount the working tree, so every git -# command in this script is env-first with error-tolerant -# fallback (`git … 2>/dev/null || true`). +# (via lib.escapeShellArg + builtins.substring at eval +# time), AND exports DEPLOY_DEPLOYER=hercules-ci-effects +# and DEPLOY_HOST=magnetite before invoking the embedded +# store path ${config.apps.deploy-docs.program}. The +# bwrap sandbox does not bind-mount the working tree +# and provides no host-PATH binaries beyond /nix/store +# ro-bind + runtimeInputs PATH, so every git/hostname/ +# whoami consumer in this script is env-first with +# bash-builtin or error-tolerant shelled fallback. # # Secret passing rule (per ADR-002): wrangler authentication flows ONLY # through inherited env vars; no authentication CLI flags are used. @@ -94,18 +138,29 @@ Flags: --help, -h Print this usage and exit 0. Environment contract (see top-of-file header for full details): - Required (secret, caller-provided): - CLOUDFLARE_API_TOKEN wrangler auth token - CLOUDFLARE_ACCOUNT_ID Cloudflare account id (account-scoped ops) + Required (secret, caller-provided from the closed 4-key effects bundle): + CLOUDFLARE_API_TOKEN wrangler auth token (CONSUMED) + CLOUDFLARE_ACCOUNT_ID Cloudflare account id (CONSUMED; account-scoped ops) + GITHUB_TOKEN bundle homogeneity (not consumed by deploy.sh) + SOPS_AGE_KEY bundle homogeneity (not consumed by deploy.sh) Required (config, injected by deploy.nix): DOCS_PAYLOAD path to the vanixiets-docs derivation output DOCS_NODE_MODULES path to vanixiets-docs-deps node_modules tree - Optional: + Optional (env-first with shelled-fallback): + GIT_REV, GIT_REV_SHORT, GIT_REV_SHORT12, GIT_BRANCH, + GIT_COMMIT_MSG, GIT_WORKTREE_STATUS + git metadata; supplied by effect preamble when no + .git is reachable; otherwise resolved via `git ...`. + DEPLOY_HOST short hostname; fallback `${HOSTNAME%%.*}` (bash + builtin, no external binary). + DEPLOY_DEPLOYER actor identity; fallback chain GITHUB_ACTOR → + `whoami 2>/dev/null` → "unknown". + Optional (caller debugging / overrides): WRANGLER, DEPLOY_DOCS_DEBUG GITHUB_ACTIONS / GITHUB_ACTOR / GITHUB_WORKFLOW - When set, the production deploy message is prefixed with - the GitHub Actions context; otherwise whoami and hostname - are used. + When GITHUB_ACTIONS is set, the production deploy + message uses the GitHub Actions context (workflow + name) instead of DEPLOY_HOST. Examples: nix run .#deploy-docs -- preview my-feature-branch @@ -193,14 +248,25 @@ commit_tag="${GIT_REV_SHORT12:-$(git rev-parse --short=12 HEAD 2>/dev/null || tr commit_short="${GIT_REV_SHORT:-$(git rev-parse --short HEAD 2>/dev/null || true)}" current_branch="${GIT_BRANCH:-$(git branch --show-current 2>/dev/null || true)}" +# Resolve deployer / deploy_host with env-first / bash-builtin / shelled-fallback +# per the DEPLOY_* env-var contract (see top-of-file header). The bwrap +# sandbox provides no host-PATH binaries beyond /nix/store ro-bind + +# runtimeInputs PATH, so unconditional `hostname -s` / `whoami` would fail +# with `command not found` (exit 127). Bash builtin `$HOSTNAME` is populated +# from gethostname(2) at shell startup and requires no external binary in +# any context; `${HOSTNAME%%.*}` mimics `hostname -s` via parameter +# expansion. `whoami` is retained as a final fallback for local shells where +# DEPLOY_DEPLOYER and GITHUB_ACTOR are both unset; it is error-tolerant +# (`2>/dev/null || echo unknown`) so a missing /etc/passwd entry surfaces +# as "unknown" rather than a non-zero exit. +deploy_host="${DEPLOY_HOST:-${HOSTNAME%%.*}}" +deployer="${DEPLOY_DEPLOYER:-${GITHUB_ACTOR:-$(whoami 2>/dev/null || echo unknown)}}" + # Compose deploy message (prefer GitHub Actions context, fall back to local). if [[ -n "${GITHUB_ACTIONS:-}" ]]; then - deployer="${GITHUB_ACTOR:-github-actions}" deploy_context="${GITHUB_WORKFLOW:-CI}" deploy_msg="Deployed by ${deployer} from ${current_branch} via ${deploy_context}" else - deployer=$(whoami) - deploy_host=$(hostname -s) deploy_msg="Deployed by ${deployer} from ${current_branch} on ${deploy_host}" fi diff --git a/modules/effects/vanixiets/herculesCI/deploy-docs.nix b/modules/effects/vanixiets/herculesCI/deploy-docs.nix index de4f6e7b3..1f0e6e2e8 100644 --- a/modules/effects/vanixiets/herculesCI/deploy-docs.nix +++ b/modules/effects/vanixiets/herculesCI/deploy-docs.nix @@ -28,17 +28,22 @@ # emits this shape). Never extracts SOPS_AGE_KEY (ADR-002 # exclusivity) and never references NPM_TOKEN. # -# Symmetric GIT_* env-var contract (m4-deploy-docs-git-env-contract): +# Symmetric env-var contract (GIT_* + DEPLOY_*) (m4-deploy-docs-git-env-contract): # Exports GIT_REV, GIT_REV_SHORT, GIT_REV_SHORT12, GIT_BRANCH, # GIT_COMMIT_MSG, GIT_WORKTREE_STATUS from herculesCI.config.repo.* -# at eval time via lib.escapeShellArg + builtins.substring. Required -# because the buildbot-effects bwrap sandbox does not bind-mount the -# working tree (upstream-by-design across all 3 reference effect -# implementations); deploy.sh's git invocations would fail with -# `fatal: not a git repository` without these exports. Symmetric to -# the secrets env-var contract: the script declares env-first / -# git-fallback on every GIT_* consumer, and this preamble supplies -# the authoritative values for the sandboxed path. +# at eval time via lib.escapeShellArg + builtins.substring AND exports +# DEPLOY_DEPLOYER=hercules-ci-effects and DEPLOY_HOST=magnetite (the +# execution host on which buildbot-master schedules effect runs). +# Required because the buildbot-effects bwrap sandbox does not +# bind-mount the working tree AND provides no host-PATH binaries +# beyond /nix/store ro-bind + writeShellApplication runtimeInputs PATH +# (upstream-by-design across all 3 reference effect implementations); +# deploy.sh's git invocations would fail with `fatal: not a git +# repository` and its `hostname -s` / `whoami` invocations would fail +# with `command not found` (exit 127) without these exports. Symmetric +# to the secrets env-var contract: the script declares env-first / +# bash-builtin / shelled-fallback on every consumer, and this preamble +# supplies the authoritative values for the sandboxed path. # # Posture A outer gate: # Fork-PR exposure is blocked by `effects_on_pull_requests = false` @@ -136,6 +141,18 @@ export GIT_COMMIT_MSG=${lib.escapeShellArg "effect deploy from rev ${toString shortRev}"} export GIT_WORKTREE_STATUS=clean + # Deploy-context env-var contract (m4-deploy-docs-git-env-contract): + # supply DEPLOY_DEPLOYER and DEPLOY_HOST so deploy.sh's bash-builtin / + # shelled-fallback chain never has to invoke `whoami` / `hostname` — + # neither binary is on PATH inside the bwrap sandbox (only + # /nix/store ro-bind + runtimeInputs). Hard-coded values are + # appropriate here because every effect run is dispatched by the + # hercules-ci-effects framework on magnetite (the buildbot-nix + # worker host); divergent values would indicate a misconfigured + # effect runner, not a per-run difference worth surfacing. + export DEPLOY_DEPLOYER=hercules-ci-effects + export DEPLOY_HOST=magnetite + # Env-var-contract guard: fail fast if the secrets bundle is # missing either Cloudflare key. Message excludes the value; # only key name is echoed (VAL-EFFECT-DEPLOYDOCS-21). From 7163add4b9711562d025af2eb10c8cb208b65b78 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Fri, 24 Apr 2026 19:47:42 -0400 Subject: [PATCH 30/77] feat(effects): add release-packages branch-dispatcher effect MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Declare herculesCI.onPush.default.outputs.effects.release-packages as a single dispatcher that iterates packages/* and runs semantic-release per package. Replaces 3 former per-job features (preview-release-version, production-release-packages-dryrun, production-release-packages-cutover). - Option Gamma store-path embedding for both list-packages-json and release flake apps (no nix run shell-out from inside effect) - Branch dispatch: primaryRepo.branch == "main" → real semantic-release; any other branch → semantic-release --dry-run (no git mutations) - Per-package atomicity: failures don't block remaining packages, but the effect exits non-zero at end iff any package failed - Pattern C'-refined secrets preamble extracts GITHUB_TOKEN only; SOPS_AGE_KEY exclusivity rule and NPM_TOKEN never-rule are documented in outer module comments (kept out of rendered effectScript to satisfy VAL-EFFECT-RELEASEPACKAGES-22 / -24 zero-match contracts) - Structured banners: RELEASE-PACKAGES-ACTION (per-run path), and RELEASE-PACKAGE-{ITERATION,OK,FAILURE} (per-package, RP-05/-06/-19 log-grep anchors) --- .../vanixiets/herculesCI/release-packages.nix | 196 ++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 modules/effects/vanixiets/herculesCI/release-packages.nix diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix new file mode 100644 index 000000000..9f46498f7 --- /dev/null +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -0,0 +1,196 @@ +# effects.release-packages — semantic-release per-package dispatcher +# (M4 feature `m4-release-packages`). Consolidates three pre-consolidation +# jobs (preview-release-version, production-release-packages-dryrun, +# production-release-packages-cutover) into a single herculesCI effect +# that iterates every package discovered under `packages/*` and dispatches +# `semantic-release` with branch-aware behaviour. +# +# Design contract (see mission AGENTS.md "ADR-002 locked decisions" and +# `.factory/validation-contract.md` VAL-EFFECT-RELEASEPACKAGES-*): +# +# Option Gamma store-path embedding: +# The effect body invokes both the `list-packages-json` and `release` +# flake apps via the nix-eval-time resolved store paths +# `${config.apps.x86_64-linux.list-packages-json.program}` and +# `${config.apps.x86_64-linux.release.program}`. The effect never +# dispatches via a flake-app shell-out (`nix run`); bwrap does not +# bind the working tree, so the .# syntax cannot resolve. +# +# Branch dispatch (exact string equality): +# Selection is driven by `primaryRepo.branch == "main"`, surfaced +# through `herculesCI.config.repo.branch` which hercules-ci-effects +# populates from the primaryRepo record at flake.herculesCI entry. +# * primaryRepo.branch == "main" → real semantic-release per package +# (semantic-release's per-package commit-analyzer decides whether +# to cut a release; tag push + GitHub Release published when so; +# npmPublish stays false — never overridden). +# * any other branch (or null) → per-package `--dry-run` +# (semantic-release prints the next-version preview only; no git +# tags pushed, no GitHub Release created, no remote git mutation). +# +# Pattern C'-refined secrets preamble: +# Extracts GITHUB_TOKEN ONLY from $HERCULES_CI_SECRETS_JSON at the +# `.GITHUB_TOKEN.data.value` envelope (see +# modules/effects/vanixiets/secrets.nix for the generator that emits +# this shape). Per ADR-002 §5.3 exclusivity audit, this effect MUST +# NOT consume SOPS_AGE_KEY (load-bearing only for k3d-bootstrap-secrets) +# and MUST NOT reference NPM_TOKEN (not part of the closed 4-key +# bundle; npmPublish=false invariant means no npm publish surface). +# +# Per-package atomicity (NOT fail-fast): +# If package A fails, the loop continues to package B. Each +# per-package failure is recorded; the dispatcher exits non-zero at +# the end iff ANY package failed. Successful per-package tags + +# GitHub Releases persist (semantic-release's own per-package +# mutations are atomic per-package); failures surface via the +# RELEASE-PACKAGE-FAILURE banners for manual re-trigger. +# +# Posture A outer gate: +# Fork-PR exposure is blocked by `effects_on_pull_requests = false` +# in `buildbot-nix.toml`, which precedes this inner branch-dispatch. +# The dispatcher logic runs only after the outer gate has passed. +# +# Structured banners (RP-05 / RP-06 / RP-19 log-grep anchors): +# * `RELEASE-PACKAGES-ACTION: dry-run|release` emitted exactly once +# per run, identifying the dispatcher path taken (analogous to the +# deploy-docs DEPLOY-DOCS-ACTION banner). +# * `RELEASE-PACKAGE-ITERATION: ` emitted before each +# per-package release invocation (count must equal +# `just list-packages-json | jq length`). +# * `RELEASE-PACKAGE-OK: ` emitted on per-package success. +# * `RELEASE-PACKAGE-FAILURE: (exit )` emitted on +# per-package failure; the loop continues regardless. +{ + config, + inputs, + lib, + withSystem, + ... +}: +{ + herculesCI = + herculesCI: + let + # primaryRepo.branch (exposed as config.repo.branch by + # hercules-ci-effects' paramModule, populated from primaryRepo.branch + # at flake.herculesCI entry). Type: nullable string — null on tag + # pushes where branch is not populated. + branch = herculesCI.config.repo.branch; + shortRev = herculesCI.config.repo.shortRev; + rev = herculesCI.config.repo.rev; + + # Branch dispatch: exact string equality. null == "main" is false + # in Nix, so tag pushes naturally fall through to the dry-run path. + isMain = branch == "main"; + + # Action banner emitted once per run (RP-05 log-grep anchor). + actionBanner = if isMain then "release" else "dry-run"; + + # Eval-time --dry-run flag injection. Empty on main, "--dry-run" + # otherwise. Ordered AFTER the package-path positional arg per + # release.sh's CLI grammar (`release [--dry-run]`). + dryRunFlag = if isMain then "" else "--dry-run"; + in + { + onPush.default.outputs.effects.release-packages = withSystem "x86_64-linux" ( + { config, pkgs, ... }: + let + hci-effects = inputs.hercules-ci-effects.lib.withPkgs pkgs; + + # Option Gamma: resolved at nix eval time to /nix/store paths + # that the bwrap sandbox can execute without a working-tree or + # nix-daemon lookup. + listPackagesProgram = config.apps.list-packages-json.program; + releaseProgram = config.apps.release.program; + in + hci-effects.mkEffect { + name = "release-packages"; + + effectScript = '' + set -euo pipefail + + echo "=== effects.release-packages (semantic-release per-package dispatcher) ===" + echo "branch: ${lib.escapeShellArg (toString branch)}" + echo "rev: ${lib.escapeShellArg (toString rev)}" + echo "shortRev: ${lib.escapeShellArg (toString shortRev)}" + echo "isMain: ${if isMain then "true" else "false"}" + + # Structured banner (RP log-grep anchor): emitted once per run + # so log-grep can distinguish dry-run vs release dispatch path. + echo "RELEASE-PACKAGES-ACTION: ${actionBanner}" + + # Secrets preamble — Pattern C'-refined (ADR-002): + # extract GITHUB_TOKEN ONLY from $HERCULES_CI_SECRETS_JSON + # at the .data.value envelope. The other bundle keys are + # intentionally NOT extracted here — exclusivity rules and + # the npm-publish-never invariant are documented in the + # outer-module doc-comments (kept out of the rendered + # effectScript to satisfy VAL-EFFECT-RELEASEPACKAGES-22 / + # -24's `rg -c` zero-match contract on effectScript output). + export GITHUB_TOKEN="$(jq -r '.GITHUB_TOKEN.data.value' "$HERCULES_CI_SECRETS_JSON")" + + # Env-var-contract guard: fail fast if the secrets bundle is + # missing GITHUB_TOKEN. Message excludes the value; only key + # name is echoed. + if [ -z "''${GITHUB_TOKEN:-}" ] || [ "$GITHUB_TOKEN" = "null" ]; then + echo "error: GITHUB_TOKEN missing from \$HERCULES_CI_SECRETS_JSON" >&2 + exit 1 + fi + + # Option Gamma store-path dispatch — both flake apps' /nix/store + # paths are embedded at eval time via the perSystem + # config.apps..program attributes. No flake-app shell-out + # (bwrap would not resolve .#). + LIST_PACKAGES=${listPackagesProgram} + RELEASE=${releaseProgram} + + # Discover packages under packages/* (jq-driven enumeration of + # the list-packages-json output). + packages_json="$("$LIST_PACKAGES")" + echo "packages discovered: $packages_json" + + # Per-package failure tracker — populated inside the loop; + # used at end-of-loop to set the aggregate exit code. + failed_packages=() + + # Per-package atomicity loop: iterate every {name, path} + # entry. Read paths into the loop via process substitution + # over `jq -r '.[].path'` (one path per line, robust to + # paths-with-spaces unlike `for pkg in $(...)`). + while IFS= read -r pkg_path; do + [ -z "$pkg_path" ] && continue + + echo "RELEASE-PACKAGE-ITERATION: $pkg_path" + + # Disable -e for the per-package invocation so a single + # package's failure does not abort the loop. We capture + # the exit code, log appropriately, and continue. + set +e + "$RELEASE" "$pkg_path" ${dryRunFlag} + rc=$? + set -e + + if [ "$rc" -eq 0 ]; then + echo "RELEASE-PACKAGE-OK: $pkg_path" + else + echo "RELEASE-PACKAGE-FAILURE: $pkg_path (exit $rc)" + failed_packages+=("$pkg_path") + fi + done < <(printf '%s\n' "$packages_json" | jq -r '.[].path') + + # Aggregate exit code: OR of all per-package results. Zero + # iff every package's invocation exited zero (or zero + # packages were discovered, which is itself a degenerate + # success). Non-zero iff any package failed; the failed + # set is enumerated in stderr for operator triage. + if [ "''${#failed_packages[@]}" -gt 0 ]; then + echo "error: ''${#failed_packages[@]} package(s) failed: ''${failed_packages[*]}" >&2 + exit 1 + fi + + echo "=== release-packages effect complete (exit 0) ===" + ''; + } + ); + }; +} From c6f742f593a1dfaf455fb5000ffb6a768b550e5a Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Fri, 24 Apr 2026 20:10:34 -0400 Subject: [PATCH 31/77] fix(apps/release,effects): extend env-var contract to RELEASE_REPO_ROOT/CI/GIT_AUTHOR/COMMITTER + harden runtimeInputs Mirrors the m4-deploy-docs-git-env-contract precedent on release.sh / release-packages.nix to bypass three buildbot-effects bwrap-sandbox failure modes that would surface when release-packages runs end-to-end: 1. `fatal: not a git repository` from `git rev-parse --show-toplevel` (no .git working-tree mount in the sandbox). 2. `error: could not lock config file .git/config` from `git config user.{name,email}` writes (.git is RO). 3. `running on a CI environment is required` from semantic-release's env-ci default (sandbox is not a recognised CI provider). release.sh: - replace `repo_root=\$(git rev-parse --show-toplevel)` with an env-first / shelled-fallback expression `\${RELEASE_REPO_ROOT:-\$(git rev-parse --show-toplevel 2>/dev/null || pwd)}`. - replace the `git config user.{name,email}` mutating-write block with `export GIT_AUTHOR_NAME/EMAIL` + `export GIT_COMMITTER_NAME/EMAIL` parameter-expansion-default chains; git honours these natively. - retain GIT_USER_NAME/GIT_USER_EMAIL as transitional aliases. - extend the top-of-file env-var contract header to declare the new vars and document caller-mechanism mappings. release-packages.nix: - export CI=true, GIT_BRANCH (eval-time from herculesCI.config.repo.branch via lib.escapeShellArg), RELEASE_REPO_ROOT="\$PWD", and the GIT_AUTHOR_NAME/EMAIL + GIT_COMMITTER_NAME/EMAIL identity quartet in the effect preamble after the GITHUB_TOKEN guard. - record the contract in the module-level doc-comment symmetric to deploy-docs.nix's "Symmetric env-var contract (GIT_* + DEPLOY_*)". release.nix: - add pkgs.gnused, pkgs.gawk, pkgs.findutils to runtimeInputs to eliminate latent host-PATH coupling for any future semantic-release plugin / node_modules helper that shells out to sed/awk/find. Verification: - `nix run .#release -- --help` exits 0 and prints the updated env-var list. - `nix eval --raw .#apps.aarch64-darwin.release.program` resolves cleanly (build side-effect succeeds). - effectScript probe: `rg -c 'export GIT_AUTHOR_NAME='` = 1, `rg -c 'CI=true'` = 1. - isolation smoke: `env -i` invocation from /tmp with all env vars set exits without 'fatal: not a git repository' / 'could not lock config file' / 'CI environment is required'. - shellcheck modules/apps/release/release.sh: clean. - `nix flake check` aarch64-darwin baseline failure on `packages.aarch64-darwin.rgContainer-aarch64` is pre-existing (documented in AGENTS.md), unrelated to these changes. Local-only verification; end-to-end buildbot-effects run on magnetite awaits orchestrator-stage install-copy invocation per AGENTS.md "Pre-push effect validation". Fulfills VAL-EFFECT-RELEASEPACKAGES-29. --- modules/apps/release/release.nix | 12 ++ modules/apps/release/release.sh | 116 ++++++++++++++---- .../vanixiets/herculesCI/release-packages.nix | 39 ++++++ 3 files changed, 146 insertions(+), 21 deletions(-) diff --git a/modules/apps/release/release.nix b/modules/apps/release/release.nix index 223e4e8d9..172fef2d9 100644 --- a/modules/apps/release/release.nix +++ b/modules/apps/release/release.nix @@ -52,6 +52,18 @@ pkgs.jq pkgs.gnugrep pkgs.coreutils + # Hardening per m4-release-packages-runtime-deps-contract: + # explicitly declare every host-PATH binary that release.sh + # OR any transitive semantic-release plugin / node_modules + # helper might shell out to. The buildbot-effects bwrap + # sandbox provides only /nix/store ro-bind + writeShellApplication + # runtimeInputs PATH (no host PATH binaries); a missing input + # surfaces only at runtime as `command not found`. Symmetric + # to the deploy-docs runtimeInputs hardening done in the + # m4-deploy-docs-git-env-contract feature. + pkgs.gnused + pkgs.gawk + pkgs.findutils ]; runtimeEnv = { DOCS_NODE_MODULES = "${config.packages.vanixiets-docs-deps}/packages/docs/node_modules"; diff --git a/modules/apps/release/release.sh b/modules/apps/release/release.sh index 734b2cf02..d883b80cd 100644 --- a/modules/apps/release/release.sh +++ b/modules/apps/release/release.sh @@ -25,7 +25,11 @@ # needed). # --help Print this usage and exit 0. # -# Env-var contract (per ADR-002 / env-var-contract-design.md §2.3): +# Env-var contract (per ADR-002 / env-var-contract-design.md §2.3, extended +# by the m4-release-packages-runtime-deps-contract feature to ALL host-PATH +# binary dependencies and to .git-write avoidance — symmetric to the +# m4-deploy-docs-git-env-contract precedent on deploy.sh). +# # Required (secret, production path only — not --dry-run): # GITHUB_TOKEN @semantic-release/github auth for tag push and # release publish. Filtered-out plugin list under @@ -33,25 +37,83 @@ # Required (config, injected by release.nix runtimeEnv): # DOCS_NODE_MODULES vanixiets-docs-deps node_modules tree hosting # node_modules/.bin/semantic-release. -# Optional (all modes): +# +# Optional (CI-mode signalling; required by env-ci on the effect path): +# CI "true" tells semantic-release / env-ci that the +# run is non-interactive CI. Required when running +# in the buildbot-effects bwrap sandbox: the +# sandbox is not a recognised CI provider, so +# semantic-release would otherwise abort with +# `running on a CI environment is required` (env-ci +# default). Set by the effect preamble; unset on +# local-shell invocations where semantic-release's +# --no-ci flag (under --dry-run) bypasses the check. +# +# Optional (repo-root resolution; env-first with shelled-fallback): +# RELEASE_REPO_ROOT absolute path to the working tree's repo root. +# Required when running inside the buildbot-effects +# bwrap sandbox: the sandbox does not bind-mount +# the working tree, so `git rev-parse +# --show-toplevel` would fail with `fatal: not a +# git repository` and abort the script. Effect +# preamble sets to "$PWD" (mkEffect cwd is the +# pristine source root). Fallback chain: +# git rev-parse --show-toplevel 2>/dev/null || pwd +# — error-tolerant so a missing .git does not +# cause non-zero exit. +# +# Optional (git identity; env-first, NO .git/config writes): +# GIT_AUTHOR_NAME semantic-release commit author name +# (semantic-release writes a CHANGELOG commit +# on the production path). git honours these +# env vars natively without writing to +# .git/config — required because the bwrap +# sandbox mounts /nix/store ro-bind, and +# `git config user.email "…"` would fail with +# `error: could not lock config file .git/config` +# when the working tree's .git is unavailable. +# Default (effect preamble): semantic-release. +# GIT_AUTHOR_EMAIL semantic-release commit author email. +# Default (effect preamble): +# semantic-release@vanixiets.local +# GIT_COMMITTER_NAME semantic-release commit committer name. +# Default (effect preamble): semantic-release +# GIT_COMMITTER_EMAIL semantic-release commit committer email. +# Default (effect preamble): +# semantic-release@vanixiets.local +# GIT_USER_NAME transitional alias — when GIT_AUTHOR_NAME and +# GIT_COMMITTER_NAME are unset, this value is +# used to seed both. Retained for callers that +# have not yet migrated to the GIT_AUTHOR_*/ +# GIT_COMMITTER_* convention. +# GIT_USER_EMAIL transitional alias for GIT_AUTHOR_EMAIL + +# GIT_COMMITTER_EMAIL. +# +# Optional (passthrough, not consumed): # SOPS_AGE_KEY reserved passthrough for sops-decrypt hooks # (no consumer in the current tree; declared but # NOT enforced via :? guard — see ADR-002, which # REJECTS SOPS_AGE_KEY as a general pattern). -# GIT_USER_NAME git identity; default: semantic-release -# GIT_USER_EMAIL git identity; default: semantic-release@vanixiets.local # # Caller mechanisms: # - Local dev dry-run: `nix run .#release -- packages/ --dry-run` -# needs no secret env (plugin filter strips github) +# needs no secret env (plugin filter strips github); +# git fallback resolves repo-root and identity +# from the local worktree's .git. # - Local dev prod: caller-side sops wrapper (decrypt # secrets/shared.yaml before the nix run) OR # direnv dotenv (.envrc `dotenv` + .env) # - GHA env: step `env:` block populates GITHUB_TOKEN from -# the repo secrets (package-release.yaml) -# - M4 effect: production-release-packages effect preamble -# extracts GITHUB_TOKEN from HERCULES_CI_SECRETS_JSON -# and exports before invoking the app program path +# the repo secrets (package-release.yaml); +# checkout action provides the .git working tree +# so RELEASE_REPO_ROOT / GIT_AUTHOR_* fallbacks +# are exercised. +# - M4 effect: release-packages effect preamble extracts +# GITHUB_TOKEN from HERCULES_CI_SECRETS_JSON and +# exports it alongside RELEASE_REPO_ROOT="$PWD", +# CI=true, GIT_BRANCH=, and the +# GIT_AUTHOR_*/GIT_COMMITTER_* identity quartet +# before invoking the app program path. # # Secret passing rule (per ADR-002): NO secrets are passed as CLI flags. # Authentication flows exclusively through the inherited environment. @@ -78,8 +140,9 @@ Flags: --help Print this usage and exit. Environment: - GITHUB_TOKEN, SOPS_AGE_KEY, DOCS_NODE_MODULES, GIT_USER_NAME, - GIT_USER_EMAIL (see release.sh header for details). + GITHUB_TOKEN, SOPS_AGE_KEY, DOCS_NODE_MODULES, RELEASE_REPO_ROOT, CI, + GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, + GIT_USER_NAME, GIT_USER_EMAIL (see release.sh header for details). EOF } @@ -175,7 +238,14 @@ if [ -z "$package_path" ]; then exit 2 fi -repo_root="$(git rev-parse --show-toplevel)" +# Repo-root resolution: env-first, then error-tolerant git fallback, then +# pwd. Required because the buildbot-effects bwrap sandbox does not bind- +# mount the working tree's .git, so `git rev-parse --show-toplevel` would +# fail with `fatal: not a git repository` (exit 128) and abort the script. +# The effect preamble sets RELEASE_REPO_ROOT="$PWD" so this branch resolves +# without invoking git. Local-shell and GHA paths set RELEASE_REPO_ROOT +# to empty, exercising the git fallback against the live worktree. +repo_root="${RELEASE_REPO_ROOT:-$(git rev-parse --show-toplevel 2>/dev/null || pwd)}" cd "$repo_root" if [ ! -d "$package_path" ]; then @@ -184,15 +254,19 @@ if [ ! -d "$package_path" ]; then exit 1 fi -# Configure a git identity if none is present so semantic-release can -# create tags/commits without a separate setup step. A pre-configured -# identity (e.g. from the caller's ~/.gitconfig) is preserved. -if [ -z "$(git config user.email 2>/dev/null || true)" ]; then - git config user.email "${GIT_USER_EMAIL:-semantic-release@vanixiets.local}" -fi -if [ -z "$(git config user.name 2>/dev/null || true)" ]; then - git config user.name "${GIT_USER_NAME:-semantic-release}" -fi +# Git identity: exported via GIT_AUTHOR_* / GIT_COMMITTER_* env vars rather +# than written to .git/config. Required because the buildbot-effects bwrap +# sandbox renders .git read-only (mounts /nix/store ro-bind only) and +# `git config user.email "…"` would fail with `error: could not lock config +# file .git/config`. git honours these env vars natively without any config +# write. Transitional aliases GIT_USER_NAME/GIT_USER_EMAIL seed the quartet +# when the new vars are unset, preserving existing local/GHA caller +# behaviour. Each export uses parameter-expansion default chaining so a +# pre-set value (effect preamble or caller env) is preserved unchanged. +export GIT_AUTHOR_NAME="${GIT_AUTHOR_NAME:-${GIT_USER_NAME:-semantic-release}}" +export GIT_AUTHOR_EMAIL="${GIT_AUTHOR_EMAIL:-${GIT_USER_EMAIL:-semantic-release@vanixiets.local}}" +export GIT_COMMITTER_NAME="${GIT_COMMITTER_NAME:-${GIT_USER_NAME:-semantic-release}}" +export GIT_COMMITTER_EMAIL="${GIT_COMMITTER_EMAIL:-${GIT_USER_EMAIL:-semantic-release@vanixiets.local}}" cd "$package_path" diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 9f46498f7..944f4915a 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -37,6 +37,23 @@ # and MUST NOT reference NPM_TOKEN (not part of the closed 4-key # bundle; npmPublish=false invariant means no npm publish surface). # +# Symmetric env-var contract (CI + GIT_AUTHOR/COMMITTER + RELEASE_REPO_ROOT) +# (m4-release-packages-runtime-deps-contract): +# Exports CI=true, GIT_BRANCH (from herculesCI.config.repo.branch at +# eval time via lib.escapeShellArg), RELEASE_REPO_ROOT="$PWD" (mkEffect +# cwd is the source root), and the GIT_AUTHOR_NAME/EMAIL + +# GIT_COMMITTER_NAME/EMAIL identity quartet. Required because the +# buildbot-effects bwrap sandbox does not bind-mount the working tree's +# .git AND provides no host-PATH binaries beyond /nix/store ro-bind + +# writeShellApplication runtimeInputs PATH (upstream-by-design across +# all 3 reference effect implementations); release.sh's +# `git rev-parse --show-toplevel` would fail with `fatal: not a git +# repository`, its `git config user.{name,email} "…"` writes would +# fail with `error: could not lock config file .git/config`, and +# semantic-release would abort with `running on a CI environment is +# required` (env-ci default) without these exports. Symmetric to +# the deploy-docs env-var contract (GIT_* + DEPLOY_*) precedent. +# # Per-package atomicity (NOT fail-fast): # If package A fails, the loop continues to package B. Each # per-package failure is recorded; the dispatcher exits non-zero at @@ -137,6 +154,28 @@ exit 1 fi + # Symmetric env-var contract (m4-release-packages-runtime-deps-contract): + # populate CI / GIT_BRANCH / RELEASE_REPO_ROOT and the + # GIT_AUTHOR_*/GIT_COMMITTER_* identity quartet so release.sh + # never has to invoke `git rev-parse --show-toplevel` or + # `git config user.{name,email} "…"` inside the bwrap sandbox. + # CI is set so env-ci recognises the run as non-interactive CI, + # bypassing semantic-release's `running on a CI environment is + # required` abort. GIT_BRANCH is interpolated at nix eval time + # from herculesCI.config.repo.branch (empty string on tag-push + # where branch is null). RELEASE_REPO_ROOT="$PWD": mkEffect cwd + # is the source root, so $PWD is the canonical repo root inside + # the sandbox. GIT_AUTHOR_*/GIT_COMMITTER_*: hard-coded identity + # for the semantic-release CHANGELOG commit; git honours these + # env vars natively without writing to .git/config. + export CI=true + export GIT_BRANCH=${lib.escapeShellArg (if branch == null then "" else toString branch)} + export RELEASE_REPO_ROOT="$PWD" + export GIT_AUTHOR_NAME=semantic-release + export GIT_AUTHOR_EMAIL=semantic-release@vanixiets.local + export GIT_COMMITTER_NAME=semantic-release + export GIT_COMMITTER_EMAIL=semantic-release@vanixiets.local + # Option Gamma store-path dispatch — both flake apps' /nix/store # paths are embedded at eval time via the perSystem # config.apps..program attributes. No flake-app shell-out From 40ad5cb2f3cc717dfa8889949aa7e4cddc7605de Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Fri, 24 Apr 2026 21:41:45 -0400 Subject: [PATCH 32/77] Update flake inputs: llm-agents --- flake.lock | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/flake.lock b/flake.lock index 756c563c8..78dcb80b2 100644 --- a/flake.lock +++ b/flake.lock @@ -781,11 +781,11 @@ "treefmt-nix": "treefmt-nix" }, "locked": { - "lastModified": 1777019177, - "narHash": "sha256-YjPvucTsKmGO9QVNz07x7sSsK11PB0jtMniRkTolbq4=", + "lastModified": 1777078377, + "narHash": "sha256-2EykR9XvDFwO2t/pWgKzfmwwjGX2YsD6LPU4D2PuWyY=", "owner": "numtide", "repo": "llm-agents.nix", - "rev": "8ff0f2a7fcd176b4547da6879ad549de2bbded41", + "rev": "f5f1cc1c90316b8ef96cf009ce3a290e6955da80", "type": "github" }, "original": { From 310592b1c88c362a5cf3d592823ffb0cce31f44b Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Fri, 24 Apr 2026 23:25:16 -0400 Subject: [PATCH 33/77] feat(effects): clone-internally-then-push preamble for release-packages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements ADR-003 §Architecture phases 1-6 in the release-packages effect to fix the `fatal: not a git repository` failure on cd-via-effects: 1. Canonicalized clone URL (https://github.com/cameronraysmith/vanixiets.git) — never the buildbot-nix-baked remoteHttpUrl (invariant #1). 2. Full clone (no shallow / since-date / blob-filter flags; semantic-release needs the full tag/notes/log surface; invariant #2). 3. Exact-rev checkout via `git checkout -B "$GIT_BRANCH" "$GIT_REV"` for force-push determinism (invariant #3). 4. Single freshness check before the package loop with `RELEASE-CLONE-STALE` exit on race (invariant #11). 5. Sanitized RELEASE-CLONE-{START,CHECKOUT,READY,STALE} banners; no token-baked URL ever appears in any banner (invariants #9, #10). 6. `GIT_CREDENTIALS=x-access-token:${GITHUB_TOKEN}` export — no host credential helper, no ~/.git-credentials write (invariants #4, #5). Single-step EXIT trap for clone cleanup (invariant #12). RELEASE_REPO_ROOT now points at $clone_dir instead of the empty mkEffect $PWD that triggered the `fatal: not a git repository` failure mode. The preamble lands at ~35 bash lines; well under the ~60-line ADR-003 §D3 helper-extraction threshold, so it is inlined in release-packages.nix. Future PR-creating mutating effects should prefer upstream hci-effects.git-update / flakeUpdate per ADR-003 §D3. D2 lock (no @semantic-release/git in plugin chains): unchanged. D4 lock (effects_on_pull_requests=true, effects_branches=["*"]): unchanged. Per-effect `branch == "main"` gate: unchanged (the production boundary). Verified locally: - nix eval drvPath for main/non-main/tag-push primaryRepo: all resolve - rendered effectScript contains all 4 RELEASE-CLONE banners, canonical clone URL, GIT_CREDENTIALS export, and `checkout -B`. Refs: m5-01a-release-packages-clone-internally-implement, VAL-RELEASE-α-PREAMBLE-001..012, VAL-RELEASE-α-AUTH-001..007, VAL-RELEASE-α-DISPATCH-001..004. --- .../vanixiets/herculesCI/release-packages.nix | 201 +++++++++++++++--- 1 file changed, 173 insertions(+), 28 deletions(-) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 944f4915a..02cb1044c 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -37,22 +37,66 @@ # and MUST NOT reference NPM_TOKEN (not part of the closed 4-key # bundle; npmPublish=false invariant means no npm publish surface). # +# ADR-003 Option α clone preamble (m5-01a-release-packages-clone-internally-implement): +# The buildbot-effects bwrap sandbox does NOT bind-mount the worker's +# git checkout (`mkEffect` only seeds `HOME=/build/home`); the previous +# iteration of this effect failed with `fatal: not a git repository` +# because semantic-release ran against an empty $PWD. Per ADR-003 +# "release-packages clone-and-push" (Option α, locked 2026-04-24; +# mission-internal note kept under repo-local `.factory/library/`, +# git-ignored via `.git/info/exclude`), +# the preamble below performs an in-sandbox clone of +# the canonical GitHub URL, checks out the exact triggering rev, +# verifies branch-tip freshness, exports `GIT_CREDENTIALS` for +# semantic-release's auth-URL builder, and points the existing +# env-var contract's `RELEASE_REPO_ROOT` at the clone (not `$PWD`). +# +# Six normative phases (ADR-003 §Architecture): +# 1. URL canonicalization — literal `https://github.com/cameronraysmith/vanixiets.git` +# (NOT `herculesCI.config.repo.remoteHttpUrl`, which bakes the +# buildbot-nix GitHub App installation token at clone time). +# 2. Full clone (no shallow / since-date / blob-filter flags): +# semantic-release needs the full tag list, full commit log +# since `lastRelease.gitHead`, and per-commit `git diff-tree` +# changed-file lookups (semantic-release-monorepo path filter). +# 3. Exact-rev checkout via `git checkout -B "$GIT_BRANCH" "$GIT_REV"` — +# force-push and rapid-merge events become deterministic; the +# effect releases what buildbot-master's nix-eval ran against. +# 4. Single freshness check before the package loop (invariant #11): +# `git fetch origin "$GIT_BRANCH"` + `rev-parse HEAD == origin/$GIT_BRANCH`; +# stale runs emit `RELEASE-CLONE-STALE` and exit non-zero. +# 5. Sanitized structured banners — `RELEASE-CLONE-{START,CHECKOUT,READY,STALE}`. +# NEVER echo a token-baked URL (invariant #9); only the +# canonicalized GitHub URL appears in any banner. +# 6. `GIT_CREDENTIALS=x-access-token:${GITHUB_TOKEN}` export for +# semantic-release's `get-git-auth-url.js` URL builder +# (in-process; no host credential helper, no ~/.git-credentials +# write per invariants #5 / VAL-RELEASE-α-AUTH-003 / -004). +# +# Helper extraction (ADR-003 §D3): the inline preamble lands at ~35 +# bash lines; well under the ~60-line threshold for factoring into +# `modules/effects/lib/mkMutatingEffect.nix`. Future PR-creating +# mutating effects (security-update-pr, dep-bump) should prefer +# upstream `hci-effects.git-update`/`flakeUpdate`, NOT a local helper. +# # Symmetric env-var contract (CI + GIT_AUTHOR/COMMITTER + RELEASE_REPO_ROOT) -# (m4-release-packages-runtime-deps-contract): +# (m4-release-packages-runtime-deps-contract; extended in m5-01a): # Exports CI=true, GIT_BRANCH (from herculesCI.config.repo.branch at -# eval time via lib.escapeShellArg), RELEASE_REPO_ROOT="$PWD" (mkEffect -# cwd is the source root), and the GIT_AUTHOR_NAME/EMAIL + -# GIT_COMMITTER_NAME/EMAIL identity quartet. Required because the -# buildbot-effects bwrap sandbox does not bind-mount the working tree's -# .git AND provides no host-PATH binaries beyond /nix/store ro-bind + -# writeShellApplication runtimeInputs PATH (upstream-by-design across -# all 3 reference effect implementations); release.sh's -# `git rev-parse --show-toplevel` would fail with `fatal: not a git -# repository`, its `git config user.{name,email} "…"` writes would -# fail with `error: could not lock config file .git/config`, and -# semantic-release would abort with `running on a CI environment is -# required` (env-ci default) without these exports. Symmetric to -# the deploy-docs env-var contract (GIT_* + DEPLOY_*) precedent. +# eval time via lib.escapeShellArg), RELEASE_REPO_ROOT=$clone_dir +# (the in-sandbox clone created by the ADR-003 preamble; previously +# "$PWD", which referred to the empty mkEffect cwd that triggered +# the `fatal: not a git repository` failure mode), and the +# GIT_AUTHOR_NAME/EMAIL + GIT_COMMITTER_NAME/EMAIL identity quartet. +# Required because the buildbot-effects bwrap sandbox does not +# bind-mount the worker's checkout AND provides no host-PATH binaries +# beyond /nix/store ro-bind + writeShellApplication runtimeInputs +# PATH (upstream-by-design across all 3 reference effect +# implementations); release.sh's `git config user.{name,email} "…"` +# writes would fail with `error: could not lock config file +# .git/config`, and semantic-release would abort with `running on a +# CI environment is required` (env-ci default) without these +# exports. Symmetric to the deploy-docs env-var contract (GIT_* + +# DEPLOY_*) precedent. # # Per-package atomicity (NOT fail-fast): # If package A fails, the loop continues to package B. Each @@ -154,23 +198,117 @@ exit 1 fi - # Symmetric env-var contract (m4-release-packages-runtime-deps-contract): - # populate CI / GIT_BRANCH / RELEASE_REPO_ROOT and the - # GIT_AUTHOR_*/GIT_COMMITTER_* identity quartet so release.sh - # never has to invoke `git rev-parse --show-toplevel` or - # `git config user.{name,email} "…"` inside the bwrap sandbox. + # === ADR-003 Option α clone preamble =========================== + # Phases 1–6 per ADR-003 §Architecture and the file-header + # doc-comment. All token-leak / freshness / authority invariants + # (#1, #2, #3, #5, #9, #11, #12) are codified here. + + # Canonicalized clone URL (ADR-003 §Architecture step 1, + # invariant #1). Hard-coded GitHub URL — NEVER derived from + # `herculesCI.config.repo.remoteHttpUrl`, which buildbot-nix + # populates with `https://git:@github.com/...` + # for GitHub-App-backed repos. Reusing that URL would bake the + # buildbot-nix App's `Contents: Read-only` token into local git + # config — exactly the wrong authority for a release mutation + # and a clear-text token-leak risk in any subsequent banner echo. + clone_url="https://github.com/cameronraysmith/vanixiets.git" + + # mkEffect's $TMPDIR is /tmp (tmpfs); mktemp here keeps the + # clone inside the bwrap-managed tmpfs which is reaped on + # sandbox exit regardless of the trap below. + clone_dir="$(mktemp -d -t release-packages-clone.XXXXXX)" + + # Single-step EXIT trap (invariant #12). Defensive hygiene only; + # the bwrap tmpfs is ephemeral. Do NOT add multi-step traps. + trap 'rm -rf "$clone_dir"' EXIT + + # Pre-compute git refs as nix-eval-time literals. Bash captures + # them as local vars so the freshness-check / checkout / banner + # phases share one canonicalized value without re-shelling-out. + GIT_REV=${lib.escapeShellArg (toString rev)} + GIT_BRANCH=${lib.escapeShellArg (if branch == null then "" else toString branch)} + + # RELEASE-CLONE-START banner (invariants #9, #10): emitted + # BEFORE the clone with the sanitized public URL. Token never + # appears in this output even though GIT_CREDENTIALS is set + # later in this effect — the URL string is the canonical one. + echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" + + # Full clone (ADR-003 §Architecture step 2, invariant #2): + # NO shallow flag, NO since-date flag, NO blob-filter flag. + # semantic-release requires `git tag --merged ` + # (lib/git.js:24-31), the full unshallowed tag fetch + # (lib/git.js:106-132), notes refs (lib/git.js:144-154), and + # the full commit log since lastRelease.gitHead + # (lib/get-commits.js:7-25); semantic-release-monorepo + # additionally needs per-commit changed-file lookups + # (src/git-utils.js:16-24). Pack size (~21.5 MiB) does not + # warrant shallow optimization at this scale. + git clone "$clone_url" "$clone_dir" + git -C "$clone_dir" fetch --tags origin + + # Exact-rev checkout (ADR-003 §Architecture step 3, invariant + # #3). When GIT_BRANCH is empty (tag-push events: hercules-ci- + # effects models tag checkouts as `branch = null`), use a + # synthetic local branch name. The dry-run gate (isMain == + # false) ensures no production push attempts in that case; + # see ADR-003 §"Tag-push event handling". + if [ -n "$GIT_BRANCH" ]; then + checkout_branch="$GIT_BRANCH" + else + checkout_branch="release-packages-detached" + fi + git -C "$clone_dir" checkout -B "$checkout_branch" "$GIT_REV" + echo "RELEASE-CLONE-CHECKOUT: $GIT_REV" + + # Freshness check (ADR-003 §Architecture step 4, invariant + # #11). Single check after checkout, before the package loop. + # Per-package re-checks are NOT added (see ADR-003 §11). On + # tag-push events (GIT_BRANCH empty) there is no branch tip + # to compare against, so the check is structurally + # inapplicable; the ADR-003 §"Tag-push event handling" gate + # forces dry-run for those runs anyway. + if [ -n "$GIT_BRANCH" ]; then + git -C "$clone_dir" fetch origin "$GIT_BRANCH" + head_rev="$(git -C "$clone_dir" rev-parse HEAD)" + remote_rev="$(git -C "$clone_dir" rev-parse "origin/$GIT_BRANCH")" + if [ "$head_rev" != "$remote_rev" ]; then + echo "RELEASE-CLONE-STALE: expected $head_rev remote $remote_rev" >&2 + exit 1 + fi + fi + + echo "RELEASE-CLONE-READY: $clone_dir" + + # Token authentication (ADR-003 §Architecture step 6, + # invariants #4, #5). semantic-release's get-git-auth-url.js + # treats GIT_CREDENTIALS as a pre-baked `user:password` pair + # and constructs the authenticated URL in-process — NO host + # credential helper installation, NO write to ~/.git- + # credentials. The PAT held in `vanixiets-effects-secrets` + # (Contents: Read+Write) is the canonical authority; the + # buildbot-nix GitHub App installation token (Contents: + # Read-only) is intentionally NOT reused for release + # mutation. The shell variable is only consumed by + # semantic-release's URL builder; it is never echoed. + export GIT_CREDENTIALS="x-access-token:''${GITHUB_TOKEN}" + + # === existing m4-release-packages env-var contract ============= + # (extended for m5-01a: RELEASE_REPO_ROOT now points at the + # in-sandbox clone, not the empty mkEffect $PWD that previously + # caused `fatal: not a git repository`). + # # CI is set so env-ci recognises the run as non-interactive CI, # bypassing semantic-release's `running on a CI environment is - # required` abort. GIT_BRANCH is interpolated at nix eval time - # from herculesCI.config.repo.branch (empty string on tag-push - # where branch is null). RELEASE_REPO_ROOT="$PWD": mkEffect cwd - # is the source root, so $PWD is the canonical repo root inside - # the sandbox. GIT_AUTHOR_*/GIT_COMMITTER_*: hard-coded identity - # for the semantic-release CHANGELOG commit; git honours these - # env vars natively without writing to .git/config. + # required` abort. GIT_BRANCH is the eval-time literal value + # already captured above (re-exported for child processes). + # GIT_AUTHOR_*/GIT_COMMITTER_* are hard-coded identities for the + # semantic-release CHANGELOG-prepare phase; git honours these + # env vars natively without writing to .git/config (which the + # bwrap /nix/store ro-bind would block anyway). export CI=true - export GIT_BRANCH=${lib.escapeShellArg (if branch == null then "" else toString branch)} - export RELEASE_REPO_ROOT="$PWD" + export GIT_BRANCH + export RELEASE_REPO_ROOT="$clone_dir" export GIT_AUTHOR_NAME=semantic-release export GIT_AUTHOR_EMAIL=semantic-release@vanixiets.local export GIT_COMMITTER_NAME=semantic-release @@ -183,6 +321,13 @@ LIST_PACKAGES=${listPackagesProgram} RELEASE=${releaseProgram} + # cd into the clone before invoking list-packages-json: that + # script calls `git rev-parse --show-toplevel`, which must + # resolve to $clone_dir (the only real git tree in this + # sandbox). release.sh's own RELEASE_REPO_ROOT consumer also + # picks up the same clone via the env export above. + cd "$clone_dir" + # Discover packages under packages/* (jq-driven enumeration of # the list-packages-json output). packages_json="$("$LIST_PACKAGES")" From 4a48f4c94175a6ebe5e7af640cdcb3c8c52e1db6 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 00:23:10 -0400 Subject: [PATCH 34/77] vars: update via generator vanixiets-effects-secrets (machine: magnetite) --- .../cloudflare-account-id/secret | 8 ++++---- .../cloudflare-api-token/secret | 8 ++++---- .../vanixiets-effects-secrets/github-token/secret | 8 ++++---- .../magnetite/vanixiets-effects-secrets/secrets/secret | 10 +++++----- .../vanixiets-effects-secrets/sops-age-key/secret | 8 ++++---- 5 files changed, 21 insertions(+), 21 deletions(-) diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/secret index a7b5ec04a..4a2328de2 100644 --- a/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/secret +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-account-id/secret @@ -1,14 +1,14 @@ { - "data": "ENC[AES256_GCM,data:AEgZtL+XHM1oGrCSUlbzKfyBtIsYwBRfgeQ03NOrvJc=,iv:bH5kvogxXCC0DiuLfPhc5ahu0lsoVqX9SY+tYstqRa4=,tag:YxUqn5pjhyuZC5MX6iDeEg==,type:str]", + "data": "ENC[AES256_GCM,data:/OBJ9hZQfSkB4hH4Vd/WwySOCRTvQ1vtOhnzphTQm/Y=,iv:UyXCBL4d6xC8vpnwRyNRftrZ1C5r+9UFAoyrszFIl8I=,tag:xjoTGWKP8ULziGw/6eiDSQ==,type:str]", "sops": { "age": [ { "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", - "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxbE45bHZ6UE5Uc2NISnZM\nRlRIMDFZVktxL1JkMTBDbC84YUU4RHdRM0JzCjdhV3N5Snd3VHZLWnNKdG5JRmc0\nY1FkeHBFM3I5cGJ2VGFwaE9aR00zWXcKLS0tIHhFSkZHeE1SMnZNNzR5S214OVpi\nZVJrcXpzeENCRmdRRVJSNXl1cDhOZmcKjTMBlJ36OfqyATLYMLry6cicewCDmJC3\nBmCbf9s1cq3os7UlmWYFrfvn5cR5mc9u6u/2Sa6djDPWXXn12m0tUA==\n-----END AGE ENCRYPTED FILE-----\n" + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqb2tPanlsYWdKSm5pZVkz\nL2t4ZmxJVlVHV2xaZ3QzSWFaWksrVEo2dUdBCittSkxSd002V0hSbHpqem04U1Ju\ndFlDeTBqSTA0bEhvNTI5VnM1K2xGTjQKLS0tIE54aFhNV2JQZnZtRnZDT3hsSjh2\ncGtHTDN5cjh6bXhaVG4rRVAxOVgrWVUKdFUa2irGlkRqsF0D7Nw/COB1ep7BYzgl\n4FbKGOfktfY5yxCBMnQ6xAjnh7H2BT4sXlwDIEmK7ER7SxuRNCwttg==\n-----END AGE ENCRYPTED FILE-----\n" } ], - "lastmodified": "2026-04-24T06:07:27Z", - "mac": "ENC[AES256_GCM,data:nF0LZlmrNffvbk7aTAzqsnZVwo7uNzPYqxd49l2xFg0w7h6SxJ5RnPGydHo49KwZ7PmBN+jef9OGf+et9MNYlo7M0aoupXsqT4EJFoCZitUMDP3KdmUy6dxc9AaitGBNohbpmh5UDnzL9AFtoGTwCJcJeHgsjgErWBY0TLrD2l0=,iv:jF8KUIwGFUgic5dss1EG/zdsLPvmXvMB10hbApezAjA=,tag:96is3O7SLbcjrS9pPq65fA==,type:str]", + "lastmodified": "2026-04-25T04:23:09Z", + "mac": "ENC[AES256_GCM,data:AuF16VAGcd6PHlW/6/8xtJnUxa3GHW/23DJHlhRI5CvyHHYC2h5/Gv10KzWvpn5yrKuR/uLQh9Es/pE58/p1p9AvWO+YT3BI65Aw+b9TpH1gZvrQdaJDsYMJWsjJG0IGwDeHh5h9+l4GdYl+0tKXnVBoD3oZhPXvMnxwm1bxtCw=,iv:xWUbnrZ23rlmTHPRpE2nYt9ZXdxAD1rNdnDbPQyKSzA=,tag:c7oL9VW3dotI3ra3aWJAsQ==,type:str]", "version": "3.12.2" } } diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/secret index 75ba61bbf..897d5e102 100644 --- a/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/secret +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/cloudflare-api-token/secret @@ -1,14 +1,14 @@ { - "data": "ENC[AES256_GCM,data:2fa05e29Y06qALSPVR4GdkpUbCbPgJF1yFtYof6nrti0Cmq7HBSprQ==,iv:fROlnLnp+n1Goh6bt1gCHlSviWxRyhX5cb1my4DKwu0=,tag:EfSl5ASs6D5/r0uMUpwvBQ==,type:str]", + "data": "ENC[AES256_GCM,data:ll/mj1o5a4fmMD/SjK06JgvBuj3kv9w71L3ulXnPiebT95QGxZVmxw==,iv:o13DSTC3My8Beo1c99tcGs2RpoONoKHK90ixG7RF5og=,tag:Vut0YlBwejcaQWCMENoCrw==,type:str]", "sops": { "age": [ { "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", - "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZTWpubXVoNUFsRDdualpz\nb3ArYVNTeURuTXNGSmplcnIvM0h2OUlwd0FrCkYwUVJNcURBM2ZvZ1NIaWRZb2JY\nMFgyejFiS25Ucjl0QWNVR3ZKVnVpdDQKLS0tIGg0WElseU1DT1hwNG9VTU9rRE0w\nb21Tc2EvRGhaVW43SEttcWFjREdPNVkK1zlIYVD04enRhykmSvFOzsrKX9HZ4W2C\nBOSERkQwh6N52VuOCfHl7lUU5Xl5emUyTcYesWc4udGbBM/u5yzdhQ==\n-----END AGE ENCRYPTED FILE-----\n" + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA4ZFlINU80TllhS1A1Z0ZW\nZ1BLZGZHR2pSNzhuSUc4RTJESmlzeFNoZ25vCnZLRHZmeDFEQVd4L1NGaDlkRmdz\nSVZpTHJhY291UDBlaFRGQmwyQjhzcGcKLS0tICt4WWlJMGNCRWZiTFE5SXFCSHF2\nOHN0RHcyWFF3eXZFZjRTR3JPVmF3T1kK3cNIRPEo+2BWR57pzJDailXt3RORYwaS\nxFthTyKc529E482sxfunbw3uOEEspmwaz+N+rd7ilmc+z5vGFfvtTQ==\n-----END AGE ENCRYPTED FILE-----\n" } ], - "lastmodified": "2026-04-24T06:07:27Z", - "mac": "ENC[AES256_GCM,data:F0fPnnhgEiFgsq72R5m64pzQuerjrlwrK/3UiJPvhcrP/IYY+4GObr60ymPZ5EyuMiPV67DEZCUgvXZI4RHOVr0JqvXSrg+w/O5l8q4naYKrIFoI3hs6M/N3jdQotapYeLQ/iDkVEjKULrqTTjoLXgw/waTe2hBw/ME6s23Bj8A=,iv:mTiO2SKYvlIi07l/cZSNJrh/HcErDwtiYLwa8KET8KY=,tag:Crd90W2grb10Ce+CBf+tdg==,type:str]", + "lastmodified": "2026-04-25T04:23:09Z", + "mac": "ENC[AES256_GCM,data:7oXBrSrNaHr59jIZPK/KjwyrZgKqfl70jEKhR59sUxO+mnmVbwasbokXFbirIqR8m96cIjO4u7xVLvnDjReYnX88J1AVpwVBQqZjuWjM0iMsAHL8P4S4JD/HeDlNZ/9Jer31IH4Hhl2Lz+u1nbGKjkI0mV6+jRMRkGb+WBaw6As=,iv:NSQiMz2IY/Ld/WhzJq3W6ReQ+/nxxUOv+UJrRy7h72A=,tag:vv21belO0aEFcnwH/fTU2Q==,type:str]", "version": "3.12.2" } } diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/secret index 7be71495e..9b1d48292 100644 --- a/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/secret +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/github-token/secret @@ -1,14 +1,14 @@ { - "data": "ENC[AES256_GCM,data:fjjkFqnu6DEY7ip6HJNSaI0FcDKTnUt+8BkbqeuPCkrMlTD0ostxQg==,iv:gKHO4qp6u3R90DFNxhgT2WQWKdlV2iLJKyolZFSd14I=,tag:loWGQn/fAAVXb8hr8ylqWg==,type:str]", + "data": "ENC[AES256_GCM,data:wXZgIB158bRujqibywxsEIqASjayNXYHcgvVHAMEworQaeEoHbViD6YeKGu5D02eFpAMOlFQhsVnD5KCb2iog6uejD3jBBYKn2ZI7rIsLk2WWKkx8xstRj++sA1n,iv:pHLNBkeu3y0a5NlF0cdWQBUGv+fNhYymu766JrULiZQ=,tag:S2qukZQIlVbQ6AqzWdvtOw==,type:str]", "sops": { "age": [ { "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", - "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmcEFDdnplL3pUTlNNY2RI\ncjQvSCtkS0RZY1JEUm80Z09aakVhRjJpTUVVCjB3eDR2Z2FtUGRmS3hVK3hQTWRD\nWmhISi8vRko4dkZieWhMeC85U2d2RFEKLS0tIFprTjZvcGlHeGNxYk00TndqSDlU\nOHNsc2RFZWRWeDcxUHEvVS8wQ3Q3eVkKb6PLRVBxGWYkY3SUaNtpUMs8v0YQ0Lwc\nmsm9+ipNzERg5jGnQZ0ip8pfv4nPbOZUA/cpsAFse30HKYnF2g+cbw==\n-----END AGE ENCRYPTED FILE-----\n" + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2MjhUMW1RcVV0L0t6U0pn\nTUNyTFV5YVE1bmNiQm9Fc3lkK2tqaThqYmxrCkJ1RVgzQ3RkdU5oWUViS0VGSWRF\nOFFtMkVVM05sQmhHY0ZwbFRudFJhN3MKLS0tIFgrN2NjNW5WWkMzOUJROWNNOTgx\nZjFKYmpjYU14bms2Q2lIMlBBY2ZrQkEKiWBY/9oxKU4Bz79v4FiRR0DvP6B2w/zN\no1K265lmrK1ME8ZpX/VKifAyPRySEMYUrRKjJWITMsJZ/4c1x4YmNg==\n-----END AGE ENCRYPTED FILE-----\n" } ], - "lastmodified": "2026-04-24T06:07:27Z", - "mac": "ENC[AES256_GCM,data:o3smM5LQObtBpwubQF9GVJJEmWWUauXo4zI2BTm5qG+LJ4bXFfOuEYvjJvvhSHcz1d4wFlbCOmY6i/F/NTh7hS4wINvztIuC7Dsac3pmnJKxfRMwwMbSsWQ9ZQS8Fq3S/DYiET9grShO61g2kPQu7frZrGPQXIbi9l6kDKQ56Ag=,iv:LC5fs6mciQEJc/odi3V+wY+irYEROMQcmcZFxAqj2KE=,tag:5oOLLDcuzTKDhWkuFN6IOQ==,type:str]", + "lastmodified": "2026-04-25T04:23:09Z", + "mac": "ENC[AES256_GCM,data:CaUOtHsYZXn2QXLpXMWlVbV0SOvSfT8pVNdgN0arNVXsGDBiOkhWejNXalpK6qW4w7l+30hu1ZdEysb9bT/jTmJBc8jbzaS4H+KsAhGcwTVPQvBWRs9o7069FZw7DwidNL2fXk88xskJsdI80AMIDgiJwaI7sGUg1QVpur14vRk=,iv:SiP9tBtnJjl8jhwi+hK5wyLdl4dr0kSaUnRyGiPjULQ=,tag:y5PzytkR+iMh6zHWrbU/cQ==,type:str]", "version": "3.12.2" } } diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/secret index 4c477f732..f2a9d38d5 100644 --- a/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/secret +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/secrets/secret @@ -1,18 +1,18 @@ { - "data": "ENC[AES256_GCM,data:qo8Ka1sKx9BoqPy5DUYJsgyEpEahSkbM60jB6nmPKQYeNSlONXITExWGp2k75Iu+2gTYqRF2D78P2xGgSr+uPncdJb2zJdBepgAtZXSuLk0tLiPj3PbvEYPSlbXzCN5MZsi721nk5P/eaSh5Cwpcz+vF77g4T4gE9Na775xMbALjosMo1CDU0+n8bqJGiwBmWf9iW5EbxqtsPSolbdg70FIG59pk8/TCAktXnGQ+6B3vJ+dKJsGYN5kMx9WxYsZtMTlNBs4dpJu3Vt3Dm2tkwPw8/bGUHlHWmi/UzIgSv2AAcMAOSbGB2KogCo3fGZvyIj0WL+qsO9Wz5pl5ZmbzjhuPdy/Ri3zBNxyHUgjcmwttXHnQT8BMAlM/zfwLgF3oRuSyUQisKBTXcW6Y549RypoXDEoSx2egc0iPj+ZVjylxpRAScx9SbquVrc7Ua3UBCeT49/qVA3AaPXwhzhfG3U6DfkHh+LyLrRCz/drLmWo57FsbZqUWsltMngNOvqqmrrQ8yTSxJstN5kS6Paay6EOIwlvPNHAD4e6ETggQivwTzAeO3hfIdLBJEjTVcKIN1kLKL+IO9skyh/vqBTHYy44+WKq0bldnaXo=,iv:cu7qkM2xwzsZ2z7OZyouP9Pk637yGT6zpGMU1vK8Kng=,tag:73gDM12N6eUPhnZvL4CbAQ==,type:str]", + "data": "ENC[AES256_GCM,data:MO/hhqhg3AN0RUuI/VRzczr+eJSGPUj869RagOJHqlhMluorHo/TXhI8ZchCC4TCtGbI438jED9Tk4/SwgkDvOlh9ImUIPwjBJ4Krf92tjHyCUvmB0RLU65TEM2Y4ZZ/a+/UdN5xI4lFWUiCbi2izvAVkAf7KOM6v0gYd5nSDqd0DnzCEX8S4bYmoK1DujWvkcuoOaSWMCbZj3thmq4UVvR/QekXSwtruIGRVn4HvbRF5jmElitezh5WZ88lMRmdpUMvHMlUBeJ6rhp3QriwiDdhnebHBGJiLJ1NKqFXMyrO+UdnNYj3NyR8rsFEkwygyAqPUCzTLwHXYH+Hdzq055Qh5tRDbNKLDqCdRoJEzi5xV5Xh9fRAMgxX9aEbphKrSmSjKp2hpel2fV46waJhkr45H8+dml9TVpMy9ABL6Z15UMm+Gul423ONGc77Gxfh8KeNl8PJ+N7UwAMIbxOaA2af7Dthf2NsY4wR4IaW8uugEAnRjwg7HBfHbqz5sHQNVKpBoxz9rlMadcP8XFBJIL9XiuYl3gy7fViTbb0iDIC/Yj3EmH4nzrdvXz6Np5EhyXscEIikhRccugkUa/bTFQDuy5P3VW1t7zKqRDcu93r7L7/6K6EHjAbF+d+BpKiJlf0oDvH4F50tyQVPeVE9lxDRPrqrokrsPyA6SD2a3g==,iv:ch1qPhSYuMcmxLWDv6e0rrnCXJuR+PEfkRlTKOFrCB4=,tag:nh3yS+YAz+NGoedW6e7y+w==,type:str]", "sops": { "age": [ { "recipient": "age1a7a70qcpjemlvk6q4uaf4k77p9eq7lj7wcal5jdj3xuetznyqdrs3mfnsf", - "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4dzA0TDloUXZsUG1jMEhr\ncG1RbTJDMXRma1M4SXYyK2M4blRqWWNocVhrClM4VW02VHgrOWloQjdKNkJRODZZ\nRVB2MXp4cmR5Ym41THA5S0dqWEFQWkEKLS0tIEN6VEE3a2ZUdmN6L1ZoNTdkRHdX\nVCtkeU1FWDJZZjM5OW81b0ZHVnU0encKSl6/rKzZz816Vo7kdmo2Ivy4007DTAgy\nKTgygkhbEQ1UnfgO94YZhmeLItjJq8f56+zope+dtjgcAnlzQ205Ow==\n-----END AGE ENCRYPTED FILE-----\n" + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsYzdHVFlPSzBadkhVSlpH\nazUyR2ZXd0pCeUVaS3c5Z3lYQ0ppWmowTTNrClYyaWNIWGxzYitad3hwRWtpTi9l\nRE5MSUNNZjBmbG41TjczY0dwelY5aUkKLS0tIDJXYllRaDRodEE0WXNycHJjWGxn\nYno4dzM2d1VUeGFJR28zNTVUM3QvdUEKipx+/dhr1CajOhxYi0AExLwiTugvlgV5\newpvtd5vPJtRlW1z6gBB3/jQ/mXKmNzZ0ye2/jgi/EXDrc46gKFUAw==\n-----END AGE ENCRYPTED FILE-----\n" }, { "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", - "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBHdENVQW1RcEJsSWYxR3BY\nNGdGNzJxSlRPRkpHelVqR0tEdUxQZEpwWUhNCjJGSDQ0TXl5TVpOYitObG0yUE5w\nN3J6UzZoSUpxUzJscnJ3bk5TVkxjcGMKLS0tIHFxSDd3bEYzdHlmVTl5YWpvMnJ5\nMUgvenN0NElwTVFvbWdrbG04alZSVDAKwGB1h5ahtW27pfIsspJVaCetzr/p1zTR\n9Kh6ZAoqRPxh6dr2iHP0kmiUP5aWDDd5f0yj07jq+y2XfO5ICxt4FA==\n-----END AGE ENCRYPTED FILE-----\n" + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpcjhYVEVxUnJKUk1IVDE5\nQnJ3ajV0eWVjNy9mQ0FPQThsVGFPaTBiV2hZCnJ0OGlXU2MvL0pMYVQyVGtPRHNy\nZGZGTlhGRGhWUlFBS0JFOFVhNFdQSWcKLS0tIEJzNHZrdUR3VVJaWTNZL2RjcnA0\neFBpUURPd1prTEh6YStsSVRRSkdoaW8KImrSlyxhBcEglg5Ng3aD8TEzmz1QuqBc\nvhQvq4fE+al5GTH8y6OI1M0fJ6U86QjgisZ+HTm+autx9uSNKFY07Q==\n-----END AGE ENCRYPTED FILE-----\n" } ], - "lastmodified": "2026-04-24T06:07:27Z", - "mac": "ENC[AES256_GCM,data:raOcQI23iIX0yTroEDhMy/wY9apTjX2lMFfnGkh6gySyF9d1OK3Dz9dbjMi1sPLViXfzQUYypfScyywZpSYTNLaKUDiOCPotpP37tecPuJPTm8YFTXxvEKql6rUrs2dO19b7K1TtZSgyPdr1MlBbIKYN9FwvHXCiZI01iYVj4y4=,iv:OsFByARwKr2I/vYP/irtYQ+BAE0H1U8UbcQ93KSL0vE=,tag:ayxNf1yukt9QhyJIvwE5aw==,type:str]", + "lastmodified": "2026-04-25T04:23:09Z", + "mac": "ENC[AES256_GCM,data:YPbRgJjvjQqhuKAFlRObDKeW8T3mfODb2SxvQhKZES0uZbYhukmB6eUL+pG/ahVvxmgEUjp7iUQilnYGwXvFiaUUBXeAvgL5nihsrOf4FqS5VpxY02Ru07EKVRlySAlzzud6qA0CUZaoHeI906Dz1ya+O660LCHjrVF5mJH2CKQ=,iv:lWG/GoD53zd+U6x/6GLKM7BqSnOEyJF3EQSWTcKJaWg=,tag:4TN+TerZQJjknRNAdD+f0w==,type:str]", "version": "3.12.2" } } diff --git a/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/secret b/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/secret index bd1e00e97..733054c62 100644 --- a/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/secret +++ b/vars/per-machine/magnetite/vanixiets-effects-secrets/sops-age-key/secret @@ -1,14 +1,14 @@ { - "data": "ENC[AES256_GCM,data:vqhuD6dBU/LSTYaQzKV6gyRVvVKPTN/Z56vq5J1zCRHilYRapEFHLc+tIGdaVK9M2owioshL+ntconal9Upg09npJ91genXQXHU=,iv:NHfF1GuC1FX+5tEYdEzkp3HuP3Da2K8byZpF1hFAy2M=,tag:1HtqkhSFu0+XLnWnwsewcQ==,type:str]", + "data": "ENC[AES256_GCM,data:xguBcKIembzF0R1jeBRZpzcMngH7dobeZ9omyA8qsDC9/KyL3xbq88WlMwg100qjOu1qX+Fstd9rDsQIfIxNRs71WxM/98D/nTQ=,iv:z2GfXbqfGm1r2RUkQOhWzkRCa4KK07j7GeQO2+U2tfo=,tag:0GlU1gyhel4vtxBnaKL3Cg==,type:str]", "sops": { "age": [ { "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", - "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrSHBKbUNDdkJlbWd5VThu\nTXhtQ2VkVm9NdkJyd3NnM0hTNEN6N2YxcjNRCmZuNU01S2UrQUtaVTQ1N0hBWXYr\nc3FkV3RVQkpOTGFueGtGVE5JT1BhYjQKLS0tIGN0eTlSbjBvM0ZjckhzenVZWmxx\nVklkdXMyVzErY2k3Z2ZiMHNvUzZNZFkK8Vb9w6D3CqOwLKRw+YRWe1QtJzkWc3xY\nxTmPWXi6ubPSnVZbUmZ4vvLGSvBwv6AidojFJRsmh3C2JBkR9sLLEA==\n-----END AGE ENCRYPTED FILE-----\n" + "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxRW5xTFBPNE4rS2s1aFB3\nZ2xPUmtpSEtYQmlwbkFxVk5OQnVDaXlDSlE4ClNFVXU1TXI5TmJ0eU51TnMwYm5C\neUhKTkg2VlpJc0cya2RNTUpuODhiZVkKLS0tIENYaE9UQk1sN3U4cGZyM01zY1g5\nOGxCeWZGakpnNHp2R2RzU3NwVER1ZzQKJvfm6jku5uDNZfCYXF+DMl4FvgxwEM9l\ni5efofA/vYKzSYv3P9JbymhfgDdBgcWaVqamj8/PsKrA01w7oby06A==\n-----END AGE ENCRYPTED FILE-----\n" } ], - "lastmodified": "2026-04-24T06:07:27Z", - "mac": "ENC[AES256_GCM,data:NaUGtQVXgnAYEfZpE7D6y4nygxB9QLeVY99wYJvM1xrMCV/B/XHhsF5GGH7vqISnwxwFVQRCZDwQKc/zowpufB0aifUWzp5ceWdBYTlLsGzR51/G3yiaCtCNS32G0gjZE6qV+l1SBOyDCcrcgPULTMdH/7u+deEUE56RtZplSho=,iv:TBuoSlst0YN746JD49rf8SXmCiuUGt1pBVRk8wTdSk4=,tag:R4dmtQryBuHJFI2U0BoTBA==,type:str]", + "lastmodified": "2026-04-25T04:23:10Z", + "mac": "ENC[AES256_GCM,data:HhA9HdbqnUYoP1qDD+K3Hz1Ax1OFj2al3uTLz8+U2gJfxuu2wvi3EENZ8frcxyjXuCaV2NfOc+5QjipfxvcOW9hR0d1uo6PCdZNodWY3Ps6Ym54KF+sWopyaQfSs3acOdnePz8TBD2HPQv/iBBczUabWW04R9VQVBLiB+CEFyKs=,iv:AnFFhyikMzgd6r7S4AUnIuNh894TuSYGZLBC528n22o=,tag:khEErihhRvFv11+74XhT1w==,type:str]", "version": "3.12.2" } } From 4f29a6d52788f0c639dee89bd91c3269ac387646 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 01:10:35 -0400 Subject: [PATCH 35/77] fix(effects): add git + runtime PATH inputs to release-packages mkEffect --- .../vanixiets/herculesCI/release-packages.nix | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 02cb1044c..34136c563 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -167,6 +167,19 @@ hci-effects.mkEffect { name = "release-packages"; + # Runtime PATH inputs for the effectScript body (m5-01d-release-packages-runtimeinputs-fix). + # mkEffect's defaultInputs (cacert + curl + jq + effectSetupHook) plus stdenvNoCC's + # bundled coreutils/bash/gnused/gnugrep/gawk/gnutar cover every directly-invoked binary + # in the effectScript EXCEPT `git`, which the ADR-003 Option α clone preamble calls + # via `git clone`, `git fetch`, `git checkout -B`, and `git rev-parse`. The flake apps + # invoked downstream (`${listPackagesProgram}`, `${releaseProgram}`) carry their own + # writeShellApplication-baked runtimeInputs PATH so internal tool resolution is + # self-contained. Adding `pkgs.git` here closes the m5-01c Phase 1 dry-run regression + # where the bwrap sandbox emitted RELEASE-CLONE-START correctly and then failed with + # `git: command not found` from stdenv-linux/setup line 1842 (`git clone "$clone_url" …`). + # `cacert` is already in defaultInputs, so HTTPS clone CA-trust resolution is unaffected. + inputs = [ pkgs.git ]; + effectScript = '' set -euo pipefail From b2ab8b4ad4b1b0405af99eea9e167c0c76cf3cad Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 10:46:39 -0400 Subject: [PATCH 36/77] feat(effects): delegate non-main release-packages dispatch to preview-version flake app --- .../vanixiets/herculesCI/release-packages.nix | 54 ++++++++++++++----- 1 file changed, 42 insertions(+), 12 deletions(-) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 34136c563..504259c23 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -16,17 +16,30 @@ # dispatches via a flake-app shell-out (`nix run`); bwrap does not # bind the working tree, so the .# syntax cannot resolve. # -# Branch dispatch (exact string equality): +# Branch dispatch (exact string equality; m5-01e Option C delegation): # Selection is driven by `primaryRepo.branch == "main"`, surfaced # through `herculesCI.config.repo.branch` which hercules-ci-effects # populates from the primaryRepo record at flake.herculesCI entry. # * primaryRepo.branch == "main" → real semantic-release per package -# (semantic-release's per-package commit-analyzer decides whether -# to cut a release; tag push + GitHub Release published when so; -# npmPublish stays false — never overridden). -# * any other branch (or null) → per-package `--dry-run` -# (semantic-release prints the next-version preview only; no git -# tags pushed, no GitHub Release created, no remote git mutation). +# via the `${releaseProgram}` flake app (release.sh production +# path; semantic-release's per-package commit-analyzer decides +# whether to cut a release; tag push + GitHub Release published +# when so; npmPublish stays false — never overridden). +# * any other branch (or null) → per-package merge-preview via +# the `${previewVersionProgram}` flake app (preview-version.sh; +# m5-01e delegation to the existing flake app, Option C, closes +# the m5-01c Phase 1 version-preview gap). preview-version.sh +# simulates merging the current branch into `main` via +# `git merge-tree --write-tree` + temporary worktree, then runs +# semantic-release with `--branches "$TARGET_BRANCH"` and the +# commit-analyzer + release-notes-generator plugin pair only +# (no `@semantic-release/github`, no tag push, no GitHub +# Release, no remote git mutation). The previous non-main path +# delegated to `release.sh --dry-run`, which short-circuited on +# the in-tree `branches: ["main"]` config before exercising +# analyzeCommits/generateNotes; preview-version's +# `--branches` override is what makes the version-preview path +# actually run for cd-via-effects and other non-main branches. # # Pattern C'-refined secrets preamble: # Extracts GITHUB_TOKEN ONLY from $HERCULES_CI_SECRETS_JSON at the @@ -160,9 +173,14 @@ # Option Gamma: resolved at nix eval time to /nix/store paths # that the bwrap sandbox can execute without a working-tree or - # nix-daemon lookup. + # nix-daemon lookup. preview-version-program added in m5-01e + # (Option C) so non-main runs can delegate to the existing + # `preview-version` flake app rather than re-using + # `release.sh --dry-run` (which short-circuits on + # branches:["main"] before exercising the analyzeCommits path). listPackagesProgram = config.apps.list-packages-json.program; releaseProgram = config.apps.release.program; + previewVersionProgram = config.apps.preview-version.program; in hci-effects.mkEffect { name = "release-packages"; @@ -327,12 +345,17 @@ export GIT_COMMITTER_NAME=semantic-release export GIT_COMMITTER_EMAIL=semantic-release@vanixiets.local - # Option Gamma store-path dispatch — both flake apps' /nix/store - # paths are embedded at eval time via the perSystem + # Option Gamma store-path dispatch — all three flake apps' + # /nix/store paths are embedded at eval time via the perSystem # config.apps..program attributes. No flake-app shell-out - # (bwrap would not resolve .#). + # (bwrap would not resolve .#). PREVIEW is unused on the main + # branch path and RELEASE is unused on the non-main branch + # path — both are exported unconditionally for log auditability + # so an operator inspecting the rendered effectScript sees the + # full set of /nix/store paths the effect was built against. LIST_PACKAGES=${listPackagesProgram} RELEASE=${releaseProgram} + PREVIEW=${previewVersionProgram} # cd into the clone before invoking list-packages-json: that # script calls `git rev-parse --show-toplevel`, which must @@ -362,8 +385,15 @@ # Disable -e for the per-package invocation so a single # package's failure does not abort the loop. We capture # the exit code, log appropriately, and continue. + # m5-01e Option C: eval-time branch the dispatch line so + # the rendered bash invokes either the production + # release.sh path (main) or the merge-preview + # preview-version.sh path (non-main). The CLI grammars + # differ — `release [--dry-run]` vs + # `preview-version [target-branch] [package-path]` — so a + # single shared variable + shared flag would not work. set +e - "$RELEASE" "$pkg_path" ${dryRunFlag} + ${if isMain then ''"$RELEASE" "$pkg_path"'' else ''"$PREVIEW" main "$pkg_path"''} rc=$? set -e From f750940e01a8f55e0221d9c89dc780dba585d5f1 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 12:31:19 -0400 Subject: [PATCH 37/77] fix(effects): handle GitHub refs/pull//merge ref form in release-packages clone preamble --- .../vanixiets/herculesCI/release-packages.nix | 178 +++++++++++++++--- 1 file changed, 149 insertions(+), 29 deletions(-) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 504259c23..186151600 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -92,6 +92,55 @@ # mutating effects (security-update-pr, dep-bump) should prefer # upstream `hci-effects.git-update`/`flakeUpdate`, NOT a local helper. # +# Three-way branch-form handling (m5-01g-release-packages-pr-merge-ref-handling): +# buildbot-nix dispatches `release-packages` with three structurally +# distinct `branch` shapes; the clone preamble distinguishes them +# eval-time and emits matched bash for each: +# +# (A) GitHub PR push event — `branch = "refs/pull//merge"` +# (the synthetic GitHub test-merge ref form). Detected +# eval-time via `builtins.match "^refs/pull/([0-9]+)/merge$"`. +# PR-detection pattern modelled on buildbot-nix's +# `buildbot_nix/buildbot_nix/build_canceller.py:16` +# (`branch.startswith((\"refs/pull/\", \"refs/merge-requests/\"))`), +# narrowed to GitHub form here. Synthetic local branch +# `pr--merge` (no slashes; unambiguous to `git rev-parse`) +# replaces the raw ref name for `git checkout -B`. Freshness +# check uses a custom refspec +# `+refs/pull//merge:refs/remotes/origin/pr--merge` so +# `git rev-parse origin/pr--merge` resolves; without this +# mapping, `git fetch origin refs/pull//merge` only +# populates FETCH_HEAD and the rev-parse fails with +# `fatal: ambiguous argument 'origin/refs/pull//merge'` +# (the production blocker captured on PR #1858, 2026-04-25). +# Refspec form modelled on buildbot-nix's +# `buildbot_nix/buildbot_nix/nix_eval.py:GitLocalPrMerge.run` +# fetch idiom. Emits `RELEASE-CLONE-PR-MERGE: ` +# BEFORE the standard `RELEASE-CLONE-START`; emits +# `RELEASE-CLONE-STALE-PR` (instead of `RELEASE-CLONE-STALE`) +# on stale rev. +# +# (B) regular branch push — `branch` non-empty, non-PR-ref. +# Pre-m5-01g flow unchanged: `checkout_branch=$GIT_BRANCH`, +# `git fetch origin $GIT_BRANCH`, +# `git rev-parse origin/$GIT_BRANCH` for freshness. +# +# (C) tag-push event — `branch = null` (hercules-ci-effects +# models tag checkouts this way). Pre-m5-01g flow unchanged: +# synthetic local branch `release-packages-detached`, +# freshness check skipped (no branch tip; dry-run gate +# ensures no production push). See ADR-003 §"Tag-push event +# handling". +# +# Local-CLI flows (`nix run .#preview-version`) are unaffected by +# this branching — they only ever run from a real working tree +# where `branch` is a regular ref. The PR-merge form arises only +# inside the buildbot-nix dispatch path. +# Full ADR-003 invariant audit (§1–§13) is preserved; the m5-01g +# refactor generalises the freshness-check tracking-ref form +# (invariant #11) without altering the single-check cadence or any +# other invariant. +# # Symmetric env-var contract (CI + GIT_AUTHOR/COMMITTER + RELEASE_REPO_ROOT) # (m4-release-packages-runtime-deps-contract; extended in m5-01a): # Exports CI=true, GIT_BRANCH (from herculesCI.config.repo.branch at @@ -157,6 +206,21 @@ # in Nix, so tag pushes naturally fall through to the dry-run path. isMain = branch == "main"; + # GitHub PR-merge ref detection (m5-01g). buildbot-nix dispatches + # `release-packages` on PR push events with `--branch` set to the + # synthetic GitHub test-merge ref form `refs/pull//merge` rather + # than the PR head branch name. This is the dominant non-main + # dispatch path in production. Pattern modelled on buildbot-nix's + # `build_canceller.py:16` PR-detection idiom (`branch.startswith + # ((\"refs/pull/\", \"refs/merge-requests/\"))`); narrowed to the + # GitHub form here because GitLab is not a vanixiets backend. + # `builtins.match` returns null on no-match and a list of capture + # groups on success, so `prMergeMatch != null` is the canonical + # eval-time predicate. + prMergeMatch = if branch == null then null else builtins.match "^refs/pull/([0-9]+)/merge$" branch; + isPrMerge = prMergeMatch != null; + prNumber = if isPrMerge then builtins.head prMergeMatch else null; + # Action banner emitted once per run (RP-05 log-grep anchor). actionBanner = if isMain then "release" else "dry-run"; @@ -258,6 +322,17 @@ # phases share one canonicalized value without re-shelling-out. GIT_REV=${lib.escapeShellArg (toString rev)} GIT_BRANCH=${lib.escapeShellArg (if branch == null then "" else toString branch)} + ${lib.optionalString isPrMerge '' + # PR-merge banner (m5-01g, VAL-RELEASE-α-PR-001) emitted + # ONLY on PR-merge dispatch BEFORE the standard + # CLONE-START banner so log-grep can distinguish the + # synthetic-ref dispatch without further parsing. PR + # number is parsed eval-time from the synthetic GitHub + # test-merge ref via `builtins.match` and embedded as a + # literal here so the rendered effectScript carries the + # parsed number directly (no bash indirection). + echo "RELEASE-CLONE-PR-MERGE: ${toString prNumber} $GIT_REV" + ''} # RELEASE-CLONE-START banner (invariants #9, #10): emitted # BEFORE the clone with the sanitized public URL. Token never @@ -279,35 +354,80 @@ git -C "$clone_dir" fetch --tags origin # Exact-rev checkout (ADR-003 §Architecture step 3, invariant - # #3). When GIT_BRANCH is empty (tag-push events: hercules-ci- - # effects models tag checkouts as `branch = null`), use a - # synthetic local branch name. The dry-run gate (isMain == - # false) ensures no production push attempts in that case; - # see ADR-003 §"Tag-push event handling". - if [ -n "$GIT_BRANCH" ]; then - checkout_branch="$GIT_BRANCH" - else - checkout_branch="release-packages-detached" - fi - git -C "$clone_dir" checkout -B "$checkout_branch" "$GIT_REV" - echo "RELEASE-CLONE-CHECKOUT: $GIT_REV" - - # Freshness check (ADR-003 §Architecture step 4, invariant - # #11). Single check after checkout, before the package loop. - # Per-package re-checks are NOT added (see ADR-003 §11). On - # tag-push events (GIT_BRANCH empty) there is no branch tip - # to compare against, so the check is structurally - # inapplicable; the ADR-003 §"Tag-push event handling" gate - # forces dry-run for those runs anyway. - if [ -n "$GIT_BRANCH" ]; then - git -C "$clone_dir" fetch origin "$GIT_BRANCH" - head_rev="$(git -C "$clone_dir" rev-parse HEAD)" - remote_rev="$(git -C "$clone_dir" rev-parse "origin/$GIT_BRANCH")" - if [ "$head_rev" != "$remote_rev" ]; then - echo "RELEASE-CLONE-STALE: expected $head_rev remote $remote_rev" >&2 - exit 1 - fi - fi + # #3) + branch-form-aware freshness check (ADR-003 §Architecture + # step 4, invariant #11; m5-01g three-way generalization). + # + # Three eval-time-distinguished cases: + # + # Case A (isPrMerge==true): GitHub PR push event with + # `branch = refs/pull//merge`. Synthetic local branch + # name `pr--merge` (no slashes; unambiguous to git ref + # resolution). Freshness check uses a custom refspec + # `+refs/pull//merge:refs/remotes/origin/pr--merge` + # to materialize the missing remote-tracking ref — + # `git fetch origin refs/pull//merge` alone updates + # FETCH_HEAD but does NOT auto-create + # `refs/remotes/origin/refs/pull//merge`, which is the + # production blocker captured on PR #1858 (`fatal: + # ambiguous argument 'origin/refs/pull/1858/merge'`). + # Refspec form modelled on buildbot-nix's + # `nix_eval.py:GitLocalPrMerge.run` fetch idiom. + # + # Case B (isPrMerge==false, GIT_BRANCH non-empty): regular + # branch push. Identical to the pre-m5-01g flow. + # + # Case C (isPrMerge==false, GIT_BRANCH empty): tag-push + # event (hercules-ci-effects models tag checkouts as + # `branch = null`). Synthetic local branch name + # `release-packages-detached`; no freshness check (no + # branch tip to compare against; the dry-run gate + # `isMain == false` ensures no production push in this + # case). Identical to the pre-m5-01g flow. + ${ + if isPrMerge then + '' + checkout_branch="pr-${toString prNumber}-merge" + git -C "$clone_dir" checkout -B "$checkout_branch" "$GIT_REV" + echo "RELEASE-CLONE-CHECKOUT: $GIT_REV" + + # Custom-refspec freshness check: create the missing + # remote-tracking ref under refs/remotes/origin/pr--merge + # so `git rev-parse origin/pr--merge` resolves + # unambiguously. Without this refspec mapping, the + # plain `git fetch origin refs/pull//merge` only + # updates FETCH_HEAD. Refspec form modelled on + # buildbot-nix's `nix_eval.py:GitLocalPrMerge.run` + # idiom; the PR number is the eval-time-parsed literal. + git -C "$clone_dir" fetch origin \ + "+refs/pull/${toString prNumber}/merge:refs/remotes/origin/pr-${toString prNumber}-merge" + head_rev="$(git -C "$clone_dir" rev-parse HEAD)" + remote_rev="$(git -C "$clone_dir" rev-parse "origin/pr-${toString prNumber}-merge")" + if [ "$head_rev" != "$remote_rev" ]; then + echo "RELEASE-CLONE-STALE-PR: expected $head_rev remote $remote_rev" >&2 + exit 1 + fi + '' + else + '' + if [ -n "$GIT_BRANCH" ]; then + checkout_branch="$GIT_BRANCH" + else + checkout_branch="release-packages-detached" + fi + git -C "$clone_dir" checkout -B "$checkout_branch" "$GIT_REV" + echo "RELEASE-CLONE-CHECKOUT: $GIT_REV" + + if [ -n "$GIT_BRANCH" ]; then + git -C "$clone_dir" fetch origin "$GIT_BRANCH" + head_rev="$(git -C "$clone_dir" rev-parse HEAD)" + remote_rev="$(git -C "$clone_dir" rev-parse "origin/$GIT_BRANCH")" + if [ "$head_rev" != "$remote_rev" ]; then + echo "RELEASE-CLONE-STALE: expected $head_rev remote $remote_rev" >&2 + exit 1 + fi + fi + '' + } echo "RELEASE-CLONE-READY: $clone_dir" From 4fbaa8f75bf5b778d7b9b09f578ea8ed91109074 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 13:18:15 -0400 Subject: [PATCH 38/77] fix(effects): pivot release-packages PR-dispatch to refs/pull//head form --- .../vanixiets/herculesCI/release-packages.nix | 241 ++++++++++-------- 1 file changed, 138 insertions(+), 103 deletions(-) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 186151600..c81eab856 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -92,33 +92,78 @@ # mutating effects (security-update-pr, dep-bump) should prefer # upstream `hci-effects.git-update`/`flakeUpdate`, NOT a local helper. # -# Three-way branch-form handling (m5-01g-release-packages-pr-merge-ref-handling): +# Three-way branch-form handling (m5-01h-release-packages-pr-head-ref-pivot; +# supersedes m5-01g-release-packages-pr-merge-ref-handling, commit 4b66343fe): # buildbot-nix dispatches `release-packages` with three structurally # distinct `branch` shapes; the clone preamble distinguishes them # eval-time and emits matched bash for each: # # (A) GitHub PR push event — `branch = "refs/pull//merge"` # (the synthetic GitHub test-merge ref form). Detected -# eval-time via `builtins.match "^refs/pull/([0-9]+)/merge$"`. -# PR-detection pattern modelled on buildbot-nix's -# `buildbot_nix/buildbot_nix/build_canceller.py:16` +# eval-time via `builtins.match "^refs/pull/([0-9]+)/merge$"` +# (eval-time predicate kept identical to m5-01g; the dispatch +# input form has not changed — only how the effect resolves +# the SHA-of-record from it). PR-detection pattern modelled +# on buildbot-nix's `buildbot_nix/buildbot_nix/build_canceller.py:16` # (`branch.startswith((\"refs/pull/\", \"refs/merge-requests/\"))`), -# narrowed to GitHub form here. Synthetic local branch -# `pr--merge` (no slashes; unambiguous to `git rev-parse`) -# replaces the raw ref name for `git checkout -B`. Freshness -# check uses a custom refspec -# `+refs/pull//merge:refs/remotes/origin/pr--merge` so -# `git rev-parse origin/pr--merge` resolves; without this -# mapping, `git fetch origin refs/pull//merge` only -# populates FETCH_HEAD and the rev-parse fails with -# `fatal: ambiguous argument 'origin/refs/pull//merge'` -# (the production blocker captured on PR #1858, 2026-04-25). -# Refspec form modelled on buildbot-nix's +# narrowed to GitHub form here. +# +# m5-01h pivot rationale — GitHub's `refs/pull//merge` is a +# SYNTHETIC, EPHEMERAL, NON-STABLE test-merge commit. GitHub +# recomputes it whenever the base branch advances, the PR head +# is updated, or its internal merge-test scheduler fires. +# buildbot-nix snapshots the merge-SHA at nix-eval time (T0) and +# passes it as `--rev`, but by effect-runtime (T1) GitHub may +# have recomputed the merge under the same ref name. The m5-01g +# production log on PR #1858 captured `RELEASE-CLONE-STALE-PR: +# expected f750940... remote 030a3498...` — a true-positive +# staleness signal exposing that the merge-ref form is +# fundamentally the wrong unit of truth for this dispatch path. +# +# m5-01h fetches `+refs/pull//head:refs/remotes/origin/pr--head` +# instead. `refs/pull//head` is the developer-pushed PR +# source branch tip, stable until the next dev push, and is +# what fast-forward-merge dry-run analysis actually wants — +# preview-version.sh's `git merge-tree --branches main` +# simulation operates from a working tree against main, so +# giving it the PR-head working tree is exactly correct. The +# SHA actually checked out is resolved at runtime via +# `head_sha=$(git rev-parse origin/pr--head)` post-fetch; +# buildbot's `--rev` (the ephemeral merge SHA) is retained +# ONLY as a forensic record in the DISPATCH banner. +# +# Synthetic local branch `pr--head` (no slashes; unambiguous +# to `git rev-parse`) replaces the raw ref name for +# `git checkout -B`. Refspec form modelled on buildbot-nix's # `buildbot_nix/buildbot_nix/nix_eval.py:GitLocalPrMerge.run` -# fetch idiom. Emits `RELEASE-CLONE-PR-MERGE: ` -# BEFORE the standard `RELEASE-CLONE-START`; emits -# `RELEASE-CLONE-STALE-PR` (instead of `RELEASE-CLONE-STALE`) -# on stale rev. +# fetch idiom (the `+ref:remote-tracking-ref` mapping form), +# reused here for the /head ref instead of the /merge ref. +# +# Emits TWO banners BEFORE the standard `RELEASE-CLONE-START`: +# * canonical positional `RELEASE-CLONE-PR-HEAD: +# ` — the SHA actually checked out and analyzed. +# * forensic key=value `RELEASE-CLONE-PR-DISPATCH: +# buildbot-rev= head=` — informational +# record of buildbot's `--rev` (the ephemeral T0 merge SHA) +# alongside the runtime-resolved head SHA. Drift between +# the two values is normal and benign. +# m5-01g's `RELEASE-CLONE-PR-MERGE` and `RELEASE-CLONE-STALE-PR` +# banners are RETIRED (no apples-to-apples freshness comparison +# is meaningful for the /head form: the head SHA is fresh by +# construction post-fetch). A trivial head-existence sanity +# check `git rev-parse --verify origin/pr--head` runs after +# checkout — non-zero only on a rare force-push race that +# removes the head ref between fetch and verify, in which case +# set -e aborts the effect. Cadence-equivalent to invariant #11 +# (single freshness check), generalised to a sanity probe. +# +# Upstream gap (skipping per user direction): `buildbot-effects` +# CLI accepts only `--rev/--branch/--repo/--secrets` (cli.py:103 +# has `# TODO: support ref`), and the bwrap sandbox strips env +# to {IN_HERCULES_CI_EFFECT, HERCULES_CI_SECRETS_JSON, +# NIX_BUILD_TOP, TMPDIR, NIX_REMOTE} so we cannot smuggle +# GitHub env in. Hand-rolled refspec inside the effectScript +# is the only path; we own it. # # (B) regular branch push — `branch` non-empty, non-PR-ref. # Pre-m5-01g flow unchanged: `checkout_branch=$GIT_BRANCH`, @@ -136,10 +181,12 @@ # this branching — they only ever run from a real working tree # where `branch` is a regular ref. The PR-merge form arises only # inside the buildbot-nix dispatch path. -# Full ADR-003 invariant audit (§1–§13) is preserved; the m5-01g -# refactor generalises the freshness-check tracking-ref form -# (invariant #11) without altering the single-check cadence or any -# other invariant. +# Full ADR-003 invariant audit (§1–§13) is preserved; m5-01h +# generalises invariant #3 (exact-rev source: `head_sha` runtime- +# resolved post-fetch is the new exact-rev for case A) and +# invariant #11 (freshness-check shape: head-existence sanity +# probe for case A) without altering the single-check cadence or +# any other invariant. # # Symmetric env-var contract (CI + GIT_AUTHOR/COMMITTER + RELEASE_REPO_ROOT) # (m4-release-packages-runtime-deps-contract; extended in m5-01a): @@ -322,93 +369,81 @@ # phases share one canonicalized value without re-shelling-out. GIT_REV=${lib.escapeShellArg (toString rev)} GIT_BRANCH=${lib.escapeShellArg (if branch == null then "" else toString branch)} - ${lib.optionalString isPrMerge '' - # PR-merge banner (m5-01g, VAL-RELEASE-α-PR-001) emitted - # ONLY on PR-merge dispatch BEFORE the standard - # CLONE-START banner so log-grep can distinguish the - # synthetic-ref dispatch without further parsing. PR - # number is parsed eval-time from the synthetic GitHub - # test-merge ref via `builtins.match` and embedded as a - # literal here so the rendered effectScript carries the - # parsed number directly (no bash indirection). - echo "RELEASE-CLONE-PR-MERGE: ${toString prNumber} $GIT_REV" - ''} - - # RELEASE-CLONE-START banner (invariants #9, #10): emitted - # BEFORE the clone with the sanitized public URL. Token never - # appears in this output even though GIT_CREDENTIALS is set - # later in this effect — the URL string is the canonical one. - echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" - - # Full clone (ADR-003 §Architecture step 2, invariant #2): - # NO shallow flag, NO since-date flag, NO blob-filter flag. - # semantic-release requires `git tag --merged ` - # (lib/git.js:24-31), the full unshallowed tag fetch - # (lib/git.js:106-132), notes refs (lib/git.js:144-154), and - # the full commit log since lastRelease.gitHead - # (lib/get-commits.js:7-25); semantic-release-monorepo - # additionally needs per-commit changed-file lookups - # (src/git-utils.js:16-24). Pack size (~21.5 MiB) does not - # warrant shallow optimization at this scale. - git clone "$clone_url" "$clone_dir" - git -C "$clone_dir" fetch --tags origin - - # Exact-rev checkout (ADR-003 §Architecture step 3, invariant - # #3) + branch-form-aware freshness check (ADR-003 §Architecture - # step 4, invariant #11; m5-01g three-way generalization). - # - # Three eval-time-distinguished cases: - # - # Case A (isPrMerge==true): GitHub PR push event with - # `branch = refs/pull//merge`. Synthetic local branch - # name `pr--merge` (no slashes; unambiguous to git ref - # resolution). Freshness check uses a custom refspec - # `+refs/pull//merge:refs/remotes/origin/pr--merge` - # to materialize the missing remote-tracking ref — - # `git fetch origin refs/pull//merge` alone updates - # FETCH_HEAD but does NOT auto-create - # `refs/remotes/origin/refs/pull//merge`, which is the - # production blocker captured on PR #1858 (`fatal: - # ambiguous argument 'origin/refs/pull/1858/merge'`). - # Refspec form modelled on buildbot-nix's - # `nix_eval.py:GitLocalPrMerge.run` fetch idiom. - # - # Case B (isPrMerge==false, GIT_BRANCH non-empty): regular - # branch push. Identical to the pre-m5-01g flow. - # - # Case C (isPrMerge==false, GIT_BRANCH empty): tag-push - # event (hercules-ci-effects models tag checkouts as - # `branch = null`). Synthetic local branch name - # `release-packages-detached`; no freshness check (no - # branch tip to compare against; the dry-run gate - # `isMain == false` ensures no production push in this - # case). Identical to the pre-m5-01g flow. + # === Branch-form-aware clone + checkout + freshness/sanity === + # ADR-003 §Architecture steps 2-5 + invariants #1, #2, #3, #9, + # #10, #11. Three eval-time-distinguished cases dispatched by + # the eval-time conditional below; see the file-header Nix + # doc-comment for the full case-by-case rationale and the + # m5-01h design pivot. ${ if isPrMerge then '' - checkout_branch="pr-${toString prNumber}-merge" - git -C "$clone_dir" checkout -B "$checkout_branch" "$GIT_REV" - echo "RELEASE-CLONE-CHECKOUT: $GIT_REV" - - # Custom-refspec freshness check: create the missing - # remote-tracking ref under refs/remotes/origin/pr--merge - # so `git rev-parse origin/pr--merge` resolves - # unambiguously. Without this refspec mapping, the - # plain `git fetch origin refs/pull//merge` only - # updates FETCH_HEAD. Refspec form modelled on - # buildbot-nix's `nix_eval.py:GitLocalPrMerge.run` - # idiom; the PR number is the eval-time-parsed literal. + # m5-01h Case A: clone first so head_sha can be resolved + # before the canonical/forensic banners are emitted. + git clone "$clone_url" "$clone_dir" + git -C "$clone_dir" fetch --tags origin + + # Custom-refspec head-fetch (modelled on buildbot-nix's + # `nix_eval.py:GitLocalPrMerge.run` idiom; the PR number + # is the eval-time-parsed literal). Materializes + # `refs/remotes/origin/pr-${toString prNumber}-head` so + # the subsequent `git rev-parse` resolves unambiguously. + # `git fetch origin refs/pull//head` alone updates + # FETCH_HEAD but does NOT auto-create the remote- + # tracking ref; the explicit `+ref:remote-tracking-ref` + # mapping closes that gap. git -C "$clone_dir" fetch origin \ - "+refs/pull/${toString prNumber}/merge:refs/remotes/origin/pr-${toString prNumber}-merge" - head_rev="$(git -C "$clone_dir" rev-parse HEAD)" - remote_rev="$(git -C "$clone_dir" rev-parse "origin/pr-${toString prNumber}-merge")" - if [ "$head_rev" != "$remote_rev" ]; then - echo "RELEASE-CLONE-STALE-PR: expected $head_rev remote $remote_rev" >&2 - exit 1 - fi + "+refs/pull/${toString prNumber}/head:refs/remotes/origin/pr-${toString prNumber}-head" + head_sha="$(git -C "$clone_dir" rev-parse origin/pr-${toString prNumber}-head)" + + # Two banners (m5-01h, VAL-RELEASE-α-PR-001) emitted + # BEFORE the standard `RELEASE-CLONE-START` line so + # log-grep can distinguish the PR-head dispatch path + # from regular-branch dispatch without further parsing. + # * canonical positional: SHA actually checked out + # and analyzed. + # * forensic key=value: buildbot's `--rev` (the + # ephemeral GitHub-computed merge SHA at eval-time + # T0) alongside the runtime-resolved head SHA. + # Drift between the two values is normal and benign + # (GitHub may have recomputed the synthetic merge + # between T0 and T1; the head SHA is the stable + # dev-pushed reference). + echo "RELEASE-CLONE-PR-HEAD: ${toString prNumber} $head_sha" + echo "RELEASE-CLONE-PR-DISPATCH: ${toString prNumber} buildbot-rev=$GIT_REV head=$head_sha" + + # Standard upstream-input record (invariants #9, #10): + # sanitized public URL only — token never appears here + # even though GIT_CREDENTIALS is exported below for + # semantic-release's URL builder. + echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" + + # Exact-rev checkout (ADR-003 §Architecture step 3, + # invariant #3 — generalised: $head_sha is the new + # exact-rev source for case A; buildbot's $GIT_REV is + # the ephemeral merge SHA and is NOT a valid checkout + # target). Synthetic local branch `pr--head` (no + # slashes; unambiguous to git ref resolution). + git -C "$clone_dir" checkout -B "pr-${toString prNumber}-head" "$head_sha" + echo "RELEASE-CLONE-CHECKOUT: $head_sha" + + # Head-existence sanity check (m5-01h; replaces m5-01g's + # STALE-PR failure mode). Trivially true post-fetch + # unless the head ref disappears (rare force-push race), + # in which case the non-zero exit propagates via set -e + # and aborts the effect. Cadence-equivalent to invariant + # #11 (single freshness check), generalised to a sanity + # probe for case A. + git -C "$clone_dir" rev-parse --verify origin/pr-${toString prNumber}-head >/dev/null '' else '' + # Cases B and C: unchanged from m5-01g (pre-m5-01h state). + echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" + + git clone "$clone_url" "$clone_dir" + git -C "$clone_dir" fetch --tags origin + if [ -n "$GIT_BRANCH" ]; then checkout_branch="$GIT_BRANCH" else From a2f8b58b6ebad53ac542e38273201cd56a5a48cf Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 14:35:48 -0400 Subject: [PATCH 39/77] fix(docs): redirect semantic-release verifyAuth to local bare clone --- modules/apps/docs/preview-version.sh | 43 +++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/modules/apps/docs/preview-version.sh b/modules/apps/docs/preview-version.sh index 5a0691319..0d6e208e7 100644 --- a/modules/apps/docs/preview-version.sh +++ b/modules/apps/docs/preview-version.sh @@ -95,6 +95,16 @@ ORIGINAL_REMOTE_HEAD="" WORKTREE_NODE_MODULES_LINK="" LOCAL_NODE_MODULES_LINK="" +# Local bare clone used to redirect semantic-release verifyAuth's +# `git push --dry-run HEAD:` away from the GitHub remote +# (which can short-circuit semantic-release on branch-protection rejection +# or token-permission mismatch — see m5-01i mission notes). +# Populated AFTER `git update-ref` so the bare's refs/heads/ +# captures TEMP_COMMIT, allowing the dry-run push to be a no-op fast-forward +# against a quiescent file:// remote with no auth and no protection. +PREVIEW_BARE_DIR="" +PREVIEW_BARE="" + # Colors for output RED='\033[0;31m' GREEN='\033[0;32m' @@ -175,6 +185,12 @@ cleanup() { git worktree prune 2>/dev/null || true fi + # Clean up local bare clone created for semantic-release verifyAuth + # redirection (see m5-01i fix). + if [ -n "$PREVIEW_BARE_DIR" ] && [ -d "$PREVIEW_BARE_DIR" ]; then + rm -rf "$PREVIEW_BARE_DIR" + fi + # Restore detached HEAD if we attached it via the CURRENT_BRANCH override path. # Gated on WE_ATTACHED_HEAD so this is a no-op in the normal attached-HEAD flow. # Must cd to REPO_ROOT first because cleanup may be triggered while cwd is @@ -288,6 +304,26 @@ git update-ref "refs/heads/$TARGET_BRANCH" "$TEMP_COMMIT" # Also update remote-tracking branch to match (so semantic-release sees them as synchronized) git update-ref "refs/remotes/origin/$TARGET_BRANCH" "$TEMP_COMMIT" +# Capture the post-update-ref state into a local bare clone so semantic-release's +# `verifyAuth` can run `git push --dry-run HEAD:` against a +# quiescent file:// remote instead of the GitHub origin (m5-01i fix). +# +# The bare must be cloned from $REPO_ROOT (cwd's local refs at clone time +# include the just-updated refs/heads/ = TEMP_COMMIT). Cloning +# from $REPO_ROOT — not WORKTREE_DIR which has not been created yet — is what +# makes verifyAuth's push a trivial no-op fast-forward. +# +# Without this redirect, semantic-release v25.0.3's `lib/git.js:205-211` +# performs a real network round-trip to GitHub, which short-circuits the run +# whenever branch protection or token-permission mismatches reject the +# dry-run push (then `lib/git.js:282-290` strict-=== compare against +# TEMP_COMMIT can never succeed and `index.js:84-100` bails with "behind +# the remote one"), preventing analyzeCommits from ever firing. +echo -e "${BLUE}creating local bare clone for semantic-release repository-url override...${NC}" +PREVIEW_BARE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/preview-bare.XXXXXX") +PREVIEW_BARE="$PREVIEW_BARE_DIR/preview.git" +git clone --quiet --bare "$REPO_ROOT" "$PREVIEW_BARE" + # Create worktree at target branch (now pointing to merge commit) echo -e "${BLUE}creating temporary worktree at ${TARGET_BRANCH}...${NC}" git worktree add --quiet "$WORKTREE_DIR" "$TARGET_BRANCH" @@ -316,7 +352,12 @@ echo -e "\n${BLUE}running semantic-release analysis...${NC}\n" # This is safe because dry-run skips publish/success/fail steps anyway PLUGINS="@semantic-release/commit-analyzer,@semantic-release/release-notes-generator" -OUTPUT=$(GITHUB_REF="refs/heads/$TARGET_BRANCH" node ./node_modules/.bin/semantic-release --dry-run --no-ci --branches "$TARGET_BRANCH" --plugins "$PLUGINS" 2>&1 || true) +# Forensic banner (m5-01i): confirms the verifyAuth-redirect bare clone is +# engaged in production logs. Stable banner namespace; kept structurally +# similar to RELEASE-CLONE-PR-HEAD / RELEASE-CLONE-PR-DISPATCH. +echo "RELEASE-PREVIEW-BARE: $PREVIEW_BARE" + +OUTPUT=$(GITHUB_REF="refs/heads/$TARGET_BRANCH" node ./node_modules/.bin/semantic-release --dry-run --no-ci --repository-url "file://$PREVIEW_BARE" --branches "$TARGET_BRANCH" --plugins "$PLUGINS" 2>&1 || true) # Display semantic-release summary (filter out verbose plugin repetition) echo "$OUTPUT" | grep -v "^$" | grep -vE "(No more plugins|does not provide step)" | \ From df9fc96e01b66a1451baa29be1898081ee89e5b6 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 40/77] chore(workflows): remove unused comments --- .github/workflows/cd.yaml | 31 ++++-------------------------- .github/workflows/deploy-docs.yaml | 5 ++--- 2 files changed, 6 insertions(+), 30 deletions(-) diff --git a/.github/workflows/cd.yaml b/.github/workflows/cd.yaml index 6c45b46bf..4b9386dcf 100644 --- a/.github/workflows/cd.yaml +++ b/.github/workflows/cd.yaml @@ -59,9 +59,6 @@ permissions: deployments: write jobs: - # job 1: set-variables - # determines deployment settings and variables based on event type - # Always runs - needed for production job routing and provides outputs set-variables: runs-on: ubuntu-latest if: | @@ -100,16 +97,12 @@ jobs: CHECKOUT_REV="${{ github.sha }}" fi - # Enable deployment on push to main (production) if [[ "${{ github.event_name }}" == "push" && "${{ github.ref }}" == "refs/heads/main" ]]; then DEPLOY_ENABLED="true" DEPLOY_ENVIRONMENT="production" fi - # Sanitize branch name for Cloudflare preview alias (must be valid subdomain component) - # - Replace / and other non-alphanumeric chars with - - # - Collapse consecutive hyphens, remove leading/trailing hyphens - # - Truncate to 40 chars (safe for subdomain label limit of 63) + # Sanitize for Cloudflare subdomain label (≤63 chars; truncate to 40 for safety) SANITIZED_BRANCH=$(echo "$CHECKOUT_REF" | tr '/' '-' | tr -c 'a-zA-Z0-9-' '-' | sed 's/--*/-/g; s/^-//; s/-$//' | cut -c1-40) echo "debug=$DEBUG" >> $GITHUB_OUTPUT @@ -134,10 +127,6 @@ jobs: uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 - # sparse-checkout: | - # packages - # justfile - # sparse-checkout-cone-mode: false - name: Setup Nix uses: ./.github/actions/setup-nix @@ -158,8 +147,6 @@ jobs: echo "packages=$PACKAGES" >> $GITHUB_OUTPUT echo "Discovered packages: $PACKAGES" - # job 2: preview-release-version - # Preview semantic-release version for each package (PR only, fast feedback) preview-release-version: needs: [set-variables] if: | @@ -257,8 +244,6 @@ jobs: path: ${{ steps.cache.outputs.cache-path }} key: ${{ steps.cache.outputs.cache-key }} - # job 3: preview-docs-deploy - # Deploy docs to preview environment (PR only, fast feedback) preview-docs-deploy: needs: [set-variables] if: | @@ -276,8 +261,6 @@ jobs: force_run: ${{ needs.set-variables.outputs.force-ci }} secrets: inherit - # job 4: bootstrap-verification - # validates Makefile bootstrap workflow on clean ubuntu system bootstrap-verification: needs: [set-variables] runs-on: ubuntu-latest @@ -365,9 +348,7 @@ jobs: path: ${{ steps.cache.outputs.cache-path }} key: ${{ steps.cache.outputs.cache-key }} - # job 7: test-cluster - # validates kubernetes manifests and local cluster integration - # informational only - does not block production releases + # Informational only - does not block production releases test-cluster: needs: [set-variables] if: | @@ -380,9 +361,7 @@ jobs: debug_enabled: ${{ needs.set-variables.outputs.debug }} secrets: inherit - # job 11: production-release-packages - # Release packages via semantic-release on main/beta branches - # Semantic-release determines if actual release is needed + # Semantic-release determines internally whether to actually release production-release-packages: needs: [set-variables] if: | @@ -408,9 +387,7 @@ jobs: secrets: SOPS_AGE_KEY: ${{ secrets.SOPS_AGE_KEY }} - # job 12: production-docs-deploy - # Documentation deployment to production (conditional) - # Depends on production-release-packages to ensure packages are released first + # Depends on production-release-packages so packages release before docs production-docs-deploy: needs: [set-variables, production-release-packages] if: | diff --git a/.github/workflows/deploy-docs.yaml b/.github/workflows/deploy-docs.yaml index 9e24355e5..c0c8cb5d3 100644 --- a/.github/workflows/deploy-docs.yaml +++ b/.github/workflows/deploy-docs.yaml @@ -103,9 +103,8 @@ jobs: if: steps.cache.outputs.should-run == 'true' id: deployment env: - # SOPS_AGE_KEY decrypts secrets/shared.yaml at step time. - # Per ADR-002 env-var contract, deploy.sh no longer calls - # `sops exec-env` internally — the wrap moves to the caller. + # SOPS_AGE_KEY decrypts secrets/shared.yaml at step time. The sops exec-env + # wrap moves to the caller (deploy.sh no longer calls sops internally). SOPS_AGE_KEY: ${{ secrets.SOPS_AGE_KEY }} GITHUB_ACTIONS: "true" GITHUB_ACTOR: ${{ github.actor }} From 6cde6d929a28d4845535c4a8c6596f3796292237 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 41/77] chore(justfile): remove unused comments --- justfile | 52 ++-------------------------------------------------- 1 file changed, 2 insertions(+), 50 deletions(-) diff --git a/justfile b/justfile index 1f43c9855..8e5f88281 100644 --- a/justfile +++ b/justfile @@ -1,18 +1,8 @@ -# This is a jusfile for the vanixiets repository. -# Sections are separated by ## and recipes are documented with a single # -# on lines preceding the recipe. - -## nix -## clan -## k3d -## secrets -## sops -## CI/CD +# justfile for vanixiets. Sections separated by ##; recipes documented with single # on the preceding line. nix_cmd := "nix --accept-flake-config" # Default command when 'just' is run without arguments -# Run 'just ' to execute a command. default: help # Display help @@ -231,7 +221,6 @@ nix-flake-io: tests_count=$(nix eval --raw .#tests --apply 'x: toString (builtins.length (builtins.attrNames x))' 2>/dev/null || echo "0") echo "(${tests_count} top-level test attrs)" - # Flake inputs printf "\n## inputs\n" nix flake metadata --json 2>/dev/null | jq -r '.locks.nodes | keys[] | select(. != "root")' @@ -351,7 +340,6 @@ bootstrap-verify *ARGS: bootstrap-setup-user *ARGS: {{nix_cmd}} run --no-warn-dirty .#setup-user -- {{ARGS}} -# nix run home-manager -- build --flake ".#{{ profile }}" # Bootstrap build home-manager with flake [group('nix-home-manager')] home-manager-bootstrap-build profile="aarch64-linux": @@ -363,7 +351,6 @@ home-manager-bootstrap-build profile="aarch64-linux": --show-trace \ --print-build-logs -# nix run home-manager -- switch --flake ".#{{ profile }}" # Bootstrap switch home-manager with flake [group('nix-home-manager')] home-manager-bootstrap-switch profile="aarch64-linux": @@ -1009,8 +996,7 @@ k3d-deploy-infrastructure: # Full k3d workflow: create cluster, bootstrap secrets, deploy all layers # Body lives in modules/apps/cluster/k3d-full.{nix,sh}; delegates back to -# the k3d-down, k3d-up, and k3d-deploy recipes above (none of which are -# flake-app converted in M1). +# the k3d-down, k3d-up, and k3d-deploy recipes above. [group('k3d')] k3d-full *ARGS: {{nix_cmd}} run --no-warn-dirty .#k3d-full -- {{ARGS}} @@ -1212,29 +1198,22 @@ hash-encrypt source_file user="crs58": #!/usr/bin/env bash set -euo pipefail - # Generate content-based hash for filename HASH=$(nix hash file --type sha256 --base64 "{{source_file}}" | cut -d'-' -f2 | head -c 32) - # Extract base filename without extension BASE_NAME=$(basename "{{source_file}}" .yaml) BASE_NAME=$(basename "$BASE_NAME" .yml) - # Create target path TARGET_DIR="secrets/users/{{user}}" TARGET_FILE="${TARGET_DIR}/${HASH}-${BASE_NAME}.yaml" - # Ensure target directory exists mkdir -p "$TARGET_DIR" - # Copy file with hash-based name cp "{{source_file}}" "$TARGET_FILE" echo "Copied {{source_file}} → $TARGET_FILE" - # Encrypt in place with sops sops encrypt --in-place "$TARGET_FILE" echo "Encrypted $TARGET_FILE with sops" - # Display verification info echo "Hash: $HASH" echo "Final path: $TARGET_FILE" @@ -1244,21 +1223,16 @@ verify-hash original_file secret_file: #!/usr/bin/env bash set -euo pipefail - # Extract hash from secret filename SECRET_BASENAME=$(basename "{{secret_file}}") EXPECTED_HASH=$(echo "$SECRET_BASENAME" | cut -d'-' -f1) - # Generate hash of original file ACTUAL_HASH=$(nix hash file --type sha256 --base64 "{{original_file}}" | cut -d'-' -f2 | head -c 32) - # Create temporary file for decrypted content TEMP_FILE=$(mktemp) trap "rm -f $TEMP_FILE" EXIT - # Decrypt secret file to temp location sops decrypt "{{secret_file}}" > "$TEMP_FILE" - # Generate hash of decrypted content DECRYPTED_HASH=$(nix hash file --type sha256 --base64 "$TEMP_FILE" | cut -d'-' -f2 | head -c 32) echo "Original file: {{original_file}}" @@ -1268,7 +1242,6 @@ verify-hash original_file secret_file: echo "Decrypted hash: $DECRYPTED_HASH" echo - # Verify original matches filename hash if [ "$ACTUAL_HASH" = "$EXPECTED_HASH" ]; then echo "Original file hash matches secret filename hash" else @@ -1276,7 +1249,6 @@ verify-hash original_file secret_file: exit 1 fi - # Verify decrypted content matches original if [ "$DECRYPTED_HASH" = "$ACTUAL_HASH" ]; then echo "Decrypted content matches original file" else @@ -1326,10 +1298,8 @@ ci-run-watch workflow="ci.yaml": echo "triggering workflow: {{workflow}} on branch: $(git branch --show-current)" gh workflow run {{workflow}} --ref $(git branch --show-current) - # wait a moment for run to start sleep 5 - # get the latest run ID RUN_ID=$(gh run list --workflow={{workflow}} --limit 1 --json databaseId --jq '.[0].databaseId') echo "watching run: $RUN_ID" @@ -1407,9 +1377,6 @@ test-flake-workflow: --matrix os:ubuntu-latest \ --container-architecture linux/amd64' -# Command to run sethvargo/ratchet to pin GitHub Actions workflows version tags to commit hashes -# If not installed, you can use docker to run the command -# ratchet_base := "docker run -it --rm -v \"${PWD}:${PWD}\" -w \"${PWD}\" ghcr.io/sethvargo/ratchet:0.9.2" ratchet_base := "ratchet" # List of GitHub Actions workflows @@ -1443,7 +1410,6 @@ cache-rosetta-builder: echo "Finding nix-rosetta-builder VM image in current system..." - # Find the rosetta-builder.yaml from current system YAML_PATH=$(nix-store --query --requisites /run/current-system | grep 'rosetta-builder.yaml$' || true) if [ -z "$YAML_PATH" ]; then @@ -1466,7 +1432,6 @@ cache-rosetta-builder: IMAGE_SIZE=$(du -h "$IMAGE_PATH" | cut -f1) echo "Size: $IMAGE_SIZE" - # Push to cachix echo "" echo "Pushing to Cachix (this may take a few minutes for ~2GB image)..." sops exec-env secrets/shared.yaml "cachix push \$CACHIX_CACHE_NAME $IMAGE_PATH" @@ -1493,7 +1458,6 @@ check-rosetta-cache: echo "Checking if nix-rosetta-builder image is cached..." - # Find the image from current system YAML_PATH=$(nix-store --query --requisites /run/current-system | grep 'rosetta-builder.yaml$' || true) if [ -z "$YAML_PATH" ]; then @@ -1512,7 +1476,6 @@ check-rosetta-cache: echo "Checking cache for: $IMAGE_PATH" - # Check if the image is in cache CACHE_NAME=$(sops exec-env secrets/shared.yaml 'echo $CACHIX_CACHE_NAME') if {{nix_cmd}} path-info --store "https://$CACHE_NAME.cachix.org" "$IMAGE_PATH" &>/dev/null; then @@ -1533,15 +1496,12 @@ test-cachix: set -euo pipefail echo "Testing cachix push/pull..." - # Build a simple derivation STORE_PATH=$({{nix_cmd}} build nixpkgs#hello --no-link --print-out-paths) echo "Built: $STORE_PATH" - # Push to cachix echo "Pushing to cachix..." sops exec-env secrets/shared.yaml "cachix push \$CACHIX_CACHE_NAME $STORE_PATH" - # Verify it's in the cache by trying to pull it from another location CACHE_NAME=$(sops exec-env secrets/shared.yaml 'echo $CACHIX_CACHE_NAME') echo "● Push completed. Verify at: https://app.cachix.org/cache/$CACHE_NAME" echo "Store path: $STORE_PATH" @@ -1558,7 +1518,6 @@ cache-darwin-system: echo "Cache: https://app.cachix.org/cache/$CACHE_NAME" echo "" - # Check if already cached FLAKE_OUTPUT=".#darwinConfigurations.$HOSTNAME.system" echo "Checking if system is already cached..." if {{nix_cmd}} path-info --store "https://$CACHE_NAME.cachix.org" "$FLAKE_OUTPUT" &>/dev/null; then @@ -1584,7 +1543,6 @@ cache-darwin-system: echo "Built: $SYSTEM_PATH" echo "" - # Push the path and all its runtime dependencies echo "Pushing system and all dependencies to cachix..." echo "(This may take several minutes depending on what's not already cached)" nix-store --query --requisites --include-outputs "$SYSTEM_PATH" | \ @@ -1764,7 +1722,6 @@ sops-load-agent: #!/usr/bin/env bash set -euo pipefail - # Check if we're on darwin if [[ "$OSTYPE" != "darwin"* ]]; then echo "⚠️ This command is only needed on macOS (darwin)" echo " Linux uses systemd instead of launchd" @@ -1773,28 +1730,23 @@ sops-load-agent: PLIST="$HOME/Library/LaunchAgents/org.nix-community.home.sops-nix.plist" - # Check if plist exists if [ ! -f "$PLIST" ]; then echo "⊘ SOPS plist not found: $PLIST" echo " Run 'just activate' first to create the plist" exit 1 fi - # Check if already loaded if launchctl list | grep -q "org.nix-community.home.sops-nix"; then echo "✓ SOPS agent already loaded" echo " Secrets directory: ~/.config/sops-nix/secrets/" exit 0 fi - # Load the agent echo "Loading SOPS launchd agent..." launchctl load "$PLIST" - # Brief wait for agent to start sleep 1 - # Verify it loaded if launchctl list | grep -q "org.nix-community.home.sops-nix"; then echo "✓ SOPS agent loaded successfully" echo " Secrets directory: ~/.config/sops-nix/secrets/" From baf411a60c4d75c6517de44a66940b2eb55fab3c Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 42/77] chore(apps/bootstrap): remove unused comments --- modules/apps/bootstrap/bootstrap.nix | 11 +---------- modules/apps/bootstrap/bootstrap.sh | 16 ++-------------- modules/apps/bootstrap/setup-user.nix | 19 ++----------------- modules/apps/bootstrap/setup-user.sh | 6 ------ modules/apps/bootstrap/verify.nix | 9 +-------- modules/apps/bootstrap/verify.sh | 13 ++----------- 6 files changed, 8 insertions(+), 66 deletions(-) diff --git a/modules/apps/bootstrap/bootstrap.nix b/modules/apps/bootstrap/bootstrap.nix index e785cf476..3a79e8a9a 100644 --- a/modules/apps/bootstrap/bootstrap.nix +++ b/modules/apps/bootstrap/bootstrap.nix @@ -1,9 +1,5 @@ # Flake app: re-run the bootstrap flow from an already-nix-ready host. # -# Usage: -# nix run .#bootstrap # install direnv if missing, confirm nix -# nix run .#bootstrap -- --help -# # Chicken-and-egg note: The repo's primary bootstrap entry point is the # Makefile (`make bootstrap`), which installs nix itself via the NixOS # community installer and only then installs direnv. This flake app, by @@ -12,12 +8,7 @@ # have nix but want to (idempotently) finish the direnv half of bootstrap # or re-verify that bootstrap has been completed. # -# Idempotent: detects existing nix and direnv via `command -v`; only -# attempts `nix profile install nixpkgs#direnv` when direnv is missing. -# Does not mutate /nix or /etc/nix; only touches the user's nix profile. -# -# Template bifurcation (writeShellApplication): PURE READFILE FORM. -# The sidecar needs no nix-eval-time path injection. +# Idempotent. { ... }: { perSystem = diff --git a/modules/apps/bootstrap/bootstrap.sh b/modules/apps/bootstrap/bootstrap.sh index 2a0fe097b..3656a5f80 100644 --- a/modules/apps/bootstrap/bootstrap.sh +++ b/modules/apps/bootstrap/bootstrap.sh @@ -1,14 +1,5 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Re-run the bootstrap flow (direnv install + status report) on a host -# that already has nix installed. -# -# Chicken-and-egg: `make bootstrap` is the real first-contact installer; -# it installs nix itself. This flake app runs UNDER nix, so by definition -# nix is already present. Treat this app as the post-nix half of -# bootstrap: it ensures direnv is installed and reports status. -# -# Idempotent: re-runs produce no new state when direnv is already present. set -euo pipefail usage() { @@ -36,8 +27,7 @@ esac printf '=== Bootstrap (nix-present host) ===\n\n' -# Step 1: confirm nix (cannot be missing since we're running under nix, -# but surface the version for parity with `make verify`). +# Confirm nix even though running under nix means it must be present. if command -v nix >/dev/null 2>&1; then printf '● nix found at %s\n' "$(command -v nix)" nix --version @@ -50,9 +40,7 @@ else fi printf '\n' -# Step 2: install direnv if missing. `nix profile install` is idempotent -# against the same attribute path; we guard with `command -v` for a -# cleaner no-op output when direnv is already on PATH. +# command -v guard yields cleaner no-op output than relying on nix profile install idempotence. if command -v direnv >/dev/null 2>&1; then printf '● direnv already installed at %s\n' "$(command -v direnv)" else diff --git a/modules/apps/bootstrap/setup-user.nix b/modules/apps/bootstrap/setup-user.nix index 3c6557f1c..78bff0261 100644 --- a/modules/apps/bootstrap/setup-user.nix +++ b/modules/apps/bootstrap/setup-user.nix @@ -1,21 +1,6 @@ -# Flake app: generate the user's age key for sops-nix secrets (first-time -# user setup only; idempotent on re-run). +# Flake app: generate the user's age key for sops-nix secrets (first-time user setup; idempotent on re-run). # -# Usage: -# nix run .#setup-user # generate key if absent; print public key -# nix run .#setup-user -- --help -# -# Chicken-and-egg note: Mirrors `make setup-user` from the repo-root -# Makefile. Requires nix to be already installed (this flake app cannot -# run before nix). For a clean-host first-contact, use `make setup-user` -# instead; both targets share the same idempotence guarantee. -# -# Idempotent: if ~/.config/sops/age/keys.txt already exists, the script -# re-prints the public key and exits 0 WITHOUT regenerating. Only the -# first invocation writes keys.txt (mode 0600). -# -# Template bifurcation (writeShellApplication): PURE READFILE FORM. -# The sidecar needs no nix-eval-time path injection. +# Idempotent: if ~/.config/sops/age/keys.txt exists, re-prints the public key and exits 0 without regenerating. { ... }: { perSystem = diff --git a/modules/apps/bootstrap/setup-user.sh b/modules/apps/bootstrap/setup-user.sh index 13deea88c..b82ca237d 100644 --- a/modules/apps/bootstrap/setup-user.sh +++ b/modules/apps/bootstrap/setup-user.sh @@ -1,10 +1,5 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Generate the user's age key at ~/.config/sops/age/keys.txt. If the key -# already exists, print the public key and exit 0 without regenerating. -# -# Idempotent: re-running on a host with an existing key file is a no-op -# aside from stdout (no mutation of the key file or its parent dir). set -euo pipefail usage() { @@ -45,7 +40,6 @@ if [ -f "$key_file" ]; then exit 0 fi -# First run: create the key file with a locked-down mode. mkdir -p "$key_dir" age-keygen -o "$key_file" chmod 600 "$key_file" diff --git a/modules/apps/bootstrap/verify.nix b/modules/apps/bootstrap/verify.nix index 724cce46c..74787d071 100644 --- a/modules/apps/bootstrap/verify.nix +++ b/modules/apps/bootstrap/verify.nix @@ -1,9 +1,5 @@ # Flake app: verify the host's nix + flakes + direnv + devShell setup. # -# Usage: -# nix run .#verify # full status report; exit nonzero on missing nix/flakes -# nix run .#verify -- --help -# # Chicken-and-egg note: Mirrors `make verify` from the repo-root Makefile. # The Makefile version is callable from a nix-free shell (it's plain # make + shell). This flake-app version assumes nix is already installed @@ -11,10 +7,7 @@ # post-bootstrap sanity checks, buildbot effects) can invoke the audit # without depending on GNU make being on PATH. # -# Idempotent / pure: only reads state. Writes nothing; touches no files. -# -# Template bifurcation (writeShellApplication): PURE READFILE FORM. -# The sidecar needs no nix-eval-time path injection. +# Read-only. { ... }: { perSystem = diff --git a/modules/apps/bootstrap/verify.sh b/modules/apps/bootstrap/verify.sh index f5a15dd59..422a0045b 100644 --- a/modules/apps/bootstrap/verify.sh +++ b/modules/apps/bootstrap/verify.sh @@ -1,11 +1,6 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Verify the host's nix installation, flakes support, direnv presence, -# and flake metadata. Mirrors `make verify` from the repo-root Makefile, -# minus the devShell build (which is expensive and duplicates what -# `nix flake check` already covers). -# -# Idempotent / pure: does not mutate state. +# Mirrors `make verify` minus the devShell build because expensive. set -euo pipefail usage() { @@ -33,7 +28,6 @@ failed=0 printf '\n=== Verifying installation ===\n\n' -# Check 1: nix binary printf 'Checking nix installation: ' if command -v nix >/dev/null 2>&1; then printf '● nix found at %s\n' "$(command -v nix)" @@ -46,7 +40,6 @@ else fi printf '\n' -# Check 2: flakes support printf 'Checking nix flakes support: ' if nix flake --help >/dev/null 2>&1; then printf '● flakes enabled\n' @@ -56,7 +49,6 @@ else fi printf '\n' -# Check 3: direnv (optional) printf 'Checking direnv installation: ' if command -v direnv >/dev/null 2>&1; then printf '● direnv found at %s\n' "$(command -v direnv)" @@ -67,7 +59,6 @@ else fi printf '\n' -# Check 4: flake metadata parseable printf 'Checking flake validity: ' if nix --accept-flake-config flake metadata . >/dev/null 2>&1; then printf '● flake is valid\n' @@ -77,7 +68,7 @@ else fi printf '\n' -# Check 5: surface /etc/nix/nix.conf for auditability (match make verify) +# Surface /etc/nix/nix.conf for auditability — parity with make verify. printf '/etc/nix/nix.conf:\n' printf '==================\n' if [ -f /etc/nix/nix.conf ]; then From 965d1655ada18b11e4615a914be5c8264c105e6a Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 43/77] chore(apps/cluster): remove unused comments --- .../apps/cluster/k3d-bootstrap-secrets.nix | 8 ---- modules/apps/cluster/k3d-bootstrap-secrets.sh | 25 ++++------ modules/apps/cluster/k3d-configure-dns.nix | 4 -- modules/apps/cluster/k3d-configure-dns.sh | 9 +--- modules/apps/cluster/k3d-full.nix | 15 ++---- modules/apps/cluster/k3d-full.sh | 8 ---- modules/apps/cluster/k3d-integration-ci.nix | 3 -- modules/apps/cluster/k3d-integration-ci.sh | 16 ++----- modules/apps/cluster/k3d-test-coverage.nix | 11 +---- modules/apps/cluster/k3d-test-coverage.sh | 47 ++----------------- modules/apps/cluster/k3d-wait-argocd-sync.nix | 4 -- modules/apps/cluster/k3d-wait-argocd-sync.sh | 7 +-- modules/apps/cluster/k3d-wait-ready.nix | 6 +-- modules/apps/cluster/k3d-wait-ready.sh | 5 -- modules/apps/cluster/list-packages-json.nix | 5 -- modules/apps/cluster/list-packages-json.sh | 9 +--- modules/apps/cluster/nixidy-bootstrap.nix | 7 --- modules/apps/cluster/nixidy-bootstrap.sh | 6 --- modules/apps/cluster/nixidy-build.nix | 4 -- modules/apps/cluster/nixidy-build.sh | 8 +--- modules/apps/cluster/nixidy-push.nix | 8 ---- modules/apps/cluster/nixidy-push.sh | 9 ---- modules/apps/cluster/nixidy-sync.nix | 4 -- modules/apps/cluster/nixidy-sync.sh | 7 --- 24 files changed, 30 insertions(+), 205 deletions(-) diff --git a/modules/apps/cluster/k3d-bootstrap-secrets.nix b/modules/apps/cluster/k3d-bootstrap-secrets.nix index 5f70cdff0..c207a8af5 100644 --- a/modules/apps/cluster/k3d-bootstrap-secrets.nix +++ b/modules/apps/cluster/k3d-bootstrap-secrets.nix @@ -1,14 +1,6 @@ # k3d-bootstrap-secrets.nix - Bootstrap sops-age-key into a running k3d cluster. # -# Usage: -# nix run .#k3d-bootstrap-secrets -# -# Template form: pure readFile (no nix-computed variable injection). # Idempotent: second invocation leaves the secret byte-identical. -# -# Supports two key-source branches: -# - SOPS_AGE_KEY env var present -> write to tmpfile, use -# - otherwise -> read $HOME/.config/sops/age/keys.txt { ... }: { perSystem = diff --git a/modules/apps/cluster/k3d-bootstrap-secrets.sh b/modules/apps/cluster/k3d-bootstrap-secrets.sh index d8a9395ed..3489413ff 100644 --- a/modules/apps/cluster/k3d-bootstrap-secrets.sh +++ b/modules/apps/cluster/k3d-bootstrap-secrets.sh @@ -1,22 +1,15 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Bootstrap the sops-age-key Kubernetes secret required by -# sops-secrets-operator to decrypt SopsSecret custom resources in the -# local-k3d cluster. Idempotent: reapplies cleanly and does not mutate -# the secret when the key source has not changed. +# Idempotent: reapplies cleanly and does not mutate the secret when the key source has not changed. # -# Usage: -# k3d-bootstrap-secrets [--help] -# -# Env-var contract (per ADR-002 / env-var-contract-design.md §2.4): +# Env-var contract: # One of the following MUST be satisfied (narrow exception; env-first): # SOPS_AGE_KEY (env) single-line AGE-SECRET-KEY-… -# body (CI / M4 effect preamble) +# body (CI / effect preamble) # $HOME/.config/sops/age/keys.txt (file) local dev pathway # # This is the ONLY flake app in modules/apps/ that intentionally consumes -# SOPS_AGE_KEY directly. Per ADR-002 ("SOPS_AGE_KEY exposure as a general -# pattern is REJECTED"), no other M4 effect or app is permitted to expose +# SOPS_AGE_KEY directly. No other effect or app is permitted to expose # it. Rationale: the k3d bootstrap flow needs an age key INSIDE the # ephemeral cluster for sops-secrets-operator to decrypt SopsSecret CRs # at runtime — this is a load-bearing narrow exception. @@ -24,7 +17,7 @@ # Caller mechanisms: # - Local dev: file-branch via $HOME/.config/sops/age/keys.txt # - GHA env: GHA `env:` block with SOPS_AGE_KEY from repo secrets -# - M4 effect: effect preamble extracts SOPS_AGE_KEY from +# - effect: effect preamble extracts SOPS_AGE_KEY from # HERCULES_CI_SECRETS_JSON and exports before invoking # the transitive caller (k3d-integration-ci) # @@ -52,11 +45,9 @@ EOF ;; esac -# Env-var contract: validate key source BEFORE any kubectl invocation so -# the failure surface points at the contract (SOPS_AGE_KEY env OR the -# keys.txt file) rather than an opaque kubectl/api error. This is the -# ordering that satisfies VAL-ENVCONTRACT-K3DBOOT-04's "fails fast" intent. -# Determine age key file: env var (CI / effect preamble) or local file (dev). +# Validate key source BEFORE any kubectl invocation so the failure surface +# points at the contract (SOPS_AGE_KEY env OR the keys.txt file) rather +# than an opaque kubectl/api error. if [ -n "${SOPS_AGE_KEY:-}" ]; then echo "Using SOPS_AGE_KEY from environment variable" KEYFILE=$(mktemp) diff --git a/modules/apps/cluster/k3d-configure-dns.nix b/modules/apps/cluster/k3d-configure-dns.nix index be784057b..e055d228b 100644 --- a/modules/apps/cluster/k3d-configure-dns.nix +++ b/modules/apps/cluster/k3d-configure-dns.nix @@ -1,9 +1,5 @@ # k3d-configure-dns.nix - Patch CoreDNS to forward sslip.io queries to public DNS. # -# Usage: -# nix run .#k3d-configure-dns -# -# Template form: pure readFile (no nix-computed variable injection). # Required because OrbStack's default DNS (192.168.107.1) cannot resolve # sslip.io wildcards used by the local ArgoCD application routes. # Idempotent: second invocation exits 0 without patching. diff --git a/modules/apps/cluster/k3d-configure-dns.sh b/modules/apps/cluster/k3d-configure-dns.sh index 1047a2b82..61e599529 100644 --- a/modules/apps/cluster/k3d-configure-dns.sh +++ b/modules/apps/cluster/k3d-configure-dns.sh @@ -1,13 +1,6 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Patch the k3d cluster's CoreDNS to forward sslip.io queries to public -# DNS resolvers so ArgoCD Application routes using .sslip.io domains -# resolve inside the cluster. Idempotent: re-running on an -# already-configured cluster detects the existing "sslip.io" block and -# exits 0 without mutation. -# -# Usage: -# k3d-configure-dns [--help] +# Idempotent via grep detection of the existing "sslip.io" block in the Corefile. set -euo pipefail case "${1:-}" in diff --git a/modules/apps/cluster/k3d-full.nix b/modules/apps/cluster/k3d-full.nix index 78f8c29d1..5d7a844cb 100644 --- a/modules/apps/cluster/k3d-full.nix +++ b/modules/apps/cluster/k3d-full.nix @@ -1,15 +1,10 @@ # k3d-full.nix - Full local-k3d lifecycle: down -> up -> deploy. # -# Usage: -# nix run .#k3d-full -# -# Template form: pure readFile (no nix-computed variable injection). -# Orchestration wrapper that delegates to the underlying justfile -# recipes for k3d-down, k3d-up, and k3d-deploy — none of which are in -# the M1 flake-app conversion scope. `just` is therefore included as a -# runtimeInput. The recipes themselves still need k3d, ctlptl, kubectl, -# etc. on PATH; those come from the user's dev environment (writeShell- -# Application prepends runtimeInputs to $PATH without stripping it). +# Delegates to the underlying justfile recipes for k3d-down, k3d-up, and +# k3d-deploy. `just` is a runtimeInput because the recipes themselves +# still need k3d/ctlptl/kubectl/etc on PATH; those come from the user's +# dev environment (writeShellApplication prepends runtimeInputs to $PATH +# without stripping it). { ... }: { perSystem = diff --git a/modules/apps/cluster/k3d-full.sh b/modules/apps/cluster/k3d-full.sh index b07cb705e..032e4b04d 100644 --- a/modules/apps/cluster/k3d-full.sh +++ b/modules/apps/cluster/k3d-full.sh @@ -1,13 +1,5 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Full local-k3d lifecycle: tear down any existing cluster, recreate it, -# and deploy foundation + infrastructure layers. Delegates to the -# original just recipes (k3d-down, k3d-up, k3d-deploy) which remain the -# single source of truth for the cluster wiring during the M1 transition -# window. -# -# Usage: -# k3d-full [--help] set -euo pipefail case "${1:-}" in diff --git a/modules/apps/cluster/k3d-integration-ci.nix b/modules/apps/cluster/k3d-integration-ci.nix index ad56f3a0f..4752885a8 100644 --- a/modules/apps/cluster/k3d-integration-ci.nix +++ b/modules/apps/cluster/k3d-integration-ci.nix @@ -1,8 +1,5 @@ # k3d-integration-ci.nix - CI-variant full integration: file:///manifests + tests. # -# Usage: -# nix run .#k3d-integration-ci -# # Template bifurcation (writeShellApplication): PURE READFILE FORM. # `text = builtins.readFile ./k3d-integration-ci.sh` — the sidecar is # consumed verbatim, no nix-eval-time string interpolation. This is the diff --git a/modules/apps/cluster/k3d-integration-ci.sh b/modules/apps/cluster/k3d-integration-ci.sh index 026f2e949..d0da42568 100644 --- a/modules/apps/cluster/k3d-integration-ci.sh +++ b/modules/apps/cluster/k3d-integration-ci.sh @@ -1,14 +1,8 @@ #!/usr/bin/env bash # shellcheck shell=bash -# CI integration driver for the local-k3d cluster. Uses the local -# file:///manifests repo URL (no GitHub credentials required) and -# orchestrates the full seven-phase flow consumed by -# .github/workflows/test-cluster.yaml's `integration` job. +# CI integration driver for the local-k3d cluster, consumed by .github/workflows/test-cluster.yaml's `integration` job. # -# Usage: -# k3d-integration-ci [--help] -# -# Env-var contract (per ADR-002 / env-var-contract-design.md §2.5): +# Env-var contract: # Transitively required (consumed by k3d-bootstrap-secrets, the leaf): # SOPS_AGE_KEY age key body for sops-secrets-operator inside the # ephemeral k3d cluster. Enforcement is deferred to @@ -25,12 +19,12 @@ # - Local dev: .envrc dotenv or file-branch ($HOME/.config/sops/...) # - GHA env: job-level `env:` block populates SOPS_AGE_KEY from # repo secrets (.github/workflows/test-cluster.yaml) -# - M4 effect: test-cluster effect preamble extracts SOPS_AGE_KEY +# - effect: test-cluster effect preamble extracts SOPS_AGE_KEY # from HERCULES_CI_SECRETS_JSON and exports before # invoking ${config.apps.k3d-integration-ci.program} # -# NB: required-env guard documented here via the `:?` idiom lives in the -# leaf k3d-bootstrap-secrets.sh; this file intentionally has no top-level +# NB: required-env guard via the `:?` idiom lives in the leaf +# k3d-bootstrap-secrets.sh; this file intentionally has no top-level # `${SOPS_AGE_KEY:?…}` enforcement (the transitive contract is surfaced # via k3d-bootstrap-secrets.sh's fail-fast behaviour when neither env nor # file is present). diff --git a/modules/apps/cluster/k3d-test-coverage.nix b/modules/apps/cluster/k3d-test-coverage.nix index 5a6957d63..3cdfe20ab 100644 --- a/modules/apps/cluster/k3d-test-coverage.nix +++ b/modules/apps/cluster/k3d-test-coverage.nix @@ -1,15 +1,8 @@ # k3d-test-coverage.nix - Run chainsaw integration tests and emit coverage report. # -# Usage: -# nix run .#k3d-test-coverage -- [--raw] [chainsaw args...] -# -# Template form: pure readFile (no nix-computed variable injection). # Subsumes scripts/k3d-test-coverage.sh (legacy root-level copy kept as -# a thin shim in M5 for backward compatibility). The coverage-report -# logic lives in-tree at modules/apps/cluster/k3d-test-coverage.sh. -# -# Resolves kubernetes/tests/local-k3d/ relative to the invoking git -# worktree via `git rev-parse --show-toplevel`. +# a thin shim for backward compatibility). The coverage-report logic +# lives in-tree at modules/apps/cluster/k3d-test-coverage.sh. { ... }: { perSystem = diff --git a/modules/apps/cluster/k3d-test-coverage.sh b/modules/apps/cluster/k3d-test-coverage.sh index d0a2c166b..767757772 100644 --- a/modules/apps/cluster/k3d-test-coverage.sh +++ b/modules/apps/cluster/k3d-test-coverage.sh @@ -1,19 +1,5 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Run chainsaw tests with coverage report showing tested vs deployed resources. -# -# Usage: k3d-test-coverage [--help] [--raw] [chainsaw args...] -# -# Options: -# --help Show this message and exit 0 -# --raw Show raw uncategorized output (original format) -# -# Environment: -# CI, GITHUB_ACTIONS, NO_COLOR - Disable colors when set -# -# Exit codes: -# 0 - All tests passed -# 1 - Tests failed or error set -euo pipefail case "${1:-}" in @@ -36,7 +22,6 @@ EOF ;; esac -# Global flag for raw output mode RAW_MODE=0 # shellcheck disable=SC2034 # Colors are used via variable expansion @@ -111,7 +96,6 @@ collect_deployed_resources() { local -n deployed_ref=$1 local -n type_counts_ref=$2 - # Workloads (Deployment, StatefulSet, DaemonSet) local line ns name kind key while IFS= read -r line; do [[ -z "$line" ]] && continue @@ -122,7 +106,6 @@ collect_deployed_resources() { deployed_ref["$key"]=1 done < <(kubectl get deploy,sts,ds -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,KIND:.kind' --no-headers 2>/dev/null | grep -v '^$') - # Gateway API resources while IFS= read -r line; do [[ -z "$line" ]] && continue ns=$(awk '{print $1}' <<< "$line") @@ -137,7 +120,6 @@ collect_deployed_resources() { deployed_ref["HTTPRoute/${ns}/${name}"]=1 done < <(kubectl get httproute -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') - # Certificates while IFS= read -r line; do [[ -z "$line" ]] && continue ns=$(awk '{print $1}' <<< "$line") @@ -145,13 +127,11 @@ collect_deployed_resources() { deployed_ref["Certificate/${ns}/${name}"]=1 done < <(kubectl get certificate -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') - # ClusterIssuers (cluster-scoped) while IFS= read -r line; do [[ -z "$line" ]] && continue deployed_ref["ClusterIssuer/-/${line}"]=1 done < <(kubectl get clusterissuer -o custom-columns='NAME:.metadata.name' --no-headers 2>/dev/null | grep -v '^$') - # Count by type for key in "${!deployed_ref[@]}"; do kind="${key%%/*}" type_counts_ref["$kind"]=$(( ${type_counts_ref["$kind"]:-0} + 1 )) @@ -172,7 +152,6 @@ collect_tested_resources() { current_ns="-" while IFS= read -r line; do - # New document resets state if [[ "$line" == "---" ]]; then if [[ -n "$current_kind" && -n "$current_name" ]]; then key="${current_kind}/${current_ns}/${current_name}" @@ -184,32 +163,28 @@ collect_tested_resources() { continue fi - # Extract kind if [[ "$line" =~ ^kind:\ *(.+)$ ]]; then current_kind="${BASH_REMATCH[1]}" fi - # Extract name (first name field is metadata.name) + # First "name:" line is metadata.name (guards against later annotation-name etc). if [[ "$line" =~ ^[[:space:]]+name:\ *(.+)$ ]]; then if [[ -z "$current_name" ]]; then current_name="${BASH_REMATCH[1]}" fi fi - # Extract namespace if [[ "$line" =~ ^[[:space:]]+namespace:\ *(.+)$ ]]; then current_ns="${BASH_REMATCH[1]}" fi done < "$file" - # Last resource in file if [[ -n "$current_kind" && -n "$current_name" ]]; then key="${current_kind}/${current_ns}/${current_name}" tested_ref["$key"]=1 fi done < <(find "$test_dir" -name "*assert*.yaml" -type f) - # Count by type for key in "${!tested_ref[@]}"; do kind="${key%%/*}" type_counts_ref["$kind"]=$(( ${type_counts_ref["$kind"]:-0} + 1 )) @@ -231,8 +206,7 @@ print_resource_table() { printf " %-15s %3d\n" "Total" "$total" } -# Categorize a resource as application, foundation, or system -# Returns: "application", "foundation", or "system" +# Categorize a resource as application, foundation, or system. categorize_resource() { local key="$1" local kind="${key%%/*}" @@ -272,13 +246,11 @@ categorize_resource() { Deployment/kube-system/cilium-operator) echo "foundation"; return ;; esac - # Application resources (our nixidy/ArgoCD managed stack) - # Includes: argocd, cert-manager, sops-secrets-operator, step-ca, - # gateway-system, plus Gateway API resources + # Application resources (nixidy/ArgoCD-managed: argocd, cert-manager, + # sops-secrets-operator, step-ca, gateway-system, Gateway API). echo "application" } -# Get human-readable description for system components get_system_description() { local key="$1" @@ -325,12 +297,10 @@ print_coverage_report() { echo "" echo "${BOLD}Coverage Analysis:${RESET}" - # Categorize resources and calculate coverage local matched=0 local untested=() local key category - # Categorized counts local app_total=0 app_tested=0 local foundation_total=0 foundation_tested=0 local system_total=0 system_tested=0 @@ -369,13 +339,12 @@ print_coverage_report() { esac done - # Calculate raw coverage (all resources) local raw_coverage=0 if [[ $deployed_count -gt 0 ]]; then raw_coverage=$(( matched * 100 / deployed_count )) fi - # Calculate managed coverage (excluding system components) + # Excluding system components. local managed_total=$(( app_total + foundation_total )) local managed_tested=$(( app_tested + foundation_tested )) local managed_coverage=0 @@ -384,7 +353,6 @@ print_coverage_report() { fi if [[ $RAW_MODE -eq 1 ]]; then - # Original raw output format local cov_color="$RED" if [[ $raw_coverage -ge 80 ]]; then cov_color="$GREEN" @@ -399,7 +367,6 @@ print_coverage_report() { if [[ ${#untested[@]} -gt 0 ]] || [[ ${#system_resources[@]} -gt 0 ]]; then echo "${BOLD}Untested Resources:${RESET}" local rest ns name - # Combine untested managed resources with untested system resources local all_untested=() for key in "${untested[@]}"; do all_untested+=("$key") @@ -424,7 +391,6 @@ print_coverage_report() { echo " ${GREEN}All deployed resources have test coverage${RESET}" fi else - # Categorized output format echo "" echo "${BOLD}═══════════════════════════════════════════════════════════════════${RESET}" echo "${BOLD} COVERAGE BY CATEGORY ${RESET}" @@ -459,7 +425,6 @@ print_coverage_report() { [[ $managed_coverage -ge 50 && $managed_coverage -lt 80 ]] && mgd_color="$YELLOW" printf " ${BOLD}Managed Resources Total: ${mgd_color}%2d/%2d (%3d%%)${RESET}\n" "$managed_tested" "$managed_total" "$managed_coverage" - # Untested managed resources if [[ ${#untested[@]} -gt 0 ]]; then echo "" echo "${BOLD}Untested Managed Resources:${RESET}" @@ -477,7 +442,6 @@ print_coverage_report() { done | sort fi - # System components section echo "" echo "${BOLD}System Components (excluded from coverage):${RESET}" local rest ns name desc @@ -501,7 +465,6 @@ print_coverage_report() { main() { setup_colors - # Parse --raw flag local args=() for arg in "$@"; do if [[ "$arg" == "--raw" ]]; then diff --git a/modules/apps/cluster/k3d-wait-argocd-sync.nix b/modules/apps/cluster/k3d-wait-argocd-sync.nix index 226907f2b..85ba4691e 100644 --- a/modules/apps/cluster/k3d-wait-argocd-sync.nix +++ b/modules/apps/cluster/k3d-wait-argocd-sync.nix @@ -1,9 +1,5 @@ # k3d-wait-argocd-sync.nix - Wait for all ArgoCD Applications to reach Synced + Healthy. # -# Usage: -# nix run .#k3d-wait-argocd-sync -# -# Template form: pure readFile (no nix-computed variable injection). # Matches the Phase-4 post-bootstrap gating from the justfile # `k3d-wait-argocd-sync` recipe. The expected-apps list mirrors the # nixidy sync-wave declarations and is the source of truth at this layer. diff --git a/modules/apps/cluster/k3d-wait-argocd-sync.sh b/modules/apps/cluster/k3d-wait-argocd-sync.sh index 2bd1b255c..1c25bbdc9 100644 --- a/modules/apps/cluster/k3d-wait-argocd-sync.sh +++ b/modules/apps/cluster/k3d-wait-argocd-sync.sh @@ -1,11 +1,6 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Block until every ArgoCD Application in the local-k3d cluster is both -# Healthy and Synced, then verify the root Gateway is Programmed by -# Cilium's Gateway API implementation. -# -# Usage: -# k3d-wait-argocd-sync [--help] +# Block until every ArgoCD Application is Healthy and Synced, then verify the root Gateway is Programmed by Cilium. set -euo pipefail case "${1:-}" in diff --git a/modules/apps/cluster/k3d-wait-ready.nix b/modules/apps/cluster/k3d-wait-ready.nix index 4abb92056..23ef3542b 100644 --- a/modules/apps/cluster/k3d-wait-ready.nix +++ b/modules/apps/cluster/k3d-wait-ready.nix @@ -1,11 +1,7 @@ # k3d-wait-ready.nix - Block until kluctl-deployed foundation + infra pods are Ready. # -# Usage: -# nix run .#k3d-wait-ready -# -# Template form: pure readFile (no nix-computed variable injection). # Mirrors the Phase 3 post-deploy gating that sat in the justfile -# `k3d-wait-ready` recipe. All kubectl waits have deterministic timeouts. +# `k3d-wait-ready` recipe. { ... }: { perSystem = diff --git a/modules/apps/cluster/k3d-wait-ready.sh b/modules/apps/cluster/k3d-wait-ready.sh index d29146d90..9ff42e293 100644 --- a/modules/apps/cluster/k3d-wait-ready.sh +++ b/modules/apps/cluster/k3d-wait-ready.sh @@ -1,10 +1,5 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Wait for the kluctl-deployed foundation (Cilium) and infrastructure -# (ArgoCD, sops-secrets-operator, step-ca) pods to reach Ready. -# -# Usage: -# k3d-wait-ready [--help] set -euo pipefail case "${1:-}" in diff --git a/modules/apps/cluster/list-packages-json.nix b/modules/apps/cluster/list-packages-json.nix index 5ae137ee0..84c488b82 100644 --- a/modules/apps/cluster/list-packages-json.nix +++ b/modules/apps/cluster/list-packages-json.nix @@ -1,10 +1,5 @@ # list-packages-json.nix - Emit a JSON matrix of workspace packages. # -# Usage: -# nix run .#list-packages-json -# -# Template form: pure readFile (no nix-computed variable injection). -# # Enumerates packages// directories containing a package.json # and emits a JSON array of {name, path} entries consumed by the # preview-release-version matrix step in cd.yaml's set-variables job. diff --git a/modules/apps/cluster/list-packages-json.sh b/modules/apps/cluster/list-packages-json.sh index a005cd2e1..e80cb4e27 100644 --- a/modules/apps/cluster/list-packages-json.sh +++ b/modules/apps/cluster/list-packages-json.sh @@ -1,13 +1,6 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Emit a JSON matrix entry per packages// with a package.json. -# -# Usage: -# list-packages-json [--help] -# -# Output: a single JSON array line of {name, path} objects on stdout. -# Resolves the repo root via `git rev-parse --show-toplevel`, so callers -# may invoke from any subdirectory of the vanixiets worktree. +# Resolves repo root via git rev-parse --show-toplevel. set -euo pipefail case "${1:-}" in diff --git a/modules/apps/cluster/nixidy-bootstrap.nix b/modules/apps/cluster/nixidy-bootstrap.nix index 6042ff007..3674a913c 100644 --- a/modules/apps/cluster/nixidy-bootstrap.nix +++ b/modules/apps/cluster/nixidy-bootstrap.nix @@ -1,11 +1,4 @@ # nixidy-bootstrap.nix - Apply the local-k3d app-of-apps bootstrap Application CR. -# -# Usage: -# nix run .#nixidy-bootstrap -# -# Template form: pure readFile (no nix-computed variable injection). -# Emits the bootstrap Application CR to stdout and pipes it into -# `kubectl apply -f -` against the live k3d cluster context. { ... }: { perSystem = diff --git a/modules/apps/cluster/nixidy-bootstrap.sh b/modules/apps/cluster/nixidy-bootstrap.sh index a5cf31667..1b6eafb85 100644 --- a/modules/apps/cluster/nixidy-bootstrap.sh +++ b/modules/apps/cluster/nixidy-bootstrap.sh @@ -1,11 +1,5 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Apply the app-of-apps bootstrap Application CR for the local-k3d -# environment: renders the manifest with `nixidy bootstrap` and pipes it -# into kubectl apply -f - against the cluster context in use. -# -# Usage: -# nixidy-bootstrap [--help] set -euo pipefail case "${1:-}" in diff --git a/modules/apps/cluster/nixidy-build.nix b/modules/apps/cluster/nixidy-build.nix index 0ab6f0670..b7e823bec 100644 --- a/modules/apps/cluster/nixidy-build.nix +++ b/modules/apps/cluster/nixidy-build.nix @@ -1,9 +1,5 @@ # nixidy-build.nix - Render nixidy manifests for local-k3d to ./result. # -# Usage: -# nix run .#nixidy-build -# -# Template form: pure readFile (no nix-computed variable injection). # The nixidy CLI is exposed via config.packages.nixidy (set in # modules/nixidy.nix) and added to runtimeInputs; the flake-app # invocation resolves the env at `.#local-k3d` using the current diff --git a/modules/apps/cluster/nixidy-build.sh b/modules/apps/cluster/nixidy-build.sh index cdce60749..9ad70b232 100644 --- a/modules/apps/cluster/nixidy-build.sh +++ b/modules/apps/cluster/nixidy-build.sh @@ -1,12 +1,6 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Build nixidy-rendered Kubernetes manifests for the local-k3d env into -# ./result. Equivalent to `nixidy build .#local-k3d`; preserved here so -# the invocation is packaged as a first-class flake app for CI effects -# and justfile wrappers. -# -# Usage: -# nixidy-build [--help] +# Consumers: CI effects + justfile wrappers. set -euo pipefail case "${1:-}" in diff --git a/modules/apps/cluster/nixidy-push.nix b/modules/apps/cluster/nixidy-push.nix index 492d6e341..8b2c9ee8c 100644 --- a/modules/apps/cluster/nixidy-push.nix +++ b/modules/apps/cluster/nixidy-push.nix @@ -1,12 +1,4 @@ # nixidy-push.nix - Rsync rendered manifests to the local-k3d private repo. -# -# Usage: -# nix run .#nixidy-push -# -# Template form: pure readFile (no nix-computed variable injection). -# The target repo path is resolved at runtime from the LOCAL_K3D_REPO -# env var (fallback: $HOME/projects/nix-workspace/local-k3d), mirroring -# the justfile `local_k3d_repo` convention. { ... }: { perSystem = diff --git a/modules/apps/cluster/nixidy-push.sh b/modules/apps/cluster/nixidy-push.sh index 625ba8d8c..593c4e16d 100644 --- a/modules/apps/cluster/nixidy-push.sh +++ b/modules/apps/cluster/nixidy-push.sh @@ -1,14 +1,5 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Sync the ./result/ tree produced by nixidy-build into the private -# local-k3d manifest repo, then commit and push any diff. -# -# Usage: -# nixidy-push [--help] -# -# Environment: -# LOCAL_K3D_REPO path to the local-k3d manifest repo -# (default: $HOME/projects/nix-workspace/local-k3d) set -euo pipefail case "${1:-}" in diff --git a/modules/apps/cluster/nixidy-sync.nix b/modules/apps/cluster/nixidy-sync.nix index cd2407a31..ba2b92cb0 100644 --- a/modules/apps/cluster/nixidy-sync.nix +++ b/modules/apps/cluster/nixidy-sync.nix @@ -1,9 +1,5 @@ # nixidy-sync.nix - Compose nixidy-build then nixidy-push. # -# Usage: -# nix run .#nixidy-sync -# -# Template form: pure readFile (no nix-computed variable injection). # Composes nixidy-build and nixidy-push by invoking them sequentially # via their published bin names (both exposed as runtimeInputs). Running # the sidecars directly—rather than going through `nix run .#...`—means diff --git a/modules/apps/cluster/nixidy-sync.sh b/modules/apps/cluster/nixidy-sync.sh index 7bb8d48b4..9b5551b52 100644 --- a/modules/apps/cluster/nixidy-sync.sh +++ b/modules/apps/cluster/nixidy-sync.sh @@ -1,12 +1,5 @@ #!/usr/bin/env bash # shellcheck shell=bash -# Build nixidy manifests then push them to the local-k3d private repo. -# Composition of `nixidy-build` followed by `nixidy-push`; both live on -# PATH as writeShellApplication-wrapped commands supplied via -# runtimeInputs in nixidy-sync.nix. -# -# Usage: -# nixidy-sync [--help] set -euo pipefail case "${1:-}" in From fc8d87e9edc57fc16fd41d5ed60ee46e89b5a7e5 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 44/77] chore(apps/docs): remove unused comments --- modules/apps/docs/deploy.nix | 25 +-- modules/apps/docs/deploy.sh | 277 +++++++------------------- modules/apps/docs/preview-version.nix | 10 +- modules/apps/docs/preview-version.sh | 119 ++++------- 4 files changed, 122 insertions(+), 309 deletions(-) diff --git a/modules/apps/docs/deploy.nix b/modules/apps/docs/deploy.nix index 51683b6e6..ae5af8629 100644 --- a/modules/apps/docs/deploy.nix +++ b/modules/apps/docs/deploy.nix @@ -3,12 +3,8 @@ # nix run .#deploy-docs -- preview # nix run .#deploy-docs -- production # -# Consumes the nix-built CF Worker payload from config.packages.vanixiets-docs -# ($out/{dist/,.wrangler/,wrangler.jsonc}) and dispatches to wrangler against -# the inherited environment per the ADR-002 env-var contract; the caller -# supplies CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID via one of -# {sops exec-env, direnv dotenv, GHA step env, M4 effect preamble reading -# HERCULES_CI_SECRETS_JSON}. See deploy.sh header for the full contract. +# Why: consumes the nix-built CF Worker payload from +# config.packages.vanixiets-docs (DOCS_PAYLOAD). # # Template bifurcation (writeShellApplication): INTERPOLATION FORM. # `text` is a nix string that injects one eval-time-computed path @@ -32,17 +28,14 @@ program = lib.getExe ( pkgs.writeShellApplication { name = "deploy-docs"; - # Per ADR-002 env-var contract: secrets flow via inherited env - # (never via `sops exec-env` inside the script), so pkgs.sops / - # pkgs.age are no longer required runtime inputs. + # Secrets flow via inherited env (never via `sops exec-env` + # inside the script), so pkgs.sops / pkgs.age are not required + # runtime inputs. # - # Hermeticity (m4-deploy-docs-git-env-contract): sed/awk/grep/find - # are explicitly declared even though hercules-ci-effects' default - # sandbox PATH supplies them implicitly. Closing this latent - # coupling makes the writeShellApplication self-sufficient under - # ANY caller context (not just the bwrap sandbox), satisfying the - # writeShellApplication invariant that PATH is exactly equal to - # runtimeInputs at runtime. + # sed/awk/grep/find are explicitly declared because the + # hercules-ci-effects bwrap sandbox PATH does not include them + # by default. Required for the writeShellApplication invariant + # that PATH equals runtimeInputs at runtime. runtimeInputs = [ pkgs.nodejs_24 pkgs.jq diff --git a/modules/apps/docs/deploy.sh b/modules/apps/docs/deploy.sh index 88fc2c4d3..89d6d9199 100644 --- a/modules/apps/docs/deploy.sh +++ b/modules/apps/docs/deploy.sh @@ -1,119 +1,40 @@ #!/usr/bin/env bash # shellcheck shell=bash # Docs deployment dispatcher invoked via `nix run .#deploy-docs`. +# See `usage()` for caller-facing usage; this header documents the +# env-var contract only. # -# Env-var contract (per ADR-002 / env-var-contract-design.md §2.1, extended -# by the m4-deploy-docs-git-env-contract feature to ALL host-PATH binary -# dependencies). 12 caller-overridable variables: 4 secret tokens (the -# closed effects bundle), 6 GIT_*, 2 DEPLOY_*. Symmetric env-first / -# shelled-fallback shape across all four caller contexts (effect preamble, -# sops exec-env, direnv, GHA env, local shell). -# -# Required (secret, provided by caller from the closed 4-key effects -# bundle — see modules/effects/vanixiets/secrets.nix): -# CLOUDFLARE_API_TOKEN wrangler auth token (CONSUMED by this -# script; required at runtime). -# CLOUDFLARE_ACCOUNT_ID Cloudflare account id (CONSUMED; wrangler -# requires this for account-scoped ops such -# as `versions upload` on a Worker attached -# to an account-level resource). -# GITHUB_TOKEN not consumed by deploy.sh; documented as -# part of the canonical effects bundle for -# homogeneity (consumed by release.sh). -# SOPS_AGE_KEY not consumed by deploy.sh; documented as -# part of the canonical effects bundle for -# homogeneity (consumed by -# k3d-bootstrap-secrets.sh). Per ADR-002, -# this script does NOT shell out to sops. -# Required (config, injected by deploy.nix): -# DOCS_PAYLOAD store path of the vanixiets-docs derivation -# ($out/{dist/, .wrangler/, wrangler.jsonc}) -# DOCS_NODE_MODULES store path of vanixiets-docs-deps node_modules -# tree (runtimeEnv of deploy.nix) -# -# Optional (git metadata; env-first with git-fallback). Extended in the -# m4-deploy-docs-git-env-contract feature to let the script run inside -# the buildbot-effects bwrap sandbox which does NOT bind-mount the -# working tree (upstream-by-design). Every GIT_* consumer below is -# expressed as `${GIT_X:-$(git … 2>/dev/null || true)}` so the three -# supported caller contexts all work: effect preamble (env pre-populated, -# no .git reachable), local shell from a git worktree (env unset, git -# fallback), and GHA after checkout (env unset, git fallback). -# GIT_REV 40-char commit SHA (fallback: git rev-parse HEAD) -# GIT_REV_SHORT 7-ish-char short SHA (fallback: git rev-parse --short HEAD) -# GIT_REV_SHORT12 12-char short SHA used for wrangler --tag / -# workers/tag cross-check (fallback: -# git rev-parse --short=12 HEAD). VAL-WRITESHELL-DOCS-010 -# commit_tag invariant sources from here. -# GIT_BRANCH current branch name (fallback: git branch --show-current) -# GIT_COMMIT_MSG HEAD subject line used in the version-message -# annotation (fallback: git log -1 --pretty=format:'%s') -# GIT_WORKTREE_STATUS literal "clean" or "dirty" (fallback: -# git diff-index --quiet HEAD -- && echo clean || echo dirty) -# -# Optional (deploy-context metadata; env-first with bash-builtin / -# shelled-fallback). Generalisation of the same pattern to ALL host-PATH -# binary dependencies, fixing the second-bug-class regression where the -# bwrap sandbox lacks `hostname`/`whoami` on PATH (only /nix/store -# ro-bind + writeShellApplication runtimeInputs are available). The -# bash builtin `$HOSTNAME` is populated from gethostname(2) at shell -# startup — no external binary required in any context. -# DEPLOY_HOST short hostname for the production deploy -# message. Fallback: ${HOSTNAME%%.*} -# (bash builtin parameter expansion; trims -# the first dot-suffix to mimic `hostname -s` -# without shelling out). -# DEPLOY_DEPLOYER actor identity for deploy/version messages. -# Fallback chain: GITHUB_ACTOR (GHA context) -# → `whoami 2>/dev/null` (local shell with -# /etc/passwd available) → "unknown". -# -# Optional (caller debugging / overrides): -# WRANGLER override binary path for test harnesses; default -# $DOCS_NODE_MODULES/.bin/wrangler -# DEPLOY_DOCS_DEBUG preserve tmpdir on exit when set -# GITHUB_ACTIONS / GITHUB_ACTOR / GITHUB_WORKFLOW -# when GITHUB_ACTIONS is set, the production -# deploy message uses GITHUB_WORKFLOW (default -# "CI") as the deploy context instead of -# DEPLOY_HOST. GITHUB_ACTOR participates in -# the DEPLOY_DEPLOYER fallback chain. -# -# Caller mechanisms (satisfy each contract slot via one of): -# - Local dev: caller-side sops wrapper (justfile `docs-deploy-*` -# recipes wrap with `sops` to decrypt secrets/shared.yaml -# and export the Cloudflare env before the nested nix run) -# OR direnv dotenv (.envrc loads .env with the Cloudflare -# env vars already exported). DEPLOY_HOST / DEPLOY_DEPLOYER -# left unset → bash-builtin / whoami fallback. -# - GHA env: deploy-docs.yaml step wraps the nix run with the same -# caller-side sops decrypt inside a nix-develop wrapper; -# the age key is provided via the step `env:` block from -# the repo secrets (see deploy-docs.yaml). GIT_* / DEPLOY_* -# left unset → git/bash-builtin/GITHUB_ACTOR fallback. -# - M4 effect: the deploy-docs dispatcher effect preamble extracts -# CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID from -# $HERCULES_CI_SECRETS_JSON, exports the six GIT_* -# variables interpolated from herculesCI.config.repo.* -# (via lib.escapeShellArg + builtins.substring at eval -# time), AND exports DEPLOY_DEPLOYER=hercules-ci-effects -# and DEPLOY_HOST=magnetite before invoking the embedded -# store path ${config.apps.deploy-docs.program}. The -# bwrap sandbox does not bind-mount the working tree -# and provides no host-PATH binaries beyond /nix/store -# ro-bind + runtimeInputs PATH, so every git/hostname/ -# whoami consumer in this script is env-first with -# bash-builtin or error-tolerant shelled fallback. -# -# Secret passing rule (per ADR-002): wrangler authentication flows ONLY -# through inherited env vars; no authentication CLI flags are used. -# No caller-side sops wrappers inside this script (the caller wraps if -# their mechanism is sops-based). -# -# Usage: -# deploy-docs preview -# deploy-docs production -# deploy-docs --help +# Required (secret, caller-provided from the closed 4-key effects bundle — +# modules/effects/vanixiets/secrets.nix): +# CLOUDFLARE_API_TOKEN wrangler auth token (CONSUMED). +# CLOUDFLARE_ACCOUNT_ID Cloudflare account id (CONSUMED; +# account-scoped ops require this). +# GITHUB_TOKEN not consumed here; bundle homogeneity +# (consumed by release.sh). +# SOPS_AGE_KEY not consumed here; bundle homogeneity +# (consumed by k3d-bootstrap-secrets.sh). +# Required (config, injected by deploy.nix): +# DOCS_PAYLOAD vanixiets-docs derivation outPath +# ($out/{dist/, .wrangler/, wrangler.jsonc}). +# DOCS_NODE_MODULES vanixiets-docs-deps node_modules tree. +# Optional (env-first with git-fallback): every GIT_* consumer is +# `${GIT_X:-$(git … 2>/dev/null || true)}` so the script runs both +# inside the buildbot-effects bwrap sandbox (no .git bind-mounted; env +# pre-populated by the effect preamble) and from a live worktree (env +# unset; git fallback resolves locally): +# GIT_REV, GIT_REV_SHORT, GIT_REV_SHORT12, GIT_BRANCH, +# GIT_COMMIT_MSG, GIT_WORKTREE_STATUS. +# Optional (env-first with bash-builtin / shelled-fallback): the bwrap +# sandbox lacks `hostname`/`whoami` on PATH, so DEPLOY_HOST falls back to +# `${HOSTNAME%%.*}` (bash builtin populated from gethostname(2)) and +# DEPLOY_DEPLOYER falls back to GITHUB_ACTOR → `whoami 2>/dev/null` +# → "unknown". +# Optional (caller debugging / overrides): +# WRANGLER, DEPLOY_DOCS_DEBUG, GITHUB_ACTIONS / GITHUB_ACTOR / +# GITHUB_WORKFLOW (when GITHUB_ACTIONS is set, the production deploy +# message uses GITHUB_WORKFLOW (default "CI") as deploy context +# instead of DEPLOY_HOST). + set -euo pipefail usage() { @@ -184,8 +105,7 @@ if [[ -z "$mode" ]]; then fi shift -# Env-var contract guards (per ADR-002 / env-var-contract-design.md §2.1.2). -# Fail fast before any wrangler / filesystem work if the contract is unmet. +# Env-var contract guards: fail fast before any wrangler / filesystem work. : "${DOCS_PAYLOAD:?DOCS_PAYLOAD not set; deploy.nix must pass the nix-built payload}" [[ -d "$DOCS_PAYLOAD" ]] || { echo "error: DOCS_PAYLOAD=$DOCS_PAYLOAD is not a directory" >&2; exit 1; } : "${DOCS_NODE_MODULES:?DOCS_NODE_MODULES not set; deploy.nix must expose vanixiets-docs-deps via runtimeEnv}" @@ -194,8 +114,8 @@ shift # Hermetic wrangler via bun-managed node_modules (vanixiets-docs-deps derivation). # The `${WRANGLER:-...}` fallback allows test harnesses (e.g. the no-op wrangler stub -# used to exercise the post-condition error paths for VAL-WRITESHELL-DOCS-010) to -# override the hermetic binary without rewriting this script. +# used to exercise the post-condition error paths) to override the hermetic +# binary without rewriting this script. export WRANGLER="${WRANGLER:-$DOCS_NODE_MODULES/.bin/wrangler}" # Invoke wrangler via real node, not the .bin/wrangler shebang: @@ -209,16 +129,13 @@ export WRANGLER="${WRANGLER:-$DOCS_NODE_MODULES/.bin/wrangler}" # Empirical: diagnosed 2026-04-22 via magnetite linux-x64 reproducer; # same machine + wrangler runs fine under real node, hangs under bun. -# Git metadata is resolved below via env-first / git-fallback (see top-of- -# file env-var contract for GIT_*). Wrangler is invoked with absolute -# `--config "$WRANGLER_CONFIG"`, so CWD is immaterial — no `cd` into the -# worktree is required (and would fail inside the buildbot-effects bwrap +# Wrangler is invoked with absolute `--config "$WRANGLER_CONFIG"`, so no +# `cd` into the worktree is required (and would fail inside the bwrap # sandbox, which does not bind-mount the working tree). -# Materialise a writable copy of the nix payload. wrangler reads -# .wrangler/deploy/config.json whose configPath ("../../dist/server/wrangler.json") -# resolves against the config file's location, and wrangler may write state to -# .wrangler/ during deploy — both require a writable tree outside /nix/store. +# Materialise a writable copy of the nix payload: wrangler reads +# .wrangler/deploy/config.json (configPath resolves against the config +# file's location) and may write state to .wrangler/ during deploy. tmpdir=$(mktemp -d -t deploy-docs.XXXXXX) if [[ -n "${DEPLOY_DOCS_DEBUG:-}" ]]; then echo "[deploy-docs] DEBUG: preserving tmpdir at $tmpdir" >&2 @@ -237,32 +154,22 @@ chmod -R u+w "$tmpdir" # present in the source wrangler.jsonc. wrangler_config="$tmpdir/dist/server/wrangler.json" -# Commit metadata shared by preview and production subcommands. Env-first / -# git-fallback per the GIT_* env-var contract (see top-of-file header). -# All `git` invocations are guarded by `2>/dev/null || true` so that a -# missing .git (e.g. buildbot-effects bwrap sandbox, no bind-mounted worktree) -# surfaces as empty strings rather than a non-zero exit; the env-first path -# supplies the authoritative values in that context. +# Commit metadata: env-first with errexit-tolerant git fallback so a +# missing .git (bwrap sandbox) surfaces as empty strings rather than +# aborting; the env-first path supplies authoritative values in that case. commit_sha="${GIT_REV:-$(git rev-parse HEAD 2>/dev/null || true)}" commit_tag="${GIT_REV_SHORT12:-$(git rev-parse --short=12 HEAD 2>/dev/null || true)}" commit_short="${GIT_REV_SHORT:-$(git rev-parse --short HEAD 2>/dev/null || true)}" current_branch="${GIT_BRANCH:-$(git branch --show-current 2>/dev/null || true)}" -# Resolve deployer / deploy_host with env-first / bash-builtin / shelled-fallback -# per the DEPLOY_* env-var contract (see top-of-file header). The bwrap -# sandbox provides no host-PATH binaries beyond /nix/store ro-bind + -# runtimeInputs PATH, so unconditional `hostname -s` / `whoami` would fail -# with `command not found` (exit 127). Bash builtin `$HOSTNAME` is populated -# from gethostname(2) at shell startup and requires no external binary in -# any context; `${HOSTNAME%%.*}` mimics `hostname -s` via parameter -# expansion. `whoami` is retained as a final fallback for local shells where -# DEPLOY_DEPLOYER and GITHUB_ACTOR are both unset; it is error-tolerant -# (`2>/dev/null || echo unknown`) so a missing /etc/passwd entry surfaces -# as "unknown" rather than a non-zero exit. +# Resolve deployer / deploy_host with env-first / bash-builtin / +# shelled-fallback. Bash builtin `$HOSTNAME` is populated from +# gethostname(2) at shell startup, so `${HOSTNAME%%.*}` mimics +# `hostname -s` without shelling out — required because the bwrap +# sandbox lacks `hostname` on PATH. deploy_host="${DEPLOY_HOST:-${HOSTNAME%%.*}}" deployer="${DEPLOY_DEPLOYER:-${GITHUB_ACTOR:-$(whoami 2>/dev/null || echo unknown)}}" -# Compose deploy message (prefer GitHub Actions context, fall back to local). if [[ -n "${GITHUB_ACTIONS:-}" ]]; then deploy_context="${GITHUB_WORKFLOW:-CI}" deploy_msg="Deployed by ${deployer} from ${current_branch} via ${deploy_context}" @@ -279,18 +186,14 @@ case "$mode" in exit 2 fi - # Sanitize branch name for Cloudflare alias (valid subdomain component): - # replace / with -, collapse runs, strip leading/trailing -, cap at 40 chars. safe_branch=$(echo "$branch" \ | tr '/' '-' \ | tr -c 'a-zA-Z0-9-' '-' \ | sed 's/--*/-/g; s/^-//; s/-$//' \ | cut -c1-40) - # Env-first / git-fallback — see top-of-file GIT_* env-var contract. - # `git log` / `git diff-index` are error-tolerant so a missing .git - # (buildbot-effects bwrap) leaves commit_msg empty; the effect preamble - # supplies GIT_COMMIT_MSG and GIT_WORKTREE_STATUS=clean in that case. + # Env-first / errexit-tolerant git fallback so a missing .git leaves + # commit_msg empty; the effect preamble supplies authoritative values. commit_msg="${GIT_COMMIT_MSG:-$(git log -1 --pretty=format:'%s' 2>/dev/null || true)}" if [[ -n "${GIT_WORKTREE_STATUS:-}" ]]; then git_status="$GIT_WORKTREE_STATUS" @@ -331,8 +234,7 @@ case "$mode" in # `version-upload` event, which carries `version_id`, `worker_tag`, # `preview_url`, and `preview_alias_url`. # - # Three post-conditions enforce the no-silent-success invariant - # (VAL-WRITESHELL-DOCS-010): + # Three post-conditions enforce the no-silent-success invariant: # (a) the NDJSON event log contains a `type == "version-upload"` entry # with a non-empty `version_id` (primary authoritative source) # (b) `wrangler versions list --json` contains an entry whose @@ -359,13 +261,7 @@ case "$mode" in # fallback version_id source when the NDJSON event stream from # WRANGLER_OUTPUT_FILE_PATH doesn't produce the expected # `type:"version-upload"` event. Retained as defense-in-depth against - # future wrangler silent-success regressions. See top-of-file rationale - # (lines ~101-110) for the diagnostic history. - # - # Diagnostic: echo the exact upload command line to stderr so the GHA log - # shows what the shell is about to invoke (CLOUDFLARE_API_TOKEN and - # CLOUDFLARE_ACCOUNT_ID are expected in the inherited env per the - # env-var contract; never printed on argv). + # future wrangler silent-success regressions. printf '>> wrangler upload command: node %s --config %s versions upload --preview-alias %s --tag %s --message %q\n' \ "$WRANGLER" "$WRANGLER_CONFIG" "b-${SAFE_BRANCH}" "$VERSION_TAG" "$VERSION_MESSAGE" >&2 @@ -382,10 +278,8 @@ case "$mode" in unset WRANGLER_OUTPUT_FILE_PATH # Post-condition (a): extract a non-empty Worker Version ID. - # Primary source: NDJSON `version-upload` event (Option A) - # Fallback source: stdout line `Worker Version ID: ` (Option B) - # The cross-check in post-condition (b) below guarantees the version - # actually persisted server-side regardless of which source produced it. + # Primary: NDJSON `version-upload` event. Fallback: stdout line + # `Worker Version ID: `. (b) cross-checks server-side persistence. version_id="" if [[ -s "$wrangler_upload_ndjson" ]]; then version_id=$( @@ -422,23 +316,18 @@ case "$mode" in echo " - confirm CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID are exported by" >&2 echo " the caller (see deploy.sh env-var contract header for caller mechanisms)" >&2 echo " - if linux-x64 regression, confirm wrangler invoked under real node and" >&2 - echo " not bun-fake-node (see top-of-file rationale at lines ~101-110)" >&2 + echo " not bun-fake-node (see deploy.sh node invocation rationale)" >&2 echo " - inspect the wrangler internal log dumped below / raw NDJSON and stdout paths above for any output" >&2 echo "" >&2 # Locate wrangler's internal log file by glob + newest mtime across - # platform-specific candidate locations. Avoids depending on wrangler's - # stdout `Writing logs to "..."` announcement (only printed under - # WRANGLER_LOG=debug, which we no longer set). The log file contains - # full HTTP request/response bodies and any internal stack traces that - # are otherwise destroyed with the GHA runner — dump it first as the - # most informative diagnostic source when NDJSON/stdout/stderr are - # empty or truncated. + # platform-specific candidate locations. The log file contains full + # HTTP request/response bodies and any internal stack traces — most + # informative diagnostic source when NDJSON/stdout/stderr are empty. wrangler_log_path="" for candidate_dir in "$HOME/.wrangler/logs" "$HOME/.config/.wrangler/logs"; do if [[ -d "$candidate_dir" ]]; then - # Filename format is `wrangler-YYYY-MM-DD_HH-MM-SS_mmm.log` — the - # embedded timestamp is zero-padded and lexicographically sortable, - # so `sort | tail -1` selects the newest without needing ls -t. + # Filename `wrangler-YYYY-MM-DD_HH-MM-SS_mmm.log` is + # zero-padded and lex-sortable, so `sort | tail -1` picks newest. newest=$(find "$candidate_dir" -maxdepth 1 -type f -name 'wrangler-*.log' 2>/dev/null | sort | tail -1 || true) if [[ -n "$newest" ]]; then wrangler_log_path="$newest" @@ -493,7 +382,6 @@ case "$mode" in exit 1 fi - # Post-condition (c): authoritative success echo with parsed Worker Version ID. echo "" echo "Version uploaded successfully" echo " Worker Version ID: ${version_id}" @@ -514,12 +402,9 @@ case "$mode" in export WRANGLER_CONFIG="$wrangler_config" # Query for an existing version uploaded from this commit (via preview). - # Capture versions list to a tempfile so post-condition verification below - # can reuse it (avoids a second API call purely for the existing-version - # lookup) and any diagnostic error messages can reference the raw JSON. - # `| cat >` is used instead of `>` to route wrangler's stdout through a - # pipe-shaped fd; see the preview subcommand's equivalent comment for the - # empirical rationale. + # Capture versions list to a tempfile so post-condition verification can + # reuse it. `| cat >` routes through a pipe-shaped fd — see the preview + # subcommand's equivalent comment for the empirical rationale. wrangler_list_json="$tmpdir/wrangler-versions-list.json" node "$WRANGLER" --config "$WRANGLER_CONFIG" versions list --json \ @@ -544,9 +429,8 @@ case "$mode" in # via `wrangler deployments list --json` before declaring success. Like # `versions upload`, `versions deploy` does NOT accept `--json` on # wrangler 4.84.x — the event log is the authoritative machine-readable - # output channel. Detects wrangler's silent-exit failure mode - # (VAL-WRITESHELL-DOCS-010 + diagnostic session 45961bc9) when the - # CI-detection branch exits 0 without actually performing the promotion. + # output channel. Detects wrangler's silent-exit failure mode when the + # CI-detection branch exits 0 without performing the promotion. deploy_ndjson="$tmpdir/wrangler-versions-deploy.ndjson" deploy_stdout="$tmpdir/wrangler-versions-deploy.stdout" : > "$deploy_ndjson" @@ -561,9 +445,6 @@ case "$mode" in unset WRANGLER_OUTPUT_FILE_PATH - # Post-condition (a): extract a non-empty Deployment ID. Primary source - # is the NDJSON `version-deploy` event (Option A); fallback is stdout - # parsing for a recognizable deployment identifier (Option B). deployment_id="" if [[ -s "$deploy_ndjson" ]]; then deployment_id=$( @@ -574,8 +455,7 @@ case "$mode" in ) fi if [[ -z "$deployment_id" ]]; then - # stdout fallback: match patterns like "Deployment ID: " or - # "deployment_id: " that wrangler prints on the console. + # stdout fallback: match `Deployment ID: ` / `deployment_id: `. deployment_id=$( grep -oiE '(Deployment ID|deployment_id)[[:space:]]*:[[:space:]]*[a-f0-9-]+' \ "$deploy_stdout" 2>/dev/null \ @@ -595,14 +475,11 @@ case "$mode" in echo " - confirm CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID are exported by" >&2 echo " the caller (see deploy.sh env-var contract header for caller mechanisms)" >&2 echo " - if linux-x64 regression, confirm wrangler invoked under real node and" >&2 - echo " not bun-fake-node (see top-of-file rationale at lines ~101-110)" >&2 + echo " not bun-fake-node (see deploy.sh node invocation rationale)" >&2 echo " - inspect the wrangler internal log dumped below / raw capture paths above for any output" >&2 exit 1 fi - # Post-condition (b): cross-check via deployments list that the deploy - # landed server-side. `| cat >` empirically required — see preview path - # comment for the rationale. deployments_list_json="$tmpdir/wrangler-deployments-list.json" node "$WRANGLER" --config "$WRANGLER_CONFIG" deployments list --json \ @@ -626,7 +503,6 @@ case "$mode" in exit 1 fi - # Post-condition (c): authoritative success echo with parsed Deployment ID. echo "" echo "successfully promoted version ${existing_version} to production" echo " Deployment ID: ${deployment_id}" @@ -646,12 +522,11 @@ case "$mode" in export DEPLOYMENT_MESSAGE="$deploy_msg" - # Fallback direct-deploy: same post-condition pattern, but the relevant - # NDJSON event is `type == "deploy"` which carries `version_id` (no - # deployment_id field on this event type — see wrangler cli.js - # writeOutput block for `deploy`). `wrangler deploy` does NOT accept - # `--json` on wrangler 4.84.x; WRANGLER_OUTPUT_FILE_PATH is the - # authoritative machine-readable channel. + # Fallback direct-deploy: same post-condition pattern, but the + # NDJSON event is `type == "deploy"` carrying `version_id` (no + # deployment_id field on this event type). `wrangler deploy` does + # NOT accept `--json` on wrangler 4.84.x; WRANGLER_OUTPUT_FILE_PATH + # is the authoritative machine-readable channel. deploy_ndjson="$tmpdir/wrangler-deploy.ndjson" deploy_stdout="$tmpdir/wrangler-deploy.stdout" : > "$deploy_ndjson" @@ -664,10 +539,6 @@ case "$mode" in unset WRANGLER_OUTPUT_FILE_PATH - # Post-condition (a): extract the just-deployed version_id. Primary - # source: NDJSON `deploy` event (Option A). Fallback: stdout grep - # (Option B) for the "Current Version ID: " or similar line - # wrangler prints on direct deploy. deploy_version_id="" if [[ -s "$deploy_ndjson" ]]; then deploy_version_id=$( @@ -697,7 +568,7 @@ case "$mode" in echo " - confirm CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID are exported by" >&2 echo " the caller (see deploy.sh env-var contract header for caller mechanisms)" >&2 echo " - if linux-x64 regression, confirm wrangler invoked under real node and" >&2 - echo " not bun-fake-node (see top-of-file rationale at lines ~101-110)" >&2 + echo " not bun-fake-node (see deploy.sh node invocation rationale)" >&2 echo " - inspect the wrangler internal log dumped below / raw capture paths above for any output" >&2 exit 1 fi diff --git a/modules/apps/docs/preview-version.nix b/modules/apps/docs/preview-version.nix index c395add08..2f2a8bc51 100644 --- a/modules/apps/docs/preview-version.nix +++ b/modules/apps/docs/preview-version.nix @@ -1,14 +1,12 @@ -# Flake app: preview the semantic-release version that would be published after -# merging the current branch into a target branch. +# Flake app: preview the semantic-release version that would be published +# after merging the current branch into a target branch. # -# Usage: # nix run .#preview-version # root package on main # nix run .#preview-version -- main packages/docs # monorepo package preview # # Hermetic: semantic-release and its plugins are provided by the -# vanixiets-docs-deps derivation (linked into the worktree at runtime); the app -# is self-contained and does not depend on a prior `bun install` or on -# pkgs.semantic-release. +# vanixiets-docs-deps derivation linked into the worktree at runtime; no +# prior `bun install` or pkgs.semantic-release dependency. # # Template bifurcation (writeShellApplication): PURE READFILE FORM. # `text = builtins.readFile ./preview-version.sh` — the sidecar is consumed diff --git a/modules/apps/docs/preview-version.sh b/modules/apps/docs/preview-version.sh index 0d6e208e7..aa7afde84 100644 --- a/modules/apps/docs/preview-version.sh +++ b/modules/apps/docs/preview-version.sh @@ -1,38 +1,18 @@ #!/usr/bin/env bash # shellcheck shell=bash -# preview-version.sh - Preview semantic-release version after merging to target branch +# preview-version.sh - Preview semantic-release version after merging to +# target branch. See `usage()` for caller-facing usage. # -# Usage: -# nix run .#preview-version -- [target-branch] [package-path] -# -# Examples: -# nix run .#preview-version # Preview root version on main -# nix run .#preview-version -- main packages/docs # Preview docs package version on main -# nix run .#preview-version -- beta packages/docs # Preview docs version on beta -# -# This script simulates merging the current branch into the target branch and -# runs semantic-release in dry-run mode to preview what version would be released. -# -# Hermetic: DOCS_NODE_MODULES (set by preview-version.nix) points to a read-only -# node_modules tree produced by the vanixiets-docs-deps derivation. This script -# links it into the worktree's package directory and invokes semantic-release -# directly via node_modules/.bin, bypassing any need for bun or a prior -# `bun install`. -# -# Env-var contract (per ADR-002 / env-var-contract-design.md §2.2): +# Env-var contract: # Required (config, injected by preview-version.nix runtimeEnv): -# DOCS_NODE_MODULES store path of the vanixiets-docs-deps node_modules -# tree (hosts node_modules/.bin/semantic-release). +# DOCS_NODE_MODULES vanixiets-docs-deps node_modules tree (hosts +# node_modules/.bin/semantic-release). # Optional (caller-provided): -# CURRENT_BRANCH bookmark/branch name to attach HEAD to when invoked -# from a jj-colocated detached-HEAD setup. +# CURRENT_BRANCH bookmark/branch name to attach HEAD to when +# invoked from jj-colocated detached HEAD. # -# This script does NOT require secret env vars (CLOUDFLARE_API_TOKEN, -# CLOUDFLARE_ACCOUNT_ID, GITHUB_TOKEN, SOPS_AGE_KEY). semantic-release is -# invoked in --dry-run with @semantic-release/github filtered out of the -# plugin list, so no secret env required for any caller (direnv dotenv, -# `sops exec-env` wrapper, GHA `env:` block, or M4 effect preamble -# reading HERCULES_CI_SECRETS_JSON). +# No secret env vars required: semantic-release runs --dry-run with +# @semantic-release/github filtered out of the plugin list. set -euo pipefail @@ -87,25 +67,23 @@ PACKAGE_PATH="${2:-}" REPO_ROOT=$(git rev-parse --show-toplevel) WORKTREE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/semantic-release-preview.XXXXXX") -# Save original target branch HEAD for restoration ORIGINAL_TARGET_HEAD="" ORIGINAL_REMOTE_HEAD="" -# Track which node_modules symlink(s) we created so cleanup can remove them. +# Track node_modules symlink(s) we created for cleanup. WORKTREE_NODE_MODULES_LINK="" LOCAL_NODE_MODULES_LINK="" # Local bare clone used to redirect semantic-release verifyAuth's # `git push --dry-run HEAD:` away from the GitHub remote # (which can short-circuit semantic-release on branch-protection rejection -# or token-permission mismatch — see m5-01i mission notes). -# Populated AFTER `git update-ref` so the bare's refs/heads/ -# captures TEMP_COMMIT, allowing the dry-run push to be a no-op fast-forward -# against a quiescent file:// remote with no auth and no protection. +# or token-permission mismatch). Populated AFTER `git update-ref` so the +# bare's refs/heads/ captures TEMP_COMMIT, allowing the +# dry-run push to be a no-op fast-forward against a quiescent file:// +# remote with no auth and no protection. PREVIEW_BARE_DIR="" PREVIEW_BARE="" -# Colors for output RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' @@ -125,11 +103,10 @@ NC='\033[0m' # No Color ORIGINAL_HEAD_SHA="" WE_ATTACHED_HEAD=0 if [ -n "${CURRENT_BRANCH:-}" ]; then - # Env-var override path. DETECTED_BRANCH=$(git branch --show-current) if [ -z "$DETECTED_BRANCH" ]; then - # HEAD is detached; attach to the provided branch so git operations that - # rely on an attached HEAD work, and remember how to restore detached state. + # HEAD detached: attach to the provided branch and remember how to + # restore detached state on cleanup. ORIGINAL_HEAD_SHA=$(git rev-parse --verify HEAD) echo -e "${BLUE}CURRENT_BRANCH=${CURRENT_BRANCH} override; attaching HEAD for duration of preview${NC}" >&2 if ! git checkout --quiet "$CURRENT_BRANCH"; then @@ -138,8 +115,7 @@ if [ -n "${CURRENT_BRANCH:-}" ]; then fi WE_ATTACHED_HEAD=1 fi - # If HEAD was already attached, we honor CURRENT_BRANCH as-is without - # performing any checkout dance (per task spec). + # HEAD already attached: honour CURRENT_BRANCH as-is, no checkout. else CURRENT_BRANCH=$(git branch --show-current) if [ -z "$CURRENT_BRANCH" ]; then @@ -152,13 +128,11 @@ else fi fi -# Cleanup function (invoked via `trap cleanup EXIT INT TERM` below) # shellcheck disable=SC2329 cleanup() { local exit_code=$? - # Remove any node_modules symlinks we created. Only unlink if still a symlink - # (guards against manual replacement mid-run). + # Only unlink if still a symlink (guards against manual replacement). if [ -n "$WORKTREE_NODE_MODULES_LINK" ] && [ -L "$WORKTREE_NODE_MODULES_LINK" ]; then rm -f "$WORKTREE_NODE_MODULES_LINK" fi @@ -166,27 +140,22 @@ cleanup() { rm -f "$LOCAL_NODE_MODULES_LINK" fi - # Always restore target branch to original state if we modified it if [ -n "$ORIGINAL_TARGET_HEAD" ]; then echo -e "\n${BLUE}restoring ${TARGET_BRANCH} to original state...${NC}" git update-ref "refs/heads/$TARGET_BRANCH" "$ORIGINAL_TARGET_HEAD" 2>/dev/null || true fi - # Always restore remote-tracking branch to original state if we modified it if [ -n "$ORIGINAL_REMOTE_HEAD" ]; then git update-ref "refs/remotes/origin/$TARGET_BRANCH" "$ORIGINAL_REMOTE_HEAD" 2>/dev/null || true fi - # Clean up worktree if [ -d "$WORKTREE_DIR" ]; then echo -e "${BLUE}cleaning up worktree...${NC}" git worktree remove --force "$WORKTREE_DIR" 2>/dev/null || true - # Prune any stale worktree references git worktree prune 2>/dev/null || true fi - # Clean up local bare clone created for semantic-release verifyAuth - # redirection (see m5-01i fix). + # Clean up the local bare clone used for verifyAuth redirection. if [ -n "$PREVIEW_BARE_DIR" ] && [ -d "$PREVIEW_BARE_DIR" ]; then rm -rf "$PREVIEW_BARE_DIR" fi @@ -205,9 +174,9 @@ cleanup() { trap cleanup EXIT INT TERM -# link_docs_node_modules : symlink DOCS_NODE_MODULES into the given -# directory's node_modules slot, guarding against clobbering a real install. -# Echoes the resulting symlink path so callers can record it for cleanup. +# link_docs_node_modules : symlink DOCS_NODE_MODULES into the +# directory's node_modules slot, refusing to overwrite a real install. +# Echoes the symlink path for cleanup tracking. link_docs_node_modules() { local target_dir="$1" local slot="$target_dir/node_modules" @@ -219,7 +188,6 @@ link_docs_node_modules() { echo "$slot" } -# Validation if [ "$CURRENT_BRANCH" == "$TARGET_BRANCH" ]; then echo -e "${YELLOW}already on target branch ${TARGET_BRANCH}${NC}" echo -e "${YELLOW}running test-release instead of preview${NC}\n" @@ -232,7 +200,6 @@ if [ "$CURRENT_BRANCH" == "$TARGET_BRANCH" ]; then exec node ./node_modules/.bin/semantic-release --dry-run --no-ci fi -# Display what we're doing echo -e "${BLUE}═══════════════════════════════════════════════════════════════${NC}" echo -e "${BLUE}semantic-release version preview${NC}" echo -e "${BLUE}═══════════════════════════════════════════════════════════════${NC}" @@ -245,19 +212,14 @@ else fi echo -e "${BLUE}───────────────────────────────────────────────────────────────${NC}\n" -# Verify target branch exists if ! git show-ref --verify --quiet "refs/heads/$TARGET_BRANCH"; then echo -e "${RED}error: target branch '${TARGET_BRANCH}' does not exist${NC}" >&2 exit 1 fi -# Save original target branch HEAD before any modifications ORIGINAL_TARGET_HEAD=$(git rev-parse "$TARGET_BRANCH") - -# Save original remote-tracking branch HEAD before any modifications ORIGINAL_REMOTE_HEAD=$(git rev-parse "origin/$TARGET_BRANCH" 2>/dev/null || echo "") -# Create merge tree to test if merge is possible echo -e "${BLUE}simulating merge of ${CURRENT_BRANCH} → ${TARGET_BRANCH}...${NC}" # Perform merge-tree operation to test if merge is possible. @@ -277,7 +239,6 @@ if ! MERGE_OUTPUT=$(git merge-tree --write-tree "$TARGET_BRANCH" "$CURRENT_BRANC exit 1 fi -# Extract tree hash from merge-tree output (first line) MERGE_TREE=$(echo "$MERGE_OUTPUT" | head -1) if [ -z "$MERGE_TREE" ]; then @@ -285,7 +246,6 @@ if [ -z "$MERGE_TREE" ]; then exit 1 fi -# Create temporary merge commit echo -e "${BLUE}creating temporary merge commit...${NC}" TEMP_COMMIT=$(git commit-tree -p "$TARGET_BRANCH" -p "$CURRENT_BRANCH" \ -m "Temporary merge for semantic-release preview" "$MERGE_TREE") @@ -295,18 +255,18 @@ if [ -z "$TEMP_COMMIT" ]; then exit 1 fi -# Temporarily update target branch to point to merge commit -# This allows semantic-release to analyze the correct commit history -# The cleanup function will ALWAYS restore the original branch HEAD +# Temporarily point target branch at the merge commit so semantic-release +# analyzes the correct history; cleanup always restores the original HEAD. echo -e "${BLUE}temporarily updating ${TARGET_BRANCH} ref for analysis...${NC}" git update-ref "refs/heads/$TARGET_BRANCH" "$TEMP_COMMIT" -# Also update remote-tracking branch to match (so semantic-release sees them as synchronized) +# Mirror onto remote-tracking so semantic-release sees them synchronized. git update-ref "refs/remotes/origin/$TARGET_BRANCH" "$TEMP_COMMIT" -# Capture the post-update-ref state into a local bare clone so semantic-release's -# `verifyAuth` can run `git push --dry-run HEAD:` against a -# quiescent file:// remote instead of the GitHub origin (m5-01i fix). +# Capture the post-update-ref state into a local bare clone so +# semantic-release's `verifyAuth` runs `git push --dry-run +# HEAD:` against a quiescent file:// remote instead of the +# GitHub origin. # # The bare must be cloned from $REPO_ROOT (cwd's local refs at clone time # include the just-updated refs/heads/ = TEMP_COMMIT). Cloning @@ -324,15 +284,12 @@ PREVIEW_BARE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/preview-bare.XXXXXX") PREVIEW_BARE="$PREVIEW_BARE_DIR/preview.git" git clone --quiet --bare "$REPO_ROOT" "$PREVIEW_BARE" -# Create worktree at target branch (now pointing to merge commit) echo -e "${BLUE}creating temporary worktree at ${TARGET_BRANCH}...${NC}" git worktree add --quiet "$WORKTREE_DIR" "$TARGET_BRANCH" -# Navigate to worktree cd "$WORKTREE_DIR" -# Link the hermetic vanixiets-docs-deps tree into the worktree's package dir. -# (bun install is no longer required here.) +# Link hermetic vanixiets-docs-deps into the worktree's package dir. if [ -n "$PACKAGE_PATH" ]; then if [ ! -d "$PACKAGE_PATH" ]; then echo -e "${RED}error: package path '${PACKAGE_PATH}' does not exist${NC}" >&2 @@ -344,28 +301,23 @@ else WORKTREE_NODE_MODULES_LINK=$(link_docs_node_modules "$WORKTREE_DIR") fi -# Run semantic-release in dry-run mode echo -e "\n${BLUE}running semantic-release analysis...${NC}\n" -# Capture output and parse version -# Exclude @semantic-release/github to avoid GitHub token requirement for preview -# This is safe because dry-run skips publish/success/fail steps anyway +# Exclude @semantic-release/github to avoid GitHub token requirement for +# preview; safe because dry-run skips publish/success/fail steps anyway. PLUGINS="@semantic-release/commit-analyzer,@semantic-release/release-notes-generator" -# Forensic banner (m5-01i): confirms the verifyAuth-redirect bare clone is -# engaged in production logs. Stable banner namespace; kept structurally -# similar to RELEASE-CLONE-PR-HEAD / RELEASE-CLONE-PR-DISPATCH. +# Forensic banner: confirms the verifyAuth-redirect bare clone is engaged +# in production logs (parallels RELEASE-CLONE-PR-HEAD / -DISPATCH). echo "RELEASE-PREVIEW-BARE: $PREVIEW_BARE" OUTPUT=$(GITHUB_REF="refs/heads/$TARGET_BRANCH" node ./node_modules/.bin/semantic-release --dry-run --no-ci --repository-url "file://$PREVIEW_BARE" --branches "$TARGET_BRANCH" --plugins "$PLUGINS" 2>&1 || true) -# Display semantic-release summary (filter out verbose plugin repetition) echo "$OUTPUT" | grep -v "^$" | grep -vE "(No more plugins|does not provide step)" | \ grep -E "(semantic-release|Running|analyzing|Found.*commits|release version|Release note|Features|Bug Fixes|Breaking Changes|Published|\*\s)" || true echo -e "\n${BLUE}═══════════════════════════════════════════════════════════════${NC}" -# Extract and display the next version if echo "$OUTPUT" | grep -q "There are no relevant changes"; then echo -e "${YELLOW}no version bump required${NC}" echo -e "no semantic commits found since last release" @@ -386,6 +338,5 @@ fi echo -e "${BLUE}═══════════════════════════════════════════════════════════════${NC}\n" -# Preview completed successfully - exit 0 regardless of whether a version bump is pending. -# "No version bump required" is a valid outcome, not an error. +# "No version bump required" is a valid outcome, not an error: exit 0. exit 0 From 8948cb223e7967702f878fda10e9440fb57ce995 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 45/77] chore(apps/release): remove unused comments --- modules/apps/release/release.nix | 33 +++--- modules/apps/release/release.sh | 174 ++++++++----------------------- 2 files changed, 54 insertions(+), 153 deletions(-) diff --git a/modules/apps/release/release.nix b/modules/apps/release/release.nix index 172fef2d9..f76c46778 100644 --- a/modules/apps/release/release.nix +++ b/modules/apps/release/release.nix @@ -1,27 +1,23 @@ # release.nix - Production semantic-release wrapper as a flake app. # -# Usage: # nix run .#release -- # nix run .#release -- --dry-run # nix run .#release -- info # nix run .#release -- --help # -# Absorbs the `production-release-packages` job body from -# .github/workflows/package-release.yaml: configures git, invokes -# semantic-release against the target monorepo package, filters -# `@semantic-release/github` out of the plugin list when `--dry-run` -# is set (so GITHUB_TOKEN is not required for previews), and provides -# a `info` subcommand that emits release info (version, tag, released) -# as JSON. +# Configures git, invokes semantic-release against the target monorepo +# package, filters `@semantic-release/github` out of the plugin list when +# `--dry-run` is set (so GITHUB_TOKEN is not required for previews), and +# provides an `info` subcommand emitting release info as JSON. # # Hermetic: semantic-release and all plugins are provided by the # vanixiets-docs-deps derivation and linked into the package directory at # runtime. Callers do not need to run `bun install`. # -# Expected caller environment (not loaded from sops; CI-only): -# GITHUB_TOKEN - required by @semantic-release/github for production releases -# SOPS_AGE_KEY - passthrough for semantic-release hooks that may decrypt -# secrets via sops (not consumed by this script directly) +# Expected caller environment (CI-only): +# GITHUB_TOKEN, CI, RELEASE_REPO_ROOT, +# GIT_AUTHOR_NAME / GIT_AUTHOR_EMAIL, +# GIT_COMMITTER_NAME / GIT_COMMITTER_EMAIL. # # Template bifurcation (writeShellApplication): PURE READFILE FORM. # `text = builtins.readFile ./release.sh` — the sidecar is consumed verbatim, @@ -52,15 +48,12 @@ pkgs.jq pkgs.gnugrep pkgs.coreutils - # Hardening per m4-release-packages-runtime-deps-contract: - # explicitly declare every host-PATH binary that release.sh - # OR any transitive semantic-release plugin / node_modules - # helper might shell out to. The buildbot-effects bwrap - # sandbox provides only /nix/store ro-bind + writeShellApplication + # Explicitly declare every host-PATH binary that release.sh OR + # any transitive semantic-release plugin / node_modules helper + # might shell out to. The buildbot-effects bwrap sandbox + # provides only /nix/store ro-bind + writeShellApplication # runtimeInputs PATH (no host PATH binaries); a missing input - # surfaces only at runtime as `command not found`. Symmetric - # to the deploy-docs runtimeInputs hardening done in the - # m4-deploy-docs-git-env-contract feature. + # surfaces only at runtime as `command not found`. pkgs.gnused pkgs.gawk pkgs.findutils diff --git a/modules/apps/release/release.sh b/modules/apps/release/release.sh index d883b80cd..75e95909c 100644 --- a/modules/apps/release/release.sh +++ b/modules/apps/release/release.sh @@ -1,122 +1,40 @@ #!/usr/bin/env bash # shellcheck shell=bash # release.sh - Production semantic-release runner for a monorepo package. +# See `usage()` for caller-facing usage; this header documents the env-var +# contract only. # -# Usage: -# release [--dry-run] [-- extra semantic-release args] -# release info [] -# release --help -# -# Subcommands: -# (default) Run semantic-release against . Tag/publish on -# success; when --dry-run is passed, runs a preview without -# the @semantic-release/github plugin so GITHUB_TOKEN is not -# required. -# info Emit a JSON object describing the most recent release for -# (or the repo root when omitted). Fields: -# { "version": "X.Y.Z", "tag": "pkg-vX.Y.Z", "released": true } -# On no prior release: { "version": "unknown", "tag": "", -# "released": false }. -# -# Flags: -# --dry-run Pass --dry-run and --no-ci to semantic-release and strip the -# @semantic-release/github plugin from the invocation plugin -# list. Mirrors the preview-version trick (no GITHUB_TOKEN -# needed). -# --help Print this usage and exit 0. -# -# Env-var contract (per ADR-002 / env-var-contract-design.md §2.3, extended -# by the m4-release-packages-runtime-deps-contract feature to ALL host-PATH -# binary dependencies and to .git-write avoidance — symmetric to the -# m4-deploy-docs-git-env-contract precedent on deploy.sh). -# -# Required (secret, production path only — not --dry-run): -# GITHUB_TOKEN @semantic-release/github auth for tag push and -# release publish. Filtered-out plugin list under -# --dry-run means no token is consulted in that mode. -# Required (config, injected by release.nix runtimeEnv): -# DOCS_NODE_MODULES vanixiets-docs-deps node_modules tree hosting -# node_modules/.bin/semantic-release. -# -# Optional (CI-mode signalling; required by env-ci on the effect path): -# CI "true" tells semantic-release / env-ci that the -# run is non-interactive CI. Required when running -# in the buildbot-effects bwrap sandbox: the -# sandbox is not a recognised CI provider, so -# semantic-release would otherwise abort with -# `running on a CI environment is required` (env-ci -# default). Set by the effect preamble; unset on -# local-shell invocations where semantic-release's -# --no-ci flag (under --dry-run) bypasses the check. -# -# Optional (repo-root resolution; env-first with shelled-fallback): -# RELEASE_REPO_ROOT absolute path to the working tree's repo root. -# Required when running inside the buildbot-effects -# bwrap sandbox: the sandbox does not bind-mount -# the working tree, so `git rev-parse -# --show-toplevel` would fail with `fatal: not a -# git repository` and abort the script. Effect -# preamble sets to "$PWD" (mkEffect cwd is the -# pristine source root). Fallback chain: -# git rev-parse --show-toplevel 2>/dev/null || pwd -# — error-tolerant so a missing .git does not -# cause non-zero exit. -# -# Optional (git identity; env-first, NO .git/config writes): -# GIT_AUTHOR_NAME semantic-release commit author name -# (semantic-release writes a CHANGELOG commit -# on the production path). git honours these -# env vars natively without writing to -# .git/config — required because the bwrap -# sandbox mounts /nix/store ro-bind, and -# `git config user.email "…"` would fail with -# `error: could not lock config file .git/config` -# when the working tree's .git is unavailable. -# Default (effect preamble): semantic-release. -# GIT_AUTHOR_EMAIL semantic-release commit author email. -# Default (effect preamble): -# semantic-release@vanixiets.local -# GIT_COMMITTER_NAME semantic-release commit committer name. -# Default (effect preamble): semantic-release -# GIT_COMMITTER_EMAIL semantic-release commit committer email. -# Default (effect preamble): -# semantic-release@vanixiets.local -# GIT_USER_NAME transitional alias — when GIT_AUTHOR_NAME and -# GIT_COMMITTER_NAME are unset, this value is -# used to seed both. Retained for callers that -# have not yet migrated to the GIT_AUTHOR_*/ -# GIT_COMMITTER_* convention. -# GIT_USER_EMAIL transitional alias for GIT_AUTHOR_EMAIL + -# GIT_COMMITTER_EMAIL. -# -# Optional (passthrough, not consumed): -# SOPS_AGE_KEY reserved passthrough for sops-decrypt hooks -# (no consumer in the current tree; declared but -# NOT enforced via :? guard — see ADR-002, which -# REJECTS SOPS_AGE_KEY as a general pattern). -# -# Caller mechanisms: -# - Local dev dry-run: `nix run .#release -- packages/ --dry-run` -# needs no secret env (plugin filter strips github); -# git fallback resolves repo-root and identity -# from the local worktree's .git. -# - Local dev prod: caller-side sops wrapper (decrypt -# secrets/shared.yaml before the nix run) OR -# direnv dotenv (.envrc `dotenv` + .env) -# - GHA env: step `env:` block populates GITHUB_TOKEN from -# the repo secrets (package-release.yaml); -# checkout action provides the .git working tree -# so RELEASE_REPO_ROOT / GIT_AUTHOR_* fallbacks -# are exercised. -# - M4 effect: release-packages effect preamble extracts -# GITHUB_TOKEN from HERCULES_CI_SECRETS_JSON and -# exports it alongside RELEASE_REPO_ROOT="$PWD", -# CI=true, GIT_BRANCH=, and the -# GIT_AUTHOR_*/GIT_COMMITTER_* identity quartet -# before invoking the app program path. -# -# Secret passing rule (per ADR-002): NO secrets are passed as CLI flags. -# Authentication flows exclusively through the inherited environment. +# Required (secret, production path only — not --dry-run): +# GITHUB_TOKEN @semantic-release/github auth for tag push and +# release publish. Filtered-out plugin list under +# --dry-run means no token is consulted in that mode. +# Required (config, injected by release.nix runtimeEnv): +# DOCS_NODE_MODULES vanixiets-docs-deps node_modules tree hosting +# node_modules/.bin/semantic-release. +# Optional (CI-mode signalling; required by env-ci on the effect path): +# CI "true" tells semantic-release / env-ci that the +# run is non-interactive CI. Required in the +# buildbot-effects bwrap sandbox (not a recognised +# CI provider; semantic-release would otherwise +# abort `running on a CI environment is required`). +# Optional (repo-root resolution; env-first with errexit-tolerant fallback): +# RELEASE_REPO_ROOT absolute path to the working tree's repo root. +# Required in the bwrap sandbox (no .git bind-mount; +# `git rev-parse --show-toplevel` would fail). +# Fallback: git rev-parse --show-toplevel || pwd. +# Optional (git identity; env-first, NO .git/config writes — bwrap mounts +# /nix/store ro-bind, so `git config user.email …` would fail to lock +# .git/config). git honours these natively without any config write. +# Defaults applied by the effect preamble: +# GIT_AUTHOR_NAME / GIT_AUTHOR_EMAIL (semantic-release@vanixiets.local) +# GIT_COMMITTER_NAME / GIT_COMMITTER_EMAIL (semantic-release@vanixiets.local) +# GIT_USER_NAME / GIT_USER_EMAIL — transitional aliases that seed the +# quartet when the GIT_AUTHOR_* / GIT_COMMITTER_* +# forms are unset. +# Optional (passthrough, not consumed): +# SOPS_AGE_KEY reserved passthrough for sops-decrypt hooks; no +# consumer in the current tree (declared but NOT +# enforced via :? guard). set -euo pipefail @@ -176,8 +94,6 @@ emit_release_info() { fi } -# Handle top-level dispatch: --help, info subcommand, or fall through -# to the default "run semantic-release" mode. if [ $# -eq 0 ]; then usage >&2 exit 2 @@ -195,9 +111,6 @@ case "$1" in ;; esac -# Default mode: semantic-release runner. -# Parse positional arg + --dry-run flag; forward remaining args through to -# node ./node_modules/.bin/semantic-release. dry_run=0 package_path="" extra_args=() @@ -270,11 +183,9 @@ export GIT_COMMITTER_EMAIL="${GIT_COMMITTER_EMAIL:-${GIT_USER_EMAIL:-semantic-re cd "$package_path" -# Production-path env-var contract guard: fail fast on missing GITHUB_TOKEN -# BEFORE any node_modules / workspace mutation so the error points at the -# contract rather than at an opaque state-mutation side effect. Gated on -# dry_run so the dry-run path (with @semantic-release/github filtered out) -# continues to work without any secret env. +# Production-path contract guard: fail fast on missing GITHUB_TOKEN +# BEFORE any node_modules mutation so the error points at the contract +# rather than at an opaque state-mutation side effect. Gated on dry_run. if [ "$dry_run" -ne 1 ]; then : "${GITHUB_TOKEN:?GITHUB_TOKEN is required for production semantic-release (see release.sh header for caller mechanisms; not needed for --dry-run)}" fi @@ -320,10 +231,9 @@ fi if [ "$dry_run" -eq 1 ]; then # Filter @semantic-release/github so GITHUB_TOKEN is not required for - # a preview. Mirrors the plugin list used by preview-version.sh plus - # the changelog + major-tag plugins that the package.json "release" - # block declares (still safe under --dry-run: prepare/publish steps - # are no-ops in dry-run mode). + # preview; safe under --dry-run (prepare/publish steps are no-ops). + # Mirrors preview-version.sh plus changelog + major-tag plugins from + # the package.json "release" block. plugins="@semantic-release/commit-analyzer,@semantic-release/release-notes-generator,@semantic-release/changelog,semantic-release-major-tag" echo "running semantic-release (dry-run, no GitHub plugin) in ${package_path}..." node ./node_modules/.bin/semantic-release \ @@ -332,10 +242,8 @@ if [ "$dry_run" -eq 1 ]; then --plugins "$plugins" \ "${extra_args[@]}" else - # Production release path: semantic-release will create a tag and - # publish a GitHub release when invoked. GITHUB_TOKEN is enforced via - # the early :? guard above (placed before node_modules setup so failure - # modes are contract-first). + # GITHUB_TOKEN is enforced via the early :? guard above (placed before + # node_modules setup so failure modes are contract-first). echo "running production semantic-release in ${package_path}..." node ./node_modules/.bin/semantic-release "${extra_args[@]}" fi From 24522837ccb2a43da42774dcb8225d1631349782 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 46/77] chore(scripts): remove unused comments --- scripts/preview-version.sh | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/scripts/preview-version.sh b/scripts/preview-version.sh index 842ed874a..6d0c86d44 100755 --- a/scripts/preview-version.sh +++ b/scripts/preview-version.sh @@ -1,16 +1,12 @@ #!/usr/bin/env bash # preview-version.sh - Thin shim over the preview-version flake app. # -# The authoritative implementation lives at modules/apps/preview-version/ (via -# modules/apps/docs/preview-version.{nix,sh}), invoked through the flake app -# `.#preview-version`. This shim is retained so out-of-tree callers that still -# reference `./scripts/preview-version.sh` (notably `package.json:18`) keep -# working. -# -# Usage: -# ./scripts/preview-version.sh [target-branch] [package-path] -# -# Forwards all arguments to `nix run .#preview-version --`. +# The authoritative implementation lives at +# modules/apps/docs/preview-version.{nix,sh} and is invoked through the +# flake app `.#preview-version`. This shim is retained so out-of-tree +# callers that still reference `./scripts/preview-version.sh` (notably +# `package.json:18`) keep working. Run `nix run .#preview-version -- --help` +# for usage and arguments. set -euo pipefail From 9750d2595d5544478341b820a40f1f6571cd93f8 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 47/77] chore(effects/vanixiets): remove unused comments --- .../vanixiets/herculesCI/deploy-docs.nix | 130 +----- .../vanixiets/herculesCI/release-packages.nix | 429 +----------------- modules/effects/vanixiets/secrets.nix | 91 +--- 3 files changed, 24 insertions(+), 626 deletions(-) diff --git a/modules/effects/vanixiets/herculesCI/deploy-docs.nix b/modules/effects/vanixiets/herculesCI/deploy-docs.nix index 1f0e6e2e8..6ad423b7e 100644 --- a/modules/effects/vanixiets/herculesCI/deploy-docs.nix +++ b/modules/effects/vanixiets/herculesCI/deploy-docs.nix @@ -1,64 +1,4 @@ -# effects.deploy-docs — docs deployment branch-dispatcher (M4 feature -# `m4-deploy-docs`). Consolidates three pre-consolidation jobs -# (preview-docs-deploy, production-docs-deploy-dryrun, -# production-docs-deploy-cutover) into a single herculesCI effect that -# dispatches on `primaryRepo.branch`. -# -# Design contract (see mission AGENTS.md "ADR-002 locked decisions" and -# `.factory/validation-contract.md` VAL-EFFECT-DEPLOYDOCS-*): -# -# Option Gamma store-path embedding: -# The effect body invokes the `deploy-docs` flake app via the -# nix-eval-time resolved store path -# `${config.apps.x86_64-linux.deploy-docs.program}`. The effect -# never dispatches via a flake-app shell-out (bwrap does not bind -# the working tree, so the .# syntax cannot resolve). -# -# Branch dispatch (exact string equality): -# Selection is driven by `primaryRepo.branch == "main"`, surfaced -# through `herculesCI.config.repo.branch` which hercules-ci-effects -# populates from the primaryRepo record at flake.herculesCI entry. -# * primaryRepo.branch == "main" → production promote path -# * any other branch (or null) → preview upload path -# -# Pattern C'-refined secrets preamble: -# Extracts CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID from -# $HERCULES_CI_SECRETS_JSON at the `..data.value` envelope -# (see modules/effects/vanixiets/secrets.nix for the generator that -# emits this shape). Never extracts SOPS_AGE_KEY (ADR-002 -# exclusivity) and never references NPM_TOKEN. -# -# Symmetric env-var contract (GIT_* + DEPLOY_*) (m4-deploy-docs-git-env-contract): -# Exports GIT_REV, GIT_REV_SHORT, GIT_REV_SHORT12, GIT_BRANCH, -# GIT_COMMIT_MSG, GIT_WORKTREE_STATUS from herculesCI.config.repo.* -# at eval time via lib.escapeShellArg + builtins.substring AND exports -# DEPLOY_DEPLOYER=hercules-ci-effects and DEPLOY_HOST=magnetite (the -# execution host on which buildbot-master schedules effect runs). -# Required because the buildbot-effects bwrap sandbox does not -# bind-mount the working tree AND provides no host-PATH binaries -# beyond /nix/store ro-bind + writeShellApplication runtimeInputs PATH -# (upstream-by-design across all 3 reference effect implementations); -# deploy.sh's git invocations would fail with `fatal: not a git -# repository` and its `hostname -s` / `whoami` invocations would fail -# with `command not found` (exit 127) without these exports. Symmetric -# to the secrets env-var contract: the script declares env-first / -# bash-builtin / shelled-fallback on every consumer, and this preamble -# supplies the authoritative values for the sandboxed path. -# -# Posture A outer gate: -# Fork-PR exposure is blocked by `effects_on_pull_requests = false` -# + `effects_branches` in `buildbot-nix.toml`, which precedes this -# inner branch-dispatch. The dispatcher logic runs only after the -# outer gate has passed. -# -# Structured banners (DD-16 log-grep anchors): -# * `DEPLOY-DOCS-ACTION: preview-upload|promote|fresh-deploy-and-promote` -# emitted exactly once per run, identifying the dispatcher path -# taken. `fresh-deploy-and-promote` is emitted post-hoc on the -# main path only when deploy.sh reports the fallback branch. -# * `DEPLOY-DOCS-PREVIEW-URL: ` emitted on the preview path -# once the wrangler-produced preview URL is parsed from deploy.sh -# stdout, enabling downstream `curl` verification of 200 OK. +# herculesCI effect: docs deployment branch-dispatcher (preview vs promote). { config, inputs, @@ -70,16 +10,11 @@ herculesCI = herculesCI: let - # primaryRepo.branch (exposed as config.repo.branch by - # hercules-ci-effects' paramModule, populated from primaryRepo.branch - # at flake.herculesCI entry). Type: nullable string — null on tag - # pushes where branch is not populated. + # Nullable: null on tag pushes (no branch). branch = herculesCI.config.repo.branch; shortRev = herculesCI.config.repo.shortRev; rev = herculesCI.config.repo.rev; - # Branch dispatch: exact string equality. null == "main" is false - # in Nix, so tag pushes naturally fall through to the preview path. isMain = branch == "main"; in { @@ -88,18 +23,10 @@ let hci-effects = inputs.hercules-ci-effects.lib.withPkgs pkgs; - # Option Gamma: resolved at nix eval time to a /nix/store path - # that the bwrap sandbox can execute without a working-tree or - # nix-daemon lookup. deployDocsProgram = config.apps.deploy-docs.program; - # Initial action banner — refined post-hoc on the main path if - # deploy.sh's fresh-deploy-and-promote fallback triggers. actionBanner = if isMain then "promote" else "preview-upload"; - # Preview branch argument: prefer the live branch name; fall - # back to the shortRev on detached / null-branch pushes so - # deploy.sh preview has a non-empty argument. previewBranchArg = if branch != null && branch != "" then branch else shortRev; in hci-effects.mkEffect { @@ -114,26 +41,11 @@ echo "shortRev: ${lib.escapeShellArg (toString shortRev)}" echo "isMain: ${if isMain then "true" else "false"}" - # Structured banner (DD-16): emitted once per run so log-grep - # can distinguish preview-upload | promote | fresh-deploy-and-promote. echo "DEPLOY-DOCS-ACTION: ${actionBanner}" - # Secrets preamble — Pattern C'-refined (ADR-002): - # extract CLOUDFLARE_API_TOKEN from $HERCULES_CI_SECRETS_JSON at .data.value envelope. export CLOUDFLARE_API_TOKEN="$(jq -r '.CLOUDFLARE_API_TOKEN.data.value' "$HERCULES_CI_SECRETS_JSON")" - # Secrets preamble — Pattern C'-refined (ADR-002): - # extract CLOUDFLARE_ACCOUNT_ID from $HERCULES_CI_SECRETS_JSON at .data.value envelope. export CLOUDFLARE_ACCOUNT_ID="$(jq -r '.CLOUDFLARE_ACCOUNT_ID.data.value' "$HERCULES_CI_SECRETS_JSON")" - # Git-metadata env-var contract (m4-deploy-docs-git-env-contract): - # interpolate the six GIT_* values from herculesCI.config.repo.* at - # eval time so deploy.sh's env-first / git-fallback consumers never - # need to shell out to `git` inside the bwrap sandbox. GIT_REV_SHORT12 - # is computed from the full rev via builtins.substring (not a runtime - # `git rev-parse --short=12`, which would fail on a missing .git). - # GIT_WORKTREE_STATUS is hard-coded "clean" because an effect run - # always dispatches from a committed revision (hercules-ci-effects - # fetches a pristine checkout). export GIT_REV=${lib.escapeShellArg (toString rev)} export GIT_REV_SHORT=${lib.escapeShellArg (toString shortRev)} export GIT_REV_SHORT12=${lib.escapeShellArg (builtins.substring 0 12 (toString rev))} @@ -141,21 +53,10 @@ export GIT_COMMIT_MSG=${lib.escapeShellArg "effect deploy from rev ${toString shortRev}"} export GIT_WORKTREE_STATUS=clean - # Deploy-context env-var contract (m4-deploy-docs-git-env-contract): - # supply DEPLOY_DEPLOYER and DEPLOY_HOST so deploy.sh's bash-builtin / - # shelled-fallback chain never has to invoke `whoami` / `hostname` — - # neither binary is on PATH inside the bwrap sandbox (only - # /nix/store ro-bind + runtimeInputs). Hard-coded values are - # appropriate here because every effect run is dispatched by the - # hercules-ci-effects framework on magnetite (the buildbot-nix - # worker host); divergent values would indicate a misconfigured - # effect runner, not a per-run difference worth surfacing. + # Why: whoami/hostname not on bwrap PATH; supply hard-coded values. export DEPLOY_DEPLOYER=hercules-ci-effects export DEPLOY_HOST=magnetite - # Env-var-contract guard: fail fast if the secrets bundle is - # missing either Cloudflare key. Message excludes the value; - # only key name is echoed (VAL-EFFECT-DEPLOYDOCS-21). if [ -z "''${CLOUDFLARE_API_TOKEN:-}" ] || [ "$CLOUDFLARE_API_TOKEN" = "null" ]; then echo "error: CLOUDFLARE_API_TOKEN missing from \$HERCULES_CI_SECRETS_JSON" >&2 exit 1 @@ -165,26 +66,13 @@ exit 1 fi - # Option Gamma store-path dispatch — the `deploy-docs` flake - # app's /nix/store path is embedded at eval time via - # the perSystem config.apps.deploy-docs.program attribute. - # No flake-app shell-out (bwrap would not resolve .#). + # Why: bwrap sandbox does not bind working tree; .# cannot resolve. Use eval-time /nix/store path. DEPLOY_DOCS=${deployDocsProgram} ${ if isMain then '' - # Main branch → promote-by-SHA path. - # deploy.sh's `production` subcommand looks up a Worker - # version whose workers/tag annotation matches the - # current commit short-SHA-12 (uploaded earlier on the - # pre-merge branch push). If found, it promotes via - # `wrangler versions deploy @100%` (no re-upload → - # VAL-EFFECT-DEPLOYDOCS-16). If absent, it falls back - # to a fresh deploy + promote, logged with the literal - # substring "falling back to direct deploy". The - # dispatcher re-emits a DEPLOY-DOCS-ACTION banner to - # disambiguate the two execution paths for log-grep. + # release.sh's production subcommand re-emits "falling back to direct deploy" on the fresh-deploy fallback; the dispatcher grep below depends on that exact substring. deploy_log="$(mktemp -t deploy-docs-prod.XXXXXX.log)" set +e "$DEPLOY_DOCS" production 2>&1 | tee "$deploy_log" @@ -200,14 +88,6 @@ '' else '' - # Non-main → preview upload path. - # deploy.sh's `preview ` subcommand uploads a - # new Cloudflare Workers version tagged with the - # commit short-SHA-12, aliased at - # b--infra-docs.sciexp.workers.dev. - # The script emits a `Preview URL:` line on success - # which we parse + re-emit as a structured banner - # (DEPLOY-DOCS-PREVIEW-URL) for downstream 200-probes. preview_log="$(mktemp -t deploy-docs-preview.XXXXXX.log)" set +e "$DEPLOY_DOCS" preview ${lib.escapeShellArg previewBranchArg} 2>&1 | tee "$preview_log" diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index c81eab856..869ccb721 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -1,235 +1,4 @@ -# effects.release-packages — semantic-release per-package dispatcher -# (M4 feature `m4-release-packages`). Consolidates three pre-consolidation -# jobs (preview-release-version, production-release-packages-dryrun, -# production-release-packages-cutover) into a single herculesCI effect -# that iterates every package discovered under `packages/*` and dispatches -# `semantic-release` with branch-aware behaviour. -# -# Design contract (see mission AGENTS.md "ADR-002 locked decisions" and -# `.factory/validation-contract.md` VAL-EFFECT-RELEASEPACKAGES-*): -# -# Option Gamma store-path embedding: -# The effect body invokes both the `list-packages-json` and `release` -# flake apps via the nix-eval-time resolved store paths -# `${config.apps.x86_64-linux.list-packages-json.program}` and -# `${config.apps.x86_64-linux.release.program}`. The effect never -# dispatches via a flake-app shell-out (`nix run`); bwrap does not -# bind the working tree, so the .# syntax cannot resolve. -# -# Branch dispatch (exact string equality; m5-01e Option C delegation): -# Selection is driven by `primaryRepo.branch == "main"`, surfaced -# through `herculesCI.config.repo.branch` which hercules-ci-effects -# populates from the primaryRepo record at flake.herculesCI entry. -# * primaryRepo.branch == "main" → real semantic-release per package -# via the `${releaseProgram}` flake app (release.sh production -# path; semantic-release's per-package commit-analyzer decides -# whether to cut a release; tag push + GitHub Release published -# when so; npmPublish stays false — never overridden). -# * any other branch (or null) → per-package merge-preview via -# the `${previewVersionProgram}` flake app (preview-version.sh; -# m5-01e delegation to the existing flake app, Option C, closes -# the m5-01c Phase 1 version-preview gap). preview-version.sh -# simulates merging the current branch into `main` via -# `git merge-tree --write-tree` + temporary worktree, then runs -# semantic-release with `--branches "$TARGET_BRANCH"` and the -# commit-analyzer + release-notes-generator plugin pair only -# (no `@semantic-release/github`, no tag push, no GitHub -# Release, no remote git mutation). The previous non-main path -# delegated to `release.sh --dry-run`, which short-circuited on -# the in-tree `branches: ["main"]` config before exercising -# analyzeCommits/generateNotes; preview-version's -# `--branches` override is what makes the version-preview path -# actually run for cd-via-effects and other non-main branches. -# -# Pattern C'-refined secrets preamble: -# Extracts GITHUB_TOKEN ONLY from $HERCULES_CI_SECRETS_JSON at the -# `.GITHUB_TOKEN.data.value` envelope (see -# modules/effects/vanixiets/secrets.nix for the generator that emits -# this shape). Per ADR-002 §5.3 exclusivity audit, this effect MUST -# NOT consume SOPS_AGE_KEY (load-bearing only for k3d-bootstrap-secrets) -# and MUST NOT reference NPM_TOKEN (not part of the closed 4-key -# bundle; npmPublish=false invariant means no npm publish surface). -# -# ADR-003 Option α clone preamble (m5-01a-release-packages-clone-internally-implement): -# The buildbot-effects bwrap sandbox does NOT bind-mount the worker's -# git checkout (`mkEffect` only seeds `HOME=/build/home`); the previous -# iteration of this effect failed with `fatal: not a git repository` -# because semantic-release ran against an empty $PWD. Per ADR-003 -# "release-packages clone-and-push" (Option α, locked 2026-04-24; -# mission-internal note kept under repo-local `.factory/library/`, -# git-ignored via `.git/info/exclude`), -# the preamble below performs an in-sandbox clone of -# the canonical GitHub URL, checks out the exact triggering rev, -# verifies branch-tip freshness, exports `GIT_CREDENTIALS` for -# semantic-release's auth-URL builder, and points the existing -# env-var contract's `RELEASE_REPO_ROOT` at the clone (not `$PWD`). -# -# Six normative phases (ADR-003 §Architecture): -# 1. URL canonicalization — literal `https://github.com/cameronraysmith/vanixiets.git` -# (NOT `herculesCI.config.repo.remoteHttpUrl`, which bakes the -# buildbot-nix GitHub App installation token at clone time). -# 2. Full clone (no shallow / since-date / blob-filter flags): -# semantic-release needs the full tag list, full commit log -# since `lastRelease.gitHead`, and per-commit `git diff-tree` -# changed-file lookups (semantic-release-monorepo path filter). -# 3. Exact-rev checkout via `git checkout -B "$GIT_BRANCH" "$GIT_REV"` — -# force-push and rapid-merge events become deterministic; the -# effect releases what buildbot-master's nix-eval ran against. -# 4. Single freshness check before the package loop (invariant #11): -# `git fetch origin "$GIT_BRANCH"` + `rev-parse HEAD == origin/$GIT_BRANCH`; -# stale runs emit `RELEASE-CLONE-STALE` and exit non-zero. -# 5. Sanitized structured banners — `RELEASE-CLONE-{START,CHECKOUT,READY,STALE}`. -# NEVER echo a token-baked URL (invariant #9); only the -# canonicalized GitHub URL appears in any banner. -# 6. `GIT_CREDENTIALS=x-access-token:${GITHUB_TOKEN}` export for -# semantic-release's `get-git-auth-url.js` URL builder -# (in-process; no host credential helper, no ~/.git-credentials -# write per invariants #5 / VAL-RELEASE-α-AUTH-003 / -004). -# -# Helper extraction (ADR-003 §D3): the inline preamble lands at ~35 -# bash lines; well under the ~60-line threshold for factoring into -# `modules/effects/lib/mkMutatingEffect.nix`. Future PR-creating -# mutating effects (security-update-pr, dep-bump) should prefer -# upstream `hci-effects.git-update`/`flakeUpdate`, NOT a local helper. -# -# Three-way branch-form handling (m5-01h-release-packages-pr-head-ref-pivot; -# supersedes m5-01g-release-packages-pr-merge-ref-handling, commit 4b66343fe): -# buildbot-nix dispatches `release-packages` with three structurally -# distinct `branch` shapes; the clone preamble distinguishes them -# eval-time and emits matched bash for each: -# -# (A) GitHub PR push event — `branch = "refs/pull//merge"` -# (the synthetic GitHub test-merge ref form). Detected -# eval-time via `builtins.match "^refs/pull/([0-9]+)/merge$"` -# (eval-time predicate kept identical to m5-01g; the dispatch -# input form has not changed — only how the effect resolves -# the SHA-of-record from it). PR-detection pattern modelled -# on buildbot-nix's `buildbot_nix/buildbot_nix/build_canceller.py:16` -# (`branch.startswith((\"refs/pull/\", \"refs/merge-requests/\"))`), -# narrowed to GitHub form here. -# -# m5-01h pivot rationale — GitHub's `refs/pull//merge` is a -# SYNTHETIC, EPHEMERAL, NON-STABLE test-merge commit. GitHub -# recomputes it whenever the base branch advances, the PR head -# is updated, or its internal merge-test scheduler fires. -# buildbot-nix snapshots the merge-SHA at nix-eval time (T0) and -# passes it as `--rev`, but by effect-runtime (T1) GitHub may -# have recomputed the merge under the same ref name. The m5-01g -# production log on PR #1858 captured `RELEASE-CLONE-STALE-PR: -# expected f750940... remote 030a3498...` — a true-positive -# staleness signal exposing that the merge-ref form is -# fundamentally the wrong unit of truth for this dispatch path. -# -# m5-01h fetches `+refs/pull//head:refs/remotes/origin/pr--head` -# instead. `refs/pull//head` is the developer-pushed PR -# source branch tip, stable until the next dev push, and is -# what fast-forward-merge dry-run analysis actually wants — -# preview-version.sh's `git merge-tree --branches main` -# simulation operates from a working tree against main, so -# giving it the PR-head working tree is exactly correct. The -# SHA actually checked out is resolved at runtime via -# `head_sha=$(git rev-parse origin/pr--head)` post-fetch; -# buildbot's `--rev` (the ephemeral merge SHA) is retained -# ONLY as a forensic record in the DISPATCH banner. -# -# Synthetic local branch `pr--head` (no slashes; unambiguous -# to `git rev-parse`) replaces the raw ref name for -# `git checkout -B`. Refspec form modelled on buildbot-nix's -# `buildbot_nix/buildbot_nix/nix_eval.py:GitLocalPrMerge.run` -# fetch idiom (the `+ref:remote-tracking-ref` mapping form), -# reused here for the /head ref instead of the /merge ref. -# -# Emits TWO banners BEFORE the standard `RELEASE-CLONE-START`: -# * canonical positional `RELEASE-CLONE-PR-HEAD: -# ` — the SHA actually checked out and analyzed. -# * forensic key=value `RELEASE-CLONE-PR-DISPATCH: -# buildbot-rev= head=` — informational -# record of buildbot's `--rev` (the ephemeral T0 merge SHA) -# alongside the runtime-resolved head SHA. Drift between -# the two values is normal and benign. -# m5-01g's `RELEASE-CLONE-PR-MERGE` and `RELEASE-CLONE-STALE-PR` -# banners are RETIRED (no apples-to-apples freshness comparison -# is meaningful for the /head form: the head SHA is fresh by -# construction post-fetch). A trivial head-existence sanity -# check `git rev-parse --verify origin/pr--head` runs after -# checkout — non-zero only on a rare force-push race that -# removes the head ref between fetch and verify, in which case -# set -e aborts the effect. Cadence-equivalent to invariant #11 -# (single freshness check), generalised to a sanity probe. -# -# Upstream gap (skipping per user direction): `buildbot-effects` -# CLI accepts only `--rev/--branch/--repo/--secrets` (cli.py:103 -# has `# TODO: support ref`), and the bwrap sandbox strips env -# to {IN_HERCULES_CI_EFFECT, HERCULES_CI_SECRETS_JSON, -# NIX_BUILD_TOP, TMPDIR, NIX_REMOTE} so we cannot smuggle -# GitHub env in. Hand-rolled refspec inside the effectScript -# is the only path; we own it. -# -# (B) regular branch push — `branch` non-empty, non-PR-ref. -# Pre-m5-01g flow unchanged: `checkout_branch=$GIT_BRANCH`, -# `git fetch origin $GIT_BRANCH`, -# `git rev-parse origin/$GIT_BRANCH` for freshness. -# -# (C) tag-push event — `branch = null` (hercules-ci-effects -# models tag checkouts this way). Pre-m5-01g flow unchanged: -# synthetic local branch `release-packages-detached`, -# freshness check skipped (no branch tip; dry-run gate -# ensures no production push). See ADR-003 §"Tag-push event -# handling". -# -# Local-CLI flows (`nix run .#preview-version`) are unaffected by -# this branching — they only ever run from a real working tree -# where `branch` is a regular ref. The PR-merge form arises only -# inside the buildbot-nix dispatch path. -# Full ADR-003 invariant audit (§1–§13) is preserved; m5-01h -# generalises invariant #3 (exact-rev source: `head_sha` runtime- -# resolved post-fetch is the new exact-rev for case A) and -# invariant #11 (freshness-check shape: head-existence sanity -# probe for case A) without altering the single-check cadence or -# any other invariant. -# -# Symmetric env-var contract (CI + GIT_AUTHOR/COMMITTER + RELEASE_REPO_ROOT) -# (m4-release-packages-runtime-deps-contract; extended in m5-01a): -# Exports CI=true, GIT_BRANCH (from herculesCI.config.repo.branch at -# eval time via lib.escapeShellArg), RELEASE_REPO_ROOT=$clone_dir -# (the in-sandbox clone created by the ADR-003 preamble; previously -# "$PWD", which referred to the empty mkEffect cwd that triggered -# the `fatal: not a git repository` failure mode), and the -# GIT_AUTHOR_NAME/EMAIL + GIT_COMMITTER_NAME/EMAIL identity quartet. -# Required because the buildbot-effects bwrap sandbox does not -# bind-mount the worker's checkout AND provides no host-PATH binaries -# beyond /nix/store ro-bind + writeShellApplication runtimeInputs -# PATH (upstream-by-design across all 3 reference effect -# implementations); release.sh's `git config user.{name,email} "…"` -# writes would fail with `error: could not lock config file -# .git/config`, and semantic-release would abort with `running on a -# CI environment is required` (env-ci default) without these -# exports. Symmetric to the deploy-docs env-var contract (GIT_* + -# DEPLOY_*) precedent. -# -# Per-package atomicity (NOT fail-fast): -# If package A fails, the loop continues to package B. Each -# per-package failure is recorded; the dispatcher exits non-zero at -# the end iff ANY package failed. Successful per-package tags + -# GitHub Releases persist (semantic-release's own per-package -# mutations are atomic per-package); failures surface via the -# RELEASE-PACKAGE-FAILURE banners for manual re-trigger. -# -# Posture A outer gate: -# Fork-PR exposure is blocked by `effects_on_pull_requests = false` -# in `buildbot-nix.toml`, which precedes this inner branch-dispatch. -# The dispatcher logic runs only after the outer gate has passed. -# -# Structured banners (RP-05 / RP-06 / RP-19 log-grep anchors): -# * `RELEASE-PACKAGES-ACTION: dry-run|release` emitted exactly once -# per run, identifying the dispatcher path taken (analogous to the -# deploy-docs DEPLOY-DOCS-ACTION banner). -# * `RELEASE-PACKAGE-ITERATION: ` emitted before each -# per-package release invocation (count must equal -# `just list-packages-json | jq length`). -# * `RELEASE-PACKAGE-OK: ` emitted on per-package success. -# * `RELEASE-PACKAGE-FAILURE: (exit )` emitted on -# per-package failure; the loop continues regardless. +# herculesCI effect: semantic-release per-package dispatcher (dry-run vs release). { config, inputs, @@ -241,39 +10,20 @@ herculesCI = herculesCI: let - # primaryRepo.branch (exposed as config.repo.branch by - # hercules-ci-effects' paramModule, populated from primaryRepo.branch - # at flake.herculesCI entry). Type: nullable string — null on tag - # pushes where branch is not populated. + # Nullable: null on tag pushes (no branch). branch = herculesCI.config.repo.branch; shortRev = herculesCI.config.repo.shortRev; rev = herculesCI.config.repo.rev; - # Branch dispatch: exact string equality. null == "main" is false - # in Nix, so tag pushes naturally fall through to the dry-run path. isMain = branch == "main"; - # GitHub PR-merge ref detection (m5-01g). buildbot-nix dispatches - # `release-packages` on PR push events with `--branch` set to the - # synthetic GitHub test-merge ref form `refs/pull//merge` rather - # than the PR head branch name. This is the dominant non-main - # dispatch path in production. Pattern modelled on buildbot-nix's - # `build_canceller.py:16` PR-detection idiom (`branch.startswith - # ((\"refs/pull/\", \"refs/merge-requests/\"))`); narrowed to the - # GitHub form here because GitLab is not a vanixiets backend. - # `builtins.match` returns null on no-match and a list of capture - # groups on success, so `prMergeMatch != null` is the canonical - # eval-time predicate. + # builtins.match returns null on no-match, list of captures on success; null-guard keeps the eval pure for non-PR pushes. prMergeMatch = if branch == null then null else builtins.match "^refs/pull/([0-9]+)/merge$" branch; isPrMerge = prMergeMatch != null; prNumber = if isPrMerge then builtins.head prMergeMatch else null; - # Action banner emitted once per run (RP-05 log-grep anchor). actionBanner = if isMain then "release" else "dry-run"; - # Eval-time --dry-run flag injection. Empty on main, "--dry-run" - # otherwise. Ordered AFTER the package-path positional arg per - # release.sh's CLI grammar (`release [--dry-run]`). dryRunFlag = if isMain then "" else "--dry-run"; in { @@ -282,13 +32,7 @@ let hci-effects = inputs.hercules-ci-effects.lib.withPkgs pkgs; - # Option Gamma: resolved at nix eval time to /nix/store paths - # that the bwrap sandbox can execute without a working-tree or - # nix-daemon lookup. preview-version-program added in m5-01e - # (Option C) so non-main runs can delegate to the existing - # `preview-version` flake app rather than re-using - # `release.sh --dry-run` (which short-circuits on - # branches:["main"] before exercising the analyzeCommits path). + # release.sh --dry-run short-circuits on in-tree branches:["main"]; preview-version is the non-main path. listPackagesProgram = config.apps.list-packages-json.program; releaseProgram = config.apps.release.program; previewVersionProgram = config.apps.preview-version.program; @@ -296,17 +40,7 @@ hci-effects.mkEffect { name = "release-packages"; - # Runtime PATH inputs for the effectScript body (m5-01d-release-packages-runtimeinputs-fix). - # mkEffect's defaultInputs (cacert + curl + jq + effectSetupHook) plus stdenvNoCC's - # bundled coreutils/bash/gnused/gnugrep/gawk/gnutar cover every directly-invoked binary - # in the effectScript EXCEPT `git`, which the ADR-003 Option α clone preamble calls - # via `git clone`, `git fetch`, `git checkout -B`, and `git rev-parse`. The flake apps - # invoked downstream (`${listPackagesProgram}`, `${releaseProgram}`) carry their own - # writeShellApplication-baked runtimeInputs PATH so internal tool resolution is - # self-contained. Adding `pkgs.git` here closes the m5-01c Phase 1 dry-run regression - # where the bwrap sandbox emitted RELEASE-CLONE-START correctly and then failed with - # `git: command not found` from stdenv-linux/setup line 1842 (`git clone "$clone_url" …`). - # `cacert` is already in defaultInputs, so HTTPS clone CA-trust resolution is unaffected. + # Why: mkEffect's defaultInputs do not include git; clone preamble below requires it. inputs = [ pkgs.git ]; effectScript = '' @@ -318,127 +52,52 @@ echo "shortRev: ${lib.escapeShellArg (toString shortRev)}" echo "isMain: ${if isMain then "true" else "false"}" - # Structured banner (RP log-grep anchor): emitted once per run - # so log-grep can distinguish dry-run vs release dispatch path. echo "RELEASE-PACKAGES-ACTION: ${actionBanner}" - # Secrets preamble — Pattern C'-refined (ADR-002): - # extract GITHUB_TOKEN ONLY from $HERCULES_CI_SECRETS_JSON - # at the .data.value envelope. The other bundle keys are - # intentionally NOT extracted here — exclusivity rules and - # the npm-publish-never invariant are documented in the - # outer-module doc-comments (kept out of the rendered - # effectScript to satisfy VAL-EFFECT-RELEASEPACKAGES-22 / - # -24's `rg -c` zero-match contract on effectScript output). export GITHUB_TOKEN="$(jq -r '.GITHUB_TOKEN.data.value' "$HERCULES_CI_SECRETS_JSON")" - # Env-var-contract guard: fail fast if the secrets bundle is - # missing GITHUB_TOKEN. Message excludes the value; only key - # name is echoed. if [ -z "''${GITHUB_TOKEN:-}" ] || [ "$GITHUB_TOKEN" = "null" ]; then echo "error: GITHUB_TOKEN missing from \$HERCULES_CI_SECRETS_JSON" >&2 exit 1 fi - # === ADR-003 Option α clone preamble =========================== - # Phases 1–6 per ADR-003 §Architecture and the file-header - # doc-comment. All token-leak / freshness / authority invariants - # (#1, #2, #3, #5, #9, #11, #12) are codified here. - - # Canonicalized clone URL (ADR-003 §Architecture step 1, - # invariant #1). Hard-coded GitHub URL — NEVER derived from - # `herculesCI.config.repo.remoteHttpUrl`, which buildbot-nix - # populates with `https://git:@github.com/...` - # for GitHub-App-backed repos. Reusing that URL would bake the - # buildbot-nix App's `Contents: Read-only` token into local git - # config — exactly the wrong authority for a release mutation - # and a clear-text token-leak risk in any subsequent banner echo. + # Why: do not use config.repo.remoteHttpUrl — buildbot-nix bakes + # the App installation token into it; would leak via banner echo. clone_url="https://github.com/cameronraysmith/vanixiets.git" - # mkEffect's $TMPDIR is /tmp (tmpfs); mktemp here keeps the - # clone inside the bwrap-managed tmpfs which is reaped on - # sandbox exit regardless of the trap below. clone_dir="$(mktemp -d -t release-packages-clone.XXXXXX)" - # Single-step EXIT trap (invariant #12). Defensive hygiene only; - # the bwrap tmpfs is ephemeral. Do NOT add multi-step traps. trap 'rm -rf "$clone_dir"' EXIT - # Pre-compute git refs as nix-eval-time literals. Bash captures - # them as local vars so the freshness-check / checkout / banner - # phases share one canonicalized value without re-shelling-out. GIT_REV=${lib.escapeShellArg (toString rev)} GIT_BRANCH=${lib.escapeShellArg (if branch == null then "" else toString branch)} - # === Branch-form-aware clone + checkout + freshness/sanity === - # ADR-003 §Architecture steps 2-5 + invariants #1, #2, #3, #9, - # #10, #11. Three eval-time-distinguished cases dispatched by - # the eval-time conditional below; see the file-header Nix - # doc-comment for the full case-by-case rationale and the - # m5-01h design pivot. ${ if isPrMerge then '' - # m5-01h Case A: clone first so head_sha can be resolved - # before the canonical/forensic banners are emitted. git clone "$clone_url" "$clone_dir" git -C "$clone_dir" fetch --tags origin - # Custom-refspec head-fetch (modelled on buildbot-nix's - # `nix_eval.py:GitLocalPrMerge.run` idiom; the PR number - # is the eval-time-parsed literal). Materializes - # `refs/remotes/origin/pr-${toString prNumber}-head` so - # the subsequent `git rev-parse` resolves unambiguously. # `git fetch origin refs/pull//head` alone updates - # FETCH_HEAD but does NOT auto-create the remote- - # tracking ref; the explicit `+ref:remote-tracking-ref` - # mapping closes that gap. + # FETCH_HEAD but does NOT auto-create the remote-tracking + # ref; the explicit `+ref:remote-tracking-ref` mapping + # closes that gap. git -C "$clone_dir" fetch origin \ "+refs/pull/${toString prNumber}/head:refs/remotes/origin/pr-${toString prNumber}-head" head_sha="$(git -C "$clone_dir" rev-parse origin/pr-${toString prNumber}-head)" - # Two banners (m5-01h, VAL-RELEASE-α-PR-001) emitted - # BEFORE the standard `RELEASE-CLONE-START` line so - # log-grep can distinguish the PR-head dispatch path - # from regular-branch dispatch without further parsing. - # * canonical positional: SHA actually checked out - # and analyzed. - # * forensic key=value: buildbot's `--rev` (the - # ephemeral GitHub-computed merge SHA at eval-time - # T0) alongside the runtime-resolved head SHA. - # Drift between the two values is normal and benign - # (GitHub may have recomputed the synthetic merge - # between T0 and T1; the head SHA is the stable - # dev-pushed reference). echo "RELEASE-CLONE-PR-HEAD: ${toString prNumber} $head_sha" echo "RELEASE-CLONE-PR-DISPATCH: ${toString prNumber} buildbot-rev=$GIT_REV head=$head_sha" - # Standard upstream-input record (invariants #9, #10): - # sanitized public URL only — token never appears here - # even though GIT_CREDENTIALS is exported below for - # semantic-release's URL builder. echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" - # Exact-rev checkout (ADR-003 §Architecture step 3, - # invariant #3 — generalised: $head_sha is the new - # exact-rev source for case A; buildbot's $GIT_REV is - # the ephemeral merge SHA and is NOT a valid checkout - # target). Synthetic local branch `pr--head` (no - # slashes; unambiguous to git ref resolution). git -C "$clone_dir" checkout -B "pr-${toString prNumber}-head" "$head_sha" echo "RELEASE-CLONE-CHECKOUT: $head_sha" - # Head-existence sanity check (m5-01h; replaces m5-01g's - # STALE-PR failure mode). Trivially true post-fetch - # unless the head ref disappears (rare force-push race), - # in which case the non-zero exit propagates via set -e - # and aborts the effect. Cadence-equivalent to invariant - # #11 (single freshness check), generalised to a sanity - # probe for case A. + # Trivially true post-fetch unless force-push race lost the head ref; set -e propagates abort. git -C "$clone_dir" rev-parse --verify origin/pr-${toString prNumber}-head >/dev/null '' else '' - # Cases B and C: unchanged from m5-01g (pre-m5-01h state). echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" git clone "$clone_url" "$clone_dir" @@ -466,32 +125,10 @@ echo "RELEASE-CLONE-READY: $clone_dir" - # Token authentication (ADR-003 §Architecture step 6, - # invariants #4, #5). semantic-release's get-git-auth-url.js - # treats GIT_CREDENTIALS as a pre-baked `user:password` pair - # and constructs the authenticated URL in-process — NO host - # credential helper installation, NO write to ~/.git- - # credentials. The PAT held in `vanixiets-effects-secrets` - # (Contents: Read+Write) is the canonical authority; the - # buildbot-nix GitHub App installation token (Contents: - # Read-only) is intentionally NOT reused for release - # mutation. The shell variable is only consumed by - # semantic-release's URL builder; it is never echoed. + # semantic-release's get-git-auth-url.js treats GIT_CREDENTIALS as user:password and constructs the authenticated URL in-process. The vanixiets-effects-secrets PAT (Read+Write) is the canonical authority — the buildbot-nix App installation token (Read-only) is NOT reused for release mutation. export GIT_CREDENTIALS="x-access-token:''${GITHUB_TOKEN}" - # === existing m4-release-packages env-var contract ============= - # (extended for m5-01a: RELEASE_REPO_ROOT now points at the - # in-sandbox clone, not the empty mkEffect $PWD that previously - # caused `fatal: not a git repository`). - # - # CI is set so env-ci recognises the run as non-interactive CI, - # bypassing semantic-release's `running on a CI environment is - # required` abort. GIT_BRANCH is the eval-time literal value - # already captured above (re-exported for child processes). - # GIT_AUTHOR_*/GIT_COMMITTER_* are hard-coded identities for the - # semantic-release CHANGELOG-prepare phase; git honours these - # env vars natively without writing to .git/config (which the - # bwrap /nix/store ro-bind would block anyway). + # CI=true bypasses semantic-release's env-ci abort. GIT_AUTHOR/COMMITTER are honoured natively without writing .git/config (which the bwrap /nix/store ro-bind would block). export CI=true export GIT_BRANCH export RELEASE_REPO_ROOT="$clone_dir" @@ -500,53 +137,26 @@ export GIT_COMMITTER_NAME=semantic-release export GIT_COMMITTER_EMAIL=semantic-release@vanixiets.local - # Option Gamma store-path dispatch — all three flake apps' - # /nix/store paths are embedded at eval time via the perSystem - # config.apps..program attributes. No flake-app shell-out - # (bwrap would not resolve .#). PREVIEW is unused on the main - # branch path and RELEASE is unused on the non-main branch - # path — both are exported unconditionally for log auditability - # so an operator inspecting the rendered effectScript sees the - # full set of /nix/store paths the effect was built against. + # Why: bwrap sandbox does not bind working tree; .# cannot resolve. Use eval-time /nix/store paths. LIST_PACKAGES=${listPackagesProgram} RELEASE=${releaseProgram} PREVIEW=${previewVersionProgram} - # cd into the clone before invoking list-packages-json: that - # script calls `git rev-parse --show-toplevel`, which must - # resolve to $clone_dir (the only real git tree in this - # sandbox). release.sh's own RELEASE_REPO_ROOT consumer also - # picks up the same clone via the env export above. + # list-packages-json calls `git rev-parse --show-toplevel` + # which must resolve to $clone_dir (the only real git tree). cd "$clone_dir" - # Discover packages under packages/* (jq-driven enumeration of - # the list-packages-json output). packages_json="$("$LIST_PACKAGES")" echo "packages discovered: $packages_json" - # Per-package failure tracker — populated inside the loop; - # used at end-of-loop to set the aggregate exit code. failed_packages=() - # Per-package atomicity loop: iterate every {name, path} - # entry. Read paths into the loop via process substitution - # over `jq -r '.[].path'` (one path per line, robust to - # paths-with-spaces unlike `for pkg in $(...)`). while IFS= read -r pkg_path; do [ -z "$pkg_path" ] && continue echo "RELEASE-PACKAGE-ITERATION: $pkg_path" - # Disable -e for the per-package invocation so a single - # package's failure does not abort the loop. We capture - # the exit code, log appropriately, and continue. - # m5-01e Option C: eval-time branch the dispatch line so - # the rendered bash invokes either the production - # release.sh path (main) or the merge-preview - # preview-version.sh path (non-main). The CLI grammars - # differ — `release [--dry-run]` vs - # `preview-version [target-branch] [package-path]` — so a - # single shared variable + shared flag would not work. + # CLI grammars differ — release [--dry-run] vs preview-version [target-branch] [pkg-path] — so a single shared dispatch line cannot work. set +e ${if isMain then ''"$RELEASE" "$pkg_path"'' else ''"$PREVIEW" main "$pkg_path"''} rc=$? @@ -560,11 +170,6 @@ fi done < <(printf '%s\n' "$packages_json" | jq -r '.[].path') - # Aggregate exit code: OR of all per-package results. Zero - # iff every package's invocation exited zero (or zero - # packages were discovered, which is itself a degenerate - # success). Non-zero iff any package failed; the failed - # set is enumerated in stderr for operator triage. if [ "''${#failed_packages[@]}" -gt 0 ]; then echo "error: ''${#failed_packages[@]} package(s) failed: ''${failed_packages[*]}" >&2 exit 1 diff --git a/modules/effects/vanixiets/secrets.nix b/modules/effects/vanixiets/secrets.nix index 8726f815f..baf5aa6bf 100644 --- a/modules/effects/vanixiets/secrets.nix +++ b/modules/effects/vanixiets/secrets.nix @@ -1,46 +1,4 @@ -# Pattern C'-refined (mic92 idiom) — per-repo effects-secrets generator. -# -# This file owns every clan-vars and buildbot-nix wire for the -# github:cameronraysmith/vanixiets repo's effects-secrets bundle. -# Consumer-repo effect declarations (preview-docs-deploy, -# production-docs-deploy, etc.) live in each consumer's own flake under -# `herculesCI.effects.*`; vanixiets owns only the secret-material side -# of the contract. -# -# When integrated into the live tree at modules/effects/vanixiets/secrets.nix: -# - The one-line wire -# services.buildbot-nix.master.effects.perRepoSecretFiles."github:cameronraysmith/vanixiets" = …; -# currently in modules/nixos/buildbot.nix MUST be removed; it is -# authoritative here. NixOS module-system semantics allow additive -# extension of `services.buildbot-nix.master.effects.perRepoSecretFiles` -# from this module without duplicating `services.buildbot-nix.master.enable`. -# - magnetite's host module (modules/machines/nixos/magnetite/default.nix) -# MUST add `effects-vanixiets-secrets` to its `with flakeModules; [ … ]` -# list so the flake-parts deferred module is included in magnetite's -# NixOS configuration. -# -# Operator runbook (routine): -# clan vars generate --regenerate \ -# --generator vanixiets-effects-secrets magnetite -# # Walks the four prompts (cloudflare-api-token, cloudflare-account-id, -# # github-token, sops-age-key); "Enter to keep, Backspace for new" per field. -# # Composed `secrets` file is re-encrypted and git-committed automatically. -# -# Operator runbook (escape hatch — single-token non-interactive rotation): -# printf '%s' "$TOK" | clan vars set magnetite \ -# vanixiets-effects-secrets/github-token -# clan vars generate --regenerate \ -# --generator vanixiets-effects-secrets magnetite -# -# Reference: ADR-002 (Pattern C'-refined) and its amendment-A; reference -# implementation is -# ~/projects/nix-workspace/mic92-clan-dotfiles/machines/eve/modules/buildbot.nix -# (`harmonia-effects-secrets`). -# -# This file contributes a flake-parts deferred NixOS module named -# `effects-vanixiets-secrets`, following the outer-lambda shape of -# modules/nixos/buildbot.nix so that import-tree can auto-discover it and -# magnetite can `imports = with flakeModules; [ … effects-vanixiets-secrets ];`. +# Per-repo effects-secrets generator for github:cameronraysmith/vanixiets. { config, inputs, @@ -55,34 +13,12 @@ ... }: { - # Per-repo effects-secrets generator for github:cameronraysmith/vanixiets. - # - # Composes the four operator-sourced tokens - # (CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, GITHUB_TOKEN, - # SOPS_AGE_KEY) into a single `secrets` file shaped as - # hercules-ci-effects-nested JSON: - # - # { - # "": { "data": { "value": "" } }, - # … - # } - # - # consumed by buildbot-nix at dispatch time as - # HERCULES_CI_SECRETS_JSON inside the bwrap sandbox. The keys below - # match the secret identifiers referenced by effect scripts in - # modules/effects/vanixiets/herculesCI/*.nix. clan.core.vars.generators.vanixiets-effects-secrets = { - # The composed JSON file deployed to magnetite and pointed at by - # services.buildbot-nix.master.effects.perRepoSecretFiles below. - # `secret = true` is the default (see clan-core/modules/clan/vars/settings-opts.nix:30-37); - # stated explicitly to mirror mic92's harmonia-effects-secrets idiom. files.secrets = { secret = true; owner = "buildbot"; }; - # --- Prompts (interactive capture, persisted for rotation UX) --- - prompts.cloudflare-api-token = { description = '' Cloudflare API token (scope: Workers/Pages:Edit + relevant zone/R2 scopes). @@ -156,20 +92,12 @@ }; }; - # Raw prompt files are auto-materialized by `persist = true` so - # their encrypted-at-rest copy lives in the repo, enabling - # per-token rotation via the "Enter to keep" UX. They must NOT be - # deployed to magnetite: only the composed `secrets` file needs to - # reach the machine's secret store at activation time. - # (Reference: clanServices/admin/root-password.nix:17-20 uses the - # same idiom for `files.password.deploy = false`.) + # Raw prompts kept in repo (encrypted-at-rest) for the "Enter to keep" rotation UX, but only the composed secrets file deploys to magnetite. files.cloudflare-api-token.deploy = false; files.cloudflare-account-id.deploy = false; files.github-token.deploy = false; files.sops-age-key.deploy = false; - # --- Composition script (mic92 harmonia-effects-secrets style) --- - runtimeInputs = [ pkgs.jq ]; script = '' @@ -187,21 +115,6 @@ ''; }; - # Wire the composed `secrets` file to buildbot-nix's per-repo - # effects-secret map. The attribute `services.buildbot-nix.master` - # is declared as an option by - # inputs.buildbot-nix.nixosModules.buildbot-master (imported by - # modules/machines/nixos/magnetite/default.nix); the NixOS module - # system merges this additive attribute with the rest of the master - # config in modules/nixos/buildbot.nix. In particular we do NOT - # redeclare master.enable, workersFile, github.*, accessMode.*, or - # any other authoritative option here — only the perRepoSecretFiles - # attribute keyed on this repo's forge identifier. - # - # This module is the sole authoritative definition of the - # `github:cameronraysmith/vanixiets` perRepoSecretFiles entry; - # m4-01c retired the legacy inline wire in modules/nixos/buildbot.nix, - # so no transitional `lib.mkDefault` priority marker is required. services.buildbot-nix.master.effects.perRepoSecretFiles."github:cameronraysmith/vanixiets" = config.clan.core.vars.generators.vanixiets-effects-secrets.files.secrets.path; }; From 6fc1c0fee3ca7ffe36ee2fe537309e89d6d6060c Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 48/77] chore(hercules-ci): remove unused comments --- modules/hercules-ci.nix | 63 ++++++++++++++--------------------------- 1 file changed, 21 insertions(+), 42 deletions(-) diff --git a/modules/hercules-ci.nix b/modules/hercules-ci.nix index 32628da9f..270d08ab9 100644 --- a/modules/hercules-ci.nix +++ b/modules/hercules-ci.nix @@ -1,25 +1,18 @@ -# Flake-level effects framework scaffolding (M2 — mission ADR-001). +# Flake-level effects framework scaffolding. # -# Imports the `hercules-ci-effects` flake-parts module so that the -# top-level flake output `herculesCI` is wired to the schema consumed -# by buildbot-nix (`flake.outputs.herculesCI(args).onPush.default.outputs.effects`, -# per `buildbot-nix/buildbot_effects/buildbot_effects/__init__.py:142-159`). +# Imports the `hercules-ci-effects` flake-parts module so the top-level +# `herculesCI` output is wired to the schema consumed by buildbot-nix at +# `flake.outputs.herculesCI(args).onPush.default.outputs.effects` (per +# `buildbot-nix/buildbot_effects/buildbot_effects/__init__.py:142-159`). # # Per-job effects land under this same `onPush.default.outputs.effects.` -# path (M3 smoke, M4 per-job cutover). Branch gating is expressed in -# `buildbot-nix.toml` (`effects_branches`, `effects_on_pull_requests`), -# not in the Nix attribute path. -# -# See `docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md` -# for the full rationale and the fixed-attribute-path contract. +# path. Branch gating is expressed in `buildbot-nix.toml` +# (`effects_branches`, `effects_on_pull_requests`), not the Nix attr path. { inputs, lib, ... }: let # Effects execute on x86_64-linux (magnetite's buildbot-worker arch). pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux; - # `lib.withPkgs` returns the hercules-ci-effects helper set - # (mkEffect, runIf, modularEffect, ...). See - # hercules-ci-effects/flake-public-outputs.nix `lib.withPkgs`. hci-effects = inputs.hercules-ci-effects.lib.withPkgs pkgs; in { @@ -31,38 +24,24 @@ in { config, ... }: { onPush.default.outputs.effects = { - # effects.smoke — minimal diagnostic effect (M3 feature `m3-deploy-smoke`). - # - # Purpose: exercise the full buildbot-nix + hercules-ci-effects - # pipeline end-to-end (flake eval → nix-eval builder discovery → - # `run-effect` builder scheduling → bwrap execution → - # `HERCULES_CI_SECRETS_JSON` read → masked key enumeration → exit 0). - # Establishes a reusable diagnostic baseline for M4 per-job effects. + # effects.smoke — minimal diagnostic effect exercising the full + # buildbot-nix + hercules-ci-effects pipeline end-to-end as a + # reusable diagnostic baseline. # # Security invariants: - # - Prints only secret KEY NAMES, never VALUES (uses - # `jq -r 'to_entries | map(.key) | @csv'`). + # - Prints only secret KEY NAMES, never VALUES. # - Default-branch-only via `effects_branches = ["main"]` + # `effects_on_pull_requests = false` (Posture A) in # `buildbot-nix.toml`. No fork-PR or feature-branch exposure. - # - # Verification: see VAL-PROVISIONING-SMOKE-00{1..9} in - # `.factory/mission/validation-contract.md`. smoke = hci-effects.mkEffect { name = "smoke"; - # buildbot-effects populates these from the push metadata. - # `toString` coerces a null tag to the empty string so Nix string - # interpolation does not throw during eval of the effect derivation. - # - # `config.repo.ref` is intentionally not referenced: buildbot-effects - # hard-codes `"ref": None` in its JSON payload (see - # buildbot_effects/__init__.py:108, `# TODO: support ref`), and + # buildbot-effects populates branch/rev/shortRev/tag from push + # metadata; `toString` coerces a null tag to "" so interpolation + # does not throw. `config.repo.ref` is intentionally NOT + # referenced: buildbot-effects hard-codes `"ref": None` while # hercules-ci-effects declares `repo.ref` as non-nullable - # `types.str` (herculesCI-attribute.nix:18), so reading it would - # fail module type-checking at eval time. Upstream's own mkEffect - # example in buildbot-nix/nix/herculesCI/flake-module.nix follows - # the same pattern (branch/tag/rev only). + # `types.str`, so reading it would fail module type-checking. effectScript = let branch = toString (config.repo.branch or ""); @@ -81,10 +60,10 @@ in echo "shortRev: ${lib.escapeShellArg shortRev}" echo "tag: ${lib.escapeShellArg tag}" - # HERCULES_CI_SECRETS_JSON is set by buildbot-nix inside the - # bwrap sandbox to the path of the JSON secrets blob produced - # by the `perRepoSecretFiles` pipeline. See - # buildbot-nix/buildbot_effects/buildbot_effects/__init__.py:250-290. + # HERCULES_CI_SECRETS_JSON is set by buildbot-nix inside + # the bwrap sandbox to the path of the JSON secrets blob + # produced by the `perRepoSecretFiles` pipeline (see + # buildbot_effects/__init__.py:250-290). echo "HERCULES_CI_SECRETS_JSON=''${HERCULES_CI_SECRETS_JSON:-}" if [ -n "''${HERCULES_CI_SECRETS_JSON:-}" ] \ @@ -93,7 +72,7 @@ in # Key-only enumeration. VALUES ARE INTENTIONALLY OMITTED. # Do not change this to `jq -r 'to_entries[] | .value'` # or equivalent — that would leak secret payloads to the - # buildbot log (VAL-PROVISIONING-SMOKE-006). + # buildbot log. echo -n "secret keys: " jq -r 'to_entries | map(.key) | @csv' \ "''${HERCULES_CI_SECRETS_JSON}" From 23c0330d844042db4b833f0c85f30664a422e927 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 49/77] chore(checks): remove unused comments --- modules/checks/nix-unit.nix | 35 ++++------------------------------- 1 file changed, 4 insertions(+), 31 deletions(-) diff --git a/modules/checks/nix-unit.nix b/modules/checks/nix-unit.nix index afe696ce0..c7195d03e 100644 --- a/modules/checks/nix-unit.nix +++ b/modules/checks/nix-unit.nix @@ -34,10 +34,7 @@ }; nix-unit.tests = { - # Metadata Test - # TC-001: Flake Structure Smoke Test - # Validates packages have required metadata (if packages exist) testMetadataFlakeOutputsExist = { expr = (builtins.hasAttr "nixosConfigurations" self) @@ -46,10 +43,7 @@ expected = true; }; - # Regression Tests - # TC-002: Terraform Module Exports Exist - # Validates that terranix module exports exist in the flake namespace testRegressionTerraformModulesExist = { expr = (builtins.hasAttr "base" self.modules.terranix) @@ -58,8 +52,7 @@ }; # TC-003: NixOS Closure Equivalence - # Validates that machine configs exist and can be referenced - # Note: Full config evaluation requires network access, so we just test existence + # Full config evaluation requires network access, so we just test existence. testRegressionNixosConfigExists = { expr = builtins.hasAttr "electrum" self.nixosConfigurations @@ -67,10 +60,7 @@ expected = true; }; - # Invariant Tests - # TC-004: Clan Inventory Structure - # Validates inventory has required fields testInvariantClanInventoryMachines = { expr = builtins.sort builtins.lessThan (builtins.attrNames self.clan.inventory.machines); expected = [ @@ -87,7 +77,6 @@ }; # TC-005: NixOS Configs Exist - # Validates all expected configs present testInvariantNixosConfigurationsExist = { expr = builtins.sort builtins.lessThan (builtins.attrNames self.nixosConfigurations); expected = [ @@ -100,7 +89,6 @@ }; # TC-006: Darwin Configs Exist - # Validates darwin configurations are created testInvariantDarwinConfigurationsExist = { expr = builtins.sort builtins.lessThan (builtins.attrNames self.darwinConfigurations); expected = [ @@ -112,7 +100,6 @@ }; # TC-007: Home Configs Exist - # Validates standalone home configurations are created testInvariantHomeConfigurationsExist = { expr = builtins.sort builtins.lessThan (builtins.attrNames self.homeConfigurations.x86_64-linux); expected = [ @@ -121,10 +108,7 @@ ]; }; - # Feature Tests - # TC-008: Module Discovery - # Validates import-tree discovers nixos modules testFeatureModuleDiscovery = { expr = (builtins.hasAttr "base" self.modules.nixos) @@ -133,7 +117,6 @@ }; # TC-009: Darwin Module Discovery - # Validates import-tree discovers darwin modules testFeatureDarwinModuleDiscovery = { expr = (builtins.hasAttr "base" self.modules.darwin) @@ -143,8 +126,7 @@ }; # TC-010: Namespace Exports - # Validates modules export to correct namespaces as valid module definitions - # NixOS module system accepts both attrsets and functions as modules + # NixOS module system accepts both attrsets and functions as modules. testFeatureNamespaceExports = { expr = let @@ -154,18 +136,14 @@ expected = true; }; - # Type-Safety Tests - # TC-011: SpecialArgs Propagation - # Validates inputs available in all machines via specialArgs testTypeSafetySpecialargsPropagation = { expr = builtins.hasAttr "inputs" self.clan.specialArgs; expected = true; }; # TC-012: Required NixOS Options - # Validates all configs have config attribute - # full option evaluation requires network access + # Full option evaluation requires network access, so we just test existence. testTypeSafetyNixosConfigStructure = { expr = builtins.all (name: builtins.hasAttr "config" self.nixosConfigurations.${name}) ( builtins.attrNames self.nixosConfigurations @@ -173,10 +151,8 @@ expected = true; }; - # Architectural Invariant Tests - # TC-013: Namespace Merging - # Validates files in same module directory auto-merge into single namespace + # Files in the same module directory auto-merge into a single namespace. testInvariantNamespaceMerging = { expr = (builtins.hasAttr "ai" self.modules.homeManager) @@ -186,7 +162,6 @@ }; # TC-014: Clan Module Integration - # Validates clan machines have corresponding flake module exports testInvariantClanModuleIntegration = { expr = let @@ -210,7 +185,6 @@ }; # TC-015: Import-Tree Completeness - # Validates import-tree discovers key modules from each namespace testFeatureImportTreeCompleteness = { expr = (builtins.hasAttr "base" self.modules.darwin) @@ -221,7 +195,6 @@ }; # TC-016: Crossplatform Home Modules - # Validates home-manager aggregates available for both darwin and linux contexts testInvariantCrossplatformHomeModules = { expr = (builtins.hasAttr "x86_64-linux" self.homeConfigurations) From d604f0fa2bddf3ab16a4c2a5aa3774929fb2eeb5 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 50/77] chore(devshells): remove unused comments --- modules/devshells/default.nix | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/modules/devshells/default.nix b/modules/devshells/default.nix index d4c516730..58ea0ac79 100644 --- a/modules/devshells/default.nix +++ b/modules/devshells/default.nix @@ -60,7 +60,7 @@ pkgs.bun inputs'.bun2nix.packages.default pkgs.nodejs_24 # semantic-release >= 24.10.0 - pkgs.fuc # (rm/cp)z + pkgs.fuc pkgs.rip2 # Language detection pkgs.github-linguist @@ -69,8 +69,7 @@ pkgs.svgo ] # buildbot-effects CLI for local dispatch of hercules-ci-effects - # (see buildbot-nix/docs/EFFECTS.md). Linux-only: the package - # depends on bwrap and is gated at buildbot-nix/packages/flake-module.nix:29. + # (see buildbot-nix/docs/EFFECTS.md). Linux-only: depends on bwrap. ++ lib.optionals pkgs.stdenv.isLinux [ inputs'.buildbot-nix.packages.buildbot-effects ]; From 3ada058f312314a84aaa7ca229f5e9d36c55c6be Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 51/77] chore(machines/magnetite): remove unused comments --- modules/machines/nixos/magnetite/default.nix | 28 ++++---------------- modules/machines/nixos/magnetite/disko.nix | 3 --- 2 files changed, 5 insertions(+), 26 deletions(-) diff --git a/modules/machines/nixos/magnetite/default.nix b/modules/machines/nixos/magnetite/default.nix index e3e41e442..7c1a926c2 100644 --- a/modules/machines/nixos/magnetite/default.nix +++ b/modules/machines/nixos/magnetite/default.nix @@ -4,12 +4,10 @@ ... }: let - # Capture outer config for use in imports flakeModules = config.flake.modules.nixos; flakeModulesHome = config.flake.modules.homeManager; in { - # Export host module to flake namespace flake.modules.nixos."machines/nixos/magnetite" = { config, @@ -42,49 +40,36 @@ in # Make flake available to all modules (required by ssh-known-hosts) _module.args.flake = inputs.self; - # System platform nixpkgs.hostPlatform = "x86_64-linux"; - # Allow unfree packages for nixosConfigurations (clan CLI path) - # perSystem.legacyPackages only affects clanInternals.machines (nom build path) + # Required for clan CLI path; perSystem.legacyPackages only affects the nom build path. nixpkgs.config.allowUnfree = true; - # Use flake.overlays.default (drupol pattern) - # All 5 overlay layers + pkgs-by-name packages exported from modules/nixpkgs.nix + # Overlays exported from modules/nixpkgs.nix (drupol pattern). nixpkgs.overlays = [ inputs.self.overlays.default ]; # ZFS device node path - more stable for cloud VMs boot.zfs.devNodes = "/dev/disk/by-path"; - # Disko disk configuration extracted to disko.nix - # Auto-merged via import-tree - # Bootloader: GRUB BIOS mode (CX53 has legacy BIOS only, not UEFI) # srvos hardware-hetzner-cloud handles GRUB BIOS configuration - # Hostname configuration networking.hostName = "magnetite"; networking.search = [ ]; - # Override state version for new deployment system.stateVersion = "25.05"; - # User configuration managed via clan inventory users service - # See: modules/clan/inventory/services/users/cameron.nix + # User configuration managed via clan inventory users service (modules/clan/inventory/services/users/cameron.nix). - # Allow wheel group sudo without password security.sudo.wheelNeedsPassword = false; - # ACME TLS certificate configuration for public-facing services security.acme = { acceptTerms = true; defaults.email = "cameron@scientistexperience.net"; }; - # Networking configuration - # srvos hardware-hetzner-cloud sets useNetworkd=true and useDHCP=false - # Configure primary interface with DHCP + # srvos hardware-hetzner-cloud sets useNetworkd=true and useDHCP=false; configure primary interface explicitly. systemd.network.networks."10-uplink" = { matchConfig.Name = "en*"; networkConfig = { @@ -98,7 +83,6 @@ in # Firewall configuration: dual-zone (public + ZeroTier) networking.firewall = { enable = true; - # Public-facing ports only allowedTCPPorts = [ 22 80 @@ -110,7 +94,6 @@ in }; }; - # SSH daemon configuration # Increase MaxAuthTries to accommodate agent forwarding with many keys # Default is 6, but Bitwarden SSH agent may have 10+ keys loaded services.openssh.settings.MaxAuthTries = 20; @@ -125,8 +108,7 @@ in # Bridge NixOS-level sops to home-manager for user secret key delivery hm-sops-bridge.users.cameron.sopsIdentity = "crs58"; - # cameron home-manager module imports - # Infrastructure settings (useGlobalPkgs, extraSpecialArgs, etc.) provided by cameron inventory service + # cameron home-manager imports; infrastructure settings provided by the cameron inventory service. home-manager.users.cameron = { imports = [ flakeModulesHome."users/crs58" diff --git a/modules/machines/nixos/magnetite/disko.nix b/modules/machines/nixos/magnetite/disko.nix index e93af59f4..785841366 100644 --- a/modules/machines/nixos/magnetite/disko.nix +++ b/modules/machines/nixos/magnetite/disko.nix @@ -2,7 +2,6 @@ { ... }: { flake.modules.nixos."machines/nixos/magnetite" = { - # Disko disk configuration for BIOS boot disko.devices = { disk.main = { type = "disk"; @@ -15,7 +14,6 @@ size = "1M"; type = "EF02"; }; - # Boot partition for GRUB grub = { size = "1G"; content = { @@ -24,7 +22,6 @@ mountpoint = "/boot"; }; }; - # ZFS partition zfs = { size = "100%"; content = { From d7dc776470f1c66057af957a034c3a0358073831 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 19:24:13 -0400 Subject: [PATCH 52/77] chore(nixos): remove unused comments --- modules/nixos/buildbot.nix | 30 +++++++++--------------------- modules/nixos/docker.nix | 30 ++++++++++-------------------- 2 files changed, 19 insertions(+), 41 deletions(-) diff --git a/modules/nixos/buildbot.nix b/modules/nixos/buildbot.nix index 43b14cc63..9f8ffc540 100644 --- a/modules/nixos/buildbot.nix +++ b/modules/nixos/buildbot.nix @@ -1,22 +1,17 @@ -# buildbot-nix CI service for magnetite +# buildbot-nix CI service for magnetite. # -# Provides clan vars generators for buildbot credentials and configures -# the buildbot-nix master with GitHub and Gitea forge backends in fullyPrivate -# access mode (oauth2-proxy gates all UI access via GitHub OAuth). -# Generators define the credential slots; values are populated via: -# - buildbot-github-app-secret-key: manual `clan vars set` (PEM key from GitHub App) -# - buildbot-github-oauth-secret: manual `clan vars set` (OAuth client secret from GitHub App) +# Credential generator catalog (slots; values populated as marked): +# - buildbot-github-app-secret-key: manual `clan vars set` (GitHub App PEM key) +# - buildbot-github-oauth-secret: manual `clan vars set` (OAuth client secret) # - buildbot-github-webhook-secret: auto-generated # - buildbot-worker: auto-generated (worker password + workers.json) # - buildbot-oauth2-cookie-secret: auto-generated (oauth2-proxy cookie encryption) # - buildbot-http-basic-auth-password: auto-generated (oauth2-proxy to buildbot internal auth) -# The effects-secrets generator and its `perRepoSecretFiles` wire for -# github:cameronraysmith/vanixiets are authoritatively declared in -# modules/effects/vanixiets/secrets.nix (flake module -# `effects-vanixiets-secrets`, opted-in by magnetite's host module). -# Gitea-specific credentials are declared in gitea.nix: +# Gitea-specific credentials live in gitea.nix: # - buildbot-gitea-token: manual `clan vars set` (API token with write:repository, write:user) # - buildbot-gitea-webhook-secret: auto-generated +# Per-repo effects secrets for github:cameronraysmith/vanixiets are wired in +# modules/effects/vanixiets/secrets.nix (flake module `effects-vanixiets-secrets`). { config, inputs, @@ -53,7 +48,6 @@ ''; }; - # GitHub webhook secret (auto-generated) clan.core.vars.generators.buildbot-github-webhook-secret = { files."secret" = { owner = "buildbot"; @@ -75,7 +69,7 @@ ''; }; - # HTTP basic auth password for oauth2-proxy to buildbot internal communication (auto-generated) + # HTTP basic auth password for oauth2-proxy to buildbot internal communication. clan.core.vars.generators.buildbot-http-basic-auth-password = { files."secret" = { owner = "buildbot"; @@ -107,7 +101,6 @@ ''; }; - # Buildbot master with GitHub forge services.buildbot-nix.master = { enable = true; domain = "buildbot.scientistexperience.net"; @@ -154,15 +147,10 @@ topic = "build-with-buildbot"; }; - # Conservative eval sizing for CX53 (16 vCPU, 32 GB RAM) — current evalWorkerCount=4 × evalMaxMemorySize=2048MB = 8 GB peak, leaving ample headroom on 32 GB host; sizing not increased at this time - # 4 workers * 2048 MB = 8 GB max, leaving headroom for niks3 + PostgreSQL + nginx + # evalWorkerCount × evalMaxMemorySize = 8 GB peak; headroom for niks3 + PostgreSQL + nginx on 32 GB CX53. evalWorkerCount = 4; evalMaxMemorySize = 2048; - # Per-repo effects secrets for github:cameronraysmith/vanixiets are - # wired authoritatively in modules/effects/vanixiets/secrets.nix - # (flake module `effects-vanixiets-secrets`). - # niks3 binary cache integration (push built paths after successful builds) # Uses public URL to support future remote workers (e.g. cinnabar) niks3 = { diff --git a/modules/nixos/docker.nix b/modules/nixos/docker.nix index e6e5e8699..f1da3810a 100644 --- a/modules/nixos/docker.nix +++ b/modules/nixos/docker.nix @@ -1,19 +1,13 @@ -# Docker runtime for magnetite +# Docker runtime for magnetite, additive to the existing podman stack used by +# gitea-actions-runner. Required by the test-cluster effect, which drives k3d +# via ctlptl invoking the `docker` binary directly (no podman support). # -# Provisions real docker as a second container runtime alongside the existing -# podman stack used by gitea-actions-runner. Required because the -# test-cluster effect drives k3d via ctlptl (~/projects/sciops-workspace/ctlptl) -# which invokes the `docker` binary directly and has no production-quality -# podman support. +# Storage: docker's native ZFS storage driver (overlay2 does not layer cleanly +# on ZFS); requires /var/lib/docker to be its own ZFS dataset, declared as +# zroot/root/docker in modules/machines/nixos/magnetite/disko.nix. # -# Storage: docker's native ZFS storage driver is used (overlay2 does not -# layer cleanly on ZFS). This requires /var/lib/docker to be its own ZFS -# dataset; see modules/machines/nixos/magnetite/disko.nix for the -# zroot/root/docker dataset declaration. -# -# The buildbot-worker user is added to the docker group so effects running -# as that user can talk to /var/run/docker.sock (e.g. the forthcoming -# test-cluster effect invoking k3d via ctlptl). +# buildbot-worker joins the docker group so effects can reach the docker +# socket without sudo. { ... }: @@ -21,17 +15,13 @@ flake.modules.nixos.docker = { ... }: { - # Real docker daemon, additive to the existing podman stack. virtualisation.docker = { enable = true; - # Native ZFS storage driver; requires /var/lib/docker to be its own - # ZFS dataset (declared in disko.nix as zroot/root/docker). + # Native ZFS storage driver; requires /var/lib/docker to be its own ZFS dataset (disko.nix zroot/root/docker). storageDriver = "zfs"; }; - # Grant the buildbot-worker runtime user access to the docker socket - # so effects executed by the worker (e.g. test-cluster via k3d+ctlptl) - # can drive the docker daemon without sudo. + # Grant buildbot-worker docker socket access so effects can drive the daemon without sudo. users.users.buildbot-worker.extraGroups = [ "docker" ]; }; } From 57536191c2adb543838e9e9027fa04e3815eb132 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 22:46:38 -0400 Subject: [PATCH 53/77] feat(buildbot-logs): include triggered effect builds in log output --- modules/home/tools/commands/_dev-tools.nix | 65 ++++++++++++++-------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/modules/home/tools/commands/_dev-tools.nix b/modules/home/tools/commands/_dev-tools.nix index ef39e543b..976666492 100644 --- a/modules/home/tools/commands/_dev-tools.nix +++ b/modules/home/tools/commands/_dev-tools.nix @@ -347,35 +347,56 @@ api() { curl -fsS -u "buildbot:$PW" "$API/$1"; } + dump_build_steps_and_logs() { + local b="$1" n="$2" + local steps_json + echo "=== STEPS ===" + steps_json=$(api "builders/$b/builds/$n/steps") + echo "$steps_json" | jq -r '.steps[] + | "\(.number)\t\(.name)\t\(.state_string // "")\tstepid=\(.stepid)\thidden=\(.hidden // false)"' + echo + + echo "$steps_json" | jq -c '.steps[]' | while read -r step; do + number=$(echo "$step" | jq -r '.number') + name=$(echo "$step" | jq -r '.name') + stepid=$(echo "$step" | jq -r '.stepid') + hidden=$(echo "$step" | jq -r '.hidden // false') + if [ "$hidden" = "true" ] && [ "$INCLUDE_HIDDEN" != "1" ]; then continue; fi + logs_json=$(api "steps/$stepid/logs" || echo '{"logs":[]}') + echo "$logs_json" | jq -c '.logs[]?' | while read -r log; do + logid=$(echo "$log" | jq -r '.logid') + logname=$(echo "$log" | jq -r '.name') + num_lines=$(echo "$log" | jq -r '.num_lines // 0') + echo "=== STEP $number: $name / LOG: $logname ($num_lines lines) ===" + api "logs/$logid/raw" || echo "(log fetch failed)" + echo + done + done + } + echo "=== BUILD $BUILDER/$BUILD ===" - api "builders/$BUILDER/builds/$BUILD" | jq '.builds[0]' || { + parent_json=$(api "builders/$BUILDER/builds/$BUILD") || { echo "Error: build $BUILDER/$BUILD not found or API unreachable" >&2 exit 4 } + echo "$parent_json" | jq '.builds[0]' echo - echo "=== STEPS ===" - steps_json=$(api "builders/$BUILDER/builds/$BUILD/steps") - echo "$steps_json" | jq -r '.steps[] - | "\(.number)\t\(.name)\t\(.state_string // "")\tstepid=\(.stepid)\thidden=\(.hidden // false)"' - echo - - echo "$steps_json" | jq -c '.steps[]' | while read -r step; do - number=$(echo "$step" | jq -r '.number') - name=$(echo "$step" | jq -r '.name') - stepid=$(echo "$step" | jq -r '.stepid') - hidden=$(echo "$step" | jq -r '.hidden // false') - if [ "$hidden" = "true" ] && [ "$INCLUDE_HIDDEN" != "1" ]; then continue; fi - logs_json=$(api "steps/$stepid/logs" || echo '{"logs":[]}') - echo "$logs_json" | jq -c '.logs[]?' | while read -r log; do - logid=$(echo "$log" | jq -r '.logid') - logname=$(echo "$log" | jq -r '.name') - num_lines=$(echo "$log" | jq -r '.num_lines // 0') - echo "=== STEP $number: $name / LOG: $logname ($num_lines lines) ===" - api "logs/$logid/raw" || echo "(log fetch failed)" - echo + dump_build_steps_and_logs "$BUILDER" "$BUILD" + + parent_buildid=$(echo "$parent_json" | jq -r '.builds[0].buildid // empty') + if [ -n "$parent_buildid" ]; then + triggered_json=$(api "builds/$parent_buildid/triggered_builds" || echo '{"builds":[]}') + echo "$triggered_json" | jq -c '.builds[]?' | while read -r child; do + cb_builder=$(echo "$child" | jq -r '.builderid') + cb_number=$(echo "$child" | jq -r '.number') + cb_buildid=$(echo "$child" | jq -r '.buildid') + effect_name=$(api "builds/$cb_buildid/properties" \ + | jq -r '.properties[0]."virtual_builder_name"[0] // ""' 2>/dev/null || echo "") + echo "=== CHILD BUILD $cb_builder/$cb_number ($effect_name) ===" + dump_build_steps_and_logs "$cb_builder" "$cb_number" done - done + fi REMOTE_SCRIPT echo "Done." >&2 From 68bd2bc211a548aea7579e1e7f8ad6c3b8f3621b Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sat, 25 Apr 2026 22:56:35 -0400 Subject: [PATCH 54/77] refactor(buildbot-logs): split writeShellApplication body into sibling .sh --- modules/home/tools/commands/_dev-tools.nix | 138 +----------------- modules/home/tools/commands/buildbot-logs.sh | 146 +++++++++++++++++++ 2 files changed, 154 insertions(+), 130 deletions(-) create mode 100755 modules/home/tools/commands/buildbot-logs.sh diff --git a/modules/home/tools/commands/_dev-tools.nix b/modules/home/tools/commands/_dev-tools.nix index 976666492..df90248e7 100644 --- a/modules/home/tools/commands/_dev-tools.nix +++ b/modules/home/tools/commands/_dev-tools.nix @@ -268,138 +268,16 @@ # On Darwin, uses /usr/bin/ssh (Apple-signed, Keychain-integrated agent) # rather than nixpkgs openssh, matching the ntfy-send precedent for # reaching ZeroTier hosts from macOS. + # Template bifurcation (writeShellApplication): INTERPOLATION FORM. + # Body lives in the sibling ./buildbot-logs.sh (directly executable for + # local debugging). The `text` preamble injects BUILDBOT_SSH_BIN — the + # eval-time-resolved ssh path — so darwin uses /usr/bin/ssh while linux + # uses PATH ssh. Standalone invocation (./buildbot-logs.sh) falls back + # to `ssh` on PATH via the BUILDBOT_SSH_BIN default in the sidecar. buildbot-logs = { text = '' - case "''${1:-}" in - -h|--help) - cat <<'HELP' - Fetch all step logs for a buildbot-nix build from the magnetite CI host - - Usage: buildbot-logs BUILDER_ID BUILD_ID - - Retrieves every non-hidden step's log (stdio plus any named logs such as - Evaluation Warnings) from the buildbot-nix master via ssh to magnetite.zt - on the ZeroTier mesh, and concatenates results to stdout with clear - step/log section headers. Intended to be redirected to a local file for - offline search, mirroring the 'gh run download -> unzip -> grep' pattern - used for GitHub Actions logs: - - buildbot-logs 48 30 > logs/buildbot-48-30.log - rg "error:" logs/buildbot-48-30.log - - Arguments: - BUILDER_ID Numeric builder id (e.g. 48 for the nix-eval builder of - cameronraysmith/vanixiets) - BUILD_ID Build number within that builder - - Environment: - BUILDBOT_SSH_HOST Override ssh target (default: magnetite.zt). - Accepts user@host form, e.g. root@magnetite.zt. - Leave unset to rely on local ~/.ssh/config. - BUILDBOT_INCLUDE_HIDDEN Set to 1 to include steps marked hidden in - buildbot (default: skip hidden steps). - - Mapping a PR check row to BUILDER_ID/BUILD_ID: - gh pr checks --json name,link \ - | jq -r '.[] | select(.name=="buildbot/nix-build") | .link' - # URL shape: /#/builders//builds/ - # (Both buildbot/nix-build and buildbot/nix-eval share this URL — the - # parent nix-eval build contains both phases' logs as separate steps.) - - Privacy: captured logs may include build output, worker names, store - paths, and buildbot-masked token references (e.g. ). - Review before sharing publicly. - HELP - exit 0 - ;; - esac - - if [ "$#" -lt 2 ]; then - echo "Error: BUILDER_ID and BUILD_ID required" >&2 - echo "Try 'buildbot-logs --help' for more information." >&2 - exit 2 - fi - - builder="$1" - build="$2" - host="''${BUILDBOT_SSH_HOST:-magnetite.zt}" - include_hidden="''${BUILDBOT_INCLUDE_HIDDEN:-0}" - - case "$builder$build" in - *[!0-9]*|"") - echo "Error: BUILDER_ID and BUILD_ID must be positive integers" >&2 - exit 2 - ;; - esac - - echo "Fetching logs for build $builder/$build from $host..." >&2 - - ${if pkgs.stdenv.isDarwin then "/usr/bin/ssh" else "ssh"} -T "$host" \ - "BUILDER=$builder BUILD=$build INCLUDE_HIDDEN=$include_hidden bash -s" \ - <<'REMOTE_SCRIPT' - set -euo pipefail - - API=http://127.0.0.1:8010/api/v2 - PW=$(sudo -n bash -c 'cat /run/secrets.d/*/vars/buildbot-http-basic-auth-password/secret' 2>/dev/null) || { - echo "Error: failed to read buildbot http basic auth password on $(hostname)" >&2 - exit 3 - } - - api() { curl -fsS -u "buildbot:$PW" "$API/$1"; } - - dump_build_steps_and_logs() { - local b="$1" n="$2" - local steps_json - echo "=== STEPS ===" - steps_json=$(api "builders/$b/builds/$n/steps") - echo "$steps_json" | jq -r '.steps[] - | "\(.number)\t\(.name)\t\(.state_string // "")\tstepid=\(.stepid)\thidden=\(.hidden // false)"' - echo - - echo "$steps_json" | jq -c '.steps[]' | while read -r step; do - number=$(echo "$step" | jq -r '.number') - name=$(echo "$step" | jq -r '.name') - stepid=$(echo "$step" | jq -r '.stepid') - hidden=$(echo "$step" | jq -r '.hidden // false') - if [ "$hidden" = "true" ] && [ "$INCLUDE_HIDDEN" != "1" ]; then continue; fi - logs_json=$(api "steps/$stepid/logs" || echo '{"logs":[]}') - echo "$logs_json" | jq -c '.logs[]?' | while read -r log; do - logid=$(echo "$log" | jq -r '.logid') - logname=$(echo "$log" | jq -r '.name') - num_lines=$(echo "$log" | jq -r '.num_lines // 0') - echo "=== STEP $number: $name / LOG: $logname ($num_lines lines) ===" - api "logs/$logid/raw" || echo "(log fetch failed)" - echo - done - done - } - - echo "=== BUILD $BUILDER/$BUILD ===" - parent_json=$(api "builders/$BUILDER/builds/$BUILD") || { - echo "Error: build $BUILDER/$BUILD not found or API unreachable" >&2 - exit 4 - } - echo "$parent_json" | jq '.builds[0]' - echo - - dump_build_steps_and_logs "$BUILDER" "$BUILD" - - parent_buildid=$(echo "$parent_json" | jq -r '.builds[0].buildid // empty') - if [ -n "$parent_buildid" ]; then - triggered_json=$(api "builds/$parent_buildid/triggered_builds" || echo '{"builds":[]}') - echo "$triggered_json" | jq -c '.builds[]?' | while read -r child; do - cb_builder=$(echo "$child" | jq -r '.builderid') - cb_number=$(echo "$child" | jq -r '.number') - cb_buildid=$(echo "$child" | jq -r '.buildid') - effect_name=$(api "builds/$cb_buildid/properties" \ - | jq -r '.properties[0]."virtual_builder_name"[0] // ""' 2>/dev/null || echo "") - echo "=== CHILD BUILD $cb_builder/$cb_number ($effect_name) ===" - dump_build_steps_and_logs "$cb_builder" "$cb_number" - done - fi - REMOTE_SCRIPT - - echo "Done." >&2 + export BUILDBOT_SSH_BIN=${if pkgs.stdenv.isDarwin then "/usr/bin/ssh" else "ssh"} + ${builtins.readFile ./buildbot-logs.sh} ''; }; diff --git a/modules/home/tools/commands/buildbot-logs.sh b/modules/home/tools/commands/buildbot-logs.sh new file mode 100755 index 000000000..343e36ada --- /dev/null +++ b/modules/home/tools/commands/buildbot-logs.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# shellcheck shell=bash +# Fetch buildbot-nix step logs (and triggered effect builds) from magnetite. +# +# Wrapped form: invoked as `buildbot-logs ` after activation +# via writeShellApplication in modules/home/tools/commands/_dev-tools.nix. +# +# Direct execution: `./modules/home/tools/commands/buildbot-logs.sh 48 154` +# requires `ssh` on PATH; curl/jq/sudo run inside the SSH heredoc on +# magnetite, so they are not needed locally. +# +# The wrapper injects BUILDBOT_SSH_BIN (eval-time path to ssh) via the +# nix-string preamble; standalone invocation falls back to `ssh` on PATH. +set -euo pipefail + +case "${1:-}" in + -h|--help) + cat <<'HELP' +Fetch all step logs for a buildbot-nix build from the magnetite CI host + +Usage: buildbot-logs BUILDER_ID BUILD_ID + +Retrieves every non-hidden step's log (stdio plus any named logs such as +Evaluation Warnings) from the buildbot-nix master via ssh to magnetite.zt +on the ZeroTier mesh, and concatenates results to stdout with clear +step/log section headers. Intended to be redirected to a local file for +offline search, mirroring the 'gh run download -> unzip -> grep' pattern +used for GitHub Actions logs: + + buildbot-logs 48 30 > logs/buildbot-48-30.log + rg "error:" logs/buildbot-48-30.log + +Arguments: + BUILDER_ID Numeric builder id (e.g. 48 for the nix-eval builder of + cameronraysmith/vanixiets) + BUILD_ID Build number within that builder + +Environment: + BUILDBOT_SSH_HOST Override ssh target (default: magnetite.zt). + Accepts user@host form, e.g. root@magnetite.zt. + Leave unset to rely on local ~/.ssh/config. + BUILDBOT_INCLUDE_HIDDEN Set to 1 to include steps marked hidden in + buildbot (default: skip hidden steps). + +Mapping a PR check row to BUILDER_ID/BUILD_ID: + gh pr checks --json name,link \ + | jq -r '.[] | select(.name=="buildbot/nix-build") | .link' + # URL shape: /#/builders//builds/ + # (Both buildbot/nix-build and buildbot/nix-eval share this URL — the + # parent nix-eval build contains both phases' logs as separate steps.) + +Privacy: captured logs may include build output, worker names, store +paths, and buildbot-masked token references (e.g. ). +Review before sharing publicly. +HELP + exit 0 + ;; +esac + +if [ "$#" -lt 2 ]; then + echo "Error: BUILDER_ID and BUILD_ID required" >&2 + echo "Try 'buildbot-logs --help' for more information." >&2 + exit 2 +fi + +builder="$1" +build="$2" +host="${BUILDBOT_SSH_HOST:-magnetite.zt}" +include_hidden="${BUILDBOT_INCLUDE_HIDDEN:-0}" +ssh_bin="${BUILDBOT_SSH_BIN:-ssh}" + +case "$builder$build" in + *[!0-9]*|"") + echo "Error: BUILDER_ID and BUILD_ID must be positive integers" >&2 + exit 2 + ;; +esac + +echo "Fetching logs for build $builder/$build from $host..." >&2 + +"$ssh_bin" -T "$host" \ + "BUILDER=$builder BUILD=$build INCLUDE_HIDDEN=$include_hidden bash -s" \ +<<'REMOTE_SCRIPT' +set -euo pipefail + +API=http://127.0.0.1:8010/api/v2 +PW=$(sudo -n bash -c 'cat /run/secrets.d/*/vars/buildbot-http-basic-auth-password/secret' 2>/dev/null) || { + echo "Error: failed to read buildbot http basic auth password on $(hostname)" >&2 + exit 3 +} + +api() { curl -fsS -u "buildbot:$PW" "$API/$1"; } + +dump_build_steps_and_logs() { + local b="$1" n="$2" + local steps_json + echo "=== STEPS ===" + steps_json=$(api "builders/$b/builds/$n/steps") + echo "$steps_json" | jq -r '.steps[] + | "\(.number)\t\(.name)\t\(.state_string // "")\tstepid=\(.stepid)\thidden=\(.hidden // false)"' + echo + + echo "$steps_json" | jq -c '.steps[]' | while read -r step; do + number=$(echo "$step" | jq -r '.number') + name=$(echo "$step" | jq -r '.name') + stepid=$(echo "$step" | jq -r '.stepid') + hidden=$(echo "$step" | jq -r '.hidden // false') + if [ "$hidden" = "true" ] && [ "$INCLUDE_HIDDEN" != "1" ]; then continue; fi + logs_json=$(api "steps/$stepid/logs" || echo '{"logs":[]}') + echo "$logs_json" | jq -c '.logs[]?' | while read -r log; do + logid=$(echo "$log" | jq -r '.logid') + logname=$(echo "$log" | jq -r '.name') + num_lines=$(echo "$log" | jq -r '.num_lines // 0') + echo "=== STEP $number: $name / LOG: $logname ($num_lines lines) ===" + api "logs/$logid/raw" || echo "(log fetch failed)" + echo + done + done +} + +echo "=== BUILD $BUILDER/$BUILD ===" +parent_json=$(api "builders/$BUILDER/builds/$BUILD") || { + echo "Error: build $BUILDER/$BUILD not found or API unreachable" >&2 + exit 4 +} +echo "$parent_json" | jq '.builds[0]' +echo + +dump_build_steps_and_logs "$BUILDER" "$BUILD" + +parent_buildid=$(echo "$parent_json" | jq -r '.builds[0].buildid // empty') +if [ -n "$parent_buildid" ]; then + triggered_json=$(api "builds/$parent_buildid/triggered_builds" || echo '{"builds":[]}') + echo "$triggered_json" | jq -c '.builds[]?' | while read -r child; do + cb_builder=$(echo "$child" | jq -r '.builderid') + cb_number=$(echo "$child" | jq -r '.number') + cb_buildid=$(echo "$child" | jq -r '.buildid') + effect_name=$(api "builds/$cb_buildid/properties" \ + | jq -r '.properties[0]."virtual_builder_name"[0] // ""' 2>/dev/null || echo "") + echo "=== CHILD BUILD $cb_builder/$cb_number ($effect_name) ===" + dump_build_steps_and_logs "$cb_builder" "$cb_number" + done +fi +REMOTE_SCRIPT + +echo "Done." >&2 From 5d461edeecdbd35e029d15bc212a5d6188972b11 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 00:38:28 -0400 Subject: [PATCH 55/77] refactor(effects): lift flakeModule import to modules/effects/, drop smoke Smoke removed because production effects (deploy-docs, release-packages) already exercise the same buildbot-nix + hercules-ci-effects pipeline. flakeModule import lifted to repo-agnostic scope so future per-repo effect domains under modules/effects// inherit it automatically. --- modules/effects/flake-module.nix | 10 ++++ modules/hercules-ci.nix | 88 -------------------------------- 2 files changed, 10 insertions(+), 88 deletions(-) create mode 100644 modules/effects/flake-module.nix delete mode 100644 modules/hercules-ci.nix diff --git a/modules/effects/flake-module.nix b/modules/effects/flake-module.nix new file mode 100644 index 000000000..8205df8a5 --- /dev/null +++ b/modules/effects/flake-module.nix @@ -0,0 +1,10 @@ +# Lifts the hercules-ci-effects flake-parts module to repo-agnostic scope so +# per-repo effect domains under modules/effects//herculesCI/*.nix can +# merge into herculesCI.onPush.default.outputs.effects. without each +# domain re-importing the flakeModule. +{ inputs, ... }: +{ + imports = [ + inputs.hercules-ci-effects.flakeModule + ]; +} diff --git a/modules/hercules-ci.nix b/modules/hercules-ci.nix deleted file mode 100644 index 270d08ab9..000000000 --- a/modules/hercules-ci.nix +++ /dev/null @@ -1,88 +0,0 @@ -# Flake-level effects framework scaffolding. -# -# Imports the `hercules-ci-effects` flake-parts module so the top-level -# `herculesCI` output is wired to the schema consumed by buildbot-nix at -# `flake.outputs.herculesCI(args).onPush.default.outputs.effects` (per -# `buildbot-nix/buildbot_effects/buildbot_effects/__init__.py:142-159`). -# -# Per-job effects land under this same `onPush.default.outputs.effects.` -# path. Branch gating is expressed in `buildbot-nix.toml` -# (`effects_branches`, `effects_on_pull_requests`), not the Nix attr path. -{ inputs, lib, ... }: -let - # Effects execute on x86_64-linux (magnetite's buildbot-worker arch). - pkgs = inputs.nixpkgs.legacyPackages.x86_64-linux; - - hci-effects = inputs.hercules-ci-effects.lib.withPkgs pkgs; -in -{ - imports = [ - inputs.hercules-ci-effects.flakeModule - ]; - - herculesCI = - { config, ... }: - { - onPush.default.outputs.effects = { - # effects.smoke — minimal diagnostic effect exercising the full - # buildbot-nix + hercules-ci-effects pipeline end-to-end as a - # reusable diagnostic baseline. - # - # Security invariants: - # - Prints only secret KEY NAMES, never VALUES. - # - Default-branch-only via `effects_branches = ["main"]` + - # `effects_on_pull_requests = false` (Posture A) in - # `buildbot-nix.toml`. No fork-PR or feature-branch exposure. - smoke = hci-effects.mkEffect { - name = "smoke"; - - # buildbot-effects populates branch/rev/shortRev/tag from push - # metadata; `toString` coerces a null tag to "" so interpolation - # does not throw. `config.repo.ref` is intentionally NOT - # referenced: buildbot-effects hard-codes `"ref": None` while - # hercules-ci-effects declares `repo.ref` as non-nullable - # `types.str`, so reading it would fail module type-checking. - effectScript = - let - branch = toString (config.repo.branch or ""); - rev = toString (config.repo.rev or ""); - shortRev = toString (config.repo.shortRev or ""); - tag = toString (config.repo.tag or ""); - in - '' - set -euo pipefail - - echo "=== effects.smoke: buildbot-nix + hercules-ci-effects pipeline smoke test ===" - - # buildbot-effects-passed args (captured at Nix eval time via config.repo). - echo "branch: ${lib.escapeShellArg branch}" - echo "rev: ${lib.escapeShellArg rev}" - echo "shortRev: ${lib.escapeShellArg shortRev}" - echo "tag: ${lib.escapeShellArg tag}" - - # HERCULES_CI_SECRETS_JSON is set by buildbot-nix inside - # the bwrap sandbox to the path of the JSON secrets blob - # produced by the `perRepoSecretFiles` pipeline (see - # buildbot_effects/__init__.py:250-290). - echo "HERCULES_CI_SECRETS_JSON=''${HERCULES_CI_SECRETS_JSON:-}" - - if [ -n "''${HERCULES_CI_SECRETS_JSON:-}" ] \ - && [ -f "''${HERCULES_CI_SECRETS_JSON}" ]; then - echo "secrets file exists: true" - # Key-only enumeration. VALUES ARE INTENTIONALLY OMITTED. - # Do not change this to `jq -r 'to_entries[] | .value'` - # or equivalent — that would leak secret payloads to the - # buildbot log. - echo -n "secret keys: " - jq -r 'to_entries | map(.key) | @csv' \ - "''${HERCULES_CI_SECRETS_JSON}" - else - echo "secrets file exists: false" - fi - - echo "=== smoke effect complete (exit 0) ===" - ''; - }; - }; - }; -} From e1d0b22ac0e0ca6b92a2d3da5b1f5b621180b961 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 00:41:12 -0400 Subject: [PATCH 56/77] refactor(effects): drop redundant header comment from flake-module.nix --- modules/effects/flake-module.nix | 4 ---- 1 file changed, 4 deletions(-) diff --git a/modules/effects/flake-module.nix b/modules/effects/flake-module.nix index 8205df8a5..856e8d474 100644 --- a/modules/effects/flake-module.nix +++ b/modules/effects/flake-module.nix @@ -1,7 +1,3 @@ -# Lifts the hercules-ci-effects flake-parts module to repo-agnostic scope so -# per-repo effect domains under modules/effects//herculesCI/*.nix can -# merge into herculesCI.onPush.default.outputs.effects. without each -# domain re-importing the flakeModule. { inputs, ... }: { imports = [ From d92ad5cb7609484492640f855488404e595e3db2 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 01:39:36 -0400 Subject: [PATCH 57/77] chore(workflows): snapshot cd.yaml to deprecated before effects-driven slimming --- .github/deprecated/cd.yaml | 407 +++++++++++++++++++++++++++++++++++++ 1 file changed, 407 insertions(+) create mode 100644 .github/deprecated/cd.yaml diff --git a/.github/deprecated/cd.yaml b/.github/deprecated/cd.yaml new file mode 100644 index 000000000..4b9386dcf --- /dev/null +++ b/.github/deprecated/cd.yaml @@ -0,0 +1,407 @@ +name: CD +on: + workflow_dispatch: + inputs: + job: + description: specific job to run (leave empty to run all) + required: false + type: string + debug_enabled: + description: "Run the workflow with tmate.io debugging enabled" + required: false + type: boolean + default: false + deploy_enabled: + description: "Deploy documentation to Cloudflare Workers" + required: false + type: boolean + default: false + force_run: + description: "Force execution even if already successful for this commit" + required: false + type: boolean + default: false + workflow_call: + inputs: + target_configs: + description: comma-separated list of configs to build + required: false + type: string + cache_control: + description: cache control (use_cache, skip_cache) + required: false + type: string + default: use_cache + job_selection: + description: comma-separated list of jobs to run + required: false + type: string + pull_request: + types: [opened, reopened, synchronize] + paths-ignore: + - "*.md" + push: + branches: + - "main" + paths-ignore: + - "*.md" + +concurrency: + group: ci-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +defaults: + run: + shell: bash + +permissions: + contents: read + deployments: write + +jobs: + set-variables: + runs-on: ubuntu-latest + if: | + !cancelled() && + (github.event_name != 'workflow_dispatch' || + inputs.job == '' || + inputs.job == 'set-variables') + outputs: + debug: ${{ steps.set-variables.outputs.debug }} + deploy_enabled: ${{ steps.set-variables.outputs.deploy_enabled }} + deploy_environment: ${{ steps.set-variables.outputs.deploy_environment }} + checkout_ref: ${{ steps.set-variables.outputs.checkout_ref }} + checkout_rev: ${{ steps.set-variables.outputs.checkout_rev }} + sanitized_branch: ${{ steps.set-variables.outputs.sanitized_branch }} + packages: ${{ steps.discover-packages.outputs.packages }} + force-ci: ${{ steps.compute-force-ci.outputs.force-ci }} + + steps: + - name: Set action variables + id: set-variables + run: | + DEBUG="false" + DEPLOY_ENABLED="false" + DEPLOY_ENVIRONMENT="preview" + + if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then + DEBUG="${{ inputs.debug_enabled }}" + DEPLOY_ENABLED="${{ inputs.deploy_enabled }}" + fi + + if [[ "${{ github.event_name }}" == "pull_request" ]]; then + CHECKOUT_REF="${{ github.event.pull_request.head.ref }}" + CHECKOUT_REV="${{ github.event.pull_request.head.sha }}" + else + CHECKOUT_REF="${{ github.ref_name }}" + CHECKOUT_REV="${{ github.sha }}" + fi + + if [[ "${{ github.event_name }}" == "push" && "${{ github.ref }}" == "refs/heads/main" ]]; then + DEPLOY_ENABLED="true" + DEPLOY_ENVIRONMENT="production" + fi + + # Sanitize for Cloudflare subdomain label (≤63 chars; truncate to 40 for safety) + SANITIZED_BRANCH=$(echo "$CHECKOUT_REF" | tr '/' '-' | tr -c 'a-zA-Z0-9-' '-' | sed 's/--*/-/g; s/^-//; s/-$//' | cut -c1-40) + + echo "debug=$DEBUG" >> $GITHUB_OUTPUT + echo "deploy_enabled=$DEPLOY_ENABLED" >> $GITHUB_OUTPUT + echo "deploy_environment=$DEPLOY_ENVIRONMENT" >> $GITHUB_OUTPUT + echo "checkout_ref=$CHECKOUT_REF" >> $GITHUB_OUTPUT + echo "checkout_rev=$CHECKOUT_REV" >> $GITHUB_OUTPUT + echo "sanitized_branch=$SANITIZED_BRANCH" >> $GITHUB_OUTPUT + + - name: Compute force-ci flag + id: compute-force-ci + run: | + # Compute once for all jobs: workflow_dispatch force_run input OR force-ci PR label + if [[ "${{ inputs.force_run }}" == "true" ]] || \ + [[ "${{ github.event_name }}" == "pull_request" && "${{ contains(github.event.pull_request.labels.*.name, 'force-ci') }}" == "true" ]]; then + echo "force-ci=true" >> $GITHUB_OUTPUT + else + echo "force-ci=false" >> $GITHUB_OUTPUT + fi + + - name: Checkout for package discovery + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + fetch-depth: 0 + + - name: Setup Nix + uses: ./.github/actions/setup-nix + with: + installer: full + system: x86_64-linux + + # alternative: extractions/setup-just@v3 (blocked on node20->node24 upgrade via extractions/setup-crate#9) + - name: Install just + uses: taiki-e/install-action@5f57d6cb7cd20b14a8a27f522884c4bc8a187458 # v2.75.19 + with: + tool: just@1 + + - name: Discover packages + id: discover-packages + run: | + PACKAGES=$(just list-packages-json) + echo "packages=$PACKAGES" >> $GITHUB_OUTPUT + echo "Discovered packages: $PACKAGES" + + preview-release-version: + needs: [set-variables] + if: | + !cancelled() && + github.event_name == 'pull_request' + strategy: + fail-fast: false + matrix: + package: ${{ fromJson(needs.set-variables.outputs.packages) }} + runs-on: ubuntu-latest + # semantic-release verifyAuth requires push permission even in dry-run mode + # https://github.com/semantic-release/semantic-release/blob/v25.0.1/index.js#L87-L98 + permissions: + contents: write + steps: + - name: Checkout repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ github.head_ref }} # Checkout actual PR branch, not merge commit + fetch-depth: 0 # Full history needed for semantic-release analysis + fetch-tags: true # Explicitly fetch all tags for version detection + + - name: Check execution cache + id: cache + uses: ./.github/actions/cached-ci-job + with: + check-name: ${{ matrix.package.name }}-preview-release + hash-sources: "packages/${{ matrix.package.name }}/**/* .github/actions/setup-nix/action.yml .github/workflows/cd.yaml flake.nix flake.lock bun.lock modules/apps/docs/**/* pkgs/by-name/vanixiets-docs/**/*" + # Always run: semantic-release analyzes commit history which changes constantly + force-run: "true" + + - name: Fetch target branch for preview + if: steps.cache.outputs.should-run == 'true' + run: | + git fetch origin + git branch -f main origin/main + + - name: Configure git identity for temporary commits + if: steps.cache.outputs.should-run == 'true' + run: | + git config --global user.name "github-actions[bot]" + git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Setup Nix + if: steps.cache.outputs.should-run == 'true' + uses: ./.github/actions/setup-nix + with: + installer: full + system: x86_64-linux + + - name: Setup tmate debug session + if: steps.cache.outputs.should-run == 'true' && needs.set-variables.outputs.debug == 'true' + uses: mxschmitt/action-tmate@c0afd6f790e3a5564914980036ebf83216678101 # v3 + + - name: Preview version for ${{ matrix.package.name }} + if: steps.cache.outputs.should-run == 'true' + env: + CURRENT_BRANCH: ${{ github.head_ref }} + PACKAGE_PATH: ${{ matrix.package.path }} + run: | + echo "::group::Preview semantic-release version" + OUTPUT=$(nix run --accept-flake-config .#preview-version -- main "$PACKAGE_PATH" 2>&1 | tee /dev/stderr) + echo "::endgroup::" + + # Extract and annotate the next version (grep returns 1 on no match, so suppress with || true) + VERSION=$(echo "$OUTPUT" | grep "next version:" | awk '{print $3}') || true + if [ -n "$VERSION" ]; then + echo "::notice title=Next Version (${{ matrix.package.name }})::$VERSION" + else + echo "::notice title=Next Version (${{ matrix.package.name }})::No release pending" + fi + + - name: Create job result marker + # Only create marker if cache didn't exist (cache-source == 'none') + # With force-run: 'true', job runs even on cache hit, but we shouldn't overwrite existing cache + if: success() && steps.cache.outputs.should-run == 'true' && steps.cache.outputs.cache-source == 'none' + shell: bash + run: | + mkdir -p "${{ steps.cache.outputs.cache-path }}" + cat > "${{ steps.cache.outputs.cache-path }}/marker" < /dev/null; then + echo "nix not found in PATH" + exit 1 + fi + echo "nix found at: $(command -v nix)" + echo "nix store path: $(readlink -f $(which nix))" + nix --version + + - name: verify direnv configured + if: steps.cache.outputs.should-run == 'true' + run: | + if ! command -v direnv &> /dev/null; then + echo "direnv not found in PATH" + exit 1 + fi + echo "direnv found at: $(command -v direnv)" + + - name: run make verify + if: steps.cache.outputs.should-run == 'true' + run: make verify + + - name: run make setup-user + if: steps.cache.outputs.should-run == 'true' + run: make setup-user + + - name: verify age key generated + if: steps.cache.outputs.should-run == 'true' + run: | + if [ ! -f ~/.config/sops/age/keys.txt ]; then + echo "age key not generated" + exit 1 + fi + echo "age key generated successfully" + . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh && \ + nix shell nixpkgs#age -c age-keygen -y ~/.config/sops/age/keys.txt + + - name: Create job result marker + if: success() && steps.cache.outputs.should-run == 'true' + shell: bash + run: | + mkdir -p "${{ steps.cache.outputs.cache-path }}" + cat > "${{ steps.cache.outputs.cache-path }}/marker" < Date: Sun, 26 Apr 2026 01:40:19 -0400 Subject: [PATCH 58/77] refactor(workflows): remove cd jobs migrated to herculesCI effects Migrated to modules/effects/vanixiets/herculesCI/{deploy-docs,release-packages}.nix on magnetite buildbot-nix. --- .github/workflows/cd.yaml | 158 -------------------------------------- 1 file changed, 158 deletions(-) diff --git a/.github/workflows/cd.yaml b/.github/workflows/cd.yaml index 4b9386dcf..2896db989 100644 --- a/.github/workflows/cd.yaml +++ b/.github/workflows/cd.yaml @@ -147,120 +147,6 @@ jobs: echo "packages=$PACKAGES" >> $GITHUB_OUTPUT echo "Discovered packages: $PACKAGES" - preview-release-version: - needs: [set-variables] - if: | - !cancelled() && - github.event_name == 'pull_request' - strategy: - fail-fast: false - matrix: - package: ${{ fromJson(needs.set-variables.outputs.packages) }} - runs-on: ubuntu-latest - # semantic-release verifyAuth requires push permission even in dry-run mode - # https://github.com/semantic-release/semantic-release/blob/v25.0.1/index.js#L87-L98 - permissions: - contents: write - steps: - - name: Checkout repository - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - with: - ref: ${{ github.head_ref }} # Checkout actual PR branch, not merge commit - fetch-depth: 0 # Full history needed for semantic-release analysis - fetch-tags: true # Explicitly fetch all tags for version detection - - - name: Check execution cache - id: cache - uses: ./.github/actions/cached-ci-job - with: - check-name: ${{ matrix.package.name }}-preview-release - hash-sources: "packages/${{ matrix.package.name }}/**/* .github/actions/setup-nix/action.yml .github/workflows/cd.yaml flake.nix flake.lock bun.lock modules/apps/docs/**/* pkgs/by-name/vanixiets-docs/**/*" - # Always run: semantic-release analyzes commit history which changes constantly - force-run: "true" - - - name: Fetch target branch for preview - if: steps.cache.outputs.should-run == 'true' - run: | - git fetch origin - git branch -f main origin/main - - - name: Configure git identity for temporary commits - if: steps.cache.outputs.should-run == 'true' - run: | - git config --global user.name "github-actions[bot]" - git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com" - - - name: Setup Nix - if: steps.cache.outputs.should-run == 'true' - uses: ./.github/actions/setup-nix - with: - installer: full - system: x86_64-linux - - - name: Setup tmate debug session - if: steps.cache.outputs.should-run == 'true' && needs.set-variables.outputs.debug == 'true' - uses: mxschmitt/action-tmate@c0afd6f790e3a5564914980036ebf83216678101 # v3 - - - name: Preview version for ${{ matrix.package.name }} - if: steps.cache.outputs.should-run == 'true' - env: - CURRENT_BRANCH: ${{ github.head_ref }} - PACKAGE_PATH: ${{ matrix.package.path }} - run: | - echo "::group::Preview semantic-release version" - OUTPUT=$(nix run --accept-flake-config .#preview-version -- main "$PACKAGE_PATH" 2>&1 | tee /dev/stderr) - echo "::endgroup::" - - # Extract and annotate the next version (grep returns 1 on no match, so suppress with || true) - VERSION=$(echo "$OUTPUT" | grep "next version:" | awk '{print $3}') || true - if [ -n "$VERSION" ]; then - echo "::notice title=Next Version (${{ matrix.package.name }})::$VERSION" - else - echo "::notice title=Next Version (${{ matrix.package.name }})::No release pending" - fi - - - name: Create job result marker - # Only create marker if cache didn't exist (cache-source == 'none') - # With force-run: 'true', job runs even on cache hit, but we shouldn't overwrite existing cache - if: success() && steps.cache.outputs.should-run == 'true' && steps.cache.outputs.cache-source == 'none' - shell: bash - run: | - mkdir -p "${{ steps.cache.outputs.cache-path }}" - cat > "${{ steps.cache.outputs.cache-path }}/marker" < Date: Sun, 26 Apr 2026 01:40:51 -0400 Subject: [PATCH 59/77] refactor(workflows): slim set-variables to retained outputs Drop deploy_enabled/deploy_environment/checkout_ref/checkout_rev/sanitized_branch/packages outputs and the package-discovery steps. Only debug (consumed by test-cluster) and force-ci (consumed by bootstrap-verification) remain. Dead push/pull_request branches removed. --- .github/workflows/cd.yaml | 76 ++------------------------------------- 1 file changed, 3 insertions(+), 73 deletions(-) diff --git a/.github/workflows/cd.yaml b/.github/workflows/cd.yaml index 2896db989..33ebeaec8 100644 --- a/.github/workflows/cd.yaml +++ b/.github/workflows/cd.yaml @@ -63,89 +63,19 @@ jobs: runs-on: ubuntu-latest if: | !cancelled() && - (github.event_name != 'workflow_dispatch' || - inputs.job == '' || - inputs.job == 'set-variables') + (inputs.job == '' || inputs.job == 'set-variables') outputs: debug: ${{ steps.set-variables.outputs.debug }} - deploy_enabled: ${{ steps.set-variables.outputs.deploy_enabled }} - deploy_environment: ${{ steps.set-variables.outputs.deploy_environment }} - checkout_ref: ${{ steps.set-variables.outputs.checkout_ref }} - checkout_rev: ${{ steps.set-variables.outputs.checkout_rev }} - sanitized_branch: ${{ steps.set-variables.outputs.sanitized_branch }} - packages: ${{ steps.discover-packages.outputs.packages }} force-ci: ${{ steps.compute-force-ci.outputs.force-ci }} steps: - name: Set action variables id: set-variables - run: | - DEBUG="false" - DEPLOY_ENABLED="false" - DEPLOY_ENVIRONMENT="preview" - - if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then - DEBUG="${{ inputs.debug_enabled }}" - DEPLOY_ENABLED="${{ inputs.deploy_enabled }}" - fi - - if [[ "${{ github.event_name }}" == "pull_request" ]]; then - CHECKOUT_REF="${{ github.event.pull_request.head.ref }}" - CHECKOUT_REV="${{ github.event.pull_request.head.sha }}" - else - CHECKOUT_REF="${{ github.ref_name }}" - CHECKOUT_REV="${{ github.sha }}" - fi - - if [[ "${{ github.event_name }}" == "push" && "${{ github.ref }}" == "refs/heads/main" ]]; then - DEPLOY_ENABLED="true" - DEPLOY_ENVIRONMENT="production" - fi - - # Sanitize for Cloudflare subdomain label (≤63 chars; truncate to 40 for safety) - SANITIZED_BRANCH=$(echo "$CHECKOUT_REF" | tr '/' '-' | tr -c 'a-zA-Z0-9-' '-' | sed 's/--*/-/g; s/^-//; s/-$//' | cut -c1-40) - - echo "debug=$DEBUG" >> $GITHUB_OUTPUT - echo "deploy_enabled=$DEPLOY_ENABLED" >> $GITHUB_OUTPUT - echo "deploy_environment=$DEPLOY_ENVIRONMENT" >> $GITHUB_OUTPUT - echo "checkout_ref=$CHECKOUT_REF" >> $GITHUB_OUTPUT - echo "checkout_rev=$CHECKOUT_REV" >> $GITHUB_OUTPUT - echo "sanitized_branch=$SANITIZED_BRANCH" >> $GITHUB_OUTPUT + run: echo "debug=${{ inputs.debug_enabled }}" >> "$GITHUB_OUTPUT" - name: Compute force-ci flag id: compute-force-ci - run: | - # Compute once for all jobs: workflow_dispatch force_run input OR force-ci PR label - if [[ "${{ inputs.force_run }}" == "true" ]] || \ - [[ "${{ github.event_name }}" == "pull_request" && "${{ contains(github.event.pull_request.labels.*.name, 'force-ci') }}" == "true" ]]; then - echo "force-ci=true" >> $GITHUB_OUTPUT - else - echo "force-ci=false" >> $GITHUB_OUTPUT - fi - - - name: Checkout for package discovery - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - with: - fetch-depth: 0 - - - name: Setup Nix - uses: ./.github/actions/setup-nix - with: - installer: full - system: x86_64-linux - - # alternative: extractions/setup-just@v3 (blocked on node20->node24 upgrade via extractions/setup-crate#9) - - name: Install just - uses: taiki-e/install-action@5f57d6cb7cd20b14a8a27f522884c4bc8a187458 # v2.75.19 - with: - tool: just@1 - - - name: Discover packages - id: discover-packages - run: | - PACKAGES=$(just list-packages-json) - echo "packages=$PACKAGES" >> $GITHUB_OUTPUT - echo "Discovered packages: $PACKAGES" + run: echo "force-ci=${{ inputs.force_run }}" >> "$GITHUB_OUTPUT" bootstrap-verification: needs: [set-variables] From e60e05be7a18ff9f4dd63e7a88fcfaf45e9b605a Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 01:41:20 -0400 Subject: [PATCH 60/77] refactor(workflows): drop push/pull_request triggers and tidy cd metadata cd.yaml is now manual-dispatch only via gh workflow run / Actions UI. Drops push, pull_request, and workflow_call triggers; removes the deploy_enabled input (no remaining consumer); narrows permissions to contents:read; simplifies concurrency group; drops vestigial github.event_name clauses from retained job ifs. --- .github/workflows/cd.yaml | 40 +++------------------------------------ 1 file changed, 3 insertions(+), 37 deletions(-) diff --git a/.github/workflows/cd.yaml b/.github/workflows/cd.yaml index 33ebeaec8..22383a0f4 100644 --- a/.github/workflows/cd.yaml +++ b/.github/workflows/cd.yaml @@ -11,43 +11,14 @@ on: required: false type: boolean default: false - deploy_enabled: - description: "Deploy documentation to Cloudflare Workers" - required: false - type: boolean - default: false force_run: description: "Force execution even if already successful for this commit" required: false type: boolean default: false - workflow_call: - inputs: - target_configs: - description: comma-separated list of configs to build - required: false - type: string - cache_control: - description: cache control (use_cache, skip_cache) - required: false - type: string - default: use_cache - job_selection: - description: comma-separated list of jobs to run - required: false - type: string - pull_request: - types: [opened, reopened, synchronize] - paths-ignore: - - "*.md" - push: - branches: - - "main" - paths-ignore: - - "*.md" concurrency: - group: ci-${{ github.event.pull_request.number || github.ref }} + group: ci-${{ github.ref }} cancel-in-progress: true defaults: @@ -56,7 +27,6 @@ defaults: permissions: contents: read - deployments: write jobs: set-variables: @@ -82,9 +52,7 @@ jobs: runs-on: ubuntu-latest if: | !cancelled() && - (github.event_name != 'workflow_dispatch' || - inputs.job == '' || - inputs.job == 'bootstrap-verification') + (inputs.job == '' || inputs.job == 'bootstrap-verification') steps: - name: checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 @@ -169,9 +137,7 @@ jobs: needs: [set-variables] if: | !cancelled() && - (github.event_name != 'workflow_dispatch' || - inputs.job == '' || - inputs.job == 'test-cluster') + (inputs.job == '' || inputs.job == 'test-cluster') uses: ./.github/workflows/test-cluster.yaml with: debug_enabled: ${{ needs.set-variables.outputs.debug }} From 7bb30af3fa8df280b2a75328f31917566600d666 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 02:22:27 -0400 Subject: [PATCH 61/77] fix(docs): correct stale workspace filter name in README The bun workspace filter was documented as @typescript-nix-template/docs, but the actual package name in package.json is @vanixiets/docs, so the documented commands silently matched nothing. --- packages/docs/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/docs/README.md b/packages/docs/README.md index 8958953f3..cffec2213 100644 --- a/packages/docs/README.md +++ b/packages/docs/README.md @@ -57,12 +57,12 @@ packages/docs/ # Start dev server just dev # or -bun run --filter '@typescript-nix-template/docs' dev +bun run --filter '@vanixiets/docs' dev # Build just build # or -bun run --filter '@typescript-nix-template/docs' build +bun run --filter '@vanixiets/docs' build ``` ### From package directory From d411d68a4c5f9b7a94ba7e83d3aeb1c4b802f7b3 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 02:24:31 -0400 Subject: [PATCH 62/77] refactor(apps/release): drop unused SOPS_AGE_KEY env-var passthrough The SOPS_AGE_KEY env var was declared as a 'reserved passthrough' with no consumer in release.sh. No caller exports it for the release path, and dropping the documentation removes a misleading entry from the env-var contract. --- modules/apps/release/release.sh | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/modules/apps/release/release.sh b/modules/apps/release/release.sh index 75e95909c..472de60fd 100644 --- a/modules/apps/release/release.sh +++ b/modules/apps/release/release.sh @@ -31,10 +31,6 @@ # GIT_USER_NAME / GIT_USER_EMAIL — transitional aliases that seed the # quartet when the GIT_AUTHOR_* / GIT_COMMITTER_* # forms are unset. -# Optional (passthrough, not consumed): -# SOPS_AGE_KEY reserved passthrough for sops-decrypt hooks; no -# consumer in the current tree (declared but NOT -# enforced via :? guard). set -euo pipefail @@ -58,7 +54,7 @@ Flags: --help Print this usage and exit. Environment: - GITHUB_TOKEN, SOPS_AGE_KEY, DOCS_NODE_MODULES, RELEASE_REPO_ROOT, CI, + GITHUB_TOKEN, DOCS_NODE_MODULES, RELEASE_REPO_ROOT, CI, GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, GIT_USER_NAME, GIT_USER_EMAIL (see release.sh header for details). EOF From aad097d3a66dde7b91479d75be66c42ed4e47b50 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 02:25:02 -0400 Subject: [PATCH 63/77] refactor(apps/release): drop transitional GIT_USER_NAME/GIT_USER_EMAIL aliases These aliases existed to bridge a prior GHA caller that exported only the GIT_USER_* form. The GHA release path is retired; the herculesCI effect preamble exports the canonical GIT_AUTHOR_*/GIT_COMMITTER_* quartet directly. No remaining caller relies on GIT_USER_*. --- modules/apps/release/release.sh | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/modules/apps/release/release.sh b/modules/apps/release/release.sh index 472de60fd..3d548a2eb 100644 --- a/modules/apps/release/release.sh +++ b/modules/apps/release/release.sh @@ -28,9 +28,6 @@ # Defaults applied by the effect preamble: # GIT_AUTHOR_NAME / GIT_AUTHOR_EMAIL (semantic-release@vanixiets.local) # GIT_COMMITTER_NAME / GIT_COMMITTER_EMAIL (semantic-release@vanixiets.local) -# GIT_USER_NAME / GIT_USER_EMAIL — transitional aliases that seed the -# quartet when the GIT_AUTHOR_* / GIT_COMMITTER_* -# forms are unset. set -euo pipefail @@ -55,8 +52,8 @@ Flags: Environment: GITHUB_TOKEN, DOCS_NODE_MODULES, RELEASE_REPO_ROOT, CI, - GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL, - GIT_USER_NAME, GIT_USER_EMAIL (see release.sh header for details). + GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL, GIT_COMMITTER_NAME, GIT_COMMITTER_EMAIL + (see release.sh header for details). EOF } @@ -168,14 +165,12 @@ fi # sandbox renders .git read-only (mounts /nix/store ro-bind only) and # `git config user.email "…"` would fail with `error: could not lock config # file .git/config`. git honours these env vars natively without any config -# write. Transitional aliases GIT_USER_NAME/GIT_USER_EMAIL seed the quartet -# when the new vars are unset, preserving existing local/GHA caller -# behaviour. Each export uses parameter-expansion default chaining so a -# pre-set value (effect preamble or caller env) is preserved unchanged. -export GIT_AUTHOR_NAME="${GIT_AUTHOR_NAME:-${GIT_USER_NAME:-semantic-release}}" -export GIT_AUTHOR_EMAIL="${GIT_AUTHOR_EMAIL:-${GIT_USER_EMAIL:-semantic-release@vanixiets.local}}" -export GIT_COMMITTER_NAME="${GIT_COMMITTER_NAME:-${GIT_USER_NAME:-semantic-release}}" -export GIT_COMMITTER_EMAIL="${GIT_COMMITTER_EMAIL:-${GIT_USER_EMAIL:-semantic-release@vanixiets.local}}" +# write. Each export uses parameter-expansion default chaining so a pre-set +# value (effect preamble or caller env) is preserved unchanged. +export GIT_AUTHOR_NAME="${GIT_AUTHOR_NAME:-semantic-release}" +export GIT_AUTHOR_EMAIL="${GIT_AUTHOR_EMAIL:-semantic-release@vanixiets.local}" +export GIT_COMMITTER_NAME="${GIT_COMMITTER_NAME:-semantic-release}" +export GIT_COMMITTER_EMAIL="${GIT_COMMITTER_EMAIL:-semantic-release@vanixiets.local}" cd "$package_path" From 0e2f47ab901dcc406c010aae68f4d5916610bdd8 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 02:25:15 -0400 Subject: [PATCH 64/77] refactor(apps/release): drop stale GHA reference from repo-root resolution comment GHA release path is retired; the only remaining callers are the herculesCI effect preamble (sets RELEASE_REPO_ROOT) and local-shell invocations (leave it unset). --- modules/apps/release/release.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/apps/release/release.sh b/modules/apps/release/release.sh index 3d548a2eb..448ff3de7 100644 --- a/modules/apps/release/release.sh +++ b/modules/apps/release/release.sh @@ -149,8 +149,8 @@ fi # mount the working tree's .git, so `git rev-parse --show-toplevel` would # fail with `fatal: not a git repository` (exit 128) and abort the script. # The effect preamble sets RELEASE_REPO_ROOT="$PWD" so this branch resolves -# without invoking git. Local-shell and GHA paths set RELEASE_REPO_ROOT -# to empty, exercising the git fallback against the live worktree. +# without invoking git. Local-shell callers leave RELEASE_REPO_ROOT unset, +# exercising the git fallback against the live worktree. repo_root="${RELEASE_REPO_ROOT:-$(git rev-parse --show-toplevel 2>/dev/null || pwd)}" cd "$repo_root" From 8123fe65d2bfbbf7663efd242697ffb7a1b6067c Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 02:25:24 -0400 Subject: [PATCH 65/77] refactor(effects/release-packages): drop unused dryRunFlag binding dryRunFlag was defined but never threaded into the dispatch. The actual non-main path delegates to preview-version (which is always dry-run by construction), so the flag is dead in both branches. --- modules/effects/vanixiets/herculesCI/release-packages.nix | 2 -- 1 file changed, 2 deletions(-) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 869ccb721..c3d7bdabe 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -23,8 +23,6 @@ prNumber = if isPrMerge then builtins.head prMergeMatch else null; actionBanner = if isMain then "release" else "dry-run"; - - dryRunFlag = if isMain then "" else "--dry-run"; in { onPush.default.outputs.effects.release-packages = withSystem "x86_64-linux" ( From b3cd01a5f7c9a3e12928297007f737238958112b Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 02:27:11 -0400 Subject: [PATCH 66/77] refactor(effects/release-packages): extract mkReleasePackagesEffect helper Factor the effect body into a helper parameterised on dryRun. The sole call site passes dryRun=false, preserving production behavior. The helper threads dryRun through effect name, action label, dispatch line, dispatch marker, and stale-rev guard, enabling a follow-up commit to wire a release-packages-dry-run rehearsal attribute that exercises the production plugin set under semantic-release's --dry-run mode. --- .../vanixiets/herculesCI/release-packages.nix | 306 ++++++++++-------- 1 file changed, 177 insertions(+), 129 deletions(-) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index c3d7bdabe..0be0c8961 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -23,92 +23,59 @@ prNumber = if isPrMerge then builtins.head prMergeMatch else null; actionBanner = if isMain then "release" else "dry-run"; - in - { - onPush.default.outputs.effects.release-packages = withSystem "x86_64-linux" ( - { config, pkgs, ... }: - let - hci-effects = inputs.hercules-ci-effects.lib.withPkgs pkgs; - - # release.sh --dry-run short-circuits on in-tree branches:["main"]; preview-version is the non-main path. - listPackagesProgram = config.apps.list-packages-json.program; - releaseProgram = config.apps.release.program; - previewVersionProgram = config.apps.preview-version.program; - in - hci-effects.mkEffect { - name = "release-packages"; - - # Why: mkEffect's defaultInputs do not include git; clone preamble below requires it. - inputs = [ pkgs.git ]; - - effectScript = '' - set -euo pipefail - - echo "=== effects.release-packages (semantic-release per-package dispatcher) ===" - echo "branch: ${lib.escapeShellArg (toString branch)}" - echo "rev: ${lib.escapeShellArg (toString rev)}" - echo "shortRev: ${lib.escapeShellArg (toString shortRev)}" - echo "isMain: ${if isMain then "true" else "false"}" - - echo "RELEASE-PACKAGES-ACTION: ${actionBanner}" - - export GITHUB_TOKEN="$(jq -r '.GITHUB_TOKEN.data.value' "$HERCULES_CI_SECRETS_JSON")" - - if [ -z "''${GITHUB_TOKEN:-}" ] || [ "$GITHUB_TOKEN" = "null" ]; then - echo "error: GITHUB_TOKEN missing from \$HERCULES_CI_SECRETS_JSON" >&2 - exit 1 - fi - - # Why: do not use config.repo.remoteHttpUrl — buildbot-nix bakes - # the App installation token into it; would leak via banner echo. - clone_url="https://github.com/cameronraysmith/vanixiets.git" - clone_dir="$(mktemp -d -t release-packages-clone.XXXXXX)" - - trap 'rm -rf "$clone_dir"' EXIT - - GIT_REV=${lib.escapeShellArg (toString rev)} - GIT_BRANCH=${lib.escapeShellArg (if branch == null then "" else toString branch)} - ${ - if isPrMerge then - '' - git clone "$clone_url" "$clone_dir" - git -C "$clone_dir" fetch --tags origin - - # `git fetch origin refs/pull//head` alone updates - # FETCH_HEAD but does NOT auto-create the remote-tracking - # ref; the explicit `+ref:remote-tracking-ref` mapping - # closes that gap. - git -C "$clone_dir" fetch origin \ - "+refs/pull/${toString prNumber}/head:refs/remotes/origin/pr-${toString prNumber}-head" - head_sha="$(git -C "$clone_dir" rev-parse origin/pr-${toString prNumber}-head)" - - echo "RELEASE-CLONE-PR-HEAD: ${toString prNumber} $head_sha" - echo "RELEASE-CLONE-PR-DISPATCH: ${toString prNumber} buildbot-rev=$GIT_REV head=$head_sha" - - echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" - - git -C "$clone_dir" checkout -B "pr-${toString prNumber}-head" "$head_sha" - echo "RELEASE-CLONE-CHECKOUT: $head_sha" - - # Trivially true post-fetch unless force-push race lost the head ref; set -e propagates abort. - git -C "$clone_dir" rev-parse --verify origin/pr-${toString prNumber}-head >/dev/null - '' + # mkReleasePackagesEffect: shared effect body parameterised on dryRun. + # + # dryRun = false (production): + # isMain → call release.sh (publish path). + # non-main → call preview-version.sh (rehearsal path that filters + # @semantic-release/github and replaces it with a local bare clone). + # + # dryRun = true (rehearsal attribute): + # ALWAYS calls release.sh with `-- --dry-run` so the production + # plugin set (including @semantic-release/github) is exercised end + # to end, but semantic-release's prepare/publish/success steps are + # no-ops. Stale-rev guard is bypassed because the rehearsal is + # invoked manually with `--branch main` against an arbitrary rev, + # so HEAD will not equal origin/main by design. + mkReleasePackagesEffect = + { dryRun }: + withSystem "x86_64-linux" ( + { config, pkgs, ... }: + let + hci-effects = inputs.hercules-ci-effects.lib.withPkgs pkgs; + + listPackagesProgram = config.apps.list-packages-json.program; + releaseProgram = config.apps.release.program; + previewVersionProgram = config.apps.preview-version.program; + + effectName = if dryRun then "release-packages-dry-run" else "release-packages"; + + actionLabel = + if dryRun then "rehearsal (production plugins, semantic-release --dry-run)" else actionBanner; + + dispatchLine = + if dryRun then + ''"$RELEASE" "$pkg_path" -- --dry-run'' + else if isMain then + ''"$RELEASE" "$pkg_path"'' + else + ''"$PREVIEW" main "$pkg_path"''; + + # Distinct dispatch marker so rehearsal logs are not mistaken for + # production runs in CI output. Under dryRun=false the marker is + # empty (suppresses the echo line) for byte-for-byte parity with + # the pre-refactor effect. + dispatchMarkerLine = if dryRun then ''echo "RELEASE-PACKAGE-DRY-RUN-DISPATCH: $pkg_path"'' else ""; + + # Stale-rev guard bypassed under dryRun: the rehearsal attribute + # is intentionally invoked against non-main revs while declaring + # `--branch main`, so HEAD will not equal origin/main. + staleRevGuard = + if dryRun then + "" else '' - echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" - - git clone "$clone_url" "$clone_dir" - git -C "$clone_dir" fetch --tags origin - - if [ -n "$GIT_BRANCH" ]; then - checkout_branch="$GIT_BRANCH" - else - checkout_branch="release-packages-detached" - fi - git -C "$clone_dir" checkout -B "$checkout_branch" "$GIT_REV" - echo "RELEASE-CLONE-CHECKOUT: $GIT_REV" - if [ -n "$GIT_BRANCH" ]; then git -C "$clone_dir" fetch origin "$GIT_BRANCH" head_rev="$(git -C "$clone_dir" rev-parse HEAD)" @@ -118,64 +85,145 @@ exit 1 fi fi - '' - } + ''; + in + hci-effects.mkEffect { + name = effectName; - echo "RELEASE-CLONE-READY: $clone_dir" + # Why: mkEffect's defaultInputs do not include git; clone preamble below requires it. + inputs = [ pkgs.git ]; - # semantic-release's get-git-auth-url.js treats GIT_CREDENTIALS as user:password and constructs the authenticated URL in-process. The vanixiets-effects-secrets PAT (Read+Write) is the canonical authority — the buildbot-nix App installation token (Read-only) is NOT reused for release mutation. - export GIT_CREDENTIALS="x-access-token:''${GITHUB_TOKEN}" + effectScript = '' + set -euo pipefail - # CI=true bypasses semantic-release's env-ci abort. GIT_AUTHOR/COMMITTER are honoured natively without writing .git/config (which the bwrap /nix/store ro-bind would block). - export CI=true - export GIT_BRANCH - export RELEASE_REPO_ROOT="$clone_dir" - export GIT_AUTHOR_NAME=semantic-release - export GIT_AUTHOR_EMAIL=semantic-release@vanixiets.local - export GIT_COMMITTER_NAME=semantic-release - export GIT_COMMITTER_EMAIL=semantic-release@vanixiets.local + echo "=== effects.${effectName} (semantic-release per-package dispatcher) ===" + echo "branch: ${lib.escapeShellArg (toString branch)}" + echo "rev: ${lib.escapeShellArg (toString rev)}" + echo "shortRev: ${lib.escapeShellArg (toString shortRev)}" + echo "isMain: ${if isMain then "true" else "false"}" - # Why: bwrap sandbox does not bind working tree; .# cannot resolve. Use eval-time /nix/store paths. - LIST_PACKAGES=${listPackagesProgram} - RELEASE=${releaseProgram} - PREVIEW=${previewVersionProgram} + echo "RELEASE-PACKAGES-ACTION: ${actionLabel}" - # list-packages-json calls `git rev-parse --show-toplevel` - # which must resolve to $clone_dir (the only real git tree). - cd "$clone_dir" + export GITHUB_TOKEN="$(jq -r '.GITHUB_TOKEN.data.value' "$HERCULES_CI_SECRETS_JSON")" - packages_json="$("$LIST_PACKAGES")" - echo "packages discovered: $packages_json" + if [ -z "''${GITHUB_TOKEN:-}" ] || [ "$GITHUB_TOKEN" = "null" ]; then + echo "error: GITHUB_TOKEN missing from \$HERCULES_CI_SECRETS_JSON" >&2 + exit 1 + fi - failed_packages=() + # Why: do not use config.repo.remoteHttpUrl — buildbot-nix bakes + # the App installation token into it; would leak via banner echo. + clone_url="https://github.com/cameronraysmith/vanixiets.git" - while IFS= read -r pkg_path; do - [ -z "$pkg_path" ] && continue + clone_dir="$(mktemp -d -t release-packages-clone.XXXXXX)" - echo "RELEASE-PACKAGE-ITERATION: $pkg_path" + trap 'rm -rf "$clone_dir"' EXIT - # CLI grammars differ — release [--dry-run] vs preview-version [target-branch] [pkg-path] — so a single shared dispatch line cannot work. - set +e - ${if isMain then ''"$RELEASE" "$pkg_path"'' else ''"$PREVIEW" main "$pkg_path"''} - rc=$? - set -e + GIT_REV=${lib.escapeShellArg (toString rev)} + GIT_BRANCH=${lib.escapeShellArg (if branch == null then "" else toString branch)} + ${ + if isPrMerge then + '' + git clone "$clone_url" "$clone_dir" + git -C "$clone_dir" fetch --tags origin - if [ "$rc" -eq 0 ]; then - echo "RELEASE-PACKAGE-OK: $pkg_path" - else - echo "RELEASE-PACKAGE-FAILURE: $pkg_path (exit $rc)" - failed_packages+=("$pkg_path") - fi - done < <(printf '%s\n' "$packages_json" | jq -r '.[].path') + # `git fetch origin refs/pull//head` alone updates + # FETCH_HEAD but does NOT auto-create the remote-tracking + # ref; the explicit `+ref:remote-tracking-ref` mapping + # closes that gap. + git -C "$clone_dir" fetch origin \ + "+refs/pull/${toString prNumber}/head:refs/remotes/origin/pr-${toString prNumber}-head" + head_sha="$(git -C "$clone_dir" rev-parse origin/pr-${toString prNumber}-head)" - if [ "''${#failed_packages[@]}" -gt 0 ]; then - echo "error: ''${#failed_packages[@]} package(s) failed: ''${failed_packages[*]}" >&2 - exit 1 - fi + echo "RELEASE-CLONE-PR-HEAD: ${toString prNumber} $head_sha" + echo "RELEASE-CLONE-PR-DISPATCH: ${toString prNumber} buildbot-rev=$GIT_REV head=$head_sha" - echo "=== release-packages effect complete (exit 0) ===" - ''; - } - ); + echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" + + git -C "$clone_dir" checkout -B "pr-${toString prNumber}-head" "$head_sha" + echo "RELEASE-CLONE-CHECKOUT: $head_sha" + + # Trivially true post-fetch unless force-push race lost the head ref; set -e propagates abort. + git -C "$clone_dir" rev-parse --verify origin/pr-${toString prNumber}-head >/dev/null + '' + else + '' + echo "RELEASE-CLONE-START: $clone_url $GIT_REV $GIT_BRANCH" + + git clone "$clone_url" "$clone_dir" + git -C "$clone_dir" fetch --tags origin + + if [ -n "$GIT_BRANCH" ]; then + checkout_branch="$GIT_BRANCH" + else + checkout_branch="release-packages-detached" + fi + git -C "$clone_dir" checkout -B "$checkout_branch" "$GIT_REV" + echo "RELEASE-CLONE-CHECKOUT: $GIT_REV" + + ${staleRevGuard} + '' + } + + echo "RELEASE-CLONE-READY: $clone_dir" + + # semantic-release's get-git-auth-url.js treats GIT_CREDENTIALS as user:password and constructs the authenticated URL in-process. The vanixiets-effects-secrets PAT (Read+Write) is the canonical authority — the buildbot-nix App installation token (Read-only) is NOT reused for release mutation. + export GIT_CREDENTIALS="x-access-token:''${GITHUB_TOKEN}" + + # CI=true bypasses semantic-release's env-ci abort. GIT_AUTHOR/COMMITTER are honoured natively without writing .git/config (which the bwrap /nix/store ro-bind would block). + export CI=true + export GIT_BRANCH + export RELEASE_REPO_ROOT="$clone_dir" + export GIT_AUTHOR_NAME=semantic-release + export GIT_AUTHOR_EMAIL=semantic-release@vanixiets.local + export GIT_COMMITTER_NAME=semantic-release + export GIT_COMMITTER_EMAIL=semantic-release@vanixiets.local + + # Why: bwrap sandbox does not bind working tree; .# cannot resolve. Use eval-time /nix/store paths. + LIST_PACKAGES=${listPackagesProgram} + RELEASE=${releaseProgram} + PREVIEW=${previewVersionProgram} + + # list-packages-json calls `git rev-parse --show-toplevel` + # which must resolve to $clone_dir (the only real git tree). + cd "$clone_dir" + + packages_json="$("$LIST_PACKAGES")" + echo "packages discovered: $packages_json" + + failed_packages=() + + while IFS= read -r pkg_path; do + [ -z "$pkg_path" ] && continue + + echo "RELEASE-PACKAGE-ITERATION: $pkg_path" + ${dispatchMarkerLine} + + # CLI grammars differ — release [-- extra-args] vs preview-version [target-branch] [pkg-path] — so a single shared dispatch line cannot work. + set +e + ${dispatchLine} + rc=$? + set -e + + if [ "$rc" -eq 0 ]; then + echo "RELEASE-PACKAGE-OK: $pkg_path" + else + echo "RELEASE-PACKAGE-FAILURE: $pkg_path (exit $rc)" + failed_packages+=("$pkg_path") + fi + done < <(printf '%s\n' "$packages_json" | jq -r '.[].path') + + if [ "''${#failed_packages[@]}" -gt 0 ]; then + echo "error: ''${#failed_packages[@]} package(s) failed: ''${failed_packages[*]}" >&2 + exit 1 + fi + + echo "=== ${effectName} effect complete (exit 0) ===" + ''; + } + ); + in + { + onPush.default.outputs.effects.release-packages = mkReleasePackagesEffect { dryRun = false; }; }; } From 867ede28be7b1d0b1bd26fffbea957d76d829fd9 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 02:27:57 -0400 Subject: [PATCH 67/77] feat(effects/release-packages): add release-packages-dry-run rehearsal attribute The new effect attribute exercises release.sh's production plugin set (including @semantic-release/github) end to end against the cloned tree, but appends `-- --dry-run` so semantic-release's prepare/publish/success steps are no-ops. This is the load-bearing rehearsal stage of the verification staircase: it validates plugin loading, GITHUB_TOKEN auth surface, and per-package dispatch without performing any tag push or release publish. Invoke manually via buildbot-effects against an arbitrary rev while declaring `--branch main` to force the main-branch dispatch; the stale-rev guard is bypassed under dryRun so HEAD need not equal origin/main. --- modules/effects/vanixiets/herculesCI/release-packages.nix | 3 +++ 1 file changed, 3 insertions(+) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 0be0c8961..5c09ec57f 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -225,5 +225,8 @@ in { onPush.default.outputs.effects.release-packages = mkReleasePackagesEffect { dryRun = false; }; + onPush.default.outputs.effects.release-packages-dry-run = mkReleasePackagesEffect { + dryRun = true; + }; }; } From 50da4bd06c275012eaf80065a020552fe5aca7ce Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 02:44:49 -0400 Subject: [PATCH 68/77] refactor(effects/release-packages): drop release-packages-dry-run rehearsal attribute The dryRun parameter on mkReleasePackagesEffect is itself the rehearsal toggle: flip dryRun to true on a feature branch, run buildbot-effects run release-packages, then revert. A persistent second attribute auto-fires on every qualifying push and produces no signal. --- modules/effects/vanixiets/herculesCI/release-packages.nix | 3 --- 1 file changed, 3 deletions(-) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 5c09ec57f..0be0c8961 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -225,8 +225,5 @@ in { onPush.default.outputs.effects.release-packages = mkReleasePackagesEffect { dryRun = false; }; - onPush.default.outputs.effects.release-packages-dry-run = mkReleasePackagesEffect { - dryRun = true; - }; }; } From f5965e4523d922efdbd664cde41e61468c3121d2 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 02:59:11 -0400 Subject: [PATCH 69/77] docs(notes): rewrite cd-to-buildbot ADR-001 as final-state working note Reframes the original 2026-04-21 ADR draft as a value-extraction-oriented working note documenting what was actually migrated (deploy-docs, release-packages effects), what was deliberately retained on GHA (bootstrap-verification, test-cluster) and why, the rehearsal-toggle pattern, the snapshot-rollback pattern, the PAT identity transition, and the verification staircase. --- .../ADR-001-cd-to-buildbot-migration.md | 335 ++++++------------ 1 file changed, 112 insertions(+), 223 deletions(-) diff --git a/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md b/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md index c11311769..5bb11a4be 100644 --- a/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md +++ b/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md @@ -1,290 +1,179 @@ -# ADR-001: cd.yaml → buildbot-nix / hercules-ci-effects migration +# cd.yaml to herculesCI effects migration — final-state notes -Status: Accepted 2026-04-21; revised 2026-04-22 to incorporate discovery resolutions and correct architectural claims. +This document was originally drafted on 2026-04-21 as an ADR (commit `d629aa7e3`), then revised on 2026-04-22 to incorporate discovery resolutions. +On reflection it was misclassified: the work it describes is concrete and bounded, not an architecture-level decision worth ADR ceremony. +What follows is the final state of the migration on branch `cd-via-effects` as of 2026-04-26, retained as a working note for value extraction — patterns and rationale worth preserving — rather than as an ADR. +The file may be deleted before the PR merges to main; the final state remains in branch git history. -## Identity +## What was migrated -Migrate `.github/workflows/cd.yaml` and everything it transitively invokes (reusable workflows, composite actions, inline script bodies) off GitHub Actions and onto magnetite's buildbot-nix deployment, using hercules-ci-effects for impure execution and `writeShellApplication` flake apps for script bodies. Scope is strictly the CD surface — `ci.yaml` migration already landed pre-epic. +Two effect modules already existed prior to this session and now own the CD jobs they correspond to: -## Context +- `modules/effects/vanixiets/herculesCI/deploy-docs.nix` — supersedes the GHA jobs `preview-docs-deploy` and `production-docs-deploy`. +- `modules/effects/vanixiets/herculesCI/release-packages.nix` — supersedes `preview-release-version` and `production-release-packages`. -Three options were considered for the caching problem that surfaced in the origin artifact (`./logs/vanixiets-2026-04-21-test-cluster-cache-strategy.txt`), where `cd.yaml`'s coarse cache invalidation in `cached-ci-job/action.yaml` caused unrelated-input bumps to re-run the full job set: +This session removed the four corresponding GHA jobs from `.github/workflows/cd.yaml` and trimmed the workflow's surrounding scaffolding to match what's left. -- **Option A — narrow the `flake.lock` cache key via `jq` node extraction.** Recompute the key from just the subset of lock-file nodes relevant to a given job. Rejected outright: `flake.lock`'s node graph is denormalized (follows-resolutions, transitive inputs), and correctly computing a per-job projection is its own correctness problem with no test harness behind it. Brittle in-place of a principled fix. +## What was decided not to migrate, and why -- **Option B — drvPath-derived cache key in `cached-ci-job/action.yaml`.** Compute `nix eval --raw '.#checks...drvPath'` once per job and key `actions/cache` on that. Principled (the drvPath is the canonical content hash for a nix build) but is bespoke GHA machinery that becomes garbage the moment buildbot-nix takes over. Retained as **designated fallback** if Option-C discovery blocks on any single job; otherwise discarded. +`bootstrap-verification` stays in `cd.yaml` on `ubuntu-latest`. +Its semantic property is "clean-host nix bootstrap works" — it tests the bootstrapping path that *makes* nix usable. +A bwrap-isolated effect on magnetite already has nix, with `/nix/store` ro-bound and no root or systemd; running the bootstrap there would be a structurally different and less faithful test of the same property name. +Keeping it on a fresh GHA runner preserves the test's meaning. -- **Option C — migrate `cd.yaml` to buildbot-nix using `writeShellApplication` flake apps + hercules-ci-effects.** Realizes "run only when pure closure changes" as the native semantic of the build system. Chosen direction for the full `cd.yaml` surface. +`test-cluster` stays in `cd.yaml`. +It needs a Docker daemon plus worktree access in ways that don't fit bwrap. +Even if `/var/run/docker.sock` were exposed into the bwrap sandbox, containers spawned by `docker run` execute in the host's mount and network namespaces and don't see bwrap-isolated paths anyway. +The cache-locality argument that motivates effects (warm `/nix/store`, niks3 push) doesn't apply because test-cluster's outputs are pass/fail signals, not store paths. +Two future-work options exist if migration becomes worthwhile — wrap as a NixOS VM `nixosTest`, or run as a buildbot-nix worker step outside bwrap — but neither was pursued in this session. -The caching question is the proximate trigger; the underlying decision is driven by `nix-7v7`, which established self-sovereign build infrastructure on magnetite: niks3 binary cache on Cloudflare R2, buildbot-nix with GitHub + Gitea forge integration, and Gitea self-hosted forge. `buildbot-nix.toml` already configures magnetite's buildbot to evaluate `checks.x86_64-linux` against vanixiets. This epic realizes the CI/CD yield of that infrastructure investment; without it, magnetite evaluates vanixiets checks but does not gate releases. +## cd.yaml refactor outcome -Hercules-ci-effects is available to buildbot-nix as a transitive flake-lock pin only. It is **not** a top-level flake input of vanixiets today, **not** imported as a flake-parts module, and no `herculesCI` / `onPush` / `mkEffect` attribute is defined anywhere in `./modules`. Phase 3 introduces all three. (Source: `flake.nix`, `flake.lock:655-678`.) Confidence: HIGH. +Branch `cd-via-effects` shows the final shape. +The four migrated jobs were deleted: `preview-release-version`, `preview-docs-deploy`, `production-release-packages`, `production-docs-deploy`. -The steady-state target on magnetite (Hetzner, a CX53 instance type) is niks3 + buildbot-nix + Gitea colocated, with hercules-ci-effects enabled and a docker-compatible container runtime provisioned for k3d-running effects. The exact CX53 shape is internally inconsistent in-repo (`modules/nixos/buildbot.nix` sets `cores = 16`; `modules/terranix/hetzner.nix:25-30` documents 16 vCPU / 32 GB / 320 GB; an inline comment in `buildbot.nix:103` says "8 vCPU / 16 GB") and requires a live `nproc` / `free -h` / `df -h` check before capacity claims become load-bearing — noted as a Phase 4 entry condition, not fabricated as a resolved value. +`set-variables` was slimmed. +Only `debug` (consumed by `test-cluster`) and `force-ci` (consumed by `bootstrap-verification`) outputs remain. +Package-discovery steps and dead push/pull_request shell branches were removed. -`cached-ci-job/action.yaml`'s hashing algorithm has been verified (D14): it hashes `flake.lock` in its entirety via a single `git hash-object` call; the `hash-sources` input is a whitespace-separated glob list iterated with `set -f` disabling shell expansion; `**`-containing patterns are expanded by shelling to `find -type f -name ` rather than bash globstar; the action auto-includes the invoking workflow file and itself, and excludes `packages/docs/src/content/docs/notes/*`. The final key is `job-result--<12-char sha256 prefix>` over concatenated `git hash-object` outputs. Confidence: HIGH. This confirms the original cache-coarseness hypothesis and informs Option B fallback design if discovery blocks. +Triggers reduced to `workflow_dispatch:` only — no `push`, no `pull_request`, no `workflow_call`. +Manual dispatch is `gh workflow run cd.yaml --ref [-f job=]`. -## Decision — target architecture +Permissions narrowed from `contents: read` + `deployments: write` to `contents: read` only. -### Module layout +`concurrency.group` simplified to `ci-${{ github.ref }}`. -Four domain-organized subdirectories under `modules/apps/` host migrated job logic: +Vestigial `github.event_name != 'workflow_dispatch'` clauses dropped from the retained jobs' `if:` expressions. -- `modules/apps/cluster/` — k3d local integration and forward-compatible Hetzner production cluster orchestration -- `modules/apps/docs/` — documentation preview/release/deploy (partially present per nix-a8g precedent) -- `modules/apps/release/` — production release-packages -- `modules/apps/bootstrap/` — bootstrap-verification +Net diff: cd.yaml went from 407 lines to 145 lines. -Each app follows the nix-a8g template: `.nix` declares `pkgs.writeShellApplication` with `runtimeInputs` for the hermetic package closure; `.sh` holds the script body ingested via `readFile`. +## Snapshot rollback pattern -Template bifurcation (per nix-a8g extraction): `modules/apps/docs/deploy.nix` uses string-interpolation form `text = "${builtins.readFile ./deploy.sh}"` because it injects nix-computed variables at eval time (`SOPS_SECRETS_FILE`, `DOCS_PAYLOAD`); `release.nix` and `preview-version.nix` use pure `text = builtins.readFile ./release.sh`. Cluster apps requiring injection of nix-computed paths (e.g., `CLUSTER_CONFIG`, `SOPS_AGE_KEY_PATH`) use the interpolation form; otherwise pure readFile. Phase 1 documents this bifurcation as part of the cluster-app template guide. Confidence: HIGH. +Before any edits, the original cd.yaml was copied to `.github/deprecated/cd.yaml` as a frozen 407-line snapshot. +Rollback if effects misbehave is a single step: `cp .github/deprecated/cd.yaml .github/workflows/cd.yaml`. +The deprecated copy stays frozen — it does not track `cd.yaml`'s evolution, it preserves the pre-migration state. -Dual-maintenance between justfile recipe and flake app is convention-only. Justfile recipes wrap flake apps via `nix run .#`; no enforced lint. Indirect safeguards: (a) CI hash-sources pin the coupling so drift surfaces as rebuild during Phase 5; (b) shellcheck at build time catches script-level regressions. Phase 1 documents dual-maintenance as a review responsibility. Confidence: HIGH. +This supersedes the original draft's "archive cd.yaml at end of Phase 6" framing. +The snapshot was taken at the *start* of editing rather than the end, which means the rollback target is the GHA-only state, not a hybrid intermediate. -### Execution model +## Cleanup commits in release.sh and release-packages.nix -Pure data jobs become package-classified derivations where the work is genuinely a nix derivation producing a consumed artifact. The GHA `set-variables` job does **not** survive as a single derivation in the target architecture; its dispatch-variable surface is distributed across three native buildbot-nix / hercules-ci-effects mechanisms rather than centralized in one emitting package. See "Trigger translation" below for the per-variable mapping: `branch` / `rev` / `shortRev` arrive as top-level arguments to each effect via buildbot-effects; `debug` is the `buildbot-effects run --debug` flag; `force-ci` has no analog (every effect run is fresh — there is no GHA-style cache-hit skipping to override); `sanitized_branch` is computed inline inside each effect that needs it; `deploy_enabled` collapses into per-effect `hci-effects.runIf` gating plus `effects_branches` configuration; the `packages` matrix becomes flake-eval-time expansion — one attribute per package under `onPush.default.outputs.effects` or under `packages.x86_64-linux.*`. No synthetic `cd-variables` package exists in the target architecture. Confidence: HIGH. +These are independent of the GHA removals but landed alongside them on the same branch. -Impure jobs become hercules-ci-effects under a **single fixed attribute path**. buildbot-nix reads `flake.outputs.herculesCI(args).onPush.default.outputs.effects` on every evaluation; the literal `default` is not a branch name — it is the one and only attribute path buildbot-nix consumes. Source of truth: `buildbot-nix/buildbot_effects/buildbot_effects/__init__.py:142-159`. Per-branch `onPush.` nodes are allowed by the hercules type system but are ignored by buildbot-nix. Branch-specific gating is expressed two ways, neither of them in the Nix attribute path: +In `modules/apps/release/release.sh`: -- **What runs:** `herculesCI.onPush.default.outputs.effects.` (always the same set per evaluation). -- **When it runs:** `effects_branches = ["main", "release/*", ...]` (glob list) and `effects_on_pull_requests = true|false` in `buildbot-nix.toml`, **always read from the default-branch copy** via `git show origin/:buildbot-nix.toml`. A PR author cannot self-authorize by modifying their PR's toml. Source: `buildbot-nix/buildbot_nix/buildbot_nix/nix_eval.py:596-632`. -- **Within the Nix expression:** `hci-effects.runIf ` gates individual effects at eval time (e.g., `runIf (args.branch == "main")`). +- Dropped unused `SOPS_AGE_KEY` env-var passthrough (no consumer). +- Dropped transitional `GIT_USER_NAME` / `GIT_USER_EMAIL` aliases (no external caller after GHA retirement). +- Updated stale GHA reference in the repo-root resolution comment. -Inter-effect dependencies are **not expressible** at the buildbot-nix surface. Every attribute under `onPush.default.outputs.effects` becomes one independent Triggerable build on the `/run-effect` builder; all effects are triggered in parallel with `waitForFinish=True, haltOnFailure=True, flunkOnFailure=True` (`nix_eval.py:708-729`). The only cross-effect ordering guarantee is "the prior `nix-build` matrix has succeeded"; there is no "effect A before effect B" edge. Former GHA `needs:` edges carrying no data are dropped; edges carrying data collapse to derivation-input references; edges demanding execution ordering are expressed via `runIf` gating on a prior effect's completion-signal derivation or by composing into a single larger effect. Confidence: HIGH. +In `modules/effects/vanixiets/herculesCI/release-packages.nix`: -### Secret pipeline (not a sidecar) +- Dropped unused `dryRunFlag` binding (never threaded into dispatch). +- Extracted `mkReleasePackagesEffect { dryRun ? false }` helper to factor out shared logic between production and rehearsal use. -There is no buildbot-effects sidecar process. `buildbot-effects` is a CLI tool in the worker's Python environment. The end-to-end secret flow for an effect build is: +Net: release.sh 250 to 240 lines; release-packages.nix 183 to 237 lines (growth from helper structure plus comments). -1. `services.buildbot-nix.master.effects.perRepoSecretFiles.":/" = ` declared on the master's NixOS config (option at `buildbot-nix/nixosModules/master.nix:716-732`). -2. Master loads the file as a systemd `LoadCredential` entry. -3. Buildbot reads it as a `SecretInAFile` via `$CREDENTIALS_DIRECTORY`. -4. For a scheduled effect build, the master writes the JSON blob to `../secrets.json` relative to the worker's build directory and invokes `buildbot-effects run --secrets ../secrets.json `. -5. `buildbot-effects` starts a `bwrap` sandbox, bind-mounts the file at `/run/secrets.json`, and sets `HERCULES_CI_SECRETS_JSON=/run/secrets.json`. -6. The effect script reads that env var and parses the JSON to obtain secrets at runtime. +## Rehearsal toggle pattern -Canonical end-to-end example: Harmonia's codecov token, wired at `~/projects/nix-workspace/mic92-clan-dotfiles/machines/eve/modules/buildbot.nix:50-66` and consumed at `~/projects/nix-workspace/harmonia/nix/herculesCI.nix:54-60` with `jq -r '.codecov.data.token // empty' "$HERCULES_CI_SECRETS_JSON"`. Confidence: HIGH. +This is the load-bearing operational knowledge from the session. -clan-infra web01 is the **secret-wiring and niks3/buildbot colocation reference** only. Exhaustive grep of `~/projects/nix-workspace/clan-infra/` for `hercules|effects|mkEffect|onPush|herculesCI` returns no matches beyond the flake-parts URL. web01 does not run hercules effects. It uses two parallel secret systems — `sops.secrets.*` for buildbot forge credentials and `clan.core.vars.generators.*` for niks3 S3 creds, signing key, and API token — both delivered to services via systemd-managed decrypted files on disk. The effects-secrets JSON file on magnetite will follow the same delivery mechanism (clan-vars preferred, consistent with magnetite's existing convention of no sops-nix usage), but its shape (flat JSON dict) and wiring (`perRepoSecretFiles`) are dictated by buildbot-nix upstream, not web01. Confidence: HIGH. +`mkReleasePackagesEffect` takes a `dryRun` parameter. +Production: `effects.release-packages = mkReleasePackagesEffect { dryRun = false; }`. -### Per-job purity mapping +To rehearse a release without remote mutations: flip `dryRun` to `true` on a feature branch, push to magnetite, run -The `set-variables` job is intentionally absent from the table below: as explained in the execution-model paragraph above, its responsibilities (dispatch variables, package matrix, debug/force flags, branch gating) have more natural homes in the buildbot-nix / hercules-ci-effects surface — effect arguments, `runIf` gating, `effects_branches` configuration, and flake-eval-time attribute expansion — rather than as a single synthetic package derivation. See "Trigger translation" below. +``` +buildbot-effects run --branch main --rev ... release-packages +``` -| Job | Classification | Confidence | Rationale | -|---|---|---|---| -| `preview-release-version` | pure → `checks.x86_64-linux.preview-release-` (or `packages.`) | HIGH | `@semantic-release/github` is explicitly filtered from `--plugins`; no `git push`; trap-restored local-only `git update-ref`; `contents: write` permission is vestigial (semantic-release `verifyAuth` requires it even in dry-run); can be kept or dropped | -| `bootstrap-verification` | effect | HIGH | Mutates `~/.config/sops/age/keys.txt`; `make bootstrap` installs nix daemon via the nix-installer + creates `nixbld` users + writes `/etc/nix/nix.conf`; `make setup-user` generates a fresh age key — all outside the nix sandbox. Cannot be subsumed under buildbot's check graph because its job is to test the bootstrapping path that *makes* nix usable | -| `test-cluster` | effect (local-only) | HIGH | Docker / k3d / ephemeral filesystem mutation; no cross-network mutation but still nix-sandbox-external | -| `preview-docs-deploy` | effect | HIGH | `wrangler versions upload` creates a preview alias on Cloudflare Workers | -| `production-docs-deploy` | effect | HIGH | `wrangler versions deploy @100%` against production `infra.cameronraysmith.net` | -| `production-release-packages` | effect | HIGH | semantic-release with `--dry-run=false`, git tag push, GitHub Release creation, `npmPublish: false`; authority via `GITHUB_TOKEN` | +on magnetite, observe semantic-release's full plugin chain run with `--dry-run`, then revert `dryRun` to `false` and commit. +The two transient commits in branch history capture the rehearsal cycle. -### Trigger translation +Under `dryRun = true` the dispatch always calls `RELEASE -- --dry-run` (full production plugin set, suppressed mutations); the stale-rev guard is bypassed (so rehearsal can run against any rev while declaring `--branch main`); distinct log markers (`RELEASE-PACKAGES-ACTION: rehearsal`, `RELEASE-PACKAGE-DRY-RUN-DISPATCH:`) make it visually unambiguous in logs. -Mapping GHA triggers to buildbot-nix / hercules-ci-effects using the two-axis model (Nix attribute path + `buildbot-nix.toml` configuration): +A *persistent* second effect attribute (e.g., `release-packages-dry-run` exposed alongside `release-packages`) was rejected. +buildbot-nix triggers every attribute under `onPush.default.outputs.effects` on every qualifying push, so a persistent rehearsal attribute would auto-fire on every commit to main, producing a no-signal rehearsal run alongside the real one. +The transient toggle pattern keeps the production-effect surface at exactly one attribute. -- **`push` on branches** — effect discovery always happens at `onPush.default.outputs.effects.*`. Execution is gated by the default-branch `buildbot-nix.toml`: default branch always runs effects; other branches run iff their name matches an `effects_branches` glob. -- **`pull_request`** — effects execute iff `effects_on_pull_requests = true` in the default-branch `buildbot-nix.toml`. `checks..*` builds run unconditionally in the Nix sandbox. See "Fork-PR posture" below. -- **`schedule`** — expressed as `herculesCI.onSchedule. = { when = { minute; hour; dayOfWeek; dayOfMonth; }; outputs.effects. = ...; }`. The schema is a structured submodule, **not** a cron string; `dayOfWeek` is a list of `"Mon".."Sun"` translated to buildbot's `0..6`. Missing fields default to deterministic-seeded values to avoid thundering herd. Schedule changes propagate on the next successful default-branch `nix-eval` and trigger a `master.reconfig()`. Source: `buildbot-nix/buildbot_nix/buildbot_nix/scheduled.py`, `buildbot-nix/checks/test-flake/flake.nix`. -- **`workflow_dispatch`** — three substitute surfaces, in priority order: - 1. **CLI over ZeroTier (primary):** `ssh magnetite.zt buildbot-effects run github:cameronraysmith/vanixiets/# [--debug] [--secrets ...]`. The CLI is verified in `buildbot-nix/buildbot_effects/buildbot_effects/cli.py`; subcommands are `list`, `run`, `list-schedules`, `run-scheduled`; flags include `--rev`, `--branch`, `--repo`, `--path`, `--debug`, `--secrets `; flakeref syntax (`github:org/repo/branch#effect`) is supported. The master runs exactly this same command on the `run-effect` builder. Packaging: `just ci-dispatch [flags]` wraps the SSH invocation. Confidence: HIGH. - 2. **Web-UI Rebuild (secondary):** The "Rebuild" button on a prior `run-effect` build gives per-effect re-run granularity at the prior rev. Available only if the effect has already run at least once at the desired rev. The web-UI "Force Build" affordance is wired only to `{project}/nix-eval` — it re-runs the whole evaluation, not a single effect; not a per-effect substitute. - 3. **Thin GHA shim (fallback only):** A `cd-dispatch.yaml` workflow with matching `inputs` that dispatches via buildbot REST or a trailer-parsed commit push. Retained as fallback for any case where operators demand a GitHub UI surface; adds a GHA layer that defeats simplification. +### release.sh's existing `--` passthrough -Path filters (`paths-ignore: '*.md'` is the only one in `cd.yaml` and exists at workflow level, not per-job) become derivation-input scoping — restricting a derivation's `src` to the relevant subtree via `lib.fileset.*`. For content-scoped `runIf` gating, hash path content and compare in the effect declaration. +The argument parser in `release.sh` already routes `--` followed by additional args into `extra_args`, which is passed verbatim to semantic-release. +So `"$RELEASE" "$pkg_path" -- --dry-run` works without any release.sh changes — this is what the rehearsal dispatch uses. +No `EFFECT_DRY_RUN` env var or other plumbing was needed. -Arguments passed to effects: buildbot-effects passes `{ name, branch, ref, tag, rev, shortRev, remoteHttpUrl, primaryRepo }` at top level, with `primaryRepo` containing the same fields. `ref` is always `null` (TODO in upstream). Fields that hercules-ci-agent natively provides (`owner`, `remoteSshUrl`, `webUrl`, `forgeType`) are **not** set by buildbot-effects; accessing them throws under the hercules flake-module unless effects are written to degrade gracefully. Effect scripts must only rely on the fields above. +### Stale-rev guard -### Fork-PR posture +For the production attribute (`dryRun = false`), the effect aborts before semantic-release runs if local HEAD differs from origin/main HEAD. +This means a manual `buildbot-effects run --branch main` against any non-main rev safely aborts with `RELEASE-CLONE-STALE`. +The actual release fires only after fast-forward of main from `cd-via-effects`. -buildbot-nix has no author/contributor allowlist for PR builds. The PR scheduler matches `category="pull"` unconditionally; `GitLocalPrMerge` fetches fork HEAD via the base-repo URL (`refs/pull//head`) and merges as normal. `userAllowlist`/`repoAllowlist` filter which *repositories buildbot manages*, not which PRs it accepts. Source: `buildbot-nix/buildbot_nix/buildbot_nix/project_config.py:85-99`, `common.py:116-150`. +## Stage 3b verification result -Under buildbot-nix defaults (`effects_on_pull_requests = false`), fork PRs receive no effect-secrets: the effects builder returns `util.SKIPPED`, and `checks..*` runs in the Nix sandbox with no wired secrets. If the flag is flipped to `true`, fork PRs receive the full `effects_per_repo_secrets` JSON with **no author allowlist, no fork-vs-same-repo differentiation, no differential privilege, and no Nix sandbox** — effects run as impure shell commands on the worker with the secrets file on disk. The upstream README (`buildbot-nix/README.md:184-190`) explicitly warns this is exploitable. Vanixiets currently has `effects.perRepoSecretFiles = {}` and the flag unset. +On 2026-04-26 the production code path was verified end-to-end via the rehearsal toggle. -**Recommended default: Posture A.** `effects_on_pull_requests = false`. Preview-* effects run only on default-branch merges. Contributors see `checks..*` feedback (safe, Nix-sandboxed) on their PRs but no contributor-triggered preview-deploy. +`[@semantic-release/github] - Verify GitHub authentication` returned `Allowed to push to the Git repository`. +This confirms the fine-grained PAT in the `vanixiets-effects-secrets` clan-vars generator has Contents: Read+Write on `cameronraysmith/vanixiets` and authenticates correctly. -**Named future option: Posture B.** Re-push contributor PR commits onto base-repo `preview/` branches; add `effects_branches = ["preview/*"]` so secrets reach only writers with base-repo push access. Mirrors GHA's `pull_request_target` trust boundary. Adoption contingent on contributor preview-feedback becoming a priority. +The production plugin chain loaded fully: changelog, github (publish + addChannel), commit-analyzer, release-notes-generator, semantic-release-major-tag. -## Phase structure +semantic-release-monorepo correctly filtered 76 commits to the 2 touching `packages/docs/`, recognized `fix(docs):` as patch-tier, and computed next version `0.5.1`. -**Phase 1 — `writeShellApplication` foundation across four domains.** Per-job script bodies plus their transitive just/shell recipes are converted to the nix-a8g template. Justfile recipes rewrite as thin wrappers invoking `nix run .#`. Phase-1 conversion set (from inventory research 03): `list-packages-json`, `k3d-integration-ci`, `k3d-full`, `k3d-bootstrap-secrets`, `k3d-configure-dns`, `k3d-wait-ready`, `k3d-wait-argocd-sync`, `k3d-test-coverage`, `nixidy-build`, `nixidy-bootstrap`, `nixidy-sync`, `nixidy-push`, and `scripts/k3d-test-coverage.sh`. Scope explicitly excludes composite-action and reusable-workflow disappearance work — that belongs to Phase 6. No production cutover; existing GHA still runs. +All `Skip ... in dry-run mode` markers fired at correct step boundaries (prepare, publish, success). +Effect exited 0. -**Phase 2 — Per-job branch-point decision.** Per `cd.yaml` job, confirm Option-C viability. Jobs may diverge: `test-cluster` may proceed to C while `release-packages` awaits secret-pipeline work. Per-job decision, not global. +## PAT identity transition (parity gap with old GHA path) -**Phase 3 — Effects wiring.** Entry conditions: (a) add `hercules-ci-effects` as a top-level flake input with `inputs.flake-parts.follows = "flake-parts"; inputs.nixpkgs.follows = "nixpkgs";`; (b) introduce a flake-level module importing `inputs.hercules-ci-effects.flakeModule` under the deferred-module composition; (c) declare at least an empty `herculesCI = { ... }: { onPush.default.outputs.effects = { }; }`. Then per job confirmed in Phase 2, populate `herculesCI.onPush.default.outputs.effects.` and gate branches via `effects_branches` in `buildbot-nix.toml`. Overlaps with Phase 5. +The old path used the GHA implicit `GITHUB_TOKEN`: identity `github-actions[bot]`, ephemeral 1-hour token, scope determined by job-level `permissions:` block. -**Phase 4 — Worker provisioning.** Entry conditions: +The new path uses a fine-grained PAT from the `vanixiets-effects-secrets` clan-vars generator: identity is the PAT owner (`cameronraysmith`), long-lived (rotation manual), with permissions scoped by what was selected at PAT creation time rather than by per-job `permissions:`. -1. **Live CX53 capacity confirmation.** `ssh magnetite.zt 'nproc && free -h && df -h /'`. Reconcile with `modules/terranix/hetzner.nix:25-30` and `modules/nixos/buildbot.nix` `cores = 16`. The stale inline comment in `buildbot.nix:103` ("CX53 (8 vCPU, 16 GB RAM)") is either corrected or confirmed; capacity claims downstream become load-bearing only after live verification. -2. **Docker runtime.** magnetite currently runs only `virtualisation.podman` (for gitea-actions-runner, storage at `zroot/root/podman`). k3d effects need docker. Enable `virtualisation.docker.enable = true;` and provision a dedicated ZFS dataset `zroot/root/docker` in `modules/machines/nixos/magnetite/disko.nix` (mirroring the podman pattern). Validate that the docker socket is reachable by the buildbot worker user. -3. **`perRepoSecretFiles` wiring.** Add a clan-vars generator emitting the effects-secret JSON blob (shape: `{ "secretName": "value", ... }` — flat dict consumable as `HERCULES_CI_SECRETS_JSON`). Wire `services.buildbot-nix.master.effects.perRepoSecretFiles."github:cameronraysmith/vanixiets" = config.clan.core.vars.generators.buildbot-effects-vanixiets.files."secrets.json".path;`. -4. **Optional cgroup isolation.** `systemd.slices.effects` with `MemoryMax` and `CPUQuota` caps, attaching buildbot-effects runs to that slice; reduce `gitea-actions-runner.numInstances` during migration window if contention surfaces. +Releases will appear under the PAT-owner identity rather than the bot. -**Phase 5 — Per-job parity validation.** Entry condition: per-job parity-N threshold locked per the rollback rubric in "Resolutions" (D9 row). Both GHA and buildbot-nix paths run simultaneously. +Branch-protection risk is low because the active plugin list does **not** include `@semantic-release/git` (no commit-back to main). +Only tag pushes occur, and tags aren't protected by default branch rules. +`@semantic-release/git`'s presence in `devDependencies` only is intentional and not cruft — it remains available if commit-back behavior is ever wanted, but isn't loaded by the active plugin chain. -- Reversible jobs (`bootstrap-verification`, `preview-release-version`, `preview-docs-deploy`, `test-cluster`): N = 2–3. (`set-variables` is absent from the migration target per "Execution model" / "Per-job purity mapping" above.) -- Irreversible jobs (`production-docs-deploy`, `production-release-packages`): N ≥ 5 with mandatory rollback rehearsal at least once. Dual-writer mitigation is mandatory during parity: keep buildbot's semantic-release in `dry-run: true` so only GHA publishes; flip to `dry-run: false` at cutover. Symmetric approach for tag push and production deploys. +## Verification staircase -Compared: success/failure consistency, timing, log quality, secret handling, observability. Abort parity if divergence rate exceeds 20% within the first 10 runs per job or if any single divergence occurs on an irreversible job. +Four stages, each exercising strictly more of the production code path: -**Phase 6 — Per-job sunset + `cd.yaml` archival + composite-action/reusable-workflow disappearance + drift cleanup.** +1. `nix run .#preview-version` from a laptop — preview-version mechanics in isolation. +2. `buildbot-effects run --branch refs/pull/N/merge ... release-packages` on magnetite — the bwrap surface plus the PR-clone preamble; dispatches preview-version per the PR-arm logic. +3. `buildbot-effects run --branch main ... release-packages` with the rehearsal toggle active — the full production code path with `--dry-run`. This is the load-bearing test; Stage 3b above documents its outcome. +4. Actual merge — the only stage that exercises buildbot's onPush trigger plumbing end-to-end. Pending at time of writing. -Disappearance cluster splits into two sub-clusters: +## Conflicts with the original draft -- **Disappears without replacement** (exactly two composite actions per the inventory): - - `.github/actions/cached-ci-job/action.yaml` — subsumed by the content-addressed nix store + binary cache. - - `.github/actions/setup-nix/action.yml` — buildbot-nix workers have nix pre-provisioned. -- **Artifact disappears, logic migrates:** - - `.github/workflows/test-cluster.yaml` — logic migrates to `modules/apps/cluster/*.{nix,sh}` plus the `test-cluster` effect definition; the workflow file is removed. - - `.github/workflows/deploy-docs.yaml` — logic migrates to `modules/apps/docs/deploy.{nix,sh}` (already present) plus the `preview-docs-deploy` and `production-docs-deploy` effects. - - `.github/workflows/package-release.yaml` — logic migrates to `modules/apps/release/*.{nix,sh}` plus the `production-release-packages` effect. +The original draft proposed a `just ci-dispatch ` recipe wrapping the SSH+CLI invocation. +This was never built; manual dispatch in the final state is the raw `ssh magnetite.zt buildbot-effects run ...` command. +The wrapper recipe could still be added, but is not present. -Per-job removal from `cd.yaml` after parity threshold met. Final archival of `cd.yaml` once all migrated jobs are confirmed — `cd.yaml` preserved in `.github/deprecated/` per existing precedent, enabling rollback by un-archiving individual jobs. +The original draft listed seven jobs and described a phased migration with parity windows. +The final state migrated four (the docs and release pairs) and explicitly excluded `bootstrap-verification` and `test-cluster` for the structural reasons described above. +`set-variables` was retained in trimmed form rather than removed. +Phase 5 ("per-job parity validation") was not run as a separate observational period; verification happened via the rehearsal toggle (Stage 3b) prior to merge rather than via dual-running both paths in production. -**Drift and dead-surface cleanup (Phase 6):** - -- `scripts/preview-version.sh` (legacy root copy, 8684 bytes, not referenced by any active workflow; consumed only by `package.json:18` and by the deprecated `.github/deprecated/*.yaml` hash-sources lines) — delete. -- `package.json:18` (`"preview-version": "./scripts/preview-version.sh"`) — repoint to `nix run .#preview-version` or drop. -- `.github/deprecated/ci-nix-fast-build.yaml` and `.github/deprecated/ci-pre-nix-check.yaml` — drop `scripts/preview-version.sh` references from `hash-sources` strings, or leave as historical if the deprecated files are themselves earmarked for deletion. -- Documentation drift: update `packages/docs/src/content/docs/about/contributing/semantic-release-preview.md` to reference `nix run .#preview-version` and `just preview-version `. (Path differs from earlier ADR drafts that said `docs/content/.../` — the actual path is `packages/docs/src/content/docs/about/contributing/semantic-release-preview.md`.) -- `cd.yaml` `workflow_call` inputs `target_configs`, `cache_control`, `job_selection` — declared, never referenced anywhere in the file. Dead input surface; strip during migration. -- `test-cluster.yaml` `env.CACHIX_BINARY_CACHE: cameronraysmith` — set, never consumed by any action. Dead env var; strip. -- `inputs.job` selector value `'docs-deploy'` vs actual job name `production-docs-deploy` — normalize during migration (rename selector to `production-docs-deploy` or document the alias). -- `permissions: contents: write` on `preview-release-version` — vestigial (semantic-release `verifyAuth` requires it even in dry-run). Document if kept; drop if the dry-run plugin filter eliminates the dependency. - -### Exit criteria (mutually exclusive) - -- **Fully-migrated.** All 7 jobs migrated. `cd.yaml` archived to `.github/deprecated/`. Composite actions and reusable workflows disappeared per Phase 6 sub-clusters. Worker provisioning complete. Rollback recoverable via un-archiving individual job definitions. -- **Hybrid-stable.** Subset migrated; remainder stays on GHA indefinitely due to blocking outcomes or coordination requirements. `cd.yaml` active for the GHA residue. Revisit trigger: quarterly review of still-on-GHA jobs against the blocker that kept them there. Prevents drift into indefinite hybrid. -- **Discovery-blocks.** A hard blocker (e.g., a job whose secret model cannot safely migrate, or a k3d-on-docker incompatibility) surfaces during Phase 3 or 4. Fall back to Option B scoped to affected jobs — `writeShellApplication` conversion from Phase 1 still lands as independently valuable infrastructure; Option B cache-key extension applied to `cached-ci-job/action.yaml` completes the fallback for the residue. - -## Resolutions - -Phase 0 is folded into Phase 3/4/5 entry conditions; the table below summarizes discovery items from the prior draft, their resolution, and the research report that resolved each. Reports are under `.factory/research/adr-001-validation/`. - -| Item | Status | Resolution | Reference | -|---|---|---|---| -| **D1** — magnetite capacity / Docker / k3d | OPEN (live check) | CX53 shape internally inconsistent in-repo; requires `ssh magnetite.zt 'nproc && free -h && df -h'` before load-bearing use. Docker not currently enabled (only podman); Phase 4 adds `virtualisation.docker.enable = true` + dedicated ZFS dataset. | research/02 | -| **D2** — hercules-ci-effects + buildbot-nix integration | RESOLVED (HIGH) | Attribute path is fixed at `herculesCI.onPush.default.outputs.effects.`; per-branch paths are ignored. Branch gating via `effects_branches` and `effects_on_pull_requests` in `buildbot-nix.toml` (read from default branch). Secrets via `perRepoSecretFiles` → JSON file → `HERCULES_CI_SECRETS_JSON` inside bwrap sandbox. hercules-ci-effects is currently only a transitive flake-lock pin on vanixiets; Phase 3 entry adds it as a top-level input + flake-parts module. | research/01, 02 | -| **D5a** — per-job secret inventory | RESOLVED (HIGH) | `set-variables`: none. `preview-release-version`: declared `contents: write` but plugin filter removes `GITHUB_TOKEN` consumption. `preview-docs-deploy`: `SOPS_AGE_KEY` (decrypts Cloudflare creds from `secrets/shared.yaml`). `bootstrap-verification`: none. `test-cluster`: `SOPS_AGE_KEY` (for k3d `sops-age-key` Kubernetes secret bootstrap). `production-release-packages`: explicit `SOPS_AGE_KEY` + implicit `GITHUB_TOKEN`. `production-docs-deploy`: `SOPS_AGE_KEY`. | research/03 | -| **D5b** — fork-PR security posture | RESOLVED (HIGH) | No author allowlist, no fork-vs-same-repo differentiation, no Nix sandbox for effects. `effects_on_pull_requests = false` default keeps fork PRs safe. Posture A (keep default) chosen; Posture B (preview/* base-repo branches) named as upgrade path. | research/04 | -| **D5c** — secret pipeline design | RESOLVED (HIGH) | "Sidecar" framing was incorrect; actual model is `perRepoSecretFiles` → systemd `LoadCredential` → JSON file in bwrap. Magnetite follows clan-vars convention (no sops-nix yet) to generate the JSON blob. | research/01, 02 | -| **D7a** — trigger-surface mapping | RESOLVED (HIGH) | See "Trigger translation" section. Two-axis model: Nix attribute path (`onPush.default.outputs.effects`) + `buildbot-nix.toml` config (`effects_branches`, `effects_on_pull_requests`) + `onSchedule..when` structured submodule + CLI for manual dispatch. | research/01, 05 | -| **D7b** — path-filter audit | RESOLVED (HIGH) | cd.yaml has a single workflow-level `paths-ignore: '*.md'`; no job-level path filters. Translates to `lib.fileset.*` scoping of derivation `src` where desired, or is dropped as trivially handled by nix content-addressing. | research/03 | -| **D7c** — workflow_dispatch substitute | RESOLVED (HIGH) | `buildbot-effects run` is verified: subcommands `list`, `run`, `list-schedules`, `run-scheduled`; flags `--rev`, `--branch`, `--repo`, `--path`, `--debug`, `--secrets`; flakeref syntax supported. Web-UI "Force Build" is only wired to `nix-eval` (whole-evaluation); per-effect "Rebuild" requires prior run. CLI over ZeroTier is primary; Rebuild is secondary; thin GHA shim is fallback-only. Confidence upgraded MEDIUM → HIGH. | research/01, 05 | -| **D8a** — per-job purity confirmation | RESOLVED (HIGH) | See "Per-job purity mapping" table. | research/03 | -| **D8b** — bootstrap-verification rubric | RESOLVED (HIGH) | EFFECT. `make bootstrap` installs nix daemon + `nixbld` users + systemd/launchd units; `make setup-user` writes `~/.config/sops/age/keys.txt`. By construction cannot be a buildbot check; either remains a minimal GHA job gated on bootstrap-relevant paths, or becomes an effect that provisions and tests a fresh worker. | research/03 | -| **D8c** — preview-release-version tag-push resolution | RESOLVED (HIGH) | PURE. Both `scripts/preview-version.sh` and `modules/apps/docs/preview-version.sh` operate in a throwaway worktree with trap-restored local-only `git update-ref`; `@semantic-release/github` plugin is explicitly filtered from `--plugins`; no `git push` anywhere. Classification is check (or package), not pure-effect or split. | research/03 | -| **D8d** — composite-action + reusable-workflow inventory | RESOLVED (HIGH) | Exactly two composite actions: `.github/actions/cached-ci-job/action.yaml` and `.github/actions/setup-nix/action.yml`. Both disappear without replacement. Reusable workflows: `deploy-docs.yaml`, `test-cluster.yaml`, `package-release.yaml` — logic migrates to `modules/apps/`. | research/03 | -| **D9** — rollback posture | RESOLVED (HIGH) | Two classes. Reversible (parity N = 2–3; fast-revert by un-archiving from `.github/deprecated/`; trigger rollback at 1–2 consecutive divergences). Irreversible (parity N ≥ 5; mandatory rollback rehearsal; dual-writer mitigation with semantic-release `dry-run: true` during parity). Automated rollback triggers depend on ntfy observability (D12); without it, detection is eyeball-only. | research/05 | -| **D10** — Ironstar history comparison | DEFERRED (out of epic scope per revision) | Research did not cover. Accretion-vs-load-bearing audit of `cd.yaml` patterns is independent of the migration mechanics and can be deferred. | — | -| **D12** — observability transition | RESOLVED (HIGH) | Tier 1 (per-effect GitHub Commit Status via `FilteredGitHubStatusPush` + `nix_status_generator.py`; each effect posts its own context `effects.`) is sufficient for mission scope. Tier 2 (ntfy `HttpStatusPush` → `https://ntfy.zt/vanixiets-ci-fail` on default-branch failures) is deferred as an operational improvement post-mission. matrix-synapse further deferred behind ntfy. Gap vs GHA: implicit email-on-failure has no default replacement within mission scope; subscribers rely on GitHub Commit Status notifications until ntfy is wired. | research/05 | -| **D13** — cost posture | OPEN (live check) | Depends on D1 CX53-shape confirmation. CX53 at public Hetzner pricing ≈ €14/month; R2 storage ≈ $7.5/month at 500 GB. If live `nproc` shows 8 vCPU / 16 GB, headroom for concurrent k3d effects is tight and CX63 or CCX33 upgrade becomes a consideration. | research/02 | -| **D14** — cached-ci-job hashing | RESOLVED (HIGH) | Hashes `flake.lock` whole via single `git hash-object`; `hash-sources` is a whitespace-separated glob list iterated with `set -f`; `**` expanded via `find -type f -name `; auto-includes workflow file + the action itself; excludes `packages/docs/src/content/docs/notes/*`; key = `job-result--<12-char sha256 prefix>`. | research/03 | - -## Organizational shape - -Single parent epic with internal clustering. Rejected alternative: parent epic + child epics per domain. - -Justification: Phase 4 worker provisioning is cross-cutting across all effectful jobs; Phase 2 per-job branch-point decisions need a single coordination view; a unified "how is the migration going" view matters for duration tracking; dependency coordination via edges is cheaper than epic-metadata overhead. - -Estimated duration: 6–12 weeks. With ~90% of discovery resolved (see Resolutions table), the range is anchored on Phase-1/3/4/5 execution time, not discovery outcomes. Phase 4 live CX53 verification may revise the upper bound if capacity forces a server upgrade. - -Internal clusters: - -- Cluster-domain app conversion (`modules/apps/cluster/`) — 13 recipes/scripts -- Docs-domain app conversion (completion + drift cleanup) -- Release-domain app conversion (`modules/apps/release/`) -- Bootstrap-domain app conversion (`modules/apps/bootstrap/`) -- Flake-level effects wiring (hercules-ci-effects input + flakeModule + `herculesCI` attribute + per-effect declarations) -- Worker provisioning (magnetite NixOS module: docker + ZFS dataset + `perRepoSecretFiles` + optional cgroup isolation; ntfy reporter deferred post-mission) -- Parity validation (per confirmed job, per N-run rubric) -- Sunset + disappearance (per-job removal, composite-action and reusable-workflow deletion, `cd.yaml` archival, drift cleanup per Phase 6 touchpoints) - -## Consequences - -Positive: - -- Self-sovereign CI execution aligned with `nix-7v7` investment. -- Enables `nix-7v7` infrastructure to gate releases, not just evaluate checks. -- Per-effect granular caching via native hercules semantics; no bespoke GHA cache machinery. -- Each effect posts its own GitHub Commit Status context (`effects.`) — observability contract for PR authors is preserved and arguably sharper. -- Domain-organized app layout supports forward-compatible Hetzner production cluster migration. -- `writeShellApplication` + `.sh` sidecar decouples shellcheck hygiene from nix string-templating. - -Negative / risks: - -- Magnetite becomes CI single-point-of-failure. **Mitigation:** `cd.yaml` is archived in `.github/deprecated/` during Phase 6, not deleted; un-archiving individual jobs restores the GHA fallback path without code rewrites. Rollback acceptance criteria for irreversible jobs (D9) include a mandatory rehearsal. -- Secret-pipeline migration has security-adjacent complexity — fork-PR secret exposure is a real footgun if `effects_on_pull_requests` is ever flipped. Posture A (default off) is the explicit guardrail. -- For irreversible jobs (`production-release-packages`, `production-docs-deploy`), the revert path cannot un-publish artifacts; it can only return publish authority to GHA for subsequent runs. **Mitigation:** dual-writer rule during Phase 5 parity — buildbot's semantic-release runs with `dry-run: true` so only GHA publishes until cutover. -- User-facing observability shifts from GHA UI to `buildbot.scientistexperience.net` with per-effect GitHub Commit Status contexts as the primary feedback channel on PRs. Implicit GHA email-on-failure has **no default replacement within mission scope** (ntfy Tier 2 is deferred as a post-mission operational improvement); subscribers rely on GitHub Commit Status notifications until the ntfy reporter is wired. -- Transient developer-ergonomics cost during hybrid state — PRs show both GHA and buildbot commit-status contexts until Phase 6 completes per job. -- `workflow_dispatch` ergonomics change (CLI substitute instead of GitHub UI). ZT access is required to trigger effects manually; non-admin contributors cannot force-run an effect. -- Effect debug UX is strictly more powerful (`buildbot-effects run --debug`) but strictly less ergonomic than `action-tmate@v3` for non-ZT contributors. Permanent ergonomic cost. -- Fixed-cost posture shift: magnetite (CX53) supersedes GHA's effectively-free public-repo CI capacity. Absolute cost minor at Hetzner pricing, but pending D1/D13 live confirmation, capacity headroom alongside niks3 + buildbot + Gitea + 2 gitea-actions-runner podman instances is not yet quantitatively validated. -- Drift between `.sh` sidecars and legacy copies (`scripts/preview-version.sh`, `package.json:18`, docs reference, deprecated workflow hash-sources) requires active Phase 6 cleanup per enumerated touchpoints. - -## Explicit deferrals - -- Individual issue bodies and beads IDs — not ADR content. -- Ironstar-style accretion-vs-load-bearing audit (D10) — out of epic scope per revision. -- Per-phase duration estimates beyond the top-level 6–12 week range. -- Posture B adoption (fork-PR preview via `preview/*` base-repo branches) — deferred until contributor preview-feedback becomes a priority. -- ntfy `HttpStatusPush` reporter wiring — operational steady-state concern, deferred as post-mission improvement. Tier 1 (per-effect GitHub Commit Status via `FilteredGitHubStatusPush`) plus PR-based validation (`gh pr checks` + `buildbot-logs`) covers the mission-scope dev loop; the email-on-failure gap is acknowledged under Consequences. -- matrix-synapse reporter wiring — deferred behind ntfy Tier 2 (which is itself post-mission). -- Phase 5 per-job parity validation is a wall-clock observational activity that begins after all mission features complete; the mission does **not** gate on parity confirmation. Parity windows run on calendar time, not on the mission's feature-completion critical path. -- `cd.yaml` archival to `.github/deprecated/`, deletion of the reusable workflows (`deploy-docs.yaml`, `test-cluster.yaml`, `package-release.yaml`), and deletion of the composite actions (`cached-ci-job/action.yaml`, `setup-nix/action.yml`) — deferred until post-mission parity observation completes (Phase 6 depends on Phase 5 exit). The mission as currently scoped stops before Phases 5 and 6 run their full course, though both phases remain the ADR's target end-state. -- Binary-cache poisoning analysis for fork-PR-triggered niks3 uploads — content-addressing makes direct collision attacks infeasible, but trust-in-cache-contents is out of scope. +The original draft's exit-criteria block named "Fully-migrated" / "Hybrid-stable" / "Discovery-blocks" outcomes. +The final state corresponds to "Hybrid-stable" with two jobs explicitly held back on architectural grounds rather than for coordination cost. ## References -Codebase: +Final-state files on branch `cd-via-effects`: -- GHA workflow authoritative source: `/Users/crs58/projects/nix-workspace/vanixiets/.github/workflows/cd.yaml` -- Reusable workflows: `.github/workflows/test-cluster.yaml`, `.github/workflows/deploy-docs.yaml`, `.github/workflows/package-release.yaml` -- Composite actions: `.github/actions/cached-ci-job/action.yaml`, `.github/actions/setup-nix/action.yml` -- nix-a8g precedent template: `modules/apps/docs/` -- Hermetic-deps derivation shape: `pkgs/by-name/vanixiets-docs-deps/package.nix` -- Legacy preview-version drift: `scripts/preview-version.sh`, `package.json:18` -- Docs-reference drift: `packages/docs/src/content/docs/about/contributing/semantic-release-preview.md` -- Buildbot worker NixOS module: `modules/nixos/buildbot.nix` -- niks3 NixOS module: `modules/nixos/niks3.nix` -- Buildbot-nix project config: `buildbot-nix.toml` -- Magnetite machine config: `modules/machines/nixos/magnetite/default.nix`, `modules/machines/nixos/magnetite/disko.nix` -- Terranix shape declaration: `modules/terranix/hetzner.nix:25-30` -- cinnabar ntfy deployment: `modules/machines/nixos/cinnabar/ntfy.nix` +- `.github/workflows/cd.yaml` (145 lines, workflow_dispatch only) +- `.github/deprecated/cd.yaml` (407-line frozen snapshot for rollback) +- `modules/effects/vanixiets/herculesCI/release-packages.nix` +- `modules/effects/vanixiets/herculesCI/deploy-docs.nix` +- `modules/apps/release/release.sh` +- `modules/apps/release/release.nix` +- `modules/apps/release/preview-version.sh` -Research reports (this revision's evidence base): +Original draft commit: `d629aa7e3 docs(notes): draft ADR-001 cd.yaml to buildbot migration (v0)`. + +Research reports from the original ADR draft (still useful for buildbot-nix mechanics; do not represent the final migration scope): - `.factory/research/adr-001-validation/01-hercules-effects-buildbot-nix-mechanics.md` - `.factory/research/adr-001-validation/02-magnetite-state-and-web01-pattern.md` - `.factory/research/adr-001-validation/03-cd-yaml-inventory.md` - `.factory/research/adr-001-validation/04-fork-pr-security.md` - `.factory/research/adr-001-validation/05-ops-triggers-observability-rollback.md` - -External sources: - -- buildbot-nix upstream: `~/projects/nix-workspace/buildbot-nix/` - - Effects CLI: `buildbot_effects/buildbot_effects/cli.py` - - Effects flake-attr reader: `buildbot_effects/buildbot_effects/__init__.py:142-159` - - Effects dispatch + gating: `buildbot_nix/buildbot_nix/nix_eval.py:596-632`, `708-729` - - Master module effects options: `nixosModules/master.nix:716-732`, `1010-1040` - - Scheduled effects: `buildbot_nix/buildbot_nix/scheduled.py`, `models.py:ScheduleWhen` - - Commit-status generator: `buildbot_nix/buildbot_nix/nix_status_generator.py` - - Security warning (fork-PR): `README.md:184-190` -- hercules-ci-effects upstream: `~/projects/nix-workspace/hercules-ci-effects/` - - flakeModule: `flake-modules/herculesCI-attribute.nix` - - `runIf`: `effects/default.nix:47-63` -- Reference implementation (secret-wiring + niks3/buildbot colocation only — **not** an effects reference): `~/projects/nix-workspace/clan-infra/` -- End-to-end effects-secrets example (codecov token): `~/projects/nix-workspace/mic92-clan-dotfiles/machines/eve/modules/buildbot.nix:50-66`, `~/projects/nix-workspace/harmonia/nix/herculesCI.nix:54-60` -- Origin artifact (caching question that triggered the epic): `./logs/vanixiets-2026-04-21-test-cluster-cache-strategy.txt` - -Skills: - -- `~/.claude/skills/preferences-nix-ci-cd-integration/SKILL.md` -- `~/.claude/skills/preferences-nix-checks-architecture/SKILL.md` -- `~/.claude/skills/preferences-secrets/SKILL.md` -- `~/.claude/skills/preferences-adaptive-planning/SKILL.md` -- `~/.claude/skills/stigmergic-convention/SKILL.md` From 943436b8b2472fd6a6206ffe4db2003d62829ffd Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 03:02:37 -0400 Subject: [PATCH 70/77] chore(notes): remove ephemeral cd-to-buildbot migration plan The plan's final state remains recoverable from this branch's git history. The ci-cd/decisions/ subdir is removed if it has no other contents. --- .../ADR-001-cd-to-buildbot-migration.md | 179 ------------------ 1 file changed, 179 deletions(-) delete mode 100644 docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md diff --git a/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md b/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md deleted file mode 100644 index 5bb11a4be..000000000 --- a/docs/notes/development/ci-cd/decisions/ADR-001-cd-to-buildbot-migration.md +++ /dev/null @@ -1,179 +0,0 @@ -# cd.yaml to herculesCI effects migration — final-state notes - -This document was originally drafted on 2026-04-21 as an ADR (commit `d629aa7e3`), then revised on 2026-04-22 to incorporate discovery resolutions. -On reflection it was misclassified: the work it describes is concrete and bounded, not an architecture-level decision worth ADR ceremony. -What follows is the final state of the migration on branch `cd-via-effects` as of 2026-04-26, retained as a working note for value extraction — patterns and rationale worth preserving — rather than as an ADR. -The file may be deleted before the PR merges to main; the final state remains in branch git history. - -## What was migrated - -Two effect modules already existed prior to this session and now own the CD jobs they correspond to: - -- `modules/effects/vanixiets/herculesCI/deploy-docs.nix` — supersedes the GHA jobs `preview-docs-deploy` and `production-docs-deploy`. -- `modules/effects/vanixiets/herculesCI/release-packages.nix` — supersedes `preview-release-version` and `production-release-packages`. - -This session removed the four corresponding GHA jobs from `.github/workflows/cd.yaml` and trimmed the workflow's surrounding scaffolding to match what's left. - -## What was decided not to migrate, and why - -`bootstrap-verification` stays in `cd.yaml` on `ubuntu-latest`. -Its semantic property is "clean-host nix bootstrap works" — it tests the bootstrapping path that *makes* nix usable. -A bwrap-isolated effect on magnetite already has nix, with `/nix/store` ro-bound and no root or systemd; running the bootstrap there would be a structurally different and less faithful test of the same property name. -Keeping it on a fresh GHA runner preserves the test's meaning. - -`test-cluster` stays in `cd.yaml`. -It needs a Docker daemon plus worktree access in ways that don't fit bwrap. -Even if `/var/run/docker.sock` were exposed into the bwrap sandbox, containers spawned by `docker run` execute in the host's mount and network namespaces and don't see bwrap-isolated paths anyway. -The cache-locality argument that motivates effects (warm `/nix/store`, niks3 push) doesn't apply because test-cluster's outputs are pass/fail signals, not store paths. -Two future-work options exist if migration becomes worthwhile — wrap as a NixOS VM `nixosTest`, or run as a buildbot-nix worker step outside bwrap — but neither was pursued in this session. - -## cd.yaml refactor outcome - -Branch `cd-via-effects` shows the final shape. -The four migrated jobs were deleted: `preview-release-version`, `preview-docs-deploy`, `production-release-packages`, `production-docs-deploy`. - -`set-variables` was slimmed. -Only `debug` (consumed by `test-cluster`) and `force-ci` (consumed by `bootstrap-verification`) outputs remain. -Package-discovery steps and dead push/pull_request shell branches were removed. - -Triggers reduced to `workflow_dispatch:` only — no `push`, no `pull_request`, no `workflow_call`. -Manual dispatch is `gh workflow run cd.yaml --ref [-f job=]`. - -Permissions narrowed from `contents: read` + `deployments: write` to `contents: read` only. - -`concurrency.group` simplified to `ci-${{ github.ref }}`. - -Vestigial `github.event_name != 'workflow_dispatch'` clauses dropped from the retained jobs' `if:` expressions. - -Net diff: cd.yaml went from 407 lines to 145 lines. - -## Snapshot rollback pattern - -Before any edits, the original cd.yaml was copied to `.github/deprecated/cd.yaml` as a frozen 407-line snapshot. -Rollback if effects misbehave is a single step: `cp .github/deprecated/cd.yaml .github/workflows/cd.yaml`. -The deprecated copy stays frozen — it does not track `cd.yaml`'s evolution, it preserves the pre-migration state. - -This supersedes the original draft's "archive cd.yaml at end of Phase 6" framing. -The snapshot was taken at the *start* of editing rather than the end, which means the rollback target is the GHA-only state, not a hybrid intermediate. - -## Cleanup commits in release.sh and release-packages.nix - -These are independent of the GHA removals but landed alongside them on the same branch. - -In `modules/apps/release/release.sh`: - -- Dropped unused `SOPS_AGE_KEY` env-var passthrough (no consumer). -- Dropped transitional `GIT_USER_NAME` / `GIT_USER_EMAIL` aliases (no external caller after GHA retirement). -- Updated stale GHA reference in the repo-root resolution comment. - -In `modules/effects/vanixiets/herculesCI/release-packages.nix`: - -- Dropped unused `dryRunFlag` binding (never threaded into dispatch). -- Extracted `mkReleasePackagesEffect { dryRun ? false }` helper to factor out shared logic between production and rehearsal use. - -Net: release.sh 250 to 240 lines; release-packages.nix 183 to 237 lines (growth from helper structure plus comments). - -## Rehearsal toggle pattern - -This is the load-bearing operational knowledge from the session. - -`mkReleasePackagesEffect` takes a `dryRun` parameter. -Production: `effects.release-packages = mkReleasePackagesEffect { dryRun = false; }`. - -To rehearse a release without remote mutations: flip `dryRun` to `true` on a feature branch, push to magnetite, run - -``` -buildbot-effects run --branch main --rev ... release-packages -``` - -on magnetite, observe semantic-release's full plugin chain run with `--dry-run`, then revert `dryRun` to `false` and commit. -The two transient commits in branch history capture the rehearsal cycle. - -Under `dryRun = true` the dispatch always calls `RELEASE -- --dry-run` (full production plugin set, suppressed mutations); the stale-rev guard is bypassed (so rehearsal can run against any rev while declaring `--branch main`); distinct log markers (`RELEASE-PACKAGES-ACTION: rehearsal`, `RELEASE-PACKAGE-DRY-RUN-DISPATCH:`) make it visually unambiguous in logs. - -A *persistent* second effect attribute (e.g., `release-packages-dry-run` exposed alongside `release-packages`) was rejected. -buildbot-nix triggers every attribute under `onPush.default.outputs.effects` on every qualifying push, so a persistent rehearsal attribute would auto-fire on every commit to main, producing a no-signal rehearsal run alongside the real one. -The transient toggle pattern keeps the production-effect surface at exactly one attribute. - -### release.sh's existing `--` passthrough - -The argument parser in `release.sh` already routes `--` followed by additional args into `extra_args`, which is passed verbatim to semantic-release. -So `"$RELEASE" "$pkg_path" -- --dry-run` works without any release.sh changes — this is what the rehearsal dispatch uses. -No `EFFECT_DRY_RUN` env var or other plumbing was needed. - -### Stale-rev guard - -For the production attribute (`dryRun = false`), the effect aborts before semantic-release runs if local HEAD differs from origin/main HEAD. -This means a manual `buildbot-effects run --branch main` against any non-main rev safely aborts with `RELEASE-CLONE-STALE`. -The actual release fires only after fast-forward of main from `cd-via-effects`. - -## Stage 3b verification result - -On 2026-04-26 the production code path was verified end-to-end via the rehearsal toggle. - -`[@semantic-release/github] - Verify GitHub authentication` returned `Allowed to push to the Git repository`. -This confirms the fine-grained PAT in the `vanixiets-effects-secrets` clan-vars generator has Contents: Read+Write on `cameronraysmith/vanixiets` and authenticates correctly. - -The production plugin chain loaded fully: changelog, github (publish + addChannel), commit-analyzer, release-notes-generator, semantic-release-major-tag. - -semantic-release-monorepo correctly filtered 76 commits to the 2 touching `packages/docs/`, recognized `fix(docs):` as patch-tier, and computed next version `0.5.1`. - -All `Skip ... in dry-run mode` markers fired at correct step boundaries (prepare, publish, success). -Effect exited 0. - -## PAT identity transition (parity gap with old GHA path) - -The old path used the GHA implicit `GITHUB_TOKEN`: identity `github-actions[bot]`, ephemeral 1-hour token, scope determined by job-level `permissions:` block. - -The new path uses a fine-grained PAT from the `vanixiets-effects-secrets` clan-vars generator: identity is the PAT owner (`cameronraysmith`), long-lived (rotation manual), with permissions scoped by what was selected at PAT creation time rather than by per-job `permissions:`. - -Releases will appear under the PAT-owner identity rather than the bot. - -Branch-protection risk is low because the active plugin list does **not** include `@semantic-release/git` (no commit-back to main). -Only tag pushes occur, and tags aren't protected by default branch rules. -`@semantic-release/git`'s presence in `devDependencies` only is intentional and not cruft — it remains available if commit-back behavior is ever wanted, but isn't loaded by the active plugin chain. - -## Verification staircase - -Four stages, each exercising strictly more of the production code path: - -1. `nix run .#preview-version` from a laptop — preview-version mechanics in isolation. -2. `buildbot-effects run --branch refs/pull/N/merge ... release-packages` on magnetite — the bwrap surface plus the PR-clone preamble; dispatches preview-version per the PR-arm logic. -3. `buildbot-effects run --branch main ... release-packages` with the rehearsal toggle active — the full production code path with `--dry-run`. This is the load-bearing test; Stage 3b above documents its outcome. -4. Actual merge — the only stage that exercises buildbot's onPush trigger plumbing end-to-end. Pending at time of writing. - -## Conflicts with the original draft - -The original draft proposed a `just ci-dispatch ` recipe wrapping the SSH+CLI invocation. -This was never built; manual dispatch in the final state is the raw `ssh magnetite.zt buildbot-effects run ...` command. -The wrapper recipe could still be added, but is not present. - -The original draft listed seven jobs and described a phased migration with parity windows. -The final state migrated four (the docs and release pairs) and explicitly excluded `bootstrap-verification` and `test-cluster` for the structural reasons described above. -`set-variables` was retained in trimmed form rather than removed. -Phase 5 ("per-job parity validation") was not run as a separate observational period; verification happened via the rehearsal toggle (Stage 3b) prior to merge rather than via dual-running both paths in production. - -The original draft's exit-criteria block named "Fully-migrated" / "Hybrid-stable" / "Discovery-blocks" outcomes. -The final state corresponds to "Hybrid-stable" with two jobs explicitly held back on architectural grounds rather than for coordination cost. - -## References - -Final-state files on branch `cd-via-effects`: - -- `.github/workflows/cd.yaml` (145 lines, workflow_dispatch only) -- `.github/deprecated/cd.yaml` (407-line frozen snapshot for rollback) -- `modules/effects/vanixiets/herculesCI/release-packages.nix` -- `modules/effects/vanixiets/herculesCI/deploy-docs.nix` -- `modules/apps/release/release.sh` -- `modules/apps/release/release.nix` -- `modules/apps/release/preview-version.sh` - -Original draft commit: `d629aa7e3 docs(notes): draft ADR-001 cd.yaml to buildbot migration (v0)`. - -Research reports from the original ADR draft (still useful for buildbot-nix mechanics; do not represent the final migration scope): - -- `.factory/research/adr-001-validation/01-hercules-effects-buildbot-nix-mechanics.md` -- `.factory/research/adr-001-validation/02-magnetite-state-and-web01-pattern.md` -- `.factory/research/adr-001-validation/03-cd-yaml-inventory.md` -- `.factory/research/adr-001-validation/04-fork-pr-security.md` -- `.factory/research/adr-001-validation/05-ops-triggers-observability-rollback.md` From 110220ed99ae2211772148a2d6768a68455bd2d5 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 03:04:41 -0400 Subject: [PATCH 71/77] chore(workflows): move package-release.yaml to deprecated subdir tree Pure orphan reusable: only callers reside in .github/deprecated/. Standalone CI behavior unchanged. --- .github/{workflows => deprecated}/package-release.yaml | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .github/{workflows => deprecated}/package-release.yaml (100%) diff --git a/.github/workflows/package-release.yaml b/.github/deprecated/package-release.yaml similarity index 100% rename from .github/workflows/package-release.yaml rename to .github/deprecated/package-release.yaml From 3dd09cb3ee8f8994aed31edfb12a0769b036ccd7 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 03:05:00 -0400 Subject: [PATCH 72/77] chore(workflows): move package-test.yaml to deprecated subdir tree Pure orphan reusable: only callers reside in .github/deprecated/. Standalone CI behavior unchanged. --- .github/{workflows => deprecated}/package-test.yaml | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .github/{workflows => deprecated}/package-test.yaml (100%) diff --git a/.github/workflows/package-test.yaml b/.github/deprecated/package-test.yaml similarity index 100% rename from .github/workflows/package-test.yaml rename to .github/deprecated/package-test.yaml From 48154c3dc388eac5350f9acb2da9f5248dd1c6fe Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 03:05:10 -0400 Subject: [PATCH 73/77] chore(workflows): move deploy-docs.yaml to deprecated subdir tree Pure orphan reusable: only callers reside in .github/deprecated/. Standalone CI behavior unchanged. --- .github/{workflows => deprecated}/deploy-docs.yaml | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .github/{workflows => deprecated}/deploy-docs.yaml (100%) diff --git a/.github/workflows/deploy-docs.yaml b/.github/deprecated/deploy-docs.yaml similarity index 100% rename from .github/workflows/deploy-docs.yaml rename to .github/deprecated/deploy-docs.yaml From 6901d50e68d2df41fb35ec0c007bebabac5b577d Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 03:06:32 -0400 Subject: [PATCH 74/77] docs(bootstrap): update comment --- modules/apps/bootstrap/bootstrap.nix | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/apps/bootstrap/bootstrap.nix b/modules/apps/bootstrap/bootstrap.nix index 3a79e8a9a..63823f7d5 100644 --- a/modules/apps/bootstrap/bootstrap.nix +++ b/modules/apps/bootstrap/bootstrap.nix @@ -1,6 +1,6 @@ -# Flake app: re-run the bootstrap flow from an already-nix-ready host. +# Flake app: re-run the bootstrap flow from a host where nix is already installed # -# Chicken-and-egg note: The repo's primary bootstrap entry point is the +# The repo's primary bootstrap entry point is the # Makefile (`make bootstrap`), which installs nix itself via the NixOS # community installer and only then installs direnv. This flake app, by # contrast, can only run once nix is already present (since `nix run` From c8e063334f9a1512d0d26930d63abe7941839b4a Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 20:09:36 -0400 Subject: [PATCH 75/77] chore(vars): remove deprecated buildbot-effects-vanixiets -> vanixiets-effects-secrets --- .../secrets.json/machines/magnetite | 1 - .../secrets.json/secret | 18 ------------------ .../secrets.json/users/cameron | 1 - 3 files changed, 20 deletions(-) delete mode 120000 vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/machines/magnetite delete mode 100644 vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/secret delete mode 120000 vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/users/cameron diff --git a/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/machines/magnetite b/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/machines/magnetite deleted file mode 120000 index 41bd9646c..000000000 --- a/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/machines/magnetite +++ /dev/null @@ -1 +0,0 @@ -../../../../../../sops/machines/magnetite \ No newline at end of file diff --git a/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/secret b/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/secret deleted file mode 100644 index e77f86a78..000000000 --- a/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/secret +++ /dev/null @@ -1,18 +0,0 @@ -{ - "data": "ENC[AES256_GCM,data:/ZNFhCcBJtRuDIqVGQAmJAKnM69Wg4mNMSrUoWn7D30Z5QaoFFpak2qUD10S7+TDiCVaHgnmmj4dlHB3pe4k6hFfqqBGO7b5EI+Us5VWiSqSO5nZjzK8v7/tW2M4jinqlQGvKqJ/Idw9Uaf2vsXtivvDxLTVDZEeP4R5lXXtlf4LgYURbGfz/VJPLk7TMIqJ72uqrvQU/RYXftPndDDQeaeWFN3rY5VSN2mKSnxtwgejiCqyXP8ak3HOKdWuMzwt32JMrqVia5A3EgJLrnZKLnqNmRnUdob8pbCAzo0i0qsYHQjJ05D/kg==,iv:sd3gvsUEaJ6C57LE4lTnDGOemHVxrQDLen81pj3YEKg=,tag:0IkJkx0rCnl2osvhITN6Cw==,type:str]", - "sops": { - "age": [ - { - "recipient": "age1a7a70qcpjemlvk6q4uaf4k77p9eq7lj7wcal5jdj3xuetznyqdrs3mfnsf", - "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxbWVMaWo1R0g5MVc4L1lO\nM2lvUi83VzZ4ZHZYVU05SlBZaUMyTnpybUZrClhsc0sybWpROWd1V011MjE2V09w\nLzY0bWZNMEl0cHo2bmx1VTRIZmZGL0EKLS0tIFhzR25JWDZ2ZkR3UXRDSHMyZDVH\na200RlZvUXBFdG1NbXFpTXgrNTJ5NWMKbaVkB1OVKiU7No+CdKZGNDzXURbhutVU\nlLApeDcsV9T3FH6pq9uHlXERNHt8KUXkMYsDIHXJue07ahfLTmQoyQ==\n-----END AGE ENCRYPTED FILE-----\n" - }, - { - "recipient": "age1vn8fpkmkzkjttcuc3prq3jrp7t5fsrdqey74ydu5p88keqmcupvs8jtmv8", - "enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuenZOVkR5OWgrVTV4a3F2\nQVR2OXc0dFQ5SndlcVBDaGFzZll5L1J1aTJZCjNLckM5MnpqSkxKc1lIWCt2d1VH\nL0RJQmlINFR5YmFkai9VSGJCTSs3YWsKLS0tIG5yM3R1THJXMWk1UXk3em5ESmwx\nNDJtTjJaVXk1R2F1YWozMWppOXNxcmsKatUi/oGyzlfdemgwetd6bfhzVt+d4wUM\nofx+Rsy/RUP39AxHiv95lIGNgjdTSApeq+KoUZqjkmDp5Qqr8+/D3Q==\n-----END AGE ENCRYPTED FILE-----\n" - } - ], - "lastmodified": "2026-04-23T02:13:17Z", - "mac": "ENC[AES256_GCM,data:XbVJVvsIaZSM/3O4UehddlPFDTvH5akoINA55R4l3JOP+YUL+7Fy0igkOxThOZFqoQqFBkgA7QUXfXd9LOtzI7UcYodnfak9UkkhU/A/US9UVUrBUoHduzMA5cAIDDuLOHTP3K3xnrsFqNxOOvse9L3x+gYnYES8sFDI+UkSyd0=,iv:vaBAyNalKFXXHWIkrjzv3E6NH6EvIM8AcGGW5zlE8Js=,tag:l5x+9vLy0I5PxzBneSNNrQ==,type:str]", - "version": "3.12.2" - } -} diff --git a/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/users/cameron b/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/users/cameron deleted file mode 120000 index 015130152..000000000 --- a/vars/per-machine/magnetite/buildbot-effects-vanixiets/secrets.json/users/cameron +++ /dev/null @@ -1 +0,0 @@ -../../../../../../sops/users/cameron \ No newline at end of file From 949ed25044867b1927f559bbff0e7f95157cca5d Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 23:04:31 -0400 Subject: [PATCH 76/77] docs(effects/release-packages): cite buildbot-nix refspec idiom --- modules/effects/vanixiets/herculesCI/release-packages.nix | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 0be0c8961..6f473c58a 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -130,7 +130,8 @@ # `git fetch origin refs/pull//head` alone updates # FETCH_HEAD but does NOT auto-create the remote-tracking # ref; the explicit `+ref:remote-tracking-ref` mapping - # closes that gap. + # closes that gap (idiom from buildbot-nix + # buildbot_nix/buildbot_nix/nix_eval.py:GitLocalPrMerge). git -C "$clone_dir" fetch origin \ "+refs/pull/${toString prNumber}/head:refs/remotes/origin/pr-${toString prNumber}-head" head_sha="$(git -C "$clone_dir" rev-parse origin/pr-${toString prNumber}-head)" From b7076d36ff6d2dac23fa757d7558883765974a83 Mon Sep 17 00:00:00 2001 From: Cameron Smith Date: Sun, 26 Apr 2026 23:04:59 -0400 Subject: [PATCH 77/77] docs(effects/release-packages): describe GitHub PR merge-ref semantics --- modules/effects/vanixiets/herculesCI/release-packages.nix | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/modules/effects/vanixiets/herculesCI/release-packages.nix b/modules/effects/vanixiets/herculesCI/release-packages.nix index 6f473c58a..60837fb5e 100644 --- a/modules/effects/vanixiets/herculesCI/release-packages.nix +++ b/modules/effects/vanixiets/herculesCI/release-packages.nix @@ -124,6 +124,12 @@ ${ if isPrMerge then '' + # GitHub's refs/pull//merge is a synthetic test-merge + # ref recomputed on base advance, head update, or + # merge-test scheduler fire; the T0 buildbot-eval SHA + # drifts from T1 runtime content. refs/pull//head + # is the dev-pushed source-branch tip, stable until + # the next dev push. git clone "$clone_url" "$clone_dir" git -C "$clone_dir" fetch --tags origin