diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 73312467..54a6763c 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -10,6 +10,11 @@ on: branches: - main workflow_dispatch: + inputs: + registry_cache_test: + description: 'Also verify authenticated GHCR cache reuse (synthetic data only)' + type: boolean + default: false permissions: contents: read @@ -33,3 +38,35 @@ jobs: - name: Lint generated GitLab Runner configuration run: bash tests/gitlab-runner-lint.sh + + - name: Verify shared build cache with isolated builders + run: bash tests/build-cache-integration.sh + + registry-cache: + name: Authenticated registry cache proof + if: github.event_name == 'workflow_dispatch' && inputs.registry_cache_test && github.ref == 'refs/heads/main' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + packages: write + steps: + - name: Checkout trusted dispatch ref + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Verify cache export and import with job-owned credentials + shell: bash + env: + REGISTRY_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + export DOCKER_CONFIG + DOCKER_CONFIG="$(mktemp -d "$RUNNER_TEMP/crf-registry-auth.XXXXXX")" + trap 'rm -rf -- "$DOCKER_CONFIG"' EXIT + printf '%s' "$REGISTRY_TOKEN" | docker login ghcr.io -u "$GITHUB_ACTOR" --password-stdin + unset REGISTRY_TOKEN + export CRF_CACHE_TEST_REPOSITORY="ghcr.io/${GITHUB_REPOSITORY,,}-build-cache" + export CRF_CACHE_TEST_TAG="proof-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}" + bash tests/build-cache-integration.sh diff --git a/.limetech/ai-review-markers/feat-shared-build-cache-8c24ca91be08.json b/.limetech/ai-review-markers/feat-shared-build-cache-8c24ca91be08.json new file mode 100644 index 00000000..c8672c06 --- /dev/null +++ b/.limetech/ai-review-markers/feat-shared-build-cache-8c24ca91be08.json @@ -0,0 +1,8 @@ +{ + "disposition": "PATCH", + "forecast_commit": "c214cf26f7868ce28f4f7db38bb9993264e3f764", + "reviewed_sha": "8b5ad92f18df6276a0dc4562f6cfbd033259f5cb", + "schema": "limetech.ai-review-marker.v2", + "stage": "final", + "unresolved_proportionality_findings": [] +} diff --git a/.mega-linter.yml b/.mega-linter.yml index 390f156e..fa450c18 100644 --- a/.mega-linter.yml +++ b/.mega-linter.yml @@ -70,3 +70,10 @@ FILTER_REGEX_EXCLUDE: (^LICENSE$|^CHANGELOG\.md$|\.git/|megalinter-reports/) # uses .json with comments rather than .jsonc), so allow comments/trailing # commas for all JSON rather than relying on file extension. JSON_JSONLINT_ARGUMENTS: "--comments --trailing-newline --trailing-commas" + +# The shipped README needs public URLs, but new docs do not exist on main +# before a PR merges. Validate this repository's main-file links against the +# candidate checkout inside MegaLinter. Missing local targets still fail. +SPELL_LYCHEE_ARGUMENTS: + - --remap + - '^https[:]//github[.]com/unraid/ci-runner-farm/blob/main/ file:///github/workspace/' diff --git a/README.md b/README.md index 5274d7c1..1130975e 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,8 @@ provider's credentials and runtime. | Concurrent runner slots | Each slot accepts one job at a time and can have CPU and memory limits, keeping CI from starving the rest of the host. | | GitHub and GitLab providers | Keep the existing GitHub Actions integration or select GitLab.com/self-managed GitLab. | | Warm shared caches | Reuse npm, yarn, pnpm, Playwright, Cargo, sccache, or custom cache directories across jobs. | +| Optional registry build-cache profile | Supply explicit CI workflows with an existing registry cache location and a smaller per-builder GC budget. No new server or shared Docker data root. See the [setup guide](docs/build-cache.md). | +| Cache-preserving Stop and Restart | Keep per-slot Docker and GitLab job caches across maintenance. Explicit `prune-cache` deletes retained caches. Permanent slot retirement still deletes that slot's data. See [cache retention](docs/build-cache.md#keep-caches-across-stop-and-restart). | | Slot-scoped Docker-in-Docker | Give each runner slot a private privileged Docker daemon without exposing Unraid's existing Docker socket by default; privileged DinD is still capable of host compromise. | | Bring your own job image | Pull a remote image or edit and build a provider-specific starter image in the plugin. | | Named runner pools | Route jobs to purpose-built pools with independent fixed capacity, labels/tags, CPU, memory, and images. | diff --git a/docs/build-cache.md b/docs/build-cache.md new file mode 100644 index 00000000..7a7c5d06 --- /dev/null +++ b/docs/build-cache.md @@ -0,0 +1,188 @@ +# Share build caches through an existing registry + +The optional **Build cache profile** lets CI jobs reuse exported Docker build +layers across runners. Each slot keeps its own Docker daemon and writable data. +The plugin supplies configuration, not a registry server or registry credentials. +The existing pull-through image mirror remains separate and cannot accept cache +exports. + +## Configure the farm + +1. Open **Settings → CI Runner Farm → Settings → Build cache profile**. +2. Select **Existing registry (workflow opt-in)**. +3. Enter a tagless repository, such as `registry.example.com/team/build-cache`. +4. Set **Local cache budget per builder (GiB)**. The default is 20 GiB. +5. Apply the settings. In classic mode, wait for runners to finish their jobs and adopt the new profile. +6. For named pools, schedule a **Fleet Restart** after active jobs finish. Apply alone does not replace those runners. +7. Wait for the replacement runners before updating build workflows as described below. + +The configuration keys are `BUILD_CACHE_MODE` (`off` or `registry`), +`BUILD_CACHE_REPOSITORY`, and `BUILD_CACHE_LOCAL_GIB` (1–1024). +Off preserves existing behavior, including runner configuration fingerprints. +Registry mode requires Docker-in-Docker. Host-socket mode is not supported. + +Each GitHub runner and GitLab Docker-executor job receives two read-only files: + +- `/etc/ci-runner-farm/build-cache/profile.env`: the repository and config path. +- `/etc/ci-runner-farm/build-cache/buildkitd.toml`: the OCI worker's GC budget. + +Profiles are immutable snapshots. Changes do not rewrite files mounted by busy +jobs. Old snapshots remain in the plugin's runtime directory until reboot. +The current snapshot is generated again during runner provisioning. + +## Use the profile in a workflow + +Use a current Buildx client and BuildKit version that supports `maxUsedSpace`. +This profile targets the OCI worker in a `docker-container` builder. It does not +configure arbitrary existing builders, remote builders, or `docker build` calls. +GitLab job images must include the Docker CLI and Buildx plugin. + +Authenticate to the cache registry inside the job, using a credential with only +the required repository permissions. The plugin does not expose its host PAT, +runner token, or image-pull credentials through this profile. Configure private +CA trust in the job/builder when required. The profile does not disable TLS or +relax the farm's network firewall. A registry blocked by strict isolation remains +blocked. + +This shell example works in a GitHub shell job or GitLab job. Set +`CACHE_SCOPE` from a stable hash of the full project, build target, platform, and +branch identity. Use the same scope on different runners to reuse their cache. +Keep output image references separate from cache references. + +```bash +set -euo pipefail +. /etc/ci-runner-farm/build-cache/profile.env +: "${CACHE_SCOPE:?Set a project/target/platform/branch-specific cache scope}" +[[ "$CACHE_SCOPE" =~ ^[a-z0-9][a-z0-9_.-]{0,100}$ ]] || exit 1 +cache_ref="${CRF_BUILD_CACHE_REPOSITORY}:${CACHE_SCOPE}" +builder="crf-$(cat /proc/sys/kernel/random/uuid)" +docker buildx create --name "$builder" --driver docker-container \ + --buildkitd-config "$CRF_BUILDKIT_CONFIG" --bootstrap +trap 'docker buildx rm "$builder"' EXIT +docker buildx build --builder "$builder" \ + --cache-from "type=registry,ref=$cache_ref" \ + --cache-to "type=registry,ref=$cache_ref,mode=max,image-manifest=true" \ + --load -t example/app:ci . +``` + +The example removes only its own builder after the job. Registry cache survives +that cleanup. A workflow may retain its own builder instead, but it must recreate +that builder when the profile changes so the new GC budget takes effect. +GitHub jobs using `container:` must also mount the profile directory read-only +into their job container. Shell jobs receive it directly. + +For GitHub's Docker actions, load `profile.env` in a shell step and write the +needed values to `$GITHUB_OUTPUT`. Pass the config path to +`docker/setup-buildx-action` as `buildkitd-config`. Pass explicitly scoped refs to +`docker/build-push-action` as `cache-from` and `cache-to`. Do not assume that the +runner's environment automatically appears in GitHub's expression `env` context. +See Docker's [builder configuration](https://docs.docker.com/build/ci/github-actions/configure-builder/) +and [registry cache examples](https://docs.docker.com/build/ci/github-actions/cache/). + +## Scope, failure, and storage limits + +Cache names are not an access-control boundary. Only trusted jobs should share +a writable cache repository. Separate projects and trust levels with registry +permissions and separate repositories. Never give untrusted pull requests a +credential that can overwrite trusted caches. Branch-specific tags avoid normal +collisions but do not stop a credential holder from choosing another tag. +Serialize exports to the same ref, or use separate refs for concurrent writers. +Use BuildKit secret mounts for secrets; never bake credentials into layers. + +The profile provides no implicit `latest` cache tag. A missing import cache is a +normal cold build under Docker's behavior. Export failures remain job failures +unless the workflow deliberately changes Docker's default error handling. Do not +add `ignore-error=true` or fall back to an unconfigured builder to hide failures. + +The local budget is a garbage-collection target, not a hard quota. Active build +data can exceed it. It does not cover Docker images, other builders, workspaces, +package caches, or the image mirror. Exported caches also need registry retention +and garbage collection, controlled by the registry owner. Reducing local budgets +trades disk use for cache downloads and depends on registry availability. + +Enabling this option does not prune old builders or immediately recover their +disk space. Migrate workflows first, then retire unused builders through their +normal owner. Remove workflow references to the profile before disabling the +option. Classic fleets remove the mount as runners drain and are replaced. +Named pools require the same scheduled Fleet Restart described above. + +## Keep caches across Stop and Restart + +**Stop** and **Restart** retain per-slot Docker data, GitLab job caches, and the +shared image mirror cache. A later Start reuses the retained data for the same +slot and cache root. This applies with the build cache profile on or off. +The plugin records the provider and baked runner configuration identity for each +retained slot. If that identity changes, including a provider, owner, project, +registry, or trust-scope change, the plugin purges that slot's retained Docker, +workspace, job-cache, socket, and log data before starting it again. The shared +image mirror remains separate and is not treated as per-slot job state. +The plugin still removes runner containers and performs provider credential +cleanup. Plugin uninstall and active GitLab runner-token removal also retain +caches because they use Stop. + +Stop does not guarantee that active jobs finish. Schedule maintenance before +using Stop or Restart. Manual scale-down, autoscale-down, and permanent slot +retirement still delete that slot's Docker data and GitLab job cache. Registry +exports are unaffected. + +To delete retained local caches, stop the fleet, then run the explicit command: + +```bash +/usr/local/emhttp/plugins/ci-runner-farm/include/runner-farm.sh prune-cache +``` + +This command deletes plugin-owned cache directories under the configured cache +root. It refuses to proceed while managed runners, sidecars, or job containers +remain. Cache retention does not reduce existing disk use. + +## Verification + +`bash tests/build-cache.sh` checks profile validation, snapshots, fingerprints, +and both providers' mount contracts without external services. +`bash tests/cache-retention.sh` checks Stop/Restart retention and explicit cache +deletion for both providers using disposable on-disk fixtures. +`bash tests/build-cache-integration.sh` uses disposable local Docker builders and +a test registry to prove cross-builder cache reuse and effective GC settings. +It does not contact a farm host or modify existing runners. + +### Verify an authenticated registry + +The **Lint** workflow has a manual **Also verify authenticated GHCR cache reuse** +option. The package-writing job accepts dispatches only from the repository's +`main` branch. Pull-request events and other refs cannot start it. Review the +exact `main` commit before dispatching. GitHub assigns its runner. + +The job uses its short-lived `GITHUB_TOKEN` with `packages: write`, not a farm +credential. It exports synthetic Busybox layers to +`ghcr.io//-build-cache:proof--`. +Two fresh builders must have different local volumes, and the second must reuse +the exported `RUN` layer. Authentication or export errors fail the test. +The test never substitutes a local registry when the external registry fails. + +Inspect the package's actual visibility, repository link, and workflow access +after the run. Do not assume that it is private. This repository's initial +synthetic proof produced a public package. Never use a public proof repository +for private build caches. A private production cache needs its own authorized +repository and access policy before the farm points to it. + +The job removes its temporary Docker credentials and local test resources. It +leaves the small proof tag for inspection; the package owner controls retention. +A successful proof against a public package establishes authenticated exports +and cache reuse, but does not establish private-package read authorization. + +For another registry, authenticate through a temporary, job-owned Docker config. +Set `CRF_CACHE_TEST_REPOSITORY` to its tagless repository and +`CRF_CACHE_TEST_TAG` to a unique tag starting with `proof-`. Then run +`bash tests/build-cache-integration.sh`. Both inputs are required together. +Do not use the farm's host PAT, runner token, or image-pull credential. + +This check uses the plugin's generated profile in an isolated fixture. It does +not prove that a deployed runner has adopted its profile or that a production +workflow uses it. Verify the deployed read-only mount separately, then configure +each authorized build workflow as described above. Enabling the farm option +alone does not change existing build commands or reduce their old cache data. + +On macOS, use `bash tests/run-linux-checks.sh` with Docker running and Homebrew +Coreutils installed (`brew install coreutils`). The wrapper runs the shell suite +in Linux. Its real-Docker integration stage uses the installed GNU tools on the +host because the production snapshot generator requires GNU `mv -T`. diff --git a/src/usr/local/emhttp/plugins/ci-runner-farm/README.md b/src/usr/local/emhttp/plugins/ci-runner-farm/README.md index 686c9df5..9e019edf 100644 --- a/src/usr/local/emhttp/plugins/ci-runner-farm/README.md +++ b/src/usr/local/emhttp/plugins/ci-runner-farm/README.md @@ -5,6 +5,12 @@ resource-capped slots, warm package/image caches, Docker-in-Docker, and optional autoscaling. One provider owns the farm at a time; GitHub remains the default so existing installations keep their behavior. +The optional **Build cache profile** supplies an existing registry cache location +and a per-builder garbage-collection budget (20 GiB by default). It is off by +default and requires Docker-in-Docker plus explicit workflow integration. +It creates no registry server, shares no writable Docker root, and passes no +host credentials to jobs. See the [build-cache setup guide](https://github.com/unraid/ci-runner-farm/blob/main/docs/build-cache.md). + Named pools work with both providers. Each pool can use its own fixed capacity, labels or tags, CPU, memory, and runner/job image. GitHub pools require organization scope. GitLab pools require a pool-specific `glrt-` runner token diff --git a/src/usr/local/emhttp/plugins/ci-runner-farm/RunnerFarmSettings.page b/src/usr/local/emhttp/plugins/ci-runner-farm/RunnerFarmSettings.page index 47b8678e..60c85aa3 100644 --- a/src/usr/local/emhttp/plugins/ci-runner-farm/RunnerFarmSettings.page +++ b/src/usr/local/emhttp/plugins/ci-runner-farm/RunnerFarmSettings.page @@ -35,6 +35,7 @@ $defaults = [ 'CACHE_ROOT'=>'/mnt/cache/github-runner', 'WORK_TMPFS_SIZE'=>'8g', 'CACHE_MOUNTS'=>'pnpm-store:/home/runner/.local/share/pnpm/store npm:/home/runner/.npm yarn:/home/runner/.cache/yarn ms-playwright:/home/runner/.cache/ms-playwright', 'DIND'=>'true', 'SHARE_DOCKER_SOCK'=>'false', 'SHARED_IMAGE_CACHE'=>'true', 'NETWORK_ISOLATION'=>'off', + 'BUILD_CACHE_MODE'=>'off', 'BUILD_CACHE_REPOSITORY'=>'', 'BUILD_CACHE_LOCAL_GIB'=>'20', 'IMAGE_AUTOUPDATE'=>'false', 'IMAGE_AUTOUPDATE_INTERVAL'=>'1800', 'IMAGE_DRAIN_TIMEOUT'=>'3600', 'DASHBOARD_WIDGET_ENABLE'=>'true', 'AUTOSCALE'=>'false', 'AUTOSCALE_MIN'=>'2', 'AUTOSCALE_MAX'=>'16', 'AUTOSCALE_MIN_IDLE'=>'2', @@ -429,6 +430,34 @@ _(Shared image cache)_: > Runs a shared `registry:2` pull-through cache (Docker-in-Docker only) so images used across the fleet are pulled from Docker Hub once, not once per runner. Bound to the Docker bridge gateway, not the LAN — so it is not exposed off the host, though (being on the bridge gateway) it is reachable unauthenticated by any container on the default Docker bridge, not only this plugin's runners. It only caches public Docker Hub images. Turn off if you don't want the extra container. The host port defaults to 5000; if that clashes with another service, set `MIRROR_PORT` in `/boot/config/plugins/ci-runner-farm/ci-runner-farm.cfg` and Restart the fleet. :end +### Build cache profile + +_(Shared build cache)_: +: + +:crf_build_cache_mode_plug: +> Requires Docker-in-Docker and an existing writable registry. No registry server is created. +:end + +_(Cache repository)_: +: + +:crf_build_cache_repository_plug: +> Enter a repository without a URL scheme, tag, digest, or credentials. Workflows must load `/etc/ci-runner-farm/build-cache/profile.env`, configure a `docker-container` builder with its BuildKit configuration, and explicitly import/export scoped cache references. Jobs supply their own registry credentials. See the [build-cache setup guide](https://github.com/unraid/ci-runner-farm/blob/main/docs/build-cache.md). +:end + +_(Local cache budget per builder (GiB))_: +: + +:crf_build_cache_plug: +> The budget controls BuildKit garbage collection, not total disk use or a hard quota. Other builders and active builds can exceed it. In classic mode, Apply updates runners after active jobs finish. Named pools require an explicit Fleet Restart. Wait for the new runners before using the profile in workflows. Existing caches are not pruned. +:end + +### Network isolation + _(Network isolation)_: :