Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ The project is in architecture definition and isolated proof-of-concept planning
- `docs/COMPLIANCE-CHECKLIST.md`: cutover gate.
- `docs/ARCHITECTURE.md`: fleet and trust boundaries.
- `docs/SECRETS.md`: credential boundaries.
- `docs/ADDING-A-HOST.md`: repeatable host and location enrollment.
- `docs/ADDING-A-PROJECT.md`: project authorization without host changes.

Agents modifying this repository or adapting a project MUST apply these documents. Do not create a project-specific exception silently. Record any necessary exception as an explicit design decision with risks and removal criteria.

Expand All @@ -37,6 +39,8 @@ Agents modifying this repository or adapting a project MUST apply these document
- Pin production dependencies and container images to reviewed versions or digests.
- Do not remove or weaken existing required CI until parallel validation and rollback verification are complete.
- Do not install project language runtimes in the generic runner image.
- Do not put repository names, project allowlists, or project-specific logic in runner images or host configuration.
- Keep project workflows independent of host, site, instance, and scale-set names; route through shared capability labels.
- Do not allow projects to select privileged runner groups through untrusted inputs.

## Verification expectations
Expand Down
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ flowchart TD
## Core model

- The fleet supplies generic ephemeral GitHub runners.
- Each runner accepts one job and is then destroyed.
- Worker images and host configuration never contain repository allowlists or project-specific logic.
- Each runner accepts one GitHub-assigned job and is then destroyed.
- Each project supplies its own test Dockerfile, services, and commands.
- Normal CI uses a shared organization-level runner pool.
- Release, deployment, repository-writing, and internal-network jobs remain separated.
Expand Down Expand Up @@ -95,6 +96,8 @@ Never commit credentials or real deployment configuration. See [SECURITY.md](SEC
- [Experimental deployment prototype](docs/DEPLOYMENT-PROTOTYPE.md)
- [Live pilot runbook](docs/LIVE-PILOT.md)
- [Host maintenance standard](docs/HOST-MAINTENANCE.md)
- [How to add a fleet host](docs/ADDING-A-HOST.md)
- [How to add a project](docs/ADDING-A-PROJECT.md)
- [Sanitized discovery summary](docs/DISCOVERY-SUMMARY.md)
- [Roadmap](docs/ROADMAP.md)
- [Secrets model](docs/SECRETS.md)
Expand All @@ -120,6 +123,10 @@ The scaffold is validated in this repository before being published as the stand

Examples are starting points. Projects must replace placeholder action references and pin reviewed container images before production use.

## Set-and-forget growth

Enroll each Docker host once with a unique instance and scale-set name and the fleet's shared routing label. Adding a future project updates that project, private fleet policy, and GitHub runner-group access; it does not modify, rebuild, or re-register existing hosts or worker containers. Additional homes, buildings, physical machines, and VPS locations use the same host runbook and join the same compatible queue.

## First live milestone

After the inert prototype passes validation, the first separately authorized live proof of concept must:
Expand Down
166 changes: 166 additions & 0 deletions docs/ADDING-A-HOST.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# How to add a fleet host

Use this runbook to add a Proxmox VM, physical computer, remote-site machine, or VPS to an existing ci-fleet installation.

## Outcome

A host is a generic, location-independent Docker appliance. Configure it once. It must not contain repository names, project runtimes, project secrets, or project-specific test logic.

Each controller has a unique instance and scale-set name. Compatible controllers share the same routing label and runner group, so GitHub can route queued work to any available location.

| Setting | Per host | Shared by compatible hosts |
| --- | --- | --- |
| `CI_FLEET_INSTANCE` | Unique | No |
| `CI_FLEET_SCALE_SET_NAME` | Unique | No |
| `CI_FLEET_LABELS` | No | Yes |
| `CI_FLEET_RUNNER_GROUP` | No | Yes |
| Capacity and resource limits | Usually | No |
| GitHub organization | No | Yes |
| Repository allowlist | Never stored here | GitHub runner-group policy |

Example:

```dotenv
# Home
CI_FLEET_INSTANCE=home-ci-01
CI_FLEET_SCALE_SET_NAME=docker-ci-home-ci-01
CI_FLEET_LABELS=docker-ci
CI_FLEET_RUNNER_GROUP=trusted-private-ci

# Remote building
CI_FLEET_INSTANCE=warehouse-ci-01
CI_FLEET_SCALE_SET_NAME=docker-ci-warehouse-ci-01
CI_FLEET_LABELS=docker-ci
CI_FLEET_RUNNER_GROUP=trusted-private-ci

# VPS
CI_FLEET_INSTANCE=vps-ci-01
CI_FLEET_SCALE_SET_NAME=docker-ci-vps-ci-01
CI_FLEET_LABELS=docker-ci
CI_FLEET_RUNNER_GROUP=trusted-private-ci
```

## 1. Choose the host and failure boundary

The host must:

- run no production or development application workload;
- have no sensitive host mounts, SSH agent, or unrelated Docker workload;
- use local or virtualized storage suitable for Docker's write load;
- have reliable DNS, time synchronization, and outbound HTTPS access to GitHub and required registries;
- permit local administration without requiring public inbound access;
- be disposable or recoverable from a documented build and backup procedure.

Runners sharing one Docker daemon share a security boundary. Add a host instead of increasing same-host concurrency when projects need stronger separation or when one site's failure must not stop the fleet.

## 2. Install the generic host

Install a supported Linux distribution, Docker Engine, Docker Compose v2, Git, Bash, `curl`, `jq`, CA certificates, the QEMU guest agent when virtualized, and basic diagnostics.

Do not install Node, PHP, Python, Java, Composer, npm, database clients, or other project runtimes. Projects bring those in their own images.

Enable unattended security updates without automatic reboots. Configure Docker log rotation and host disk alerts. Follow [Host maintenance](HOST-MAINTENANCE.md).

## 3. Record backup and network readiness

Before placing credentials on the host:

1. verify console or local recovery access;
2. verify the host has a unique name and address;
3. create a VM backup, snapshot, or equivalent recoverable baseline;
4. record the backup destination and successful timestamp outside this public repository;
5. verify the host can reach GitHub and required container/package registries.

A remote site needs outbound connectivity; it does not need project-specific inbound access. If one location is offline, compatible jobs remain queued for another available host.

## 4. Install a pinned ci-fleet checkout

Use a reviewed immutable commit:

```bash
sudo install -d -m 0755 /opt/ci-fleet
# Populate /opt/ci-fleet from a reviewed ci-fleet commit.
git -C /opt/ci-fleet rev-parse HEAD
```

Do not follow a moving branch for unattended controller updates. Fleet image changes use reviewed rolling replacement and keep the previous version available for rollback.

## 5. Create host-local configuration

Start from `deploy/ci-fleet.env.example`:

```bash
sudo install -d -m 0700 /etc/ci-fleet/secrets
sudo install -m 0600 deploy/ci-fleet.env.example /etc/ci-fleet/ci-fleet.env
```

Set:

- a unique `CI_FLEET_INSTANCE`;
- a unique `CI_FLEET_SCALE_SET_NAME`;
- the fleet's shared `CI_FLEET_LABELS`;
- the shared `CI_FLEET_RUNNER_GROUP`;
- host-specific maximum concurrency and runner CPU/memory limits;
- `CI_FLEET_MIN_RUNNERS=0` so idle runners do not persist.

Repository names do not belong in this file.

Provision the GitHub App private key directly into `/etc/ci-fleet/secrets/github-app.pem` with mode `0600`. Prefer a separately generated App private key per host so one location can be revoked without replacing every host key. Never expose the key to runner or project containers.

## 6. Validate before starting

```bash
cd /opt/ci-fleet
set -a
. /etc/ci-fleet/ci-fleet.env
set +a
scripts/preflight.sh
docker compose -f deploy/compose.yaml build runner-image controller
scripts/preflight.sh
```

Continue only when preflight reports `PREFLIGHT_OK warnings=0`.

## 7. Start at zero and prove one job

Start with maximum concurrency one even if the host will eventually run more:

```bash
docker compose -f deploy/compose.yaml up -d --no-deps controller
docker compose -f deploy/compose.yaml logs --tail=100 controller
```

Confirm no runner exists while idle. Dispatch one manual, read-only job from an authorized private project and verify:

- one ephemeral runner appears;
- it accepts one job;
- it is destroyed afterward;
- no job-owned container, network, volume, or workspace remains;
- `scripts/healthcheck.sh` passes;
- scoped cleanup identifies no expired resource.

Follow [Live pilot](LIVE-PILOT.md) for the complete proof and rollback.

## 8. Enable unattended operations

Install the health and cleanup timers from [Host maintenance](HOST-MAINTENANCE.md). Monitor controller state, disk thresholds, Docker health, last successful job, cleanup failures, and pending reboot state.

The steady-state host should need no project-specific edits. Adding a project changes its repository, private fleet configuration, and GitHub runner-group policy—not this host.

## 9. Increase capacity safely

Raise `CI_FLEET_MAX_RUNNERS` only after measuring CPU, memory, disk, network, cache growth, collisions, cancellation, and cleanup. Keep explicit per-runner limits.

Use another host or location when it improves failure tolerance. Give every added host a new instance and scale-set name while retaining the shared routing label.

## 10. Drain, replace, or remove a host

1. Stop new capacity by setting maximum capacity to zero or stopping the controller.
2. Confirm no managed runner is active.
3. Run cleanup in dry-run mode, review candidates, then apply scoped cleanup.
4. Verify the host's unique scale set is absent from GitHub.
5. Revoke that host's GitHub App private key.
6. Securely delete host-local credentials.
7. Remove or restore the machine according to the site's recovery policy.

Other hosts and project workflows continue using the shared routing label. Replacing one host must not require editing any project workflow.
122 changes: 122 additions & 0 deletions docs/ADDING-A-PROJECT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# How to add a project

Use this runbook to authorize a trusted private project to consume an existing ci-fleet.

## Outcome

Project onboarding does not change, rebuild, restart, or re-register any fleet host or worker container. Generic runners learn the repository only when GitHub assigns a job.

A project is onboarded in three control planes:

1. the project repository supplies its Docker test contract and workflow;
2. private fleet configuration declares policy and expected capability;
3. GitHub runner-group access authorizes the repository.

Repository names are never baked into the public runner image, controller image, host environment, or individual ephemeral runner.

## 1. Confirm the trust boundary

The shared validation pool is for trusted private repositories only. A job with Docker access is host-root-equivalent.

Do not authorize:

- public repositories;
- untrusted fork pull-request code;
- deployment or internal-management jobs;
- repository-writing or release jobs;
- a project whose maintainers are not trusted for the shared Docker-host boundary.

Use separate runner groups and hosts for privileged or differently trusted workloads.

## 2. Choose the runner-group repository policy

GitHub runner groups can authorize selected repositories or all organization repositories. Keep GitHub's public-repository access override disabled.

### Selected repositories

Use this when private repositories have different maintainers or trust levels. Add each project once to the runner-group allowlist. This is a GitHub policy change, not a host or container change.

The private fleet configuration should remain the reviewable source of intended access. Until policy synchronization is automated, compare it with GitHub during onboarding and periodic audits.

### All private repositories

Use this only when every present and future private repository in the organization is trusted to execute host-privileged Docker workloads. It minimizes onboarding steps but expands the effect of a compromised or accidentally unsafe repository.

Changing between these policies never requires rebuilding fleet hosts.

## 3. Add private fleet policy

In the organization-owned private configuration repository:

- declare the repository;
- select the shared validation pool and routing label;
- keep public-repository access false;
- declare required secret names without values;
- keep deployment and release pools separate;
- validate the configuration before merge.

Never store credentials, private keys, tokens, real environment files, or host inventories in configuration.

## 4. Implement the project contract

The project must provide:

```text
scripts/ci/plan.json
scripts/ci/run.sh <task> --shard INDEX/TOTAL
scripts/ci/run.sh fast
scripts/ci/run.sh full
```

Follow [Project CI Standard](PROJECT-STANDARD.md). Existing projects must follow [Migrating Existing CI](MIGRATING-EXISTING-CI.md) and complete the [Compliance Checklist](COMPLIANCE-CHECKLIST.md).

Application runtimes, dependencies, service containers, migrations, test fixtures, task selection, and run-owned cleanup belong to the project. The fleet host must not provide them.

## 5. Add the read-only workflow

Start from `examples/workflows/parallel-ci.yml.example` and:

- pin ci-fleet actions to a reviewed immutable commit;
- declare `permissions: contents: read`;
- select only the normal shared CI label;
- expand the project task plan;
- set five-minute timeouts on ordinary jobs;
- pass only explicitly required test secrets;
- keep release, branch mutation, deployment, and internal-network access out of the workflow.

The workflow should not name a host, site, controller, instance, or scale set. It selects the shared capability label, allowing GitHub to route work to any healthy location.

## 6. Prove the migration

1. Validate all direct task/shard commands through project containers.
2. Run fast and full aggregates locally.
3. Dispatch one harmless manual shard.
4. Compare old and new CI on the same commits.
5. Verify success, failure, cancellation, timeout, cleanup, and repeated concurrency.
6. Record shard timings and rebalance work approaching the ceiling.
7. Keep the old required path available until rollback is proven.
8. Cut over required checks only after the compliance checklist passes.

Adding more hosts during this process requires no project workflow change because every compatible host shares the routing label.

## 7. Remove a project

1. Remove the repository from the GitHub runner-group allowlist, or move it outside the scope of an all-private policy.
2. Remove or disable its fleet workflow.
3. Remove its entry from private fleet configuration.
4. Revoke project test secrets that are no longer needed.
5. Confirm no queued job still targets the shared label.

Do not edit or rebuild fleet hosts. Other projects and locations continue operating normally.

## Onboarding checklist

- [ ] Repository is private and trusted for the shared Docker-host boundary.
- [ ] Private configuration declares it and validates.
- [ ] GitHub runner-group policy authorizes it.
- [ ] Public repository access remains disabled.
- [ ] Project Docker contract passes locally.
- [ ] Ordinary workflow is read-only and uses the shared label.
- [ ] Privileged jobs use separate routing.
- [ ] Manual proof, parallel comparison, cleanup tests, and rollback pass.
- [ ] Existing required CI remains until cutover is complete.
12 changes: 11 additions & 1 deletion docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,16 @@ Status: design draft

Provide a portable fleet of identical self-hosted GitHub Actions runners that can serve multiple explicitly authorized repositories. Each project owns its test environment; the fleet owns runner lifecycle, host maintenance, capacity, and common workflow behavior.

## Set-and-forget invariants

- A host is enrolled once with a unique instance and scale-set identity and a shared capability label.
- Host configuration and worker images contain no repository names, allowlists, project runtimes, or project test logic.
- GitHub runner-group policy authorizes repositories; private fleet configuration records intended organization policy.
- Adding or removing a project never requires changing, rebuilding, restarting, or re-registering a compatible host.
- Adding a location never requires changing a project workflow; GitHub routes its shared label to any healthy compatible scale set.
- Losing or draining one site leaves compatible work available to other sites.
- Privileged workloads remain on separate labels, groups, hosts, and network boundaries.

## System topology

```mermaid
Expand Down Expand Up @@ -145,4 +155,4 @@ Host-wide pruning must not run as an uncoordinated per-job operation. Hard cance

## Project adoption

All projects must follow the [Project CI Standard](PROJECT-STANDARD.md). Existing workflows must use the staged process in [Migrating Existing CI](MIGRATING-EXISTING-CI.md).
All projects must follow the [Project CI Standard](PROJECT-STANDARD.md). Existing workflows must use the staged process in [Migrating Existing CI](MIGRATING-EXISTING-CI.md). Operators use [How to add a fleet host](ADDING-A-HOST.md) and [How to add a project](ADDING-A-PROJECT.md) without introducing project-specific host state.