Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
252 changes: 252 additions & 0 deletions docs/QUICKSTART.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,252 @@
# Quickstart: your first fleet job

This is the beginner path. It takes you from zero to one verified job on
your own ephemeral CI fleet, then points at advanced material. Every step
states where it runs and whether it changes anything.

Advanced topics — schema-v3 internals, capacity promotion, health
monitoring, multi-site fleets — are linked at the end. You do not need
them for this path.

## Step 0: What ci-fleet does

ci-fleet runs your GitHub Actions jobs on your own Docker hosts using
**ephemeral** runners: a small controller service on each host watches
GitHub for queued jobs, starts a throwaway runner container for exactly
one job, and destroys it afterward. Runners are **host-privileged**:
each job gets the host Docker socket, which is host-root-equivalent. A
compromised job can therefore reach everything on the host, including
host-local controller credentials — this is why only trusted private
repositories may ever be authorized, and why host-local identity files
stay root-owned. Jobs also receive short-lived registration
configuration and may receive `GITHUB_TOKEN` or explicitly configured
project secrets. Projects bring their own toolchains in their own containers.
Capacity (how many jobs run at once) lives in reviewed private Git
configuration, never in application workflows.

One controller on one Docker host is a complete fleet.

## Step 1: Install one controller

Runs on: split between a management workstation (Git and GitHub work)
and the fresh Linux Docker host (installation only). Changes state: yes.
Comment thread
Nickfost marked this conversation as resolved.
Keep repository-writing credentials off the fleet host: anything with
write access retained there is reachable by later host-root-equivalent
jobs through the Docker socket.

1. Check the host qualifies: Docker, Compose v2, Git, Bash, outbound
HTTPS to GitHub, no project workloads, and a declared failure
Comment thread
Nickfost marked this conversation as resolved.
boundary (disposable or recoverable). See
[Adding a host](ADDING-A-HOST.md) sections 1-3. The installer itself
Comment thread
Nickfost marked this conversation as resolved.
additionally requires a systemd-based host with `python3`, `tar`,
`install`, `flock`, and standard coreutils — it fails closed with a
named missing command if one is absent.
2. On the **fleet host**, clone the public engine repository and check
out the exact reviewed engine commit you intend to pin — later steps
copy templates from it and run its installer as root, so it must be
the reviewed commit, not a moving branch. The pin must be a commit
merged into and reachable from the engine's public default branch;
never pin an unmerged local branch tip. Cloning the public engine
needs no repository-writing credential.

```bash
git clone https://github.com/RandomDevelopment/ci-fleet.git
git -C ci-fleet checkout PINNED_ENGINE_COMMIT
git -C ci-fleet merge-base --is-ancestor HEAD origin/main && echo merged
```

3. On the **management workstation** (GitHub web UI), create the GitHub
App and runner group the controller will use. The App must be
installed on the organization with organization-level
**Self-hosted runners: Read and write** permission; the concrete
bootstrap settings are in [Live pilot runbook](LIVE-PILOT.md)
sections 2-3, with one exception: create the runner group **without
selecting repositories** — repository authorization happens in Step 2
only after the proof workflow is staged, so no previously queued or
push-triggered job can become the first job. Then, on the
**fleet host**, place the host-local
identity files (root-owned, mode `0600`) from the checked-out
engine's templates as in [Adding a host](ADDING-A-HOST.md) section 5.
4. On the **management workstation**, create your private configuration
repository from the configuration template **versioned inside the
pinned engine checkout** — the schema, initializer, and validator the
pinned engine actually enforces live under its
`templates/config-repository`, while the standalone public template
may have advanced past your engine pin:

On the management workstation, clone the same pinned engine commit
(the previous clone lives on the fleet host), then copy its vendored
template:

```bash
git clone -q https://github.com/RandomDevelopment/ci-fleet.git
git -C ci-fleet checkout PINNED_ENGINE_COMMIT
cp -r ci-fleet/templates/config-repository my-fleet-config
Comment thread
Nickfost marked this conversation as resolved.
cd my-fleet-config
# Seed the vendored template on main so the initialization PR has a base:
git init -q -b main
git add -A && git commit -qm "Import ci-fleet configuration template"
git checkout -qb initialize-fleet
./scripts/init.sh \
--organization YOUR-ORG \
--project YOUR-APP-SLUG \
--repository YOUR-ORG/ACTUAL-REPOSITORY-NAME \
--controller YOUR-CONTROLLER-ID \
--location primary-site \
--runner-group YOUR-CREATED-RUNNER-GROUP \
--capacity-budget 1 \
--max-runners 1 \
--engine-ref PINNED_ENGINE_COMMIT
# Validate before committing so a rejected file or accidental value
# never enters branch history, then commit the initialized result:
./scripts/validate.sh --strict
git add -A && git commit -qm "Initialize fleet configuration"
```

`--project` is a logical lowercase slug; pass the real GitHub
repository name separately with `--repository` so
`allowed_repositories` names a repository that exists. Push both
branches to a new **private** GitHub repository, open a pull request
from `initialize-fleet` to `main`, and **merge it**. Managed controller managed controller
lifecycle is permitted only from a reviewed, merged private
configuration commit. Resolve and record that merge commit SHA
(`RESOLVED_CONFIG_COMMIT` below); do not install from an unmerged
branch.
5. Give the fleet host a way to read that private repository **as
root**: the installer fetches under `sudo` and the recurring drift
check runs as `User=root`, so a credential in an operator's
user-scoped Git configuration is invisible to both. Either configure
a narrowly scoped read-only credential in root's Git configuration,
or transfer a pinned local checkout from your management workstation
via a temporary Git bundle (the installer accepts a local checkout
path as `--config-repo`). Without one of these, the noninteractive
fetch fails closed. If you use the local-checkout path, keep that
checkout in place afterward — the recorded path is reused by
scheduled drift checks — and remove only the bundle.
6. On the **fleet host**, apply the merged configuration from the
reviewed engine checkout (`--install` for a fresh host; `--adopt` is
only for converting an existing manually installed controller):

```bash
sudo ci-fleet/scripts/install-worker-controller.sh \
--install \
--config-repo OWNER/PRIVATE-CONFIG-REPO \
Comment thread
Nickfost marked this conversation as resolved.
--ref RESOLVED_CONFIG_COMMIT \
--controller YOUR-CONTROLLER-ID
```

7. Verify: the same command with `--check` reports `CHECK_OK`, and the
GitHub runner group shows the scale set idle at zero runners.

## Step 2: Connect one repository

Runs on: management workstation (GitHub + the project repository +
private configuration). Changes state: yes, but no host changes —
onboarding never touches a fleet host.

1. Stage the proof workflow first: commit the dispatch-only job below
to the project repository's **default branch** as
`.github/workflows/fleet-proof.yml` (GitHub only offers
`workflow_dispatch` for workflows present on the default branch).
Do this **before** authorization so the first eligible job is the
controlled proof, never a push-triggered or previously queued job.
Comment thread
Nickfost marked this conversation as resolved.
Comment thread
Nickfost marked this conversation as resolved.

```yaml
name: Fleet first-job proof
on:
workflow_dispatch:
permissions:
contents: read
jobs:
proof:
runs-on: docker-ci # your pool's shared routing label
timeout-minutes: 5
steps:
- run: echo "fleet proof on ${RUNNER_NAME:-unknown}"
Comment thread
Nickfost marked this conversation as resolved.
```
2. Confirm the repository you passed as `--repository` to `init.sh` is
in the pool's `allowed_repositories` (the initializer put it there;
add it only if you skipped initialization), and validate with
`./scripts/validate.sh --strict`.
3. Authorize the repository in the GitHub runner group.

Full contract: [Adding a project](ADDING-A-PROJECT.md).

## Step 3: Run and verify one job

Runs on: dispatched from GitHub, but steps execute inside an ephemeral
runner on your fleet host with host-root-equivalent Docker access.
Changes state: yes, transiently — the job creates a runner container and
may create Docker containers, networks, and volumes; the proof below
verifies all of it is cleaned up.

1. Dispatch the staged proof workflow. Its explicit
`permissions: contents: read` and `timeout-minutes: 5` keep the job
from inheriting a read-write `GITHUB_TOKEN` default or occupying the
single runner past the ordinary-CI ceiling. Do not copy a nested
runner image reference into a job: image tags derive from each
controller's engine pin, and a workflow hard-coding one host's tag
can be routed to a host where that image does not exist. (The older
`examples/workflows/live-pilot.yml.example` targets the experimental
label and the `:dev` runner image and is not a managed-install
starter.)
2. Verify the job ran on your fleet: the Actions job metadata names the
runner, and the controller host's logs show that runner was created
by your scale set. (Runner names derive from the controller ID, not
necessarily the scale-set name, so use controller or scale-set
evidence rather than a name-prefix guess.)
3. On the fleet host, prove cleanup and health explicitly — job success
plus a returned-to-zero runner count alone does not prove the host
is clean:

```bash
# No ephemeral runner containers remain (the long-lived controller
# container is also fleet-managed, so filter by kind=runner):
sudo docker ps -aq \
--filter label=io.randomdevelopment.ci-fleet.managed=true \
--filter label=io.randomdevelopment.ci-fleet.kind=runner
# No run-owned project resources remain. Compliant jobs name Compose
# projects ci-<repo>-<run-id>-<attempt>-<task>-<shard>; the
# controller's own persistent ci-fleet_* resources are expected and
# excluded (runner containers are already covered by the label filter
# above):
sudo docker ps -a --filter "name=^ci-" --format '{{.Names}}' | grep -vx 'ci-fleet-controller-1'
Comment thread
Nickfost marked this conversation as resolved.
Comment thread
Nickfost marked this conversation as resolved.
sudo docker network ls --filter "name=^ci-" --format '{{.Name}}' | grep -vx 'ci-fleet_default'
sudo docker volume ls -q --filter "name=^ci-"
Comment thread
Nickfost marked this conversation as resolved.
# (No compliant job creates controller-prefixed volumes; the exact-name
# exclusions above avoid hiding project containers from a repository
# named 'fleet' or 'fleet-*', whose Compose names also start ci-fleet-.)
# Run a fresh health evaluation, then check installed state:
sudo systemctl start ci-fleet-health.service
sudo systemctl is-failed ci-fleet-health.service # expect: inactive/failed must NOT be failed
sudo ci-fleet/scripts/install-worker-controller.sh \
--check --config-repo OWNER/PRIVATE-CONFIG-REPO \
--ref RESOLVED_CONFIG_COMMIT --controller YOUR-CONTROLLER-ID
```

The health service is a timer-driven oneshot — `systemctl status`
alone only shows its previous run, so start it explicitly after the
proof job. Every resource listing above must be empty and the check
must report `CHECK_OK`. The [Live pilot runbook](LIVE-PILOT.md)
documents the full isolated first-job proof including read-only job
permissions.
4. Verify the controller returned to zero idle runners in the GitHub
runner group.

If the job stays queued: compare the job's complete `runs-on` expression
against the scale set's configured routing label, then check runner-group
repository authorization. A label or group mismatch is a configuration
bug, not a capacity problem.

## Step 4: Next steps (advanced)

- [Git-authored controller desired state](DESIRED-STATE.md) — the
schema-v3 model, lifecycle states, checkpoints, and rollback.
- [Capacity promotion](CAPACITY-PROMOTION.md) — raising
`max_runners`/budgets safely.
- [Fleet health monitoring](HEALTH-MONITORING.md) — heartbeats and
missed-heartbeat detection.
- [Migrating existing CI](MIGRATING-EXISTING-CI.md) — moving a real
project's test suite onto the fleet.
- [Architecture](ARCHITECTURE.md) and
[Project CI standard](PROJECT-STANDARD.md) — the full contract.
3 changes: 3 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,11 @@ Use this index to find ci-fleet concepts, requirements, examples, and step-by-st

## Start here

New operator? Follow the [Quickstart](QUICKSTART.md): what ci-fleet does, install one controller, connect one repository, run and verify one job. Everything below is reference.

| I want to… | Read |
| --- | --- |
| Complete the beginner path end to end | [Quickstart](QUICKSTART.md) |
| Understand the whole system | [Architecture](ARCHITECTURE.md) |
| Understand how ephemeral runners are created | [Runner controller design](CONTROLLER-DESIGN.md) |
| Decide whether it fits my infrastructure | [Architecture](ARCHITECTURE.md) and the root [README](../README.md) |
Expand Down