Redesign the public README and documentation

docs/CONTROLLER-DESIGN.md

@@ -0,0 +1,69 @@
+# Runner controller design
+
+## Purpose
+
+The controller turns GitHub job demand into disposable Dockerized GitHub Actions runners. It lets trusted private repositories share idle capacity without installing project runtimes directly on fleet hosts.
+
+A persistent runner service per VM fragments capacity and retains job state. This controller instead creates one runner for one job and destroys it afterward.
+
+## How it works
+
+```mermaid
+flowchart TD
+  A[GitHub App private key] -->|read-only mounted secret| C[Controller]
+  G[GitHub job queue] --> C
+  C -->|short-lived registration| R[Ephemeral runner]
+  C -->|create, observe, log, destroy| R
+  R -->|Docker socket| D[Host Docker Engine]
+  D --> P[Project-owned test containers]
+  P -. no fleet credential .-> X[No controller credential access]
+```
+
+The controller:
+
+1. authenticates to GitHub with a GitHub App;
+2. watches demand for its uniquely named scale set;
+3. generates short-lived just-in-time runner configuration;
+4. creates a runner container with resource limits, rotated logs, ownership labels, and the Docker socket;
+5. observes one assigned job;
+6. retains final diagnostics outside the disposable runner;
+7. destroys the runner and its writable state;
+8. reconciles capacity with current demand.
+
+On restart, it recovers only stale runners carrying the same fleet-instance label. It does not manage unrelated Docker workloads or another host's scale set.
+
+## Security boundary
+
+Docker socket access is host-root-equivalent. This pool is therefore limited to trusted repositories and trusted workflow revisions. Containers provide repeatability and cleanup; they do not make hostile workflow code safe.
+
+The GitHub App private key exists only as a file-mounted controller secret. A runner receives only its short-lived registration configuration. Project secrets remain in GitHub repository or environment settings and are supplied only to jobs that explicitly require them.
+
+## Choosing a runner model
+
+ci-fleet is optimized for organizations that already operate Linux and Docker hosts, trust the repositories using the pool, and want idle capacity shared across projects. Other runner models may fit different requirements.
+
+| Runner model | Good fit | Important tradeoff |
+| --- | --- | --- |
+| Persistent self-hosted runner | A small setup with one trusted repository and minimal orchestration | Workspaces and process state can survive between jobs, and idle capacity stays tied to that runner |
+| Repository-specific runners sharing one host | Projects that require distinct GitHub registration or routing | Multiple services may compete for the same CPU, memory, disk, ports, and Docker daemon |
+| Ephemeral Docker runners with ci-fleet | Multiple trusted repositories sharing Linux or Docker capacity across one or many locations | Runners are disposable, but jobs sharing a Docker daemon remain inside one host security boundary |
+| Actions Runner Controller on Kubernetes | Organizations that already operate Kubernetes and want Kubernetes-native scaling | Requires a Kubernetes control plane and its associated operations |
+| Disposable VM per job | Higher-isolation or higher-risk workloads | Stronger separation costs more startup time, storage, and provisioning infrastructure |
+| GitHub-hosted runners | Teams that prefer managed capacity and do not require self-hosted resources or networks | Provides less control over hardware, locality, caching, and private infrastructure access |
+
+The current controller uses the official `actions/scaleset` client at a pinned revision. Updates require a reviewed build and live validation before rollout.
+
+## Operations and rollback
+
+Hosts must be treated as disposable infrastructure, patched automatically, monitored for disk pressure, and restricted to trusted jobs.
+
+To roll back an experimental controller:
+
+1. stop new capacity;
+2. confirm no managed job is active;
+3. stop the controller;
+4. run scoped cleanup for that fleet instance;
+5. verify that host's scale set is absent from GitHub;
+6. restore the last reviewed image and configuration if needed.
+
+Existing project-specific CI remains available until migration validation is complete.

docs/PUBLIC-PRIVATE-CONFIGURATION.md

@@ -0,0 +1,91 @@
+# Public projects, private delivery, and private configuration
+
+ci-fleet is public so operators can inspect, reuse, review, and improve the engine. A real installation contains organization-specific topology and policy that should not be published. Secret values do not belong in either public or private Git history.
+
+The fleet supports both private application repositories and public applications that use a separate private repository for privileged CI, release, and deployment.
+
+## Supported repository models
+
+```mermaid
+flowchart LR
+  F[Public ci-fleet engine] --> T[Public configuration template]
+  T --> C[Private organization configuration]
+
+  subgraph PublicProject[Public application]
+    PS[Public source repository] --> GH[GitHub-hosted or unprivileged PR checks]
+    PS -->|approved commit or tag| PD[Private delivery repository]
+  end
+
+  subgraph PrivateProject[Private application]
+    PR[Authorized private source repository]
+  end
+
+  C --> PD
+  C --> PR
+  PD --> RG[Restricted private runner group]
+  PR --> RG
+  RG --> W[Ephemeral fleet workers]
+  W --> TEST[Dockerized tests]
+  W --> ENV[Protected staging or production environment]
+```
+
+A private application repository may use the restricted runner group directly. A public application repository does not. Its private delivery repository selects an approved public commit, runs the privileged work, and owns the deployment policy and secrets.
+
+## What belongs where?
+
+| Location | Contains |
+| --- | --- |
+| Public fleet repository | Schemas, generic controller and deployment code, validation, examples, standards, and reusable interfaces |
+| Public configuration template | Fictional examples and the expected private-repository structure |
+| Public application repository | Source code, public tests, public Dockerfiles, and unprivileged workflows |
+| Private application repository | Source code and workflows authorized to use an appropriate private runner group |
+| Private delivery repository for a public project | Approved source mappings, CI and deployment workflows, environment policy, promotion rules, and required secret names |
+| Private organization configuration | Repository mappings, logical host groups, environment policy, capacity, image names, and internal operating notes |
+| Secret manager, GitHub environment, or host-local protected file | Actual keys, tokens, passwords, and deployment credentials |
+
+Each organization generates its own private installation configuration from the public [configuration template](../templates/config-repository/README.md). The resulting private repository is an implementation detail of that organization and is not required to be accessible to users of the public project.
+
+## Public source with private CI and deployment
+
+A public project can use fleet workers without granting the public repository access to them:
+
+1. Pull requests run on GitHub-hosted runners or another unprivileged environment.
+2. A maintainer, protected workflow, release tag, or trusted automation selects an approved commit.
+3. The private delivery repository receives or records the exact public repository and full commit SHA.
+4. It verifies that the repository, revision, branch or tag policy, and requested operation are allowed.
+5. Its private workflow checks out that exact revision and runs the project's Dockerized tests on the fleet.
+6. Deployment runs only through a protected private environment with the minimum required secrets and approvals.
+7. Results and release metadata may be published back without exposing runner or deployment credentials.
+
+The private delivery repository is the authorized GitHub Actions identity. Fleet hosts and worker images remain generic and do not contain the public project's name or credentials.
+
+## Security requirements for public projects
+
+- Never authorize the public source repository or its forks for a privileged self-hosted runner group.
+- Never allow pull-request code to choose an arbitrary repository URL, Git ref, shell command, runner label, environment, or deployment target.
+- Use a full immutable commit SHA and verify that it belongs to the approved public repository.
+- Treat artifacts produced by an untrusted public workflow as untrusted; rebuild them privately or verify their provenance before signing or deployment.
+- Keep the private delivery workflow, repository allowlist, promotion policy, and environment protections outside the public repository's control.
+- Keep validation runners and deployment runners separated when deployment credentials or internal network access are involved.
+- Give workflows and GitHub Apps only the permissions required for the selected operation.
+- Ensure logs and published results cannot reveal private configuration or credentials.
+
+## General rules
+
+- Public repositories never receive direct access to privileged self-hosted runner groups.
+- Reusable public workflow references are pinned to reviewed immutable commits.
+- Public examples use fictional organizations, domains, repositories, hosts, and addresses.
+- Private configuration may declare required secret names but never their values.
+- GitHub App keys, tokens, project secrets, and deployment credentials stay outside Git history.
+- Real private configuration is validated against the same public schema used by examples.
+- A host remains generic; adding a repository changes policy and project configuration, not every worker image.
+
+## Secret locations
+
+Use one of these mechanisms according to the credential's scope:
+
+- GitHub repository or environment secrets for job-specific credentials;
+- root-owned host-local files for a small single-host controller;
+- an external secret manager for a distributed or higher-assurance fleet.
+
+See the [secrets model](SECRETS.md) and [security policy](../SECURITY.md) before configuring a live installation.

docs/README.md

@@ -0,0 +1,86 @@
+# Documentation
+
+Use this index to find ci-fleet concepts, requirements, examples, and step-by-step procedures. Repository Markdown is the authoritative documentation and is versioned with the implementation it describes.
+
+## Start here
+
+| I want to… | Read |
+| --- | --- |
+| Understand the whole system | [Architecture](ARCHITECTURE.md) |
+| Understand how ephemeral runners are created | [Runner controller design](CONTROLLER-DESIGN.md) |
+| Decide whether it fits my infrastructure | [Architecture](ARCHITECTURE.md) and the root [README](../README.md) |
+| Try the experimental implementation safely | [Live pilot runbook](LIVE-PILOT.md) |
+| Add another Docker host, VM, computer, or VPS | [Adding a host](ADDING-A-HOST.md) |
+| Add a private project to the shared runner pool | [Adding a project](ADDING-A-PROJECT.md) |
+| Convert an existing GitHub Actions workflow | [Migrating existing CI](MIGRATING-EXISTING-CI.md) |
+| Make a project compliant | [Project CI standard](PROJECT-STANDARD.md) and [compliance checklist](COMPLIANCE-CHECKLIST.md) |
+| Split tests across parallel workers | [Project CI standard](PROJECT-STANDARD.md) and the [parallel workflow example](../examples/workflows/parallel-ci.yml.example) |
+| Configure automatic updates and cleanup | [Host maintenance](HOST-MAINTENANCE.md) |
+| Handle GitHub App, workflow, or deployment secrets | [Secrets model](SECRETS.md) and [security policy](../SECURITY.md) |
+| Run private CI or deployment for a public project | [Public projects, private delivery, and private configuration](PUBLIC-PRIVATE-CONFIGURATION.md) |
+| Review current priorities | [Roadmap](ROADMAP.md) |
+| See what informed the design | [Discovery summary](DISCOVERY-SUMMARY.md) |
+
+## Concepts
+
+| Term | Meaning |
+| --- | --- |
+| Fleet host | A generic Linux machine, VM, or VPS running Docker and a controller. It contains no project runtime. |
+| Controller | Host-side service that watches GitHub demand and creates or removes ephemeral runners. |
+| Ephemeral runner | A disposable GitHub Actions runner container that accepts one job and is destroyed. |
+| Project test container | The project-owned Docker image containing its Node, PHP, Python, database, or other test environment. |
+| Runner group | GitHub organization policy that controls which repositories may use a runner pool. |
+| Scale set | One controller's uniquely named runner capacity advertised to GitHub. |
+| Shared routing label | The stable `runs-on` capability used by compatible projects, regardless of which physical host accepts the job. |
+| Test shard | One bounded slice of a larger test suite, designed to run independently and usually finish within five minutes. |
+| Private delivery configuration | Organization names, repository allowlists, host inventory, capacity, network policy, and credentials kept outside this public repository. |
+
+## Design and boundaries
+
+- [System architecture](ARCHITECTURE.md)
+- [Runner controller design](CONTROLLER-DESIGN.md)
+- [Public projects, private delivery, and private configuration](PUBLIC-PRIVATE-CONFIGURATION.md)
+- [Secrets model](SECRETS.md)
+- [Security policy](../SECURITY.md)
+
+## Standards and contracts
+
+These pages are normative for compatible projects and hosts:
+
+- [Project CI standard](PROJECT-STANDARD.md)
+- [Migration procedure](MIGRATING-EXISTING-CI.md)
+- [Compliance checklist](COMPLIANCE-CHECKLIST.md)
+- [Host maintenance standard](HOST-MAINTENANCE.md)
+- [Secrets model](SECRETS.md)
+- [Security policy](../SECURITY.md)
+
+## Operator how-tos
+
+- [Run the live pilot](LIVE-PILOT.md)
+- [Add a host](ADDING-A-HOST.md)
+- [Add a project](ADDING-A-PROJECT.md)
+- [Deploy the current experimental prototype](DEPLOYMENT-PROTOTYPE.md)
+- [Maintain, drain, clean, update, and reboot hosts](HOST-MAINTENANCE.md)
+- [Use the public configuration-repository scaffold](../templates/config-repository/README.md)
+
+The accepted target is a single target-host command, `sudo ./setup.sh`. Until that installer is implemented and released, follow the versioned pilot and deployment procedures above rather than assuming the future interface exists.
+
+## Project integration examples
+
+- [Read-only experimental workflow](../examples/workflows/experimental-smoke.yml.example)
+- [Private-repository live pilot](../examples/workflows/live-pilot.yml.example)
+- [Parallel five-minute job matrix](../examples/workflows/parallel-ci.yml.example)
+- [Project task plan](../examples/project/scripts/ci/plan.json)
+- [Standard project CI entrypoint](../examples/project/scripts/ci/run.sh)
+- [Project test Dockerfile](../examples/project/Dockerfile.test)
+- [Isolated Docker Compose configuration](../examples/project/compose.ci.yaml)
+
+Examples use fictional values and are starting points. Pin reviewed actions and images before production use.
+
+## Documentation rules
+
+- Mandatory instructions live in this repository.
+- Implementation changes update affected documentation in the same pull request.
+- Real secrets, hostnames, internal addresses, and private repository inventories never appear in public examples.
+- The GitHub Wiki is not an independent source of operational truth.
+- A future generated documentation site may improve browsing, but repository Markdown remains its source.