Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 41 additions & 2 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,18 @@ jobs:
- name: Validate shell and Python syntax
run: |
bash -n scripts/init.sh scripts/validate.sh
python3 -m py_compile scripts/init.py scripts/validate.py scripts/test_policy.py scripts/scan_committed_secrets.py
python3 -m py_compile scripts/init.py scripts/validate.py scripts/test_policy.py scripts/test_core_compatibility.py scripts/test_release_update.py scripts/scan_committed_secrets.py

- name: Prove forbidden configurations fail closed
run: python3 scripts/test_policy.py

- name: Verify exact core compatibility and release update flow
run: |
python3 scripts/test_core_compatibility.py
if [[ "$GITHUB_REPOSITORY" == RandomDevelopment/ci-fleet-config-template ]]; then
python3 scripts/test_release_update.py
fi

- name: Scan committed file contents for secrets
env:
EVENT_NAME: ${{ github.event_name }}
Expand All @@ -56,8 +63,33 @@ jobs:
done

- name: Validate reference configurations
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
./scripts/validate.sh
args=()
if [[ "$GITHUB_REPOSITORY" != RandomDevelopment/ci-fleet-config-template ]]; then
args+=(--strict)
fi
if [[ -f next-engine-rollout-evidence.json ]]; then
args+=(--next-engine-rollout-evidence next-engine-rollout-evidence.json)
fi
if [[ -n "$BASE_SHA" && ! "$BASE_SHA" =~ ^0+$ ]] && git cat-file -e "$BASE_SHA:fleet.json" 2>/dev/null; then
git show "$BASE_SHA:fleet.json" >"$RUNNER_TEMP/previous-fleet.json"
args+=(--previous-config "$RUNNER_TEMP/previous-fleet.json")
if git cat-file -e "$BASE_SHA:engine-rollout-evidence.json" 2>/dev/null; then
git show "$BASE_SHA:engine-rollout-evidence.json" >"$RUNNER_TEMP/previous-rollout-evidence.json"
args+=(--previous-rollout-evidence "$RUNNER_TEMP/previous-rollout-evidence.json")
fi
if git cat-file -e "$BASE_SHA:next-engine-rollout-evidence.json" 2>/dev/null; then
git show "$BASE_SHA:next-engine-rollout-evidence.json" >"$RUNNER_TEMP/previous-next-engine-rollout-evidence.json"
args+=(--previous-next-engine-rollout-evidence "$RUNNER_TEMP/previous-next-engine-rollout-evidence.json")
fi
./scripts/validate.sh "${args[@]}"
else
printf '{"controllers":{}}\n' >"$RUNNER_TEMP/previous-fleet.json"
args+=(--previous-config "$RUNNER_TEMP/previous-fleet.json")
./scripts/validate.sh "${args[@]}"
fi
./scripts/validate.sh --config examples/multi-host/fleet.json

- name: Prove initialized configurations pass strict policy
Expand Down Expand Up @@ -88,3 +120,10 @@ jobs:
--engine-ref "${engine_ref}" \
--output "${temporary_directory}/fleet.json"
test "$(stat -c '%a' "${temporary_directory}/fleet.json")" = 644
./scripts/validate.sh --strict --skip-path-scan --config "${temporary_directory}/fleet.json"
python3 -c 'import json,sys; p=sys.argv[1]; value=json.load(open(p, encoding="utf-8")); next(iter(value["controllers"].values()))["docker_network_policy"]={"networks_per_runner": 1, "reserve_subnets": 1, "default_address_pools": [{"base": "198.51.100.0/24", "size": 28}]}; json.dump(value, open(sys.argv[2], "w", encoding="utf-8"), indent=2)' "${temporary_directory}/fleet.json" "${temporary_directory}/rfc-5737.json"
if ./scripts/validate.sh --strict --skip-path-scan --config "${temporary_directory}/rfc-5737.json" 2>"${temporary_directory}/strict-errors"; then
echo 'RFC 5737 fixture unexpectedly passed strict validation' >&2
exit 1
fi
grep -F 'reviewed operational Docker pool CIDR' "${temporary_directory}/strict-errors"
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,13 @@ Before committing configuration changes, run:
## Hard rules

- Never add real `.env` files, credentials, tokens, private keys, cookies, or passwords.
- Never add addresses, VM IDs, storage identifiers, backup identifiers, SSH details, or rendered runtime configuration.
- Never add addresses, VM IDs, storage identifiers, backup identifiers, SSH details, or rendered runtime configuration. A reviewed Docker `default_address_pools[].base` CIDR is the sole address exception after engine support is staged.
- Do not weaken `public_repositories: false` for Docker-socket runner pools.
- Infrastructure configuration owns capacity. Application workflows submit all independent jobs and do not use `max-parallel` to model fleet size.
- Each GitHub runner group belongs to exactly one runner pool; do not create ambiguous cross-pool assignments.
- The sum of active and drained controller maxima must not exceed the pool capacity budget.
- Controller engine revisions and reusable workflows must be pinned to full reviewed commit SHAs.
- Upgrade an engine with active optional capabilities in two integrated commits: first keep engine A active and stage reviewed engine B evidence in optional adopter-owned `next-engine-rollout-evidence.json`; then change `engine_ref` to engine B, promote the matching sidecar record to active evidence, and remove the sidecar when empty.
- Production environments must require approval and must not deploy automatically.
- CI runner hosts and application deployment hosts are separate roles.
- Image promotion uses immutable digests; do not rebuild separately for production.
Expand Down
15 changes: 14 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This is the public, secret-free starting point for an organization's private `ci-fleet` configuration repository. It records which trusted projects may use each CI pool, the reviewed desired state for controller machines, infrastructure capacity budgets, logical deployment environments, and the standardized commands every project must expose.

It does **not** contain runner registration tokens, deploy credentials, private keys, host addresses, VM IDs, storage names, backup identifiers, or `.env` files.
It does **not** contain runner registration tokens, deploy credentials, private keys, host addresses, VM IDs, storage names, backup identifiers, or `.env` files. The sole address exception is a reviewed Docker `default_address_pools[].base` CIDR after an adopter has staged engine support. It is fleet capacity policy, not a host or service address. Public examples and fixtures use RFC 5737 ranges only.

```mermaid
flowchart LR
Expand Down Expand Up @@ -50,6 +50,15 @@ flowchart LR

The initializer refuses to replace a configured file unless `--force` is explicit. Run `./scripts/init.sh --help` for repository, registry, runner-group, controller, location, capacity, resource, and output options.

## Reviewed compatibility and release state

`template-compatibility.json` pins the reviewed embedded template to ci-fleet commit `0aed0d7e85e10050028b7d11fb12b84b3619e638`. It records schema and validator/initializer support, optional staged capabilities, the standalone new-controller safety check, and the unchanged fictional example engine pin. `engine-rollout-evidence.json` records active per-controller capability evidence in the shape consumed by that pinned core. New status-reporting or Docker network-policy fields require prior-state evidence. Upgrade an engine with active optional capabilities in exactly two integrated commits:

1. Keep `fleet.json` and `engine-rollout-evidence.json` on engine A. Add reviewed engine B capability evidence to optional adopter-owned `next-engine-rollout-evidence.json`, using the same schema-version-1 record shape.
2. Change `engine_ref` to engine B, promote the matching sidecar record to `engine-rollout-evidence.json`, and remove the sidecar when it has no remaining staged records.

The committed examples and initializer omit both optional fields because their engine capability has not been staged. When a private adopter later stages `docker_network_policy`, it must replace the RFC 5737 review fixture with a reviewed operational pool before strict validation passes. Initial release preparation and exclusions are in `docs/RELEASE.md`; no tag or GitHub release is published by this repository change.

## Schema v3: Git-authored controller desired state

`fleet.json` is the reviewed authority for logical controller state. Each entry in `controllers` has a unique ID and declares:
Expand Down Expand Up @@ -143,13 +152,17 @@ Adding workers reduces wall-clock time only while independent shards remain queu
|---|---|
| `fleet.json` | Fictional, valid schema-v3 configuration with one controller |
| `fleet.schema.json` | JSON Schema draft 2020-12 editor contract |
| `template-compatibility.json` | Exact reviewed core/template compatibility and prepared release state |
| `engine-rollout-evidence.json` | Active optional-capability evidence, empty in the public template |
| `next-engine-rollout-evidence.json` | Optional adopter-owned next-engine evidence sidecar; not committed by this public template |
| `scripts/init.sh` | Safe first-project and first-controller initializer |
| `scripts/validate.sh` | Structural, relational, capacity, and secret-boundary validation |
| `scripts/test_policy.py` | Regression tests proving unsafe configurations fail closed |
| `examples/multi-host/fleet.json` | Fictional two-project, two-location controller topology |
| `SECURITY.md` | Secret handling and vulnerability reporting |
| `AGENTS.md` | Non-negotiable rules for humans and coding agents |
| `docs/UPDATING.md` | Keeping a derived private repository current: schema vs template versions, releases, migrations, Dependabot |
| `docs/RELEASE.md` | Prepared initial release notes and immutable-tag publication checklist |

## Public and private boundary

Expand Down
16 changes: 16 additions & 0 deletions docs/RELEASE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Initial template release preparation

The `v1.0.0` template release is prepared but not published. Creating the tag or GitHub release requires separate authorization after this change merges.

The candidate synchronizes the standalone schema-v3 contract with `templates/config-repository` at reviewed ci-fleet commit `0aed0d7e85e10050028b7d11fb12b84b3619e638`. It adds optional staged status-reporting and Docker network-policy support. Existing configurations that omit either field remain valid.

The fictional examples retain engine commit `8df97cc7575f47696fa82a179bbe39cd2874b1ca` and omit both optional fields. Network-policy validation fixtures use RFC 5737 ranges only. A private adopter may add a reviewed operational `default_address_pools[].base` CIDR only after the compatible engine and capability evidence have existed in prior integrated states.

When release is authorized:

1. Confirm the release commit passes every required check.
2. Create annotated tag `v1.0.0` at that exact commit.
3. Publish release notes that name the tag object ID, peeled template commit, and reviewed core commit above.
4. Never retarget the tag. Correct a bad release with a new higher tag and mark the old release as superseded.

The compatibility details are machine-readable in `template-compatibility.json`. No tag or GitHub release exists as part of this preparation change.
43 changes: 32 additions & 11 deletions docs/UPDATING.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ the same rewrite-detection baseline; a local-only ref cannot do that.
Treat template files as vendored code: update them deliberately, not
casually.

The initial `v1.0.0` release is currently prepared, not published. Do not use it until the repository has an immutable tag and GitHub release. If a published release is wrong, leave its tag object untouched, publish a new higher tag, and mark the old release as superseded. Never retarget or recreate an existing release tag.

## GitHub template repositories have no fork ancestry

A repository created with GitHub's "Use this template" button is **not**
Expand All @@ -37,17 +39,19 @@ a fork: it has no git ancestry link to the template, so
Updating is an explicit operation:

1. Start from a clean tree — no uncommitted or unstaged changes,
especially to `fleet.json`; the procedure restores `fleet.json` from
the recorded pre-merge commit and would silently discard an
uncommitted edit. Then add the template as a remote and fetch its
tags into that remote's tracking namespace. `--no-tags` prevents Git
from also creating adopter-visible tags. A retargeted upstream tag
especially to `fleet.json`, `engine-rollout-evidence.json`, or the optional
`next-engine-rollout-evidence.json`; the
procedure restores adopter-owned files from the recorded pre-merge
commit and would silently discard an uncommitted edit. Then add the
template as a remote and fetch its tags into that remote's tracking
namespace. `--no-tags` prevents Git from also creating adopter-visible
tags. A retargeted upstream tag
updates a `refs/remotes/*` ref silently, so verify the reviewed
object ID yourself before using any previously fetched tag ref:

```bash
# Hard stop on any uncommitted state; a dirty tree would be silently
# overwritten by the fleet.json restore below.
# overwritten by the adopter-owned state restore below.
test -z "$(git status --porcelain)" || { echo "clean the tree first" >&2; exit 1; }
# one-time setup; on later runs require the existing remote to be the
# template, not an unrelated remote that happens to share the name:
Expand Down Expand Up @@ -100,19 +104,34 @@ Updating is an explicit operation:

If Git reports conflicts, leave the merge in progress and continue.
Whether or not it conflicted, restore the adopter-owned configuration
from the recorded pre-merge commit, then resolve and stage every other
conflict:
from the recorded pre-merge commit. Preserve rollout evidence from that
commit when it exists; otherwise keep the evidence introduced by the new
template release. Then resolve and stage every other conflict:

```bash
git restore --source="$ADOPTER_HEAD" --staged --worktree -- fleet.json
if git cat-file -e "$ADOPTER_HEAD:engine-rollout-evidence.json" 2>/dev/null; then
git restore --source="$ADOPTER_HEAD" --staged --worktree -- engine-rollout-evidence.json
fi
if git cat-file -e "$ADOPTER_HEAD:next-engine-rollout-evidence.json" 2>/dev/null; then
git restore --source="$ADOPTER_HEAD" --staged --worktree -- next-engine-rollout-evidence.json
fi
git status --short
```

If the release keeps the same `schema_version`, prove `fleet.json`
still has no staged change:
If the release keeps the same `schema_version`, prove the adopter-owned
configuration and rollout evidence still have no staged changes:

```bash
git diff --cached --exit-code -- fleet.json
git diff --cached --exit-code "$ADOPTER_HEAD" -- fleet.json
if git cat-file -e "$ADOPTER_HEAD:engine-rollout-evidence.json" 2>/dev/null; then
git diff --cached --exit-code "$ADOPTER_HEAD" -- engine-rollout-evidence.json
else
git diff --cached --exit-code "$MERGE_SOURCE" -- engine-rollout-evidence.json
fi
if git cat-file -e "$ADOPTER_HEAD:next-engine-rollout-evidence.json" 2>/dev/null; then
git diff --cached --exit-code "$ADOPTER_HEAD" -- next-engine-rollout-evidence.json
fi
```

3. Review the complete staged result — including any changes the merge
Expand Down Expand Up @@ -160,6 +179,8 @@ pinned contract. Releases that introduce a new `schema_version` are the
exception: import the new schema/validator in phase 2 of the two-phase
rollout below, after every deployed controller runs the new engine.

Migration programs use `scripts/migrate-v<old>-to-v<new>.py`. Release notes name the exact path. If a release changes `schema_version` but does not ship that named program, stop the update.

## Dependabot update PRs

Keep a `.github/dependabot.yml` in the private repository covering
Expand Down
4 changes: 4 additions & 0 deletions engine-rollout-evidence.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"schema_version": 1,
"status_reporting_engine_capabilities": {}
}
34 changes: 33 additions & 1 deletion fleet.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,39 @@
"engine_ref": {"type": "string", "pattern": "^(?!0{40}$)[0-9a-f]{40}$"},
"min_runners": {"const": 0},
"max_runners": {"type": "integer", "minimum": 1},
"runner_resources": {"$ref": "#/$defs/runner_resources"}
"runner_resources": {"$ref": "#/$defs/runner_resources"},
"docker_network_policy": {
"type": "object",
"additionalProperties": false,
"required": ["default_address_pools", "networks_per_runner", "reserve_subnets"],
"properties": {
"networks_per_runner": {"type": "integer", "minimum": 1},
"reserve_subnets": {"type": "integer", "minimum": 1},
"default_address_pools": {
"type": "array",
"minItems": 1,
"maxItems": 64,
"items": {
"type": "object",
"additionalProperties": false,
"required": ["base", "size"],
"properties": {
"base": {"type": "string"},
"size": {"type": "integer", "minimum": 0, "maximum": 29}
}
}
}
}
},
"status_reporting": {
"type": "object",
"additionalProperties": false,
"required": ["enabled", "config_file"],
"properties": {
"enabled": {"type": "boolean"},
"config_file": {"const": "/etc/ci-fleet/monitoring.env"}
}
}
}
},
"runner_resources": {
Expand Down
2 changes: 1 addition & 1 deletion scripts/init.py
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ def main() -> int:
finally:
temporary.unlink(missing_ok=True)
print(f"Initialized {output}")
print("Next: review controller capacity, configure GitHub policy, and keep every secret value outside Git.")
print("Next: review controller capacity, run ./scripts/validate.sh --strict, and keep every secret value outside Git.")
return 0


Expand Down
Loading