Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 4 additions & 11 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# Tag main with a semantic version and render the changelog. Deployments pin
# this template by tag, and the changelog commit lands untagged after it.
# Tags main and renders the changelog (CONTRIBUTING.md "Commands that prove a change").
name: release

on:
Expand Down Expand Up @@ -28,8 +27,7 @@ jobs:
with:
python-version: "3.15"

# Gate before tagging. A tag is permanent, so a broken governance tree must
# not get one -- the same content validate-governance.yaml runs on PRs.
# A tag is permanent, so the governance gate runs first.
- name: Install pinned PyYAML
run: python3 -m pip install --require-hashes -r tools/requirements-validate.txt

Expand All @@ -47,9 +45,7 @@ jobs:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: npx semantic-release

# The tag is the only reliable signal that a release happened:
# semantic-release exits 0 either way, and most pushes here are docs or
# config changes that earn no bump.
# semantic-release exits 0 whether or not it tagged, so the tag is the signal.
- name: Did it release?
id: rel
run: |
Expand Down Expand Up @@ -84,10 +80,7 @@ jobs:
git config user.email "ci@hyperi.io"
git add CHANGELOG.md
git commit -m "chore(release): ${{ steps.rel.outputs.tag || 'changelog entries' }} [skip ci]"
# The concurrency group serialises releases, not other pushes to main,
# so a commit landing meanwhile makes this push non-fast-forward. Only
# this job writes CHANGELOG.md, so replaying the commit on the new tip
# cannot conflict.
# Another push can land meanwhile, so retry on a rebase. A conflicting CHANGELOG.md edit stops the job.
for attempt in 1 2 3 4 5; do
if git push origin HEAD:main; then
exit 0
Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/validate-governance.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
# Validate the governance/ tree (shipped actions + protected-var policies) on
# every PR and push. Structural mirror of dfe-engine's rules - a broken dial
# never merges. Chart-path drift is dfe-infra's chart validation's job.
# The governance/ gate (CONTRIBUTING.md). A change outside governance/ and tools/ runs no check.
name: validate-governance

on:
Expand Down
47 changes: 20 additions & 27 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,26 +6,22 @@ Depth, and the reasoning behind the layout: [docs/architecture.md](docs/architec

### What this is

The template a deployment's git store is seeded FROM -- never itself a live
deploy repo, so nothing here is reconciled to a cluster. A deployment takes its
own read-write copy and Argo CD, dfe-engine and operators read and write that.
Not the base either: charts and ApplicationSets live in dfe-infra, pulled by
reference at the `pins.yaml` tag. One validator aside, there is no code here and
no Kubernetes manifest.
The template a deployment's git repo is copied FROM -- never itself a live deploy repo, so nothing here is reconciled to a cluster. A deployment takes a read-write copy, and Argo CD, dfe-engine and operators read and write that. Not the base either: charts and ApplicationSets live in dfe-infra, which Argo CD renders at the cluster secret's `target_revision`. The code is in `tools/`, two scripts and their tests, and there is no Kubernetes manifest.

### Where things live

| Path | Holds |
|---|---|
| `pins.yaml` | the dfe-infra tag selecting the whole certified stack, the release `channel`, per-component `overrides` |
| `pins.yaml` | the dfe-infra stack the deployment runs, the release `channel`, per-component `overrides` |
| `values/` | one `{service}-{instance}-values.yaml` per APP instance -- file presence enables the app |
| `infra/` | `common.yaml` plus one `<chart>.yaml` per data or platform chart |
| `config/` | hot-read config-as-code: `sources/`, `schemas/`, `rules/`, `hunts/`. The engine adds `compat-profiles/` and `library/` on first write |
| `governance/` | five curated dials in `actions/`, protected-var patterns in `policies/` |
| `infra/` | `common.yaml` plus one `<app>.yaml` per data, platform or edge chart |
| `config/` | config-as-code dfe-engine reads: `sources/`, `schemas/`, `rules/`, `hunts/`. The engine adds `compat-profiles/` and `library/` on first write |
| `governance/` | curated dials in `actions/`, protected-var patterns in `policies/`, byte-identical to the copies dfe-engine seeds |
| `sizing/` | the sizing resolver's committed output, none in the template |
| `tools/validate_governance.py` | the structural validator for `governance/` |
| `tools/changelog_backfill.py` | adds a `CHANGELOG.md` entry for any release tag the file misses, run by `release.yaml` |

Every directory carries its own README. Read `values/README.md` and
`infra/README.md` before a first edit.
Every directory has a README. Read `values/README.md` and `infra/README.md` before a first edit.

### Commands that prove a change

Expand All @@ -34,13 +30,10 @@ python3 tools/validate_governance.py
python3 -m unittest discover -s tools
```

The whole local gate: the required fields of each action and policy, closed param constraints, reference wiring
and the guard against an action changing a governance class. How it lies:
The whole local gate: the required fields of each action and policy, closed param constraints, declared param references, known resource classes, pinned image refs, and the guard against an action changing a governance class. How it lies:

- `validate-governance.yaml` filters `paths:` on BOTH `push` and `pull_request`,
to `governance/**` and `tools/**`. A PR touching only `values/`,
`config/`, `pins.yaml` or a README shows no checks -- did not run, not passed.
- `release.yaml` carries no path filter and re-runs the gate before tagging, so a
- `validate-governance.yaml` filters `paths:` on BOTH `push` and `pull_request`, to `governance/**` and `tools/**`. A PR touching nothing but `values/`, `config/`, `pins.yaml` or a README shows CodeQL alone: the governance gate did not run, which is not a pass.
- `release.yaml` has no path filter and re-runs the gate before tagging, so a
docs change is checked on merge rather than in review. semantic-release exits 0
either way, so it reads `git tag --points-at HEAD` instead.
- Nothing validates `values/`, `infra/` or `config/`. Chart-var drift belongs to
Expand All @@ -51,17 +44,17 @@ and the guard against an action changing a governance class. How it lies:
| Don't | Do | Why |
|---|---|---|
| Park YAML under `values/`, subdirectories included | Put anything that is not an app instance in `infra/` | Argo hands the `values/*-values.yaml` glob to `git ls-files` as a pathspec, where `*` matches `/` -- so a subdirectory file becomes a phantom Argo app (`99289a4`, `976a898`) |
| Fork this repo | Use the template, or clone at the tag and re-home | A fork of a public repo cannot be private, and base updates arrive by bumping `pins.yaml`, never by merging template history (`9b800f1`) |
| Omit `deploy:` from an instance file | Name `deploy.service` and `deploy.instance` | The ApplicationSet reads it to name the Application, so the file yields no app and no error (`1874994`) |
| Expect a Job to apply `config/schemas/` | Treat it as the engine's additive overlay | There is no Job -- the engine applies every object at its own startup. `config/README.md` claimed one until `e3b2ca9` (#4) |
| Expect an untouched `dfe-receiver` to take traffic | Name the senders in `exposure.groups.ingest.allow` in `infra/common.yaml` | On the load balancer door an EMPTY allow list renders no load balancer at all, and a `/0` opens it to everyone. The retired `exposure.public.loadBalancerSourceRanges` fails the render |
| Set `env:` or `transformFiles` in an instance file | Use `otel.serviceName` and `fileSets.<set>.files` | `env:` is not a key the thin charts read and renders nothing. `transformFiles` fails the render by name |
| Fork this repo | Use the template, or clone at the tag and re-home | A fork of a public repo cannot be private, and base updates arrive through `pins.yaml` and `dfe-ops upgrade`, never by merging template history (`9b800f1`) |
| Omit `deploy:` from an instance file | Name `deploy.service` and `deploy.instance` | The ApplicationSet reads it to name the Application and renders with `missingkey=error`, so the file yields no app (`1874994`) |
| Expect a Job to apply `config/schemas/` | Treat it as the engine's additive overlay | There is no Job -- the engine applies every object at its startup (`e3b2ca9`) |
| Expect an untouched `dfe-receiver` on a cloud to take traffic | Name the senders in `exposure.groups.ingest.allow` in `infra/common.yaml` | On the load balancer door, the cloud default, an EMPTY allow list renders no load balancer at all, and a `/0` opens it to everyone. The retired `exposure.public.loadBalancerSourceRanges` fails the render |
| Set `env:` or `transformFiles` in an instance file | Use `extraEnv`, `otel.serviceName` and `fileSets.<set>.files` | `env` is the environment name the ApplicationSet sets as a parameter, so a variable map there renders nothing. `transformFiles` fails the render by name |

### Where this sits

| Repo | Direction | Kind | Mechanism |
|---|---|---|---|
| dfe-infra | inbound | `version-pin` | `pins.yaml` `base.dfe-infra` names one tag whose `versions.yaml` is the manifest for every app image, operator and data service. Pulled by reference, never vendored. `pins.yaml` also calls `dfe-infra/scripts/dfe-stack` by path. Move it, then re-run the validation |
| dfe-schemas | inbound | none | No pin, deliberately, since `e3b2ca9` (2026-09-18). The wheel rides inside the dfe-engine image, so the dfe-infra pin already selects the schema release |
| dfe-infra | outbound | `version-pin` | `versions.yaml` `content.dfe-deploy` pins this repo's tag per stack and bootstrap seeds from it, so a change lands in a deployment only once a stack pins a newer tag |
| dfe-engine | outbound | commits into a copy | Writes a deployment's COPY through gitcrud -- `helmvars` for `values/`, `governance/`, and the config classes. Never the base, never this repo. Not declared in `dfe-infra/suite.yaml` |
| dfe-infra | inbound | `version-pin` | `pins.yaml` `base.dfe-infra` names the stack. Its `versions.yaml` lists every app image, operator and data service. `dfe-ops upgrade` and `dfe-stack resolve` read it, and Argo CD renders from the cluster secret's `target_revision`. The `pins.yaml` usage comment calls `dfe-infra/scripts/dfe-stack` by path: move the script, then update the comment |
| dfe-schemas | inbound | none | No pin, deliberately, since `e3b2ca9` (2026-09-18). The wheel rides inside the dfe-engine image, so the dfe-infra pin selects the schema release |
| dfe-infra | outbound | `version-pin` | `versions.yaml` `content.dfe-deploy` pins this repo's tag per stack, and `dfe-stack release-gate` fails a stack when the pinned template's `pins.yaml` names another stack. Bootstrap does not seed from it yet ([dfe-infra#13](https://github.com/hyperi-io/dfe-infra/issues/13)) |
| dfe-engine | outbound | commits into a copy | Writes a deployment's COPY through gitcrud -- `helmvars` for `values/`, `infravars` for `infra/`, the governance classes and the config classes -- and seeds any shipped action or policy file the copy lacks. Never the base, never this repo. Not declared in `dfe-infra/suite.yaml` |
41 changes: 29 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,44 +6,61 @@ The template a DFE deployment's own git repo is seeded from.

## What it does

- Each deployment takes its own read-write copy of this repo. That copy holds everything local to the deployment, and the base, dfe-infra, is pulled by reference at the tag `pins.yaml` names, never vendored in.
- `pins.yaml`: one dfe-infra tag selects the certified stack as the baseline, with the release channel and any per-component overrides beside it. Updating the base is a one-line bump that leaves your files alone.
- `values/`: one Helm values file per app instance. A file named for the instance and carrying its `deploy` block is what turns the app on.
- Each deployment takes a read-write copy of this repo, which holds everything local to the deployment. The base, dfe-infra, stays in its repo and Argo CD reads it by reference.
- `pins.yaml`: the dfe-infra stack the deployment runs, with the release channel and any per-component overrides beside it.
- `values/`: one Helm values file per app instance. A file named for the instance, with its `deploy` block, is what turns the app on.
- `infra/`: an optional `common.yaml` plus one optional values file per data or platform chart.
- `config/`: sources, detection rules and hunts as config-as-code, read hot by the DFE apps and dfe-engine, and `config/schemas/`, the additive schema overlay dfe-engine applies at startup.
- `config/`: sources, detection rules and hunts as config-as-code, read by dfe-engine, and `config/schemas/`, the additive schema overlay dfe-engine applies at startup.
- `governance/`: the dials and protected-variable policies, checked by `tools/validate_governance.py`.
- `sizing/`: the sizing resolver's committed output.

dfe-engine writes only this copy, by commit, so the audit trail is its git history. It never writes the base.
dfe-engine writes to this copy by commit, so the audit trail is its git history. It never writes the base.

## Quick start

Take a copy, never a fork. A fork of a public repo cannot be private, and base updates arrive by bumping `pins.yaml`, never by merging template history. Three ways, in order of preference:
Take a copy, never a fork. A fork of a public repo cannot be private, and base updates arrive through `pins.yaml`, never by merging template history. Three ways:

1. Let dfe-infra's bootstrap seed the deploy repo in the bundled in-cluster Forgejo from this template at the stack's tag. This is the default and works air-gapped.
2. GitHub "Use this template" or GitLab "new from template", for a fresh private repo with no upstream lineage.
3. Clone at the release tag, remove `origin`, push to your own remote.
1. GitHub "Use this template" or GitLab "new from template", for a fresh private repo with no upstream lineage.
2. Clone at the release tag, remove `origin`, push to your remote.
3. Leave the deploy repo unset and dfe-infra's bootstrap creates one in the bundled in-cluster Forgejo. That repo holds `sizing/` and a values file per app, not this template's files ([dfe-infra#13](https://github.com/hyperi-io/dfe-infra/issues/13)).

Then check the governance files before the first commit:

```bash
python3 tools/validate_governance.py
```

It prints `OK:` with the action and policy counts, and exits 1 on any error or on an empty `governance/`.

## Configuration

Bump the dfe-infra version in `pins.yaml` to update the base. Your `values/` and `config/` are untouched, and the schema release moves with the dfe-engine image that stack pins, because the dfe-schemas wheel rides inside it. Who writes what: `pins.yaml` is yours, `values/` is yours and dfe-engine's in the same files, `config/` is dfe-engine's and your analysts'. Per-user console state (saved searches, dashboards) is not here. It lives in the app databases.
`dfe-ops upgrade apply --deploy <your copy> --push` moves a deployment to a newer stack along a path `dfe-stack check-upgrade` reports as `SUPPORTED`. It bumps `base.dfe-infra` in `pins.yaml`, pushes the commit and moves the cluster secret's `target_revision` with it. Without `--push` it rejects a move that needs the retarget, and a hand edit of the pin alone moves no chart. Between two releases there is no verified path, so a release is a fresh deploy (dfe-infra [docs/deployment/upgrades.md](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/upgrades.md)).

The schema release moves with the dfe-engine image the stack pins, because the dfe-schemas wheel rides inside it, so `pins.yaml` has no dfe-schemas pin.

`channel` records the maturity the deployment tracks (`alpha`, `beta`, `rc` or `release`), and no tool selects anything by it. `overrides` moves one app off the certified set, by `tag@sha256` and nothing looser, and dfe-engine logs a SUPPORT-DRIFT notice at startup for each entry:

```yaml
overrides:
apps:
dfe-receiver: "v1.16.0@sha256:..."
```

`dfe-stack resolve --pins pins.yaml --emit-values -` prints the values fragment to merge into that service's existing instance file, never a separate file.

Who writes what: `pins.yaml` is yours, `values/` and `infra/` are yours and dfe-engine's in the same files, `config/` is dfe-engine's and your analysts', and `governance/` is dfe-engine's and your admins'. Per-user console state (saved searches, dashboards) is not here. It lives in the app databases.

## Documentation

- [docs/architecture.md](docs/architecture.md): how a deployment repo is laid out, and why.
- [values/README.md](values/README.md), [infra/README.md](infra/README.md) and [config/README.md](config/README.md): each directory's own rules, to read before a first edit.
- [values/README.md](values/README.md), [infra/README.md](infra/README.md), [config/README.md](config/README.md), [governance/README.md](governance/README.md) and [sizing/README.md](sizing/README.md): each directory's rules, to read before a first edit.
- dfe-infra [docs/deployment/state-and-repos.md](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/state-and-repos.md) and [docs/deployment/stack-versioning.md](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/stack-versioning.md): the canonical model and the diagrams.

The suite's published docs are at <https://dfe.docs.hyperi.io>, and this repository's pages at <https://dfe.docs.hyperi.io/dfe-deploy/>.

## Help and contributing

How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in a new issue.
How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in an issue.

Open bugs: [this repo](https://github.com/hyperi-io/dfe-deploy/issues?q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor) | [whole DFE suite](https://github.com/search?type=issues&q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor+repo%3Ahyperi-io%2Fdfe-infra+repo%3Ahyperi-io%2Fdfe-docker+repo%3Ahyperi-io%2Fdfe-deploy+repo%3Ahyperi-io%2Fdfe-engine+repo%3Ahyperi-io%2Fdfe-loader+repo%3Ahyperi-io%2Fdfe-receiver+repo%3Ahyperi-io%2Fdfe-fetcher+repo%3Ahyperi-io%2Fdfe-archiver+repo%3Ahyperi-io%2Fdfe-transform-vrl+repo%3Ahyperi-io%2Fdfe-transform-vector+repo%3Ahyperi-io%2Fdfe-transform-elastic+repo%3Ahyperi-io%2Fdfe-ui+repo%3Ahyperi-io%2Fdfe-hyperdx+repo%3Ahyperi-io%2Fscalo-rs+repo%3Ahyperi-io%2Fscalo-py+repo%3Ahyperi-io%2Flogreducer+repo%3Ahyperi-io%2Fdfe-schemas+repo%3Ahyperi-io%2Fvector-vrl+repo%3Ahyperi-io%2Fculvert+repo%3Ahyperi-io%2Fdfe-clickhouse-rs+repo%3Ahyperi-io%2Ffactbook-rs)

Expand Down
Loading
Loading