From 1da905f3ba1743b52583586ca69ec5e220f3ada4 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 14:45:06 +1100 Subject: [PATCH 1/2] docs(deploy-repo): fix claims about dfe-infra and dfe-engine A pins.yaml bump does not move the base. Argo CD renders from the cluster secret's target_revision. dfe-ops upgrade apply moves the pin and target_revision. Bootstrap creates the bundled Forgejo repo without this template (dfe-infra#13). - the schema overlay variable is DFE_ENGINE_SCHEMAS_OVERLAY_DIR - an invoke against a missing instance file creates it with its deploy block, so it deploys the instance - dfe-engine and its hunt runner read config/, not the other apps - the data ApplicationSet reads sizing/ straight from this repo, under infra/ - the infra/ file list matches the ApplicationSets - dfe-schema.yaml is removed - the values cascade lists every layer the ApplicationSets stack - each storage model states its storage.md status - the AWS bucket takes precedence over an infra/ overlay - engine seeding of governance/ and .seed-ignore are documented The repo's top diagram is added to docs/README.md. Config comments in pins.yaml, both workflows and the validator requirements are cut to pointers and short reasons. The governance/ YAML stays byte-identical to dfe-engine's seeded copies. --- .github/workflows/release.yaml | 15 +- .github/workflows/validate-governance.yaml | 4 +- CONTRIBUTING.md | 47 +++--- README.md | 41 ++++-- config/README.md | 57 +++---- docs/README.md | 38 ++++- docs/architecture.md | 163 ++++++--------------- governance/README.md | 53 ++----- infra/README.md | 133 ++++------------- pins.yaml | 41 +----- sizing/README.md | 10 +- tools/requirements-validate.txt | 3 +- values/README.md | 116 ++++----------- 13 files changed, 234 insertions(+), 487 deletions(-) diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 2f5f755..3646b18 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -1,5 +1,4 @@ -# Tag main with a semantic version and render the changelog. Deployments pin -# this template by tag, and the changelog commit lands untagged after it. +# Tags main and renders the changelog (CONTRIBUTING.md "Commands that prove a change"). name: release on: @@ -28,8 +27,7 @@ jobs: with: python-version: "3.15" - # Gate before tagging. A tag is permanent, so a broken governance tree must - # not get one -- the same content validate-governance.yaml runs on PRs. + # A tag is permanent, so the governance gate runs first. - name: Install pinned PyYAML run: python3 -m pip install --require-hashes -r tools/requirements-validate.txt @@ -47,9 +45,7 @@ jobs: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: npx semantic-release - # The tag is the only reliable signal that a release happened: - # semantic-release exits 0 either way, and most pushes here are docs or - # config changes that earn no bump. + # semantic-release exits 0 whether or not it tagged, so the tag is the signal. - name: Did it release? id: rel run: | @@ -84,10 +80,7 @@ jobs: git config user.email "ci@hyperi.io" git add CHANGELOG.md git commit -m "chore(release): ${{ steps.rel.outputs.tag || 'changelog entries' }} [skip ci]" - # The concurrency group serialises releases, not other pushes to main, - # so a commit landing meanwhile makes this push non-fast-forward. Only - # this job writes CHANGELOG.md, so replaying the commit on the new tip - # cannot conflict. + # Another push can land meanwhile, so retry on a rebase. A conflicting CHANGELOG.md edit stops the job. for attempt in 1 2 3 4 5; do if git push origin HEAD:main; then exit 0 diff --git a/.github/workflows/validate-governance.yaml b/.github/workflows/validate-governance.yaml index 98db6eb..9eae885 100644 --- a/.github/workflows/validate-governance.yaml +++ b/.github/workflows/validate-governance.yaml @@ -1,6 +1,4 @@ -# Validate the governance/ tree (shipped actions + protected-var policies) on -# every PR and push. Structural mirror of dfe-engine's rules - a broken dial -# never merges. Chart-path drift is dfe-infra's chart validation's job. +# The governance/ gate (CONTRIBUTING.md). A change outside governance/ and tools/ runs no check. name: validate-governance on: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6023e7d..7f3bcbb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,26 +6,22 @@ Depth, and the reasoning behind the layout: [docs/architecture.md](docs/architec ### What this is -The template a deployment's git store is seeded FROM -- never itself a live -deploy repo, so nothing here is reconciled to a cluster. A deployment takes its -own read-write copy and Argo CD, dfe-engine and operators read and write that. -Not the base either: charts and ApplicationSets live in dfe-infra, pulled by -reference at the `pins.yaml` tag. One validator aside, there is no code here and -no Kubernetes manifest. +The template a deployment's git repo is copied FROM -- never itself a live deploy repo, so nothing here is reconciled to a cluster. A deployment takes a read-write copy, and Argo CD, dfe-engine and operators read and write that. Not the base either: charts and ApplicationSets live in dfe-infra, which Argo CD renders at the cluster secret's `target_revision`. The code is in `tools/`, two scripts and their tests, and there is no Kubernetes manifest. ### Where things live | Path | Holds | |---|---| -| `pins.yaml` | the dfe-infra tag selecting the whole certified stack, the release `channel`, per-component `overrides` | +| `pins.yaml` | the dfe-infra stack the deployment runs, the release `channel`, per-component `overrides` | | `values/` | one `{service}-{instance}-values.yaml` per APP instance -- file presence enables the app | -| `infra/` | `common.yaml` plus one `.yaml` per data or platform chart | -| `config/` | hot-read config-as-code: `sources/`, `schemas/`, `rules/`, `hunts/`. The engine adds `compat-profiles/` and `library/` on first write | -| `governance/` | five curated dials in `actions/`, protected-var patterns in `policies/` | +| `infra/` | `common.yaml` plus one `.yaml` per data, platform or edge chart | +| `config/` | config-as-code dfe-engine reads: `sources/`, `schemas/`, `rules/`, `hunts/`. The engine adds `compat-profiles/` and `library/` on first write | +| `governance/` | curated dials in `actions/`, protected-var patterns in `policies/`, byte-identical to the copies dfe-engine seeds | +| `sizing/` | the sizing resolver's committed output, none in the template | | `tools/validate_governance.py` | the structural validator for `governance/` | +| `tools/changelog_backfill.py` | adds a `CHANGELOG.md` entry for any release tag the file misses, run by `release.yaml` | -Every directory carries its own README. Read `values/README.md` and -`infra/README.md` before a first edit. +Every directory has a README. Read `values/README.md` and `infra/README.md` before a first edit. ### Commands that prove a change @@ -34,13 +30,10 @@ python3 tools/validate_governance.py python3 -m unittest discover -s tools ``` -The whole local gate: the required fields of each action and policy, closed param constraints, reference wiring -and the guard against an action changing a governance class. How it lies: +The whole local gate: the required fields of each action and policy, closed param constraints, declared param references, known resource classes, pinned image refs, and the guard against an action changing a governance class. How it lies: -- `validate-governance.yaml` filters `paths:` on BOTH `push` and `pull_request`, - to `governance/**` and `tools/**`. A PR touching only `values/`, - `config/`, `pins.yaml` or a README shows no checks -- did not run, not passed. -- `release.yaml` carries no path filter and re-runs the gate before tagging, so a +- `validate-governance.yaml` filters `paths:` on BOTH `push` and `pull_request`, to `governance/**` and `tools/**`. A PR touching nothing but `values/`, `config/`, `pins.yaml` or a README shows CodeQL alone: the governance gate did not run, which is not a pass. +- `release.yaml` has no path filter and re-runs the gate before tagging, so a docs change is checked on merge rather than in review. semantic-release exits 0 either way, so it reads `git tag --points-at HEAD` instead. - Nothing validates `values/`, `infra/` or `config/`. Chart-var drift belongs to @@ -51,17 +44,17 @@ and the guard against an action changing a governance class. How it lies: | Don't | Do | Why | |---|---|---| | Park YAML under `values/`, subdirectories included | Put anything that is not an app instance in `infra/` | Argo hands the `values/*-values.yaml` glob to `git ls-files` as a pathspec, where `*` matches `/` -- so a subdirectory file becomes a phantom Argo app (`99289a4`, `976a898`) | -| Fork this repo | Use the template, or clone at the tag and re-home | A fork of a public repo cannot be private, and base updates arrive by bumping `pins.yaml`, never by merging template history (`9b800f1`) | -| Omit `deploy:` from an instance file | Name `deploy.service` and `deploy.instance` | The ApplicationSet reads it to name the Application, so the file yields no app and no error (`1874994`) | -| Expect a Job to apply `config/schemas/` | Treat it as the engine's additive overlay | There is no Job -- the engine applies every object at its own startup. `config/README.md` claimed one until `e3b2ca9` (#4) | -| Expect an untouched `dfe-receiver` to take traffic | Name the senders in `exposure.groups.ingest.allow` in `infra/common.yaml` | On the load balancer door an EMPTY allow list renders no load balancer at all, and a `/0` opens it to everyone. The retired `exposure.public.loadBalancerSourceRanges` fails the render | -| Set `env:` or `transformFiles` in an instance file | Use `otel.serviceName` and `fileSets..files` | `env:` is not a key the thin charts read and renders nothing. `transformFiles` fails the render by name | +| Fork this repo | Use the template, or clone at the tag and re-home | A fork of a public repo cannot be private, and base updates arrive through `pins.yaml` and `dfe-ops upgrade`, never by merging template history (`9b800f1`) | +| Omit `deploy:` from an instance file | Name `deploy.service` and `deploy.instance` | The ApplicationSet reads it to name the Application and renders with `missingkey=error`, so the file yields no app (`1874994`) | +| Expect a Job to apply `config/schemas/` | Treat it as the engine's additive overlay | There is no Job -- the engine applies every object at its startup (`e3b2ca9`) | +| Expect an untouched `dfe-receiver` on a cloud to take traffic | Name the senders in `exposure.groups.ingest.allow` in `infra/common.yaml` | On the load balancer door, the cloud default, an EMPTY allow list renders no load balancer at all, and a `/0` opens it to everyone. The retired `exposure.public.loadBalancerSourceRanges` fails the render | +| Set `env:` or `transformFiles` in an instance file | Use `extraEnv`, `otel.serviceName` and `fileSets..files` | `env` is the environment name the ApplicationSet sets as a parameter, so a variable map there renders nothing. `transformFiles` fails the render by name | ### Where this sits | Repo | Direction | Kind | Mechanism | |---|---|---|---| -| dfe-infra | inbound | `version-pin` | `pins.yaml` `base.dfe-infra` names one tag whose `versions.yaml` is the manifest for every app image, operator and data service. Pulled by reference, never vendored. `pins.yaml` also calls `dfe-infra/scripts/dfe-stack` by path. Move it, then re-run the validation | -| dfe-schemas | inbound | none | No pin, deliberately, since `e3b2ca9` (2026-09-18). The wheel rides inside the dfe-engine image, so the dfe-infra pin already selects the schema release | -| dfe-infra | outbound | `version-pin` | `versions.yaml` `content.dfe-deploy` pins this repo's tag per stack and bootstrap seeds from it, so a change lands in a deployment only once a stack pins a newer tag | -| dfe-engine | outbound | commits into a copy | Writes a deployment's COPY through gitcrud -- `helmvars` for `values/`, `governance/`, and the config classes. Never the base, never this repo. Not declared in `dfe-infra/suite.yaml` | +| dfe-infra | inbound | `version-pin` | `pins.yaml` `base.dfe-infra` names the stack. Its `versions.yaml` lists every app image, operator and data service. `dfe-ops upgrade` and `dfe-stack resolve` read it, and Argo CD renders from the cluster secret's `target_revision`. The `pins.yaml` usage comment calls `dfe-infra/scripts/dfe-stack` by path: move the script, then update the comment | +| dfe-schemas | inbound | none | No pin, deliberately, since `e3b2ca9` (2026-09-18). The wheel rides inside the dfe-engine image, so the dfe-infra pin selects the schema release | +| dfe-infra | outbound | `version-pin` | `versions.yaml` `content.dfe-deploy` pins this repo's tag per stack, and `dfe-stack release-gate` fails a stack when the pinned template's `pins.yaml` names another stack. Bootstrap does not seed from it yet ([dfe-infra#13](https://github.com/hyperi-io/dfe-infra/issues/13)) | +| dfe-engine | outbound | commits into a copy | Writes a deployment's COPY through gitcrud -- `helmvars` for `values/`, `infravars` for `infra/`, the governance classes and the config classes -- and seeds any shipped action or policy file the copy lacks. Never the base, never this repo. Not declared in `dfe-infra/suite.yaml` | diff --git a/README.md b/README.md index 736ac05..b293c80 100644 --- a/README.md +++ b/README.md @@ -6,22 +6,23 @@ The template a DFE deployment's own git repo is seeded from. ## What it does -- Each deployment takes its own read-write copy of this repo. That copy holds everything local to the deployment, and the base, dfe-infra, is pulled by reference at the tag `pins.yaml` names, never vendored in. -- `pins.yaml`: one dfe-infra tag selects the certified stack as the baseline, with the release channel and any per-component overrides beside it. Updating the base is a one-line bump that leaves your files alone. -- `values/`: one Helm values file per app instance. A file named for the instance and carrying its `deploy` block is what turns the app on. +- Each deployment takes a read-write copy of this repo, which holds everything local to the deployment. The base, dfe-infra, stays in its repo and Argo CD reads it by reference. +- `pins.yaml`: the dfe-infra stack the deployment runs, with the release channel and any per-component overrides beside it. +- `values/`: one Helm values file per app instance. A file named for the instance, with its `deploy` block, is what turns the app on. - `infra/`: an optional `common.yaml` plus one optional values file per data or platform chart. -- `config/`: sources, detection rules and hunts as config-as-code, read hot by the DFE apps and dfe-engine, and `config/schemas/`, the additive schema overlay dfe-engine applies at startup. +- `config/`: sources, detection rules and hunts as config-as-code, read by dfe-engine, and `config/schemas/`, the additive schema overlay dfe-engine applies at startup. - `governance/`: the dials and protected-variable policies, checked by `tools/validate_governance.py`. +- `sizing/`: the sizing resolver's committed output. -dfe-engine writes only this copy, by commit, so the audit trail is its git history. It never writes the base. +dfe-engine writes to this copy by commit, so the audit trail is its git history. It never writes the base. ## Quick start -Take a copy, never a fork. A fork of a public repo cannot be private, and base updates arrive by bumping `pins.yaml`, never by merging template history. Three ways, in order of preference: +Take a copy, never a fork. A fork of a public repo cannot be private, and base updates arrive through `pins.yaml`, never by merging template history. Three ways: -1. Let dfe-infra's bootstrap seed the deploy repo in the bundled in-cluster Forgejo from this template at the stack's tag. This is the default and works air-gapped. -2. GitHub "Use this template" or GitLab "new from template", for a fresh private repo with no upstream lineage. -3. Clone at the release tag, remove `origin`, push to your own remote. +1. GitHub "Use this template" or GitLab "new from template", for a fresh private repo with no upstream lineage. +2. Clone at the release tag, remove `origin`, push to your remote. +3. Leave the deploy repo unset and dfe-infra's bootstrap creates one in the bundled in-cluster Forgejo. That repo holds `sizing/` and a values file per app, not this template's files ([dfe-infra#13](https://github.com/hyperi-io/dfe-infra/issues/13)). Then check the governance files before the first commit: @@ -29,21 +30,37 @@ Then check the governance files before the first commit: python3 tools/validate_governance.py ``` +It prints `OK:` with the action and policy counts, and exits 1 on any error or on an empty `governance/`. + ## Configuration -Bump the dfe-infra version in `pins.yaml` to update the base. Your `values/` and `config/` are untouched, and the schema release moves with the dfe-engine image that stack pins, because the dfe-schemas wheel rides inside it. Who writes what: `pins.yaml` is yours, `values/` is yours and dfe-engine's in the same files, `config/` is dfe-engine's and your analysts'. Per-user console state (saved searches, dashboards) is not here. It lives in the app databases. +`dfe-ops upgrade apply --deploy --push` moves a deployment to a newer stack along a path `dfe-stack check-upgrade` reports as `SUPPORTED`. It bumps `base.dfe-infra` in `pins.yaml`, pushes the commit and moves the cluster secret's `target_revision` with it. Without `--push` it rejects a move that needs the retarget, and a hand edit of the pin alone moves no chart. Between two releases there is no verified path, so a release is a fresh deploy (dfe-infra [docs/deployment/upgrades.md](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/upgrades.md)). + +The schema release moves with the dfe-engine image the stack pins, because the dfe-schemas wheel rides inside it, so `pins.yaml` has no dfe-schemas pin. + +`channel` records the maturity the deployment tracks (`alpha`, `beta`, `rc` or `release`), and no tool selects anything by it. `overrides` moves one app off the certified set, by `tag@sha256` and nothing looser, and dfe-engine logs a SUPPORT-DRIFT notice at startup for each entry: + +```yaml +overrides: + apps: + dfe-receiver: "v1.16.0@sha256:..." +``` + +`dfe-stack resolve --pins pins.yaml --emit-values -` prints the values fragment to merge into that service's existing instance file, never a separate file. + +Who writes what: `pins.yaml` is yours, `values/` and `infra/` are yours and dfe-engine's in the same files, `config/` is dfe-engine's and your analysts', and `governance/` is dfe-engine's and your admins'. Per-user console state (saved searches, dashboards) is not here. It lives in the app databases. ## Documentation - [docs/architecture.md](docs/architecture.md): how a deployment repo is laid out, and why. -- [values/README.md](values/README.md), [infra/README.md](infra/README.md) and [config/README.md](config/README.md): each directory's own rules, to read before a first edit. +- [values/README.md](values/README.md), [infra/README.md](infra/README.md), [config/README.md](config/README.md), [governance/README.md](governance/README.md) and [sizing/README.md](sizing/README.md): each directory's rules, to read before a first edit. - dfe-infra [docs/deployment/state-and-repos.md](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/state-and-repos.md) and [docs/deployment/stack-versioning.md](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/stack-versioning.md): the canonical model and the diagrams. The suite's published docs are at , and this repository's pages at . ## Help and contributing -How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in a new issue. +How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in an issue. Open bugs: [this repo](https://github.com/hyperi-io/dfe-deploy/issues?q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor) | [whole DFE suite](https://github.com/search?type=issues&q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor+repo%3Ahyperi-io%2Fdfe-infra+repo%3Ahyperi-io%2Fdfe-docker+repo%3Ahyperi-io%2Fdfe-deploy+repo%3Ahyperi-io%2Fdfe-engine+repo%3Ahyperi-io%2Fdfe-loader+repo%3Ahyperi-io%2Fdfe-receiver+repo%3Ahyperi-io%2Fdfe-fetcher+repo%3Ahyperi-io%2Fdfe-archiver+repo%3Ahyperi-io%2Fdfe-transform-vrl+repo%3Ahyperi-io%2Fdfe-transform-vector+repo%3Ahyperi-io%2Fdfe-transform-elastic+repo%3Ahyperi-io%2Fdfe-ui+repo%3Ahyperi-io%2Fdfe-hyperdx+repo%3Ahyperi-io%2Fscalo-rs+repo%3Ahyperi-io%2Fscalo-py+repo%3Ahyperi-io%2Flogreducer+repo%3Ahyperi-io%2Fdfe-schemas+repo%3Ahyperi-io%2Fvector-vrl+repo%3Ahyperi-io%2Fculvert+repo%3Ahyperi-io%2Fdfe-clickhouse-rs+repo%3Ahyperi-io%2Ffactbook-rs) diff --git a/config/README.md b/config/README.md index ef68439..512449f 100644 --- a/config/README.md +++ b/config/README.md @@ -1,62 +1,39 @@ -# config/ -- config-as-code (hot-read) +# config/ -- config-as-code -dfe-* apps and dfe-engine **read this path directly** (hot-read / config-store -cascade). Argo CD does **not** apply it as k8s manifests. No Job here applies -ClickHouse DDL: dfe-engine is the only thing that creates a database, table, -view, role or Kafka topic, and it does it at its own startup. +dfe-engine reads this tree from its clone of the repo, and its hunt runner reads `hunts/` and `rules/` from a git-sync volume. Argo CD does **not** apply it as Kubernetes manifests. No Job here applies ClickHouse DDL: dfe-engine creates every database, table, view, role and Kafka topic at its startup. -This is everything that is config-as-code -- versioned, reviewable, promotable, -auditable. It is composed over the pinned `dfe-schemas`/built-in base (this -overlay adds/extends). +Everything here is config-as-code: versioned, reviewable and auditable in git. It adds to the core the engine ships in its `dfe-schemas` wheel, and never replaces it. | Subdir | Holds | Written by | |--------|-------|-----------| | `sources/` | source definitions (all-in-one doc, gitcrud `sources` class) | humans + dfe-engine | -| `schemas/` | this deployment's own schema overlay -- see below -- and `derived/`, the column selections sources name | humans + dfe-engine | +| `schemas/` | this deployment's schema overlay, below, and `derived/`, the column selections sources name | humans + dfe-engine | | `rules/` | detection rules, read by the hunt runner | analysts + dfe-engine | | `hunts/` | hunts, read by the hunt runner | analysts + dfe-engine | -The engine also creates `compat-profiles/` and `library/` here on first write. +The engine creates `compat-profiles/` and `library/` here on first write. -NOT here: per-user saved searches / dashboards / preferences -- those are user -state in the app databases (the HyperDX DocumentDB store), never git. Transform files are -inline in the instance file (`fileSets.transforms.files`, see `values/README.md`). -Alert destinations live on the engine's config volume, OIDC providers in the -engine's auth directory, and RBAC accounts, groups and roles under `governance/`. +NOT here: per-user saved searches, dashboards and preferences, which are user state in the app databases (HyperDX's is DocumentDB), never git. Transform files are inline in the instance file (`fileSets.transforms.files`, see `values/README.md`). Alert destinations sit in the engine's secrets backend, OIDC providers in the engine's auth directory, and RBAC accounts, groups and roles under `governance/rbac/`. -## `schemas/` -- the deployment's own overlay +## `schemas/` adds to the core and never redefines it -The core schema is the `dfe-schemas` wheel pinned inside the dfe-engine image, -read-only. This directory is the second tree, and it is **additive**: it may -declare objects the core does not, and it may never redefine one the core does. +The core schema is the `dfe-schemas` wheel pinned inside the dfe-engine image, read-only. This directory is the second tree, and it is **additive**: it may declare objects the core does not, and it may never redefine one the core does. -No chart sets `DFE_SCHEMAS_OVERLAY_DIR`, so the engine ignores this directory -until the engine's instance file points it at the engine's clone of this repo: +No chart sets `DFE_ENGINE_SCHEMAS_OVERLAY_DIR`, so the engine ignores this directory until the engine's instance file points it at the engine's clone of this repo: ```yaml # values/dfe-engine-default-values.yaml extraEnv: - DFE_SCHEMAS_OVERLAY_DIR: /var/run/dfe/deploy-repo/config/schemas + DFE_ENGINE_SCHEMAS_OVERLAY_DIR: /var/run/dfe/deploy-repo/config/schemas ``` -Lay it out the way dfe-schemas is laid out -- a `manifest.yaml` naming every object, and the -definition files it points at. The engine renders both trees for the topology it -senses and applies them in one pass. +Lay it out the way dfe-schemas is laid out: a `manifest.yaml` naming every object, and the definition files it points at. The engine renders both trees for the platform it probes and applies them in one pass. The engine checks each rule at apply: -The rules, each enforced at apply rather than written down and hoped for: +- an object whose definition sits under a core directory (`common-header/`, `registries/`, `roles/`, `tables/core/`, `tables/internal/`, `tables/meta/`, `tables/otel/`, `tables/usage/`, `topics/`, `views/`) is REJECTED by name +- an object with an id the core manifest declares is REJECTED by name +- a rejection does not stop the core apply: the rest converges and the rejection is reported +- the overlay is optional, and a directory with no `manifest.yaml` is ignored. An install with no deploy repo boots core-only. -- an object whose definition sits under a core directory (`common-header/`, - `registries/`, `roles/`, `tables/core/`, `tables/internal/`, `tables/meta/`, - `tables/otel/`, `tables/usage/`, `topics/`, `views/`) is REFUSED by name -- an object whose id the core manifest already declares is REFUSED by name -- a refusal does not stop the core apply: the rest converges and the refusal is - reported -- the overlay is optional. An install with no deploy repo boots core-only. +Every rejection, and every object the pass applied, is on `GET /api/v1/system/schema` on the engine, with the reason. -Every refusal, and every object the pass applied, is on -`GET /api/v1/system/schema` on the engine, with the reason. - -There is no schema pin here. The wheel rides inside the engine image, so the -engine pin selects the schema release -- `GET /api/v1/system/version` reports it -as `schemas`. A deployment that wants a different schema tree uses this overlay, -not a pin. +There is no schema pin here. The wheel rides inside the engine image, so the engine pin selects the schema release, and `GET /api/v1/system/version` reports it as `schemas`. A deployment that wants a different schema tree uses this overlay, not a pin. diff --git a/docs/README.md b/docs/README.md index 1d8d77b..0fa848f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,39 @@ # dfe-deploy documentation -The architecture of dfe-deploy. +dfe-deploy is the template for a deployment's git repo, the one Argo CD deploys a DFE stack from. Each deployment keeps its copy, and HyperI ships the charts and ApplicationSets separately in dfe-infra. -- [dfe-deploy architecture](architecture.md) +Operators, analysts and dfe-engine all write to the deployment repo, and Argo CD reads it over dfe-infra's charts to deploy the cluster. + +```mermaid +flowchart LR + accTitle: L1 Context -- Deployment repo + accDescr: A deployment repo starts as a copy of the dfe-deploy template. Operators, analysts and dfe-engine commit to it. Argo CD reads its overlays together with the charts and ApplicationSets from dfe-infra and applies the result to the cluster. + + template["dfe-deploy
template"] + people["Operators,
analysts"] + engine["dfe-engine"] + repo[("Deployment
repo")] + infra["dfe-infra"] + argo["Argo CD"] + cluster["Cluster"] + + template -->|copied once| repo + people -->|pull requests| repo + engine -->|commits| repo + repo -->|overlays| argo + infra -->|charts| argo + argo --> cluster + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + class repo core + class people,cluster ext +``` + +*Figure 1. L1 Context -- Deployment repo. Dashed boxes are outside DFE.* Up: top of dfe-deploy. + +The overlays are `values/`, `infra/` and `sizing/`. dfe-infra's charts render at the revision the cluster secret's `dfe.hyperi.io/target_revision` names, and `pins.yaml` records the stack that revision belongs to. dfe-infra's bootstrap creates the bundled Forgejo repo without this template ([dfe-infra#13](https://github.com/hyperi-io/dfe-infra/issues/13) tracks seeding it from the stack's tag), so the template copy is the path for a repo on GitHub, GitLab or any other forge. + +- [Architecture](architecture.md): why the repo is split from its base, the values cascade, governed ops and the rules the files do not show. +- [values/](../values/README.md), [infra/](../infra/README.md), [config/](../config/README.md), [governance/](../governance/README.md) and [sizing/](../sizing/README.md): each directory's rules, to read before a first edit. diff --git a/docs/architecture.md b/docs/architecture.md index df452b4..bf97ea2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,150 +1,73 @@ # dfe-deploy architecture -Why this repo is laid out the way it is, and the rules that are enforced rather -than written down and hoped for. The per-directory READMEs carry the detail for -each tree -- this page is the argument and the cross-tree invariants. - -## The problem +A deployment keeps its state in git and reads dfe-infra's charts by reference, and the layout below follows from that split. Each directory's README has the detail for its tree, and [Figure 1](README.md) shows who reads and writes the repo. A DFE deployment holds two kinds of state that change on different clocks. -HyperI ships one kind: charts, ApplicationSets, images, operators, tier -defaults. The deployment creates the other: which apps run and how many, how big -each one is, which sources feed it, which rules and hunts analysts wrote, which -OIDC providers authenticate people. +HyperI ships one kind: charts, ApplicationSets, images, operators, tier defaults. The deployment creates the other: which apps run and how many, how big each one is, which sources feed it, which rules and hunts analysts wrote, which OIDC providers authenticate people. -Put both in one repository and a base upgrade either overwrites the deployment's -own choices or has to be hand-merged forever. Vendor the base into the -deployment's repository and every upgrade is a merge conflict against files -nobody in that deployment wrote. +Put both in one repository and a base upgrade either overwrites the deployment's choices or has to be hand-merged forever. Copy the base into the deployment's repository and every upgrade is a merge conflict against files nobody in that deployment wrote. -There is a second requirement on top. Every operational change has to be -auditable, and the deployment has to stay manageable when the engine and the UI -are both down. That rules out a database as the authority and it rules out the -API as the only way in. +Every operational change has to be auditable as well, and the deployment has to stay manageable when the engine and the UI are both down. That rules out a database as the authority and it rules out the API as the one way in. -## The split that answers it +## Git holds the deployment's half Split the two kinds of state, and make git the authority for the second. -- The base is REFERENCED by version. `pins.yaml` names a dfe-infra tag and the - base is pulled at that tag by Argo multi-source, a Helm dependency or OCI. It - is never copied in and never a submodule, so there is nothing to merge. -- The deployment's own state is OWNED outright, in a read-write repository per - environment. A base upgrade is a one-line edit to `pins.yaml`. -- Every mutation is a commit, whoever makes it. dfe-engine commits, operators - commit by pull request, and the git history is the audit trail either way. - Editing the files directly still works, which is what keeps a deployment - manageable with the control plane down. - -This repository is only the seed for the second half. - -```mermaid -flowchart LR - T["dfe-deploy
(this template)"] -->|"seeded once at a tag"| D["a deployment's own
read-write repo"] - I["dfe-infra"] -->|"pins the template tag
(versions.yaml content block)"| T - D -->|"pins one dfe-infra tag
(pins.yaml)"| I - E["dfe-engine"] -->|"commits: values, governance, config"| D - O["operators, analysts"] -->|"commits by PR"| D - D -->|"reconciled"| C["the cluster"] - I -->|"charts and appsets,
by reference at the pinned tag"| C -``` - -Seeding happens three ways -- the bootstrap-created in-cluster Forgejo repo, -a forge template copy, or a clone at the release tag with the remote re-homed. -Never a fork: a fork of a public repo cannot be private, and fork lineage -invites merging template history, which is the one thing the pin model exists to -avoid. +- The base is REFERENCED, never copied in and never a submodule. Argo CD renders dfe-infra's charts at the cluster secret's `dfe.hyperi.io/target_revision`, and each app's chart over OCI by a digest that revision's ApplicationSet pins. +- `pins.yaml` records which stack that revision is. `dfe-ops upgrade apply --push` bumps `base.dfe-infra`, pushes it and moves `target_revision` in step, and dfe-engine reports the stack on `GET /api/v1/system/version`. A hand edit of the pin alone leaves the charts where they were. +- The deployment's state is OWNED outright, in a read-write repository per environment. +- Every mutation is a commit, whoever makes it. dfe-engine commits, operators commit by pull request, and the git history is the audit trail either way. Editing the files directly still works, which keeps a deployment manageable with the control plane down. + +This repository is the seed for the second half. A copy is made three ways: dfe-infra's bootstrap creates a bundled Forgejo repo, a forge copies the template, or someone clones at a release tag and re-homes the remote. The bundled repo is not seeded from this template yet ([dfe-infra#13](https://github.com/hyperi-io/dfe-infra/issues/13)). Never fork: a fork of a public repo cannot be private, and fork lineage invites merging template history, which the pin model exists to avoid. ## Why `values/` and `infra/` are separate trees -The dfe-infra ApplicationSet's git files generator globs `values/*-values.yaml`, -and Argo passes that glob to `git ls-files` as a pathspec. A pathspec without -`:(glob)` magic lets `*` match `/`. So `values/infra/kafka-values.yaml` matches -the glob and becomes an Argo application of its own. +The dfe-infra ApplicationSet's git files generator globs `values/*-values.yaml`, and Argo passes that glob to `git ls-files` as a pathspec. A pathspec without `:(glob)` magic lets `*` match `/`. So `values/infra/kafka-values.yaml` matches the glob and becomes an Argo application. -That single fact drives the layout. `values/` holds app instances and nothing -else, not even in a subdirectory, because a subdirectory does not hide a file -from the generator. `infra/` therefore sits at the repository root, where a -pathspec anchored on `values/` cannot reach it, and takes the values for -everything that is not an app instance -- ClickHouse, Kafka, PostgreSQL, -DocumentDB, the OTel collector, the network policies, the gateway config. +That single fact drives the layout. `values/` holds app instances and nothing else, not even in a subdirectory, because a subdirectory does not hide a file from the generator. `infra/` therefore sits at the repository root, where a pathspec anchored on `values/` cannot reach it, and takes the values for everything that is not an app instance: ClickHouse, Kafka, DocumentDB, the OTel collector, the network policies, the gateway config. ## The values cascade -Helm applies `-f` files in order and the last wins. The ApplicationSets stack -six layers, three from the base and three from the deployment: +Helm applies `-f` files in order and the last wins. Each ApplicationSet stacks dfe-infra's defaults first and this repo's files last: -| Layer | Source | Scope | +| Layer | Repo | Read by | |---|---|---| -| `argocd/values/common.yaml` | dfe-infra | the product defaults | -| `argocd/values/.yaml` | dfe-infra | cloud specifics | -| `argocd/values/profile-.yaml` | dfe-infra | the tier default | -| `infra/common.yaml` | here | deployment-wide facts every appset reads | -| `infra/.yaml` | here | one chart, platform and data appsets only | -| `values/--values.yaml` | here | one app instance | +| `argocd/values/common.yaml` | dfe-infra | every appset below | +| `argocd/values/.yaml` | dfe-infra | every appset below | +| `argocd/values/profile-.yaml` | dfe-infra | every appset below | +| `argocd/values/apps/...` | dfe-infra | apps: `_common.yaml`, then the app's `values.yaml`, `.yaml` and `profile-.yaml` | +| `sizing/.data-charts.values.yaml`, `sizing/.values.yaml` | here | data and platform | +| `infra/common.yaml` | here | apps, data, platform and edge | +| `infra/.yaml` | here | data, platform and edge | +| `values/--values.yaml` | here | apps | -A profile file is a tier DEFAULT, not a lock, so a deployment can declare -`clickhouse.mode: external` in `infra/common.yaml` and beat -`profile-scale.yaml`. Before this layer existed the only way there was editing -dfe-infra. +The edge appset adds `argocd/values/edge-.yaml` after the cloud file. Facts the cluster secret holds, such as the domain and the registry, arrive as Helm parameters and beat every file. -A fact more than one chart consumes belongs in `infra/common.yaml` once, so the -network policy, the ClickHouse chart and the loader cannot disagree about it. +A profile file is a tier DEFAULT, not a lock, so a deployment can declare `clickhouse.mode: external` in `infra/common.yaml` and beat `profile-scale.yaml`. A fact more than one chart consumes belongs in `infra/common.yaml` once, so the network policy, the ClickHouse chart and the loader cannot disagree about it. ## Governed ops -`governance/` is the content behind the engine's governed-operations API, and it -is built so that a curated dial cannot become a general-purpose write. - -- An action is a named bundle of var changes applied atomically in one commit. - An operator granted `action:invoke:` can pull that one lever without - holding raw `helmvars:write`, and every invoke supports a dry run that returns - the diff and commits nothing. -- Params are closed by construction. An enum carries its full value list, a - numeric carries both bounds, and there is no free-string param type. Values - substitute whole-value only, never into `cls`, `name` or `path`. -- An action may never change a governance class. That is a - privilege-escalation guard, enforced at both validate and invoke, and - independently by `tools/validate_governance.py`. -- A policy is a list of `cls:name:path` fnmatch patterns that locks matching vars - against Tier-1 edits and against actions, unless the caller holds - `helmvars:override`. The shipped `baseline` policy locks `image.*`, because - image references move through stack pins rather than dials. - -A policy governs the API, not the repository. A direct git commit still lands, -which is deliberate -- git is the authority and the API is a window over it. +`governance/` is the content behind the engine's governed-operations API, and it is built so that a curated dial cannot become a general-purpose write. + +- An action is a bundle of var changes, invoked by name and applied atomically in one commit. An operator granted `action:invoke:` can pull that one lever without holding raw `helmvars:write`, and every invoke supports a dry run that returns the diff and commits nothing. +- Params are closed by construction. An enum lists every value, a numeric has both bounds, and there is no free-string param type. Values substitute whole-value, never into `cls`, `name` or `path`. +- An action may never change a governance class. That is a privilege-escalation guard, enforced at both validate and invoke, and independently by `tools/validate_governance.py`. +- A policy is a list of `cls:name:path` fnmatch patterns that locks matching vars against Tier-1 edits and against actions, unless the caller holds `helmvars:override`. The shipped `baseline` policy locks `image.*`, because image references move through stack pins rather than dials. + +A policy governs the API, not the repository. A direct git commit still lands, which is deliberate: git is the authority and the API is a window over it. ## Invariants These are the ones reading the files will not tell you. -1. Any `values/*-values.yaml` file IS an application. Creating one enables the - app, deleting one removes it, and a scratch file spawns a phantom app. -2. An instance file without a `deploy:` block produces nothing at all. The - ApplicationSet reads `deploy.service` and `deploy.instance` to name the Argo - Application, so a file missing them yields no app and no error. -3. `config/schemas/` is ADDITIVE. An object whose definition sits under a core - directory, or whose id the core manifest already declares, is refused by - name. A refusal does not stop the core apply -- the rest converges and the - refusal is reported on `GET /api/v1/system/schema`. -4. There is no DDL Job. dfe-engine applies every ClickHouse object and every - bootstrap Kafka topic at its own startup, from the `dfe-schemas` wheel inside - its image. That is why there is no `dfe-schemas` pin. -5. The data-store mode and storage model are chosen at deploy and refused - afterwards, and replica counts are accepted upward only. Changing either on a - live deployment is a data migration, not a values edit. -6. Object-store credentials never appear in these files. Both data charts read - them from the environment, wired from a Secret the secrets store materialises. -7. Validation here is structural only. A chart renaming a dial var is caught by - dfe-infra's chart validation, which owns the charts these paths point into. -8. Zero governance files is a FAILURE, not a pass. An empty result means the - tree moved or the path is wrong, and a validator reporting success having - checked nothing is the exact fault it exists to catch. - -## What is deliberately not here - -Per-user UI state -- saved searches, dashboards, preferences -- lives in the app -databases (dfe-hyperdx's DocumentDB store), never in git. Secrets live in the secrets -store. Nothing in `config/` is a Kubernetes manifest, and Argo never applies it -as one: the apps and the engine read that tree directly off disk. +1. Any `values/*-values.yaml` file IS an application. Creating one enables the app, deleting one removes it, and a scratch file spawns a phantom app. +2. An instance file without a `deploy:` block produces no application. The ApplicationSet names the Argo Application from `deploy.service` and `deploy.instance` and renders with `missingkey=error`, so a missing key is a render error and no Application appears. +3. `config/schemas/` is ADDITIVE. An object whose definition sits under a core directory, or whose id the core manifest declares, is rejected by name. A rejection does not stop the core apply: the rest converges and the rejection is reported on `GET /api/v1/system/schema`. +4. There is no DDL Job. dfe-engine applies every ClickHouse object and every bootstrap Kafka topic at its startup, from the `dfe-schemas` wheel inside its image. That is why there is no `dfe-schemas` pin. +5. The data-store mode and storage model are chosen at deploy and rejected by the engine's API afterwards, and node and broker counts can rise but not fall. Changing either on a live deployment is a data migration, not a values edit. +6. Object-store credentials never appear in these files. Both data charts read them from the environment, wired from a Secret the secrets store materialises. +7. Validation here is structural. A chart renaming a dial var is caught by dfe-infra's chart validation, which owns the charts these paths point into. +8. Zero governance files is a FAILURE, not a pass. An empty result means the tree moved or the path is wrong, and a validator reporting success having checked nothing is the fault it exists to catch. + +Some state is deliberately not here. Per-user UI state (saved searches, dashboards, preferences) lives in the app databases, dfe-hyperdx's in DocumentDB, never in git. Secrets live in the secrets store. Nothing in `config/` is a Kubernetes manifest, and Argo never applies it as one: dfe-engine reads the tree from its clone, and its hunt runner reads `hunts/` and `rules/` from a git-sync volume. diff --git a/governance/README.md b/governance/README.md index b0b9c31..aef39ab 100644 --- a/governance/README.md +++ b/governance/README.md @@ -1,66 +1,35 @@ # governance/ -- curated actions + protected-var policies -The Governed Ops content for this deployment. dfe-engine reads and commits -this tree (RBAC class `governance`); operators and admins can equally edit it -by PR. If the engine and UI are both down, DFE stays fully manageable by -editing these files directly - git is the authority. +The Governed Ops content for this deployment. dfe-engine reads and commits this tree (RBAC class `governance`), and operators and admins can edit it by PR. If the engine and UI are both down, DFE stays manageable by editing these files directly: git is the authority. + +The files here are byte-identical to the library dfe-engine ships. On start the engine commits any shipped action or policy file the repo lacks and leaves every existing file alone, edited or not. A deleted file comes back on the next start unless its name is a line in `governance/.seed-ignore`. ## actions/ -- the shipped operational dials -Each file is one curated action: a named, RBAC-gated bundle of var changes -applied atomically in ONE commit via -`POST /api/v1/governance/actions/{name}/invoke` (or `dfe-api`). Operators are -granted per-dial `action:invoke:` handles - they can pull a lever -without holding raw `helmvars:write`. Every invoke supports -`?dry_run=true` (returns the diff, commits nothing). +Each file is one curated action: an RBAC-gated bundle of var changes, invoked by name and applied atomically in ONE commit via `POST /api/v1/governance/actions/{name}/invoke`. Operators are granted per-dial `action:invoke:` handles, so they can pull a lever without holding raw `helmvars:write`. Every invoke supports `?dry_run=true`, which returns the diff and commits nothing. Shipped dials: | action | what it does | params | |---|---|---| | `receiver-surge` | raise the receiver KEDA ceiling for an ingest surge | `level`: 2x / 4x | -| `receiver-normal` | return to the tier's own ceiling by removing the override | - | +| `receiver-normal` | return to the tier's ceiling by removing the override | - | | `hunts-pause` | stop the hunt runner (shed ClickHouse load) | - | | `hunts-resume` | restart the hunt runner (resumes from checkpoints) | - | | `hunts-throttle` | cap concurrent hunt runs | `cap`: 1-32 | -Params are CONSTRAINED by design - an enum carries its full value list, a -numeric carries both bounds. There is no free-string param type. Values -substitute whole-value only (`{$param: name}` or a per-value `map`), never -into `cls`/`name`/`path`. +Params are CONSTRAINED by design: an enum lists every value it takes, and a numeric has both bounds. There is no free-string param type. A param substitutes a whole value (`{$param: name}` or a per-value `map`), never into `cls`, `name` or `path`. A value of `{$unset: true}` removes the key instead of setting it, so the var returns to the layer beneath the overlay. `receiver-normal` uses it to hand the KEDA ceiling back to the tier profile. -The `changes[].name` entries reference the per-instance values files in -`../values/` (`{service}-{instance}-values` - see `values/README.md`). Every -shipped action names the `default` instance. If that instance file does not -exist, an invoke creates one holding only the dial, with no `deploy:` block, -which deploys nothing, so the dial reports success and changes nothing. Point -actions only at instances this deployment actually runs. +The `changes[].name` entries reference the per-instance values files in `../values/` (`{service}-{instance}-values`, see `values/README.md`). Every shipped action names the `default` instance. If that instance file does not exist, an invoke creates it with its `deploy:` block, which deploys that instance, after the multiplicity and ceiling checks the instance API applies. Point actions at instances this deployment runs. -An action that switches a component off also switches off the engine's secret -group for it (`secrets..enabled`, as `hunts-pause` does), and its -counterpart switches both back on. The component's Secret is only minted while -it is on, and the engine pod waits on any Secret it still reads. +An action that switches a component off switches off the engine's secret group for it as well (`secrets..enabled`, as `hunts-pause` does), and its counterpart switches both back on. The component's Secret exists while the component is on, and the engine pod waits on any Secret it reads. -Authoring a new action: validate first with -`POST /api/v1/governance/admin/actions/validate` (returns every violation and -the would-be diff without committing), then define via -`POST /api/v1/governance/admin/actions` (`governance:write`). Actions may -never change the `governance` class itself - that is a privilege-escalation -guard, enforced at both validate and invoke. +To author an action, validate first with `POST /api/v1/governance/admin/actions/validate`, which returns every violation and the would-be diff without committing, then define it with `POST /api/v1/governance/admin/actions` (`governance:write`). Actions may never change a governance class. That is a privilege-escalation guard, enforced at both validate and invoke. ## policies/ -- protected-var policies -`cls:name:path` fnmatch patterns that lock vars against Tier-1 edits AND -actions unless the caller holds `helmvars:override`. The shipped `baseline` -policy locks `image.*` - image references move through stack pins -(`dfe-stack resolve`), not dials. - -## Validation +`cls:name:path` fnmatch patterns that lock vars against Tier-1 edits AND actions unless the caller holds `helmvars:override`. The shipped `baseline` policy locks `image.*`, because image references move through stack pins (`dfe-stack resolve`), not dials. `storage-layout` and `sizing-locks` hold the keys chosen at deploy, described in [infra/README.md](../infra/README.md) and [sizing/README.md](../sizing/README.md). -`tools/validate_governance.py` structurally validates this tree (schema -shape, closed param constraints, reference wiring, no governance-class -changes) and runs in CI on every PR. Chart-path drift (a chart renaming a -dial var) is covered by dfe-infra's chart validation, which owns the charts -these paths point into. +`tools/validate_governance.py` checks this tree's structure: required fields, closed param constraints, declared param references, known resource classes, pinned image refs, and no governance-class changes. CI runs it on a PR or push that touches `governance/` or `tools/`, and the release workflow runs it before every tag. Chart-path drift (a chart renaming a dial var) is covered by dfe-infra's chart validation, which owns the charts these paths point into. diff --git a/infra/README.md b/infra/README.md index 085e01c..b68cf79 100644 --- a/infra/README.md +++ b/infra/README.md @@ -1,65 +1,25 @@ # infra/ -- the deployer's overlay for everything that is not an app instance -`values/` holds one file per APP instance and each of those files becomes an Argo -application. The substrate and platform charts are not app instances - ClickHouse, -Kafka, PostgreSQL, DocumentDB, the OTel collector, the network policies, the gateway -config. dfe-infra decides they exist; this directory is where a deployment says -what shape they take. +`values/` holds one file per APP instance, and each of those files becomes an Argo application. The data and platform charts are not app instances: ClickHouse, Kafka, DocumentDB, the OTel collector, Kafbat, the links page, the network policies, the gateway config. dfe-infra decides they exist, and this directory is where a deployment configures them. dfe-engine writes here as the `infravars` class. Two kinds of file, both optional: -- **`common.yaml`** - deployment-wide facts, read by EVERY appset (platform, data - and apps). This is the deployer's own `argocd/values/common.yaml`. A fact more - than one chart consumes - `clickhouse.mode`, `kafka.mode` - is declared here - once, so the network policy, the ClickHouse chart and the loader cannot - disagree about it. -- **`.yaml`** - one chart's own values, read by the platform and data - appsets only. The name is the dfe-infra chart directory: - `clickhouse-cluster.yaml`, `kafka.yaml`, `documentdb.yaml`, - `otel-collector.yaml`, `dfe-schema.yaml`, `kafbat.yaml`, `links.yaml`, - `network-policies.yaml`, `envoy-gateway-config.yaml`. +- `common.yaml`: deployment-wide facts, read by the app, data, platform and edge ApplicationSets. A fact more than one chart consumes, such as `clickhouse.mode` or `kafka.mode`, is declared here once, so the network policy, the ClickHouse chart and the loader cannot disagree about it. +- `.yaml`: one chart's values, read by the data, platform and edge ApplicationSets. The name is the ApplicationSet element's `app`, or its `overlay` where one is set, never the chart directory: `clickhouse-cluster.yaml`, `kafka.yaml`, `documentdb.yaml`, `otel-collector.yaml`, `kafbat.yaml`, `links.yaml`, `network-policies.yaml`, `dfe-toolbox.yaml`, `envoy-gateway-config.yaml`, and on AWS `node-pools.yaml`, which the `karpenter-pools` Application reads. -Content is plain chart values, same as `values/`. There is no `deploy:` block -here - nothing in this directory creates an application. +Content is plain chart values, the same as `values/`. There is no `deploy:` block here, and nothing in this directory creates an application. It sits at the repo root rather than under `values/` because the app generator's pathspec reaches into every subdirectory there ([values/README.md](../values/README.md)). -## Why not a subdirectory of values/ +## These files beat the tier defaults -Because a subdirectory of `values/` is not hidden from the app generator. Argo -passes the `values/*-values.yaml` glob to `git ls-files` as a pathspec, and a -pathspec without `:(glob)` magic lets `*` match `/`. `values/infra/kafka-values.yaml` -matches, and a phantom application appears. `infra/` at the repo root cannot -match a pathspec anchored on `values/`. See [`values/README.md`](../values/README.md). +Helm applies `-f` files in order and the last one wins. `common.yaml` and `.yaml` come after dfe-infra's common, cloud and profile files and after `sizing/`, and an app's instance file comes after `common.yaml`. The full order is the values cascade in [docs/architecture.md](../docs/architecture.md). -## Where these files land in the cascade +The profile file is a tier DEFAULT, not a lock. `profile-scale.yaml` declares `clickhouse.mode: cluster`, and declaring `clickhouse.mode: external` in `infra/common.yaml` beats it, and the network policy opens egress to the external service with it. -Helm applies `-f` files in order and the last one wins. The appsets stack them: +Moving a mode means moving its endpoint in the same file, `clickhouse.host` for ClickHouse and `kafka.bootstrapServers` for Kafka, because the profile pins those to the in-cluster service names. -``` -argocd/values/common.yaml dfe-infra the product SSoT -argocd/values/.yaml dfe-infra cloud specifics -argocd/values/profile-.yaml dfe-infra the TIER default -infra/common.yaml HERE deployment-wide -infra/.yaml HERE this chart (platform + data) -values/--values.yaml values/ this instance (apps) -``` - -The profile file is a tier DEFAULT, not a lock. `profile-scale.yaml` declares -`clickhouse.mode: cluster`; declaring `clickhouse.mode: external` in -`infra/common.yaml` beats it, and the derived external-service egress opens with -it. Before this layer existed a deployer had to edit dfe-infra to get there. - -Moving a mode means moving its endpoint in the same file - `clickhouse.host` for -ClickHouse, `kafka.bootstrapServers` for Kafka - because the profile pins those -to the in-cluster service names. +## The storage model is chosen once, at deploy -## The storage model, and the keys the engine refuses to change - -Both data stores take a storage model that is chosen at deploy and refused -afterwards. Models are named `-`: the family says whether data -MOVES to the bulk store or is COPIED to it, the bulk half says what that store -is. Kafka defaults to `local`, which changes nothing. ClickHouse defaults to `auto`, -which derives `cached-object` from an `objectStore.endpoint` and `local` from none, -so an endpoint with no `storageModel` beside it is not a local deployment. +Both data stores take a storage model that is chosen at deploy and rejected afterwards. A model's name is `-`: the family says whether data MOVES to the bulk store or is COPIED to it, and the bulk half says what that store is. Kafka defaults to `local`, which changes nothing. ClickHouse defaults to `auto`, which derives `cached-object` from an `objectStore.endpoint` and `local` from none, so an endpoint with no `storageModel` beside it is not a local deployment. ```yaml # infra/clickhouse-cluster.yaml @@ -82,69 +42,36 @@ kafka: storage.s3.bucket.name: my-kafka-bucket ``` -`cached-object` puts ClickHouse parts on an object-store disk behind a local -read-through cache, so the PVC stops being the capacity ceiling and the cache -stays disposable. `tiered-block` ranks a hot SSD volume over a cold bulk volume -and demotes the oldest parts as the hot one fills, needing no object store - -data MOVES, so both volumes must be durable. `tiered-object` moves closed Kafka -segments to object storage (KIP-405 tiered storage), so the PVC sizes the hot -window; Strimzi ships no RemoteStorageManager, so it needs a broker image -carrying a plugin, and the chart fails the render rather than deploying a broker -that looks configured and tiers nothing. - -Every combination, whether it ships and how well proven it is: dfe-infra -`docs/deployment/storage.md`. - -Object-store CREDENTIALS never appear in these files. Both charts read them from -the environment, wired from a Secret the secrets store materialises - seed -`//clickhouse/s3` and `//kafka/tiered` with -`access_key_id` and `secret_access_key`. A credential that already lives at a -different path or under different field names binds without reseeding: -`clickhouse.objectStore.remoteKey` / `kafka.objectStore.remoteKey` name the path, -`accessKeyProperty` / `secretKeyProperty` name the fields, and -`secretStoreName` selects the store that mounts them. - -The engine holds these as protected vars and refuses a post-deploy edit with the -policy that blocked it (`governance/policies/storage-layout.yaml`): +`cached-object` puts ClickHouse parts on an object-store disk behind a local read-through cache, so the PVC stops being the capacity ceiling and the cache stays disposable. `tiered-block` ranks a hot SSD volume over a cold bulk volume and demotes the oldest parts as the hot one fills, with no object store. Data MOVES, so both volumes must be durable. `tiered-object` moves closed Kafka segments to object storage (KIP-405 tiered storage), so the PVC sizes the hot window. Strimzi ships no RemoteStorageManager, so it needs a broker image with a plugin, and the chart fails the render rather than deploying a broker that looks configured and tiers nothing. + +`cached-object` and `tiered-object` are opt-in and render-verified, not live-proven. `tiered-block` is live-proven in cluster mode. Every combination, with its status and evidence, is in dfe-infra [docs/deployment/storage.md](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/storage.md). + +On AWS the cluster secret names a ClickHouse bucket, so a DFE-run ClickHouse on `auto` derives `cached-object`. The bucket, the storage model and the cache placement then arrive as inline Helm values, which beat this overlay. A deployment that wants `local` on AWS says so before the first deploy, in the deployment dial (`sizing.storage_model: local`). + +## Credentials stay out, and the layout keys lock + +Object-store CREDENTIALS never appear in these files. Both charts read them from the environment, wired from a Secret the secrets store materialises: seed `//clickhouse/s3` and `//kafka/tiered` with `access_key_id` and `secret_access_key`. A credential that lives at a different path or under different field names binds without reseeding: `clickhouse.objectStore.remoteKey` and `kafka.objectStore.remoteKey` name the path, `accessKeyProperty` and `secretKeyProperty` name the fields, and `secretStoreName` selects the store that mounts them. ClickHouse on AWS can take `clickhouse.objectStore.usePodIdentity: true` and use no static key. + +The engine holds these as protected vars and rejects a post-deploy edit with the policy that blocked it (`governance/policies/storage-layout.yaml`): | key | why it locks | |---|---| -| `clickhouse.mode`, `kafka.mode` | where the data lives; moving it strands every existing row | +| `clickhouse.mode`, `kafka.mode` | where the data lives, and moving it strands every existing row | | `clickhouse.storageModel`, `kafka.storageModel` | the on-disk layout, chosen once | | `clickhouse.objectStore.*`, `kafka.objectStore.*` | the object-store location the existing parts and segments are in | -| `clickhouse.tieredBlock.*`, `kafka.tieredObject.*` | the cold volume holding the only copy of what it carries, and the plugin that reaches the bucket | +| `clickhouse.tieredBlock.*`, `kafka.tieredObject.*` | the cold volume holding the sole copy of its data, and the plugin that reaches the bucket | | `clickhouse.storage.size`, `.storageClass` and the kafka pair | `volumeClaimTemplates` are immutable, so no sync applies it | -A deployment created before this vocabulary keeps its `storage-model.yaml` -alongside the new file - the engine never rewrites a policy it already seeded - -and the shipped lock carries the older key spellings so nothing goes unprotected -while a deployment is still on an older chart. +A deployment created before this vocabulary keeps its `storage-model.yaml` beside the current file, because the engine never rewrites a policy file that exists. The shipped lock lists the older key spellings as well, so nothing goes unprotected while a deployment is on an older chart. -A holder of `helmvars:override` can still make a deliberate exception, and a -direct git commit still gets through - the policy governs the API, not the repo. -Changing any of these on a live deployment is a data migration, not a values edit. +A holder of `helmvars:override` can still make a deliberate exception, and a direct git commit still gets through: the policy governs the API, not the repo. Changing any of these on a live deployment is a data migration, not a values edit. -## Node and broker counts go up, never down - -`clickhouse.replicas`, `clickhouse.keeper.replicas` and `kafka.replicas` are -accepted upward and refused downward, and not out of caution. Removing a -ClickHouse node drops a copy of the data, or the data itself when the cluster is -sharded; a Kafka broker takes every partition replica it held unless those are -reassigned first; a Keeper ensemble can lose its Raft quorum, which takes every -replicated table read-only with it. Do the move, then lower the count. +Growing a PVC needs `allowVolumeExpansion: true` on the StorageClass and, because `volumeClaimTemplates` are immutable, a StatefulSet recreate. `local-path` has no resize support at all. That is what the storage model exists to avoid, so reach for a non-`local` model before reaching for a bigger disk. The size and class keys are locked for that reason. -The refusal compares what the overlay stack DECLARES before and after, so nothing -declared means nothing to compare and the write is accepted. CPU and memory move -freely in both directions. +dfe-docker is the other way round: `DFE_DATA_ROOT` picks WHICH disk, and a bind mount takes whatever that disk has. Kubernetes locks size and class, and docker locks location. -## Storage size is not the answer +## Node and broker counts go up, never down -Growing a PVC needs `allowVolumeExpansion: true` on the StorageClass and, because -`volumeClaimTemplates` are immutable, a StatefulSet recreate. `local-path` has no -resize support at all. That is what the storage model exists to avoid, so reach -for a non-`local` model before reaching for a bigger disk. The size and class -keys are protected for the same reason. +`clickhouse.replicas`, `clickhouse.keeper.replicas` and `kafka.replicas` are accepted upward and rejected downward. Removing a ClickHouse node drops a copy of the data, or the data itself when the cluster is sharded. A Kafka broker takes every partition replica it held unless those are reassigned first. A Keeper ensemble can lose its Raft quorum, which takes every replicated table read-only with it. Do the move, then lower the count. -dfe-docker is the other way round: `DFE_DATA_ROOT` picks WHICH disk, and a bind -mount takes whatever that disk has. k8s locks size and class; docker locks -location. +The check compares what the overlay stack DECLARES before and after, so nothing declared means nothing to compare and the write is accepted. A write that leaves a declared count undeclared is rejected, because the chart default can be lower. CPU and memory move freely in both directions. diff --git a/pins.yaml b/pins.yaml index 92c52e3..49e4a31 100644 --- a/pins.yaml +++ b/pins.yaml @@ -1,42 +1,11 @@ -# pins.yaml -- the HyperI-provided base, referenced by PINNED version. -# -# This is the only coupling to the vendor repos. The base is pulled by reference -# (Argo CD multi-source / Helm dependency / OCI) at these versions -- never -# vendored in, never a git submodule. To update the base: bump a version here and -# commit; your values/ and config/ are untouched. -# -# Pin policy (see dfe-infra docs/deployment/stack-versioning.md): adopt a version only once it is >= 7 days old -# (supply-chain cooldown) and prefer an LTS line where one exists. +# The dfe-infra stack this deployment runs. README.md "Configuration" explains each key. base: - dfe-infra: "2.2.2" # charts, appsets, defaults, deployment machinery. - # ONE pin selects the WHOLE certified stack: dfe-infra's - # versions.yaml at this tag is the stack manifest (every - # dfe-* app image, operator, data service). See - # dfe-infra docs/deployment/stack-versioning.md. -# -# There is deliberately NO dfe-schemas pin. The wheel rides inside the dfe-engine -# image and the engine applies it at startup, so the engine pin already selects -# the schema release -- GET /api/v1/system/version reports it as `schemas`. A -# pin here would be a third version number for one artefact and could name a -# release the running engine does not carry. A deployment that wants a different -# schema tree adds config/schemas/ as an overlay, which is additive by design. + dfe-infra: "2.2.2" # selects dfe-infra's versions.yaml at this tag +# No dfe-schemas pin: the wheel rides inside the dfe-engine image. -# Release channel this environment tracks: alpha | beta | rc | release (the -# maturity ladder of dfe-infra's versions.yaml). It is a record for operators: -# no tool resolves it, and base.dfe-infra above is what selects the stack. -# Dev -> alpha or rc, staging -> rc, production -> release. +# alpha | beta | rc | release. A record for operators: no tool resolves it. channel: "release" -# Per-component overrides -- move ONE component off the certified set, at your -# own documented risk. Precedence: overrides > the certified manifest; use -# tag@sha256 digests, never floating tags (rejected otherwise). dfe-engine -# prints a SUPPORT-DRIFT notice at startup for every entry here. To apply an -# override to the cluster, emit the per-service fragments and MERGE each into -# that service's existing values/{service}-{instance}-values.yaml (never a new -# standalone file - the Argo files generator treats every values/*-values.yaml -# as an app): +# overrides: tag@sha256 only, format in README.md. Emit the values to merge with # dfe-infra/scripts/dfe-stack resolve --pins pins.yaml --emit-values - -# overrides: -# apps: -# dfe-receiver: "v1.16.0@sha256:..." diff --git a/sizing/README.md b/sizing/README.md index 5567e3c..4e64369 100644 --- a/sizing/README.md +++ b/sizing/README.md @@ -1,9 +1,11 @@ # sizing/ -- the resolver's committed output for this deployment -`sizing/resolved.yaml` is written by dfe-infra's sizing resolver (`scripts/resolve_sizing.py`) and committed here. It records the tier, focus, cloud, region and ingest estimate the deployment was sized from, its compute cost bucket wherever cloud shapes were resolved, and a `locked:` section holding the current value of each field the resolver refuses to move without `--migrate`: partition count, storage model, MSK broker type, KRaft controller mode, cloud and AZ count. +`sizing/resolved.yaml` is written by dfe-infra's sizing resolver (`scripts/resolve_sizing.py`) and committed here. It records the tier, focus, cloud, region and ingest estimate the deployment was sized from, its compute cost bucket wherever cloud shapes were resolved, and a `locked:` section. A re-resolve with `--previous` compares each field present in both files and rejects a change without `--migrate`. The fields are partition count, storage model, Kafka provider, KRaft controller mode, cloud and, on a cloud, AZ count. -It is committed on purpose. `dfe-ops upgrade plan --deploy --dial deployment.yaml --live` re-resolves and diffs the result against this file, so a re-size is checked against what is already deployed rather than derived from scratch. Re-run the resolver and commit what it writes rather than hand-editing this file. +It is committed on purpose. `dfe-ops upgrade plan --deploy --dial deployment.yaml --live` re-resolves and diffs the result against this file, so a re-size is checked against what is deployed rather than derived from scratch. Re-run the resolver and commit what it writes rather than hand-editing this file. -The same run writes `sizing/.values.yaml` and `sizing/.report.md`. The `kafka` and `clickhouse` blocks of the values file take effect once they are in `infra/kafka.yaml` and `infra/clickhouse-cluster.yaml`, and the report says what was sized, from which ratio, at which confidence and in which cost bucket. The OpenTofu inputs, `sizing.auto.tfvars.json`, belong beside the OpenTofu root module that reads them. +Each run writes `sizing/.values.yaml`, `sizing/.report.md` and, wherever Kafka or ClickHouse is sized, `sizing/.data-charts.values.yaml`. dfe-infra's data ApplicationSet reads that last file straight from this repo, under the `infra/` overlays, so a value in `infra/kafka.yaml` or `infra/clickhouse-cluster.yaml` beats the resolver's. Its platform ApplicationSet reads `sizing/.values.yaml` the same way, such as the `node-pools.values.yaml` that `resolve_sizing.py images` writes. -`governance/policies/sizing-locks.yaml` is the engine's side of the same locks: it refuses an API edit to the chart keys behind them unless the caller holds `helmvars:override`, and `storage-layout.yaml` does the same for the storage model. The MSK broker type and the AZ count are OpenTofu inputs with no chart key, so only the resolver guards those. Neither policy governs a direct git commit. +A report says what was sized, from which ratio, at which confidence and in which cost bucket. OpenTofu's inputs, `sizing.auto.tfvars.json`, belong beside the OpenTofu root module that reads them. + +`governance/policies/sizing-locks.yaml` is the engine's side of the same locks: it rejects an API edit to the chart keys behind them unless the caller holds `helmvars:override`, and `storage-layout.yaml` does the same for the storage model. Neither policy holds the Kafka provider or the AZ count, so the resolver alone guards those. A direct git commit gets past both. diff --git a/tools/requirements-validate.txt b/tools/requirements-validate.txt index 6f78eaf..83f0e64 100644 --- a/tools/requirements-validate.txt +++ b/tools/requirements-validate.txt @@ -1,6 +1,5 @@ # Hash-pinned dependency set for tools/validate_governance.py (CI). -# PyYAML 6.0.3 (released 2025-09-25) - every published artifact hash so -# --require-hashes resolves on any runner arch. +# Every published artefact hash, so --require-hashes resolves on any runner arch. pyyaml==6.0.3 \ --hash=sha256:214ed4befebe12df36bcc8bc2b64b396ca31be9304b8f59e25c11cf94a4c033b \ --hash=sha256:02ea2dfa234451bbb8772601d7b8e426c2bfa197136796224e50e35a78777956 \ diff --git a/values/README.md b/values/README.md index eb84848..2269931 100644 --- a/values/README.md +++ b/values/README.md @@ -1,65 +1,35 @@ # values/ -- THE helm-values home for APP INSTANCES (hard standard) -**Every helm value for an app instance in this environment lives here, one file -per service instance: `{service}-{instance}-values.yaml`. Human and engine edits -land in the SAME files.** (dfe-infra `docs/deployment/state-and-repos.md`.) +**Every helm value for an app instance in this environment lives here, one file per service instance: `{service}-{instance}-values.yaml`. Human and engine edits land in the SAME files.** (dfe-infra [docs/deployment/state-and-repos.md](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/state-and-repos.md).) -The substrate and platform charts - ClickHouse, Kafka, the network policies, the -gateway - are NOT app instances and do not belong here. They have their own -overlay directory, [`infra/`](../infra/README.md), for the reason in the next -section. +The data and platform charts (ClickHouse, Kafka, the network policies, the gateway) are NOT app instances and do not belong here. Their overlay directory is [`infra/`](../infra/README.md). -## Nothing but instance files, and nothing in a subdirectory +- The dfe-infra ApplicationSet's git files generator globs `values/*-values.yaml` in this repo: **one file = one deployed app instance**. Creating `dfe-receiver-default-values.yaml` enables a receiver, and deleting it removes the app. Do NOT park scratch YAML here, because any `*-values.yaml` file becomes an Argo application. +- Argo CD passes that glob to `git ls-files` as a pathspec, and a pathspec without `:(glob)` magic lets `*` match `/`. A file in a subdirectory MATCHES too, so a subdirectory under `values/` is not a safe parking spot for anything, and `infra/` sits at the repo root, out of the pathspec's reach. +- File CONTENT is plain chart values for that service (top-level keys, such as `image.tag`, `resources`, `keda`), layered over dfe-infra's defaults and `infra/common.yaml` ([the values cascade](../docs/architecture.md)). The environment's cloud and profile come from cluster annotations, not from a values file. +- dfe-engine writes these files by commit (the gitcrud `helmvars` class, RBAC `helmvars:write`), and operators edit the same files by PR. Git history is the audit trail either way. +- Per-component version overrides (`pins.yaml` `overrides:`) are applied by merging the emitted fragment into the SERVICE'S EXISTING values file here (`dfe-stack resolve --emit-values` prints per-service fragments with the target filename), never as a standalone file, which would spawn a phantom app. -The ApplicationSet's git files generator globs `values/*-values.yaml`, and Argo -CD passes that to `git ls-files` as a pathspec. A pathspec without `:(glob)` -magic lets `*` match `/`, so `values/anything/foo-values.yaml` MATCHES and -becomes an Argo application too. Proven: +The pathspec in a scratch checkout, with one file in a subdirectory: -``` +```console $ git ls-files --full-name -- 'values/*-values.yaml' values/dfe-receiver-default-values.yaml values/infra/clickhouse-cluster-values.yaml <- a subdirectory does not hide it ``` -So a subdirectory under `values/` is not a safe parking spot for anything. That -is why `infra/` sits at the repo root: the pathspec is anchored on `values/`, so -nothing under `infra/` can ever match it. - -How it works: - -- The dfe-infra ApplicationSet's git files generator globs - `values/*-values.yaml` in this repo: **one file = one deployed app - instance**. Creating `dfe-receiver-default-values.yaml` enables a receiver; - deleting it removes the app. Do NOT park scratch YAML here - any - `*-values.yaml` file becomes an Argo application. -- File CONTENT is plain chart values for that service (top-level keys, - e.g. `image.tag`, `resources`, `keda`), attached to the app as a - `$values` file over the pinned base + cloud + profile layers. The - environment's cloud/profile selection comes from cluster annotations, - not from a values file. -- dfe-engine writes these files by commit (the gitcrud `helmvars` class, - RBAC `helmvars:write`); operators edit the same files by PR. Git history - is the audit trail either way. -- Per-component version overrides (`pins.yaml` `overrides:`) are applied by - merging the emitted fragment into the SERVICE'S EXISTING values file here - (`dfe-stack resolve --emit-values` prints per-service fragments with the - target filename) - never as a new standalone file, which would spawn a - phantom app. - -## What an instance file must carry - -The `deploy` block is not optional: the ApplicationSet reads it to name the -Argo Application, so a file without it produces nothing at all. +## What an instance file must have + +The `deploy` block is not optional: the ApplicationSet reads it to name the Argo Application, so a file without it produces no application. ```yaml deploy: service: dfe-transform-vrl instance: edge -# An app that runs once per source also needs its own object names, consumer +# An app that runs once per source needs distinct object names, a consumer # group and OTel service.name. Without them two instances of one app collide on # names and are indistinguishable in the otel tables. A stack-wide app such as -# dfe-receiver carries the deploy block alone. +# dfe-receiver needs the deploy block alone. fullnameOverride: dfe-transform-vrl-edge component: transform-vrl-edge otel: @@ -75,14 +45,23 @@ resources: limits: {cpu: 500m, memory: 512Mi} ``` +Apps that read content files, the transforms, keep them inline here, as a list of `{name, content}` under `fileSets..files`. The chart renders the list into a ConfigMap and mounts it. Content lives in the values rather than as loose files in this repo because Helm cannot read a raw file out of an Argo `$values` source. The old `transformFiles` key fails the render. + +```yaml +fileSets: + transforms: + files: + - name: 000_parse.vrl + content: | + . = parse_json!(.message) + .ts = to_timestamp!(.timestamp) +``` + +A block scalar keeps the bytes verbatim, so VRL stays exact and a Vector transform keeps its comments and its nested `source: |` block. The ceiling is the ~1 MiB ConfigMap limit. Large shipped payloads and binary enrichment tables belong in an image or object store, not here. + ## The receiver's ingest door -The ingest door is `exposure.groups.ingest`, a deployment-wide setting -read by the receiver and the gateway alike, so it lives in -[`infra/common.yaml`](../infra/README.md), not in the receiver's instance -file. On `door: loadbalancer` (the cloud default) an EMPTY `allow` renders -no load balancer at all, so a fresh deploy has no public ingest until the -senders are named: +The ingest door is `exposure.groups.ingest`, a deployment-wide setting read by the receiver and the gateway alike, so it lives in [`infra/common.yaml`](../infra/README.md), not in the receiver's instance file. On `door: loadbalancer` (the cloud default) an EMPTY `allow` renders no load balancer at all, so a fresh deploy has no public ingest until the senders are named: ```yaml # infra/common.yaml @@ -93,39 +72,6 @@ exposure: - 203.0.113.0/24 ``` -A `/0` in `allow` is open to every address. The allow list is not -authentication: a listener no ingest credential lists runs -`server.auth.mode: none`, and the engine switches it to `bearer` once a -credential names it. - -A deployment with no LoadBalancer provisioner sets `door: gateway` -instead, which routes ingest through the cluster Gateway on -`receiver.{domain}`. That path carries HTTP only, so any other listener -left `exposed: true` fails the chart render rather than deploying a port -nothing reaches. The retired keys `exposure.mode` and -`exposure.public.loadBalancerSourceRanges` fail the render by name. The -full model is dfe-infra `docs/ingest-edge.md`. - -## Files an app reads off disk - -Apps that read their own content files - the transforms - carry it inline -here, as a list of `{name, content}` under `fileSets..files`. The -chart renders the list into a ConfigMap and mounts it. Content lives in -the values rather than as loose files in this repo because Helm cannot -read a raw file out of an Argo `$values` source. The old `transformFiles` -key fails the render. - -```yaml -fileSets: - transforms: - files: - - name: 000_parse.vrl - content: | - . = parse_json!(.message) - .ts = to_timestamp!(.timestamp) -``` +A `/0` in `allow` is open to every address. The allow list is not authentication: a listener no ingest credential lists runs `server.auth.mode: none`, and the engine switches it to `bearer` once a credential names it. -A block scalar keeps the bytes verbatim, so VRL stays exact and a Vector -transform keeps its comments and its own nested `source: |` block. The -ceiling is the ~1 MiB ConfigMap limit; large shipped payloads and binary -enrichment tables belong in an image or object store, not here. +The on-prem cloud files set `door: gateway`, and a cloud deployment with no LoadBalancer provisioner sets it in `infra/common.yaml`. That door routes ingest through the cluster Gateway on `receiver.{domain}`. The Gateway routes the `http` listener alone, so any other listener left `exposed: true` fails the chart render rather than deploying a port nothing reaches. The retired keys `exposure.mode` and `exposure.public.loadBalancerSourceRanges` fail the render by name. The full model is dfe-infra [docs/ingest-edge.md](https://github.com/hyperi-io/dfe-infra/blob/main/docs/ingest-edge.md). From f3e39c01e3a93f2111565f07cded9cd08ee5e716 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 15:06:58 +1100 Subject: [PATCH 2/2] docs(architecture): list the edge tunnel in the values cascade The table lists argocd/values/apps/ and the culvert instance files as inputs to the edge tunnel ApplicationSet, matching the apps ApplicationSet (dfe-infra layer2-edge.yaml). The edge gateway alone reads infra/.yaml. --- docs/architecture.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index bf97ea2..56e2692 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -36,13 +36,13 @@ Helm applies `-f` files in order and the last wins. Each ApplicationSet stacks d | `argocd/values/common.yaml` | dfe-infra | every appset below | | `argocd/values/.yaml` | dfe-infra | every appset below | | `argocd/values/profile-.yaml` | dfe-infra | every appset below | -| `argocd/values/apps/...` | dfe-infra | apps: `_common.yaml`, then the app's `values.yaml`, `.yaml` and `profile-.yaml` | +| `argocd/values/apps/...` | dfe-infra | apps and the edge tunnel: `_common.yaml`, then the app's `values.yaml`, `.yaml` and `profile-.yaml` | | `sizing/.data-charts.values.yaml`, `sizing/.values.yaml` | here | data and platform | | `infra/common.yaml` | here | apps, data, platform and edge | -| `infra/.yaml` | here | data, platform and edge | -| `values/--values.yaml` | here | apps | +| `infra/.yaml` | here | data, platform and the edge gateway | +| `values/--values.yaml` | here | apps, and the edge tunnel for `values/culvert-*-values.yaml` | -The edge appset adds `argocd/values/edge-.yaml` after the cloud file. Facts the cluster secret holds, such as the domain and the registry, arrive as Helm parameters and beat every file. +Both edge ApplicationSets add `argocd/values/edge-.yaml` after the cloud file. Facts the cluster secret holds, such as the domain and the registry, arrive as Helm parameters and beat every file. A profile file is a tier DEFAULT, not a lock, so a deployment can declare `clickhouse.mode: external` in `infra/common.yaml` and beat `profile-scale.yaml`. A fact more than one chart consumes belongs in `infra/common.yaml` once, so the network policy, the ClickHouse chart and the loader cannot disagree about it.