From 39ba29528a93e3f2618a133231b5107467f12832 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 13:43:17 +1100 Subject: [PATCH 1/2] docs(engine): update claims and redraw diagrams Update claims across every page under docs/, the README and CONTRIBUTING against the code and sibling repos. Pages covered by #1177 are unchanged. The engine applies ClickHouse DDL itself and commits no ddl/ or envoy/ tree. The fork's create-rule button posts to its route. It sends no postMessage to dfe-ui. scalo rejects MSK IAM. `main` is a core source the API will not modify. `If-Match` is optional on a write. Redraw all 58 Mermaid diagrams to the diagram standard. Replace text diagrams with Mermaid or lists where no picture is needed. All 62 diagrams pass the checker on Mermaid 12.3.0. architecture.md holds L2 Services -- DFE and L3 Components -- dfe-engine. Cut config comments in 17 files to pointers and short reasons. Move the retained explanations to the docs. No value changes. --- .github/workflows/sync-dfe-engine-types.yml | 16 +- .gitleaks.toml | 18 +- .hyperi-ci.yaml | 10 +- CONTRIBUTING.md | 94 ++- Dockerfile | 64 +- Makefile | 49 +- README.md | 19 +- docker-compose.dev.yaml | 19 +- docs/README.md | 8 +- docs/architecture.md | 449 ++++++++------ docs/archive/dfe-infra-snapshot.md | 70 +-- docs/archive/monorepo-migration.md | 30 +- docs/auth-envoy-topology.md | 64 +- docs/control-plane/README.md | 227 +++---- docs/control-plane/dfe-cli-design.md | 28 +- docs/control-plane/gitops-commit-standard.md | 155 ++--- docs/control-plane/governed-ops-design.md | 493 ++++++--------- docs/control-plane/idp-setup/README.md | 2 +- docs/control-plane/idp-setup/entra.md | 32 +- docs/control-plane/idp-setup/google.md | 26 +- docs/control-plane/idp-setup/okta.md | 20 +- docs/control-plane/oidc-rbac-architecture.md | 371 ++++++------ docs/control-plane/rbac-engine-reads.md | 28 +- docs/control-plane/rbac-groups.md | 117 ++-- docs/control-plane/rbac-sign-in.md | 150 +++-- docs/control-plane/rbac-vocabulary.md | 81 ++- docs/control-plane/rbac.md | 572 +++++++++--------- docs/control-plane/synthetic-data.md | 93 ++- docs/control-plane/type-safety-sync.md | 72 +-- docs/control-plane/ui-api-guide.md | 60 +- docs/data-evolution-acceptance.md | 27 +- docs/data-evolution.md | 165 ++--- docs/data-plane/README.md | 40 +- docs/data-plane/expressions-cel.md | 73 +-- docs/data-plane/external-pipelines.md | 45 +- docs/data-plane/field-mapping.md | 17 +- docs/data-plane/hunt-match-delivery.md | 21 +- docs/data-plane/query-api-python.md | 15 +- docs/data-plane/query-api.md | 200 +++--- docs/data-plane/schema-bootstrap.md | 46 +- docs/data-plane/schema-classes.md | 36 +- docs/data-plane/schema-sync.md | 42 +- docs/data-plane/schema.md | 135 ++--- docs/data-plane/source-definition.md | 47 +- docs/data-plane/source-detections.md | 38 +- docs/data-plane/source-flow.md | 175 +++--- docs/data-plane/source-registry.md | 32 +- docs/data-plane/source-routing.md | 22 +- docs/data-plane/source.md | 17 +- docs/data-plane/topic-contract.md | 24 +- docs/deployment/README.md | 172 +++--- docs/deployment/backing-services.md | 181 ++++-- docs/deployment/e2e-acceptance.md | 113 ++-- docs/deployment/keda-scaling.md | 67 +- docs/deployment/observability-standard.md | 118 ++-- docs/deployment/supply-chain-pinning.md | 65 +- docs/deployment/transports.md | 106 +++- docs/integrations/README.md | 2 +- docs/integrations/dfe-ui-guide.md | 44 +- docs/local-dev.md | 19 +- pyproject.toml | 77 +-- src/dfe_engine/appmgmt/queries.yaml | 97 +-- src/dfe_engine/backing/providers.yaml | 39 +- src/dfe_engine/defaults.yaml | 24 +- src/dfe_engine/keda_shim/queries.yaml | 40 +- .../pipeline/pipeline_template.yaml | 3 - .../default_configs/archiver-default.yaml | 4 +- .../default_configs/archiver-production.yaml | 12 +- .../default_configs/loader-default.yaml | 16 +- .../default_configs/loader-production.yaml | 18 +- src/dfe_engine/usage/queries.yaml | 14 +- tests/e2e/README-live.md | 88 ++- 72 files changed, 2900 insertions(+), 3073 deletions(-) diff --git a/.github/workflows/sync-dfe-engine-types.yml b/.github/workflows/sync-dfe-engine-types.yml index b07b2ada2..a16a6a8bd 100644 --- a/.github/workflows/sync-dfe-engine-types.yml +++ b/.github/workflows/sync-dfe-engine-types.yml @@ -2,20 +2,7 @@ # File: .github/workflows/sync-dfe-engine-types.yml # Purpose: Sync the OpenAPI specs to the dfe-ui types package # -# Creates a PR in: https://github.com/hyperi-io/dfe-ui -# Target paths. The two specs are COPIED; both type files are then -# regenerated by `yarn generate` from the copied specs, so neither is listed as an -# input and both change in the PR: -# packages/dfe-engine-types/specs/openapi.json -# packages/dfe-engine-types/specs/openapi.e2e.json -# packages/dfe-engine-types/types/index.d.ts (generated) -# packages/dfe-engine-types/types/e2e.d.ts (generated) -# -# Auth: a GitHub App installation token (vars.GH_APP_CLIENT_ID + secrets.GH_APP_PRIVATE_KEY), -# scoped to the dfe-ui repo with contents + pull-requests write, which covers push, -# PR open/close and deleting superseded sync branches (--delete-branch on close) -# Optional var: DFE_UI_REPO - repo as owner/name (default: hyperi-io/dfe-ui) -# Note: Git clone URLs require owner/name; numeric repo IDs are not supported. +# How it runs and what it needs: docs/control-plane/type-safety-sync.md. name: Sync dfe-engine-types @@ -88,7 +75,6 @@ jobs: exit 0 fi - # Close any prior auto-sync PRs so only one types sync PR is open at a time export GH_TOKEN="${DFE_UI_TOKEN}" while IFS= read -r pr_num; do [ -z "$pr_num" ] && continue diff --git a/.gitleaks.toml b/.gitleaks.toml index 49052eb66..bd5551750 100644 --- a/.gitleaks.toml +++ b/.gitleaks.toml @@ -1,10 +1,4 @@ -# Project-specific gitleaks allowlist -# Extends the CI base configuration (ci/.gitleaks.toml) -# -# The CI runs gitleaks to detect secrets in your code. This file lets you -# exclude false positives specific to your project. -# -# Documentation: https://github.com/gitleaks/gitleaks#configuration +# Project false positives for gitleaks: https://github.com/gitleaks/gitleaks#configuration [extend] useDefault = true @@ -12,27 +6,21 @@ useDefault = true [allowlist] description = "Project-specific false positive allowlist" -# Paths to exclude from scanning (regex patterns) paths = [ - # Test files often contain fake tokens/secrets '''tests/.*\.py''', '''tests/fixtures/.*''', - # Example configuration files '''.env.example''', ] -# Regex patterns for known false positives regexes = [ '''test-token''', '''example\.com''', '''llamatron''', - # DFE API key format used in documentation examples + # The API key in the docs' examples. '''dfe_ak_live_8a3f''', - # Makefile e2e ClickHouse wipe: curl -u takes shell variable references, never a - # literal credential. Both forms stay -- gitleaks scans the whole history. + # The Makefile's curl -u takes shell variables, not a credential. History holds both forms. '''DFE_ENGINE_CLICKHOUSE_USERNAME:-default\}:\$\$\{DFE_ENGINE_CLICKHOUSE_PASSWORD''', '''\$\$ch_user:\$\$ch_pass''', ] -# Stopwords - findings containing these are likely false positives stopwords = ["example", "sample", "test", "mock", "fake", "dummy", "placeholder"] diff --git a/.hyperi-ci.yaml b/.hyperi-ci.yaml index d73eff4ed..a03a5e5bc 100644 --- a/.hyperi-ci.yaml +++ b/.hyperi-ci.yaml @@ -26,11 +26,11 @@ release: library: "2.14.15" test: - min_coverage: 70 # Current: 75%. Target: 80%. Ratchet up as coverage improves. + min_coverage: 70 native_tools: [helm] python: parallel: auto - # Peak RSS of one process in a full core run was 1206 MiB (/usr/bin/time -v, 11 workers). + # Just over one worker's peak RSS in a full core run. worker_mem_mb: 1280 full: # A release waits on a green full-tier run. @@ -51,11 +51,7 @@ test: quality: python: - # semgrep: warn matches hyperi-ci's own default (quality/semgrep.py:36), so - # this is a deliberate pin rather than a downgrade. Its 38 findings are - # Flask and Django rules firing on a FastAPI codebase -- the SQL-injection - # hit is api/v1/queries.py:232, a raw-query endpoint behind - # require_action(query_execute), where taking a caller's SQL is the point. + # hyperi-ci's default. Its findings are Flask and Django rules on a FastAPI app, and the raw-query endpoint takes SQL on purpose. semgrep: warn # Above hyperi-ci's warn default: these three are at zero, so a new finding fails the PR. vulture: blocking diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 99cc3168b..a66e459dd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -17,7 +17,7 @@ versioning and changelog generation. All commits must follow this format: [optional footer(s)] ``` -### Types +The types and the release each one cuts: | Type | Description | Version Bump | |------|-------------|--------------| @@ -33,8 +33,6 @@ versioning and changelog generation. All commits must follow this format: | `chore` | Other changes that don't modify src or test files | None | | `revert` | Reverts a previous commit | Varies | -### Breaking Changes - For breaking changes that require a major version bump, add `!` after the type or include `BREAKING CHANGE:` in the footer: @@ -44,7 +42,7 @@ feat!: remove deprecated API endpoints BREAKING CHANGE: The /v1/users endpoint has been removed. Use /v2/users instead. ``` -### Examples +Examples: ```text feat(auth): add OAuth2 support for Google login @@ -58,35 +56,24 @@ refactor(core)!: restructure module exports BREAKING CHANGE: Named exports are now used instead of default exports. ``` -### Scope - Scope is optional but recommended. Use it to indicate the area of the codebase affected (e.g., `api`, `auth`, `core`, `cli`, `docs`). -## Semantic Versioning - -This project follows [Semantic Versioning 2.0.0](https://semver.org/): - -- **MAJOR** (X.0.0): Breaking changes that require users to modify their code -- **MINOR** (0.X.0): New features that are backwards-compatible -- **PATCH** (0.0.X): Bug fixes and minor improvements - -Versions are automatically determined by semantic-release based on commit -messages. Do not manually update version numbers. +semantic-release works out the version from the commit types, following [Semantic Versioning 2.0.0](https://semver.org/). Never edit a version number by hand. -## Developer Certificate of Origin +## Developer Certificate of Origin and sign-off -This project uses the Developer Certificate of Origin (DCO) to ensure that -contributors have the right to submit their contributions. +This project uses the Developer Certificate of Origin (DCO), so every +contributor certifies they have the right to submit their contribution. By making a contribution to this project, you certify that: 1. The contribution was created in whole or in part by you and you have the - right to submit it under the license indicated in the file; or + right to submit it under the license indicated in the file, or 2. The contribution is based upon previous work that, to the best of your knowledge, is covered under an appropriate open source license and you - have the right under that license to submit that work with modifications; + have the right under that license to submit that work with modifications, or 3. The contribution was provided directly to me by some other person who @@ -98,8 +85,6 @@ By making a contribution to this project, you certify that: and may be redistributed consistent with this project or the license(s) involved. -## How to Sign Off Your Commits - You must sign off each commit to indicate your acceptance of the DCO. Combine the signoff with your conventional commit message: @@ -134,18 +119,16 @@ anniversary of its release. ## Running locally To run the engine as a plain local process (a `uv` virtualenv, no containers or -Kubernetes) -- including the no-dependencies auth spike and how it wires to -dfe-hyperdx / dfe-ui / a local dfe-deploy clone -- see [docs/local-dev.md](docs/local-dev.md). +Kubernetes), with or without ClickHouse, and wire it to dfe-hyperdx, dfe-ui and a +local dfe-deploy clone, see [docs/local-dev.md](docs/local-dev.md). ## How to Contribute -1. **Fork the repository** and create your branch from `main` -2. **Make your changes** following the commit message format above -3. **Sign off your commits** with the DCO -4. **Test your changes** to ensure they work as expected -5. **Submit a pull request** with a clear description of what you've done +Fork the repository and branch from `main`. Commit in the format above, sign off +every commit, test the change, and open a pull request that says what it does. +A question that is not a bug goes in an issue. -### Pull Request Checklist +Before you open it, check: - [ ] Commits follow the conventional commit format - [ ] All commits are signed off (DCO) @@ -159,7 +142,7 @@ Open every pull request against `main`. `.github/workflows/ci.yml` fires `pull_r If you stack anyway, two things will bite: - Retarget every child to `main` BEFORE merging its parent, then push the child so the `pull_request` run fires. Merging a parent with `--delete-branch` deletes the child's base and GitHub closes the child. The two states deadlock: you cannot reopen a PR whose base branch is gone, and you cannot retarget a closed PR. Recovery is to recreate the base ref at any commit, reopen, retarget to `main`, then delete the ref again. -- A squash merge rewrites the history the child sits on, so replay only the child's own commits: `git rebase --onto main `. +- A squash merge rewrites the history the child sits on, so replay the child's commits and nothing else: `git rebase --onto main `. ## Testing your branch in a full stack @@ -169,13 +152,13 @@ see running, gets a full stack: on one host with [dfe-docker](https://github.com/hyperi-io/dfe-docker), or on a Kubernetes cluster with [dfe-infra](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/README.md). Both take the image tag or branch you nominate for each app. Where that stack -runs is your own concern and is never committed here. +runs is your concern and is never committed here. ## Run the checks locally before you push HyperI projects gate every push through CI (lint, format, tests, secret scan, dependency audit, build). Run the SAME checks locally first so your change lands -green instead of bouncing: +pass CI the first time: ```bash hyperi-ci check @@ -188,21 +171,17 @@ the same suite CI runs -- and reports what to fix. Run it before every push and pull request. It is the single best way to give your change the best chance of surviving CI. -## CI/CD Workflow - -When your pull request is merged to `main`: - -1. **semantic-release** analyses commit messages since the last release -2. Determines the next version number based on commit types -3. Generates/updates the CHANGELOG -4. Creates a new GitHub release with release notes -5. Publishes the package (if applicable) - -This happens automatically - no manual intervention required. +## Releases -## Questions +A merge to `main` runs CI and releases nothing. A release runs when the merge +commit has a `Release: true` trailer, or when a maintainer dispatches the CI +workflow (`.github/workflows/ci.yml`) with `from-head=true` to release HEAD or +`tag=` to publish that tag again. hyperi-ci then: -If you have questions about contributing, please open an issue or contact us. +1. Works out the next version from the commit types since the last release. +2. Tags it and writes the GitHub release with its notes. +3. Publishes the container image and the Helm chart to GHCR. No wheel goes to + PyPI (`.hyperi-ci.yaml`, `release.destinations`). ## Context @@ -225,7 +204,6 @@ System design and why it is built this way: | `src/dfe_engine/schema/` | ClickHouse DDL planning and apply | | `src/dfe_engine/source/` | Source definitions. Receiver and loader routing is COMPILED from these | | `src/dfe_engine/gitops/` | every mutation is a git commit, and the deploy repo is the authority | -| `docs/superpowers/` | specs and plans, including designs that are NOT built yet | ### Commands that prove a change @@ -239,12 +217,12 @@ hyperi-ci check --quick # what CI will say, faster `ruff format` is available but the repo has not adopted it, and there is no `black`. **`uv run pytest` does not run everything.** The default `addopts` deselects the -`integration`, `live` and `upstream` markers, so a green local run has not +`integration` and `live` markers, so a passing local run has not touched a backing service. Pass `-m integration` to include them. -Green locally is not green in CI. **A CI run is not atomic against hyperi-ci**: +A local pass is not a CI pass. **A CI run is not atomic against hyperi-ci**: reusable workflows resolve `@main` per JOB, so a push-event run reports success -with every real job SKIPPED. Read the `pull_request` run, per job, and check the +with every check job SKIPPED. Read the `pull_request` run, per job, and check the durations -- a test job that "passed" in 0s did not run. ### What tends to bite @@ -252,15 +230,15 @@ durations -- a test job that "passed" in 0s did not run. | Don't | Do | Why | |---|---|---| | Hardcode an app name in engine code | Add it to `dfe-infra/apps.yaml` | What an app IS is DATA. Adding one is a manifest edit plus a chart, never an engine release. | -| Treat this as an internal tool | Code to the injected config and interface | It is a PRODUCT other organisations deploy. Env-specifics -- cluster refs, cred paths, our fleet -- belong in private config repos, never here. | +| Treat this as an internal tool | Code to the injected config and interface | It is a PRODUCT other organisations deploy. Env-specifics -- cluster refs, cred paths, a deployment's hosts -- belong in that deployment's private config repo, never here. | | Hand-edit compiled routing | Change the Source | Receiver and loader routing is derived, so the API reports a hand edit as DRIFT and re-syncs over it. | -| Mock a backing service | testcontainers, a real one | No mocks as proof. `pass` and `TODO` are not functionality. | +| Mock a backing service | testcontainers, a running one | No mocks as proof. `pass` and `TODO` are not functionality. | | `from typing import List, Dict, Optional` | `list[str]`, `str \| None` | Built-in generics, and `requires-python = ">=3.14"`. | | `datetime.utcnow()` | `datetime.now(UTC)` | Deprecated since 3.12. | | stdlib `logging` | `from scalo.logger import logger` | House rule, so every service emits one structured JSON format. | | Raw `httpx` with hand-rolled retry | `scalo.http.HttpClient` | House rule. Importing httpx for its exception and response TYPES is correct and expected. | | `yaml.safe_load` | `dfe_engine.yaml_utils` | ruamel, YAML 1.2. YAML 1.1 reads `no` as boolean false. | -| `except Exception:` then a silent fallback | Catch the specific error, log with context, re-raise | 12 sites already return `False`/`None`/`[]` with no trace, out of 251 `except Exception:` sites in `src/`. An operator cannot tell failure from empty. | +| `except Exception:` then a silent fallback | Catch the specific error, log with context, re-raise | A bare fallback to `False`, `None` or `[]` leaves no trace, and an operator cannot tell failure from empty. | | Assume a config key does something | Grep for its consumer | Config that parses and is never read is a recurring defect here. | | Trust `gh issue list` | `gh issue view ` | It caps at 30 without `--limit`, so absence is not evidence of closure. | @@ -284,9 +262,9 @@ moves. | repo | how it interacts | |---|---| -| dfe-infra | Deploys this engine, and owns `apps.yaml` -- a byte copy is vendored here at `appmgmt/apps.yaml` as the last resort in the resolution order. Lockstep: a catalogue entry the engine does not carry is not reflected by its management API. | +| dfe-infra | Deploys this engine, and owns `apps.yaml` -- a byte copy sits here at `src/dfe_engine/appmgmt/apps.yaml`, read when `DFE_ENGINE_APP_CATALOGUE_FILE` names no other file. Lockstep: a catalogue entry the engine does not carry is not reflected by its management API. | | dfe-schemas | Ships the schema and DDL trees as a PACKAGE, not a submodule. Read out of the installed wheel unless `DFE_ENGINE_SCHEMAS_DIR` points elsewhere. | -| scalo-py | The shared Python library, plus a contract guard: this repo overrides scalo's runtime base image with its own literal, and its own test fails when the committed Dockerfile stops matching. | +| scalo-py | The shared Python library, plus a contract guard: this repo sets scalo's runtime base image as a literal, and a test here fails when the committed Dockerfile stops matching. | | the Rust apps | Each emits its config schema (` config-schema`), and the deployment mounts it at `DFE_ENGINE_APP_CONTRACT_DIR`. `/api/v1/services` takes every default and type from it, so nothing of an app's config is restated here. | | logreducer | A declared dependency (`logreducer[enhanced]`). The sampler's `smart` mode, its default, and `anomaly` mode run it, imported lazily so the scikit-learn stack loads only when one of them does. | @@ -295,10 +273,10 @@ moves. | repo | how it interacts | |---|---| | dfe-infra | Pins this engine's image in TWO places: the dfe-engine chart and the hyperdx chart, which runs the engine image as its dashboards init container. Both move together. There is no schema-apply workload, because the engine applies schema at its own startup. | -| dfe-ui | Vendors this repo's OpenAPI spec and generates its scope types from an engine module. A workflow here re-vendors it by PR on a push to main. | +| dfe-ui | Copies this repo's OpenAPI specs into its `dfe-engine-types` package and generates its types from them. `.github/workflows/sync-dfe-engine-types.yml` opens that PR on dfe-ui when a push to main changes a spec. | **The consequence, and it is the reason this section exists:** dfe-engine is the ONLY controller of ClickHouse objects and Kafka topics. A change to schema, source compilation or topic naming is a SUITE-WIDE move, not a local one -- every Rust service reads configuration this engine writes, and infra and docker -wait on it. A one-line change here can land in five other repositories. +wait on it. diff --git a/Dockerfile b/Dockerfile index 19356b29d..291739a26 100644 --- a/Dockerfile +++ b/Dockerfile @@ -5,69 +5,39 @@ # License: BUSL-1.1 # Copyright: (c) 2026 HYPERI PTY LIMITED # -# The runtime stage below aligns with the scalo-py deployment contract -# (src/dfe_engine/deployment_contract.py). validate_dockerfile in -# tests/unit/test_deployment/test_contract.py guards drift (runtime base image, -# EXPOSE, venv PATH, entrypoint). The builder stage is project-specific (uv venv -# build from the public PyPI index). - -# --- Builder stage (uv venv) --- -# Base image is a build-arg with a digest-pinned default (#106). Trixie both -# ends so glibc(runtime) >= glibc(builder). Pinned so neither tag can float and -# invert that relationship silently. Re-resolve digests on a bump. Renovate -# maintains them. +# The runtime stage follows the scalo-py deployment contract, and +# tests/unit/test_deployment/test_contract.py guards it. + +# --- Builder stage --- +# Trixie in both stages, so the runtime glibc is never older than the builder's. ARG BUILDER_IMAGE=ghcr.io/astral-sh/uv:python3.14-trixie-slim@sha256:d8e732839dfc4225a1ff698803b478b2223c05636c055d3add41ab9dbec0f315 FROM ${BUILDER_IMAGE} AS builder WORKDIR /app -# Compile .pyc at build so the cost is paid once here, not on every cold start -# (#106). ~1s off startup for a larger image. The platform ordering ranks -# startup first, image disk last. +# Compile .pyc once here rather than on every cold start. ENV UV_COMPILE_BYTECODE=1 -# Phase 1 -- dependencies only, so a source edit does not reinstall the graph -# (#106). UV_NO_BUILD=1 requires a prebuilt wheel for every third-party dep -# (#106: a silent compile-from-source is the failure mode we most want to be -# loud). Scoped to this phase because a blanket ENV breaks uv's own editable -# self-install (dfe-engine has no wheel of its own). README.md is read during -# metadata resolution (pyproject sets readme = "README.md"). +# Dependencies first, so a source edit does not reinstall them. UV_NO_BUILD=1 +# fails a dependency with no wheel, and only here: uv builds the project itself. COPY pyproject.toml uv.lock README.md ./ RUN UV_NO_BUILD=1 uv sync --frozen --no-dev --no-install-project -# Phase 2 -- our own source + the editable project install (built, not a wheel). COPY src/ src/ RUN uv sync --frozen --no-dev -# /app/schemas-seed is copied out of the installed dfe-schemas wheel, so the -# image ships exactly the version uv.lock pins. bootstrap.py copies it into -# DFE_ENGINE_SCHEMAS_DIR on the first run of each engine version. It stays root-owned -# and read-only, as the COPY it replaced was. -# -# The rest are writable dirs owned by the runtime UID. /app/secrets is the -# scalo.secrets root (#106): the engine mints an ES384 JWT signing key on first -# boot and, with no writable secrets dir and no runtime WORKDIR, it defaulted to -# ./.secrets under / and crash-looped as non-root. COPY --from preserves this -# ownership into the runtime stage. -# -# /app/app-config is where a deployment with no chart has the engine render each -# app's own config file. It exists in the image so a volume mounted there -# inherits this ownership. A mountpoint the image does not own lands root-owned -# and the engine runs as UID 1000. +# The schema seed bootstrap.py copies on first run (README.md, Configuration), and +# the writable dirs the runtime UID owns. /app/app-config takes the app configs a +# deployment with no chart has the engine render. RUN /app/.venv/bin/python -c \ "import dfe_schemas, shutil; shutil.copytree(dfe_schemas.schemas_root(), '/app/schemas-seed')" \ && mkdir -p /app/schemas /app/config /app/secrets /app/app-config \ && chown -R 1000:1000 /app/schemas /app/config /app/secrets /app/app-config -# --- Runtime stage (aligned with scalo-py deployment contract) --- -# Digest-pinned runtime base (#106). python:3.14-slim is already Debian 13 -# trixie. Kept literal (not an ARG) because scalo's validate_dockerfile matches -# `FROM ` by substring, so ARG-parameterising the runtime base needs a -# scalo-py change (hyperi-io/scalo-py#4) -- tracked, not worked around by deleting -# the drift test. +# --- Runtime stage --- +# A literal, not an ARG: test_contract.py matches `FROM ` as written. FROM python:3.14-slim@sha256:35caba83af2fe6da9a5d922f9b0156f21d1246a48278c58a08bd253303ba42e8 AS runtime -# Static OCI labels (from contract) LABEL org.opencontainers.image.title="dfe-engine" LABEL org.opencontainers.image.description="DFE Engine -- REST API and config control plane" LABEL org.opencontainers.image.vendor="HYPERI PTY LIMITED" @@ -79,7 +49,6 @@ LABEL io.hyperi.profile="production" RUN apt-get update && apt-get install -y --no-install-recommends \ ca-certificates curl netcat-openbsd iputils-ping \ && rm -rf /var/lib/apt/lists/* -# Dynamic OCI labels (injected by CI at build time) ARG OCI_SOURCE="" ARG OCI_REVISION="" ARG OCI_VERSION="" @@ -89,14 +58,12 @@ LABEL org.opencontainers.image.revision="${OCI_REVISION}" LABEL org.opencontainers.image.version="${OCI_VERSION}" LABEL org.opencontainers.image.created="${OCI_CREATED}" -# Copy the uv-built virtualenv from the builder and put it on PATH. COPY --from=builder /app /app # The hyperi-ci container build adds /licenses/LICENSE and the Build's dist/THIRD-PARTY-NOTICES to this final stage. COPY LICENSE NOTICE /app/ ENV PATH="/app/.venv/bin:$PATH" -# Runtime WORKDIR (#106): without it CWD is / and any relative default path -# (e.g. the secrets root) resolves under /, unwritable by the non-root user. +# The relative secrets default resolves here, not under an unwritable /. WORKDIR /app # DFE_ENGINE_SECRETS_PATH holds a directory path, not a secret value, and the engine and chart read it by name. @@ -108,10 +75,9 @@ ENV DFE_CONFIG_DIR=/app/config \ RUN useradd --create-home --uid 1000 appuser USER 1000 -# 9090 = observability (health + /metrics, scalo ServiceApp); 8000 = API traffic. EXPOSE 9090 8000 -# Probe the observability port (#106): health answers on 9090, not 8000. +# 9090 serves health and /metrics, 8000 the API. HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ CMD ["curl", "-sf", "-o", "/dev/null", "http://localhost:9090/livez"] diff --git a/Makefile b/Makefile index fd110bb1f..3f9633368 100644 --- a/Makefile +++ b/Makefile @@ -19,10 +19,7 @@ build: check: hyperi-ci check -# Local dev API server. First time: `cp .env.example .env`. Loads .env (the app -# does not read it itself), defaults the config dir, and serves Swagger UI at -# /docs. DFE_ENGINE_SCHEMAS_DIR is left unset on purpose: the schemas then come from -# the installed dfe-schemas package, which is the version uv.lock pins. +# Local dev API server: docs/local-dev.md. The app does not read .env itself. dev: @test -f .env || { echo "Create .env first: cp .env.example .env"; exit 1; } @echo "Swagger UI -> http://localhost:$${DFE_ENGINE_API_PORT:-8003}/docs" @@ -30,34 +27,7 @@ dev: : "$${DFE_CONFIG_DIR:=./config}"; export DFE_CONFIG_DIR; \ uv run dfe-engine run -# Persisted host-run API for Playwright (dfe-ui `yarn test:e2e`). -# Fresh slate on every start: YAML config, schema writes, minted secrets and -# artifacts live under tmp/e2e (gitignored) so the installed schema trees and -# ./config are never touched, and every ClickHouse database the engine creates -# is namespaced under the $(E2E_CH_PREFIX) prefix so the wipe can sweep it by -# prefix rather than by a hand-maintained list. Wiping on START (not on stop) is -# deliberate: a killed or crashed run leaves nothing behind for the next one. -# Pass E2E_KEEP=1 to reuse the last run instead of wiping. -# Playwright must poll /readyz (not /health/ready, which 404s on this binary): -# DFE_ENGINE_READY_URL=http://127.0.0.1:8003/readyz yarn test:e2e -# Startup parity: a plain `dfe-engine run`, so the process comes up with exactly -# what auth/bootstrap.py seeds (roles.yaml, the four dfe-* groups, break-glass -# admin) and no baseline beyond it. Fixture data is the suite's job, seeded over -# the API from dfe-ui. -# Hunt log and checkpoint paths are pinned in too: they are not among the -# DFE_CONFIG_DIR-derived subdirs, so their defaults land in the CWD and would -# outlive the wipe. The gitops deploy repo is pinned for the same reason, and -# only takes effect when .env.e2e sets DFE_ENGINE_GITOPS_ENABLED=true -- the -# app-management seeds write their instances, artefacts and sources into it. -# NOT swept: ClickHouse users/roles, which are server-global rather than -# per-database. Only the governance reconcile creates those, and it runs -# unless DFE_ENGINE_TENANT_ISOLATION_ENABLED is false. -# E2E_NO_CLICKHOUSE=1 runs the engine with no ClickHouse at all: no table -# bootstrap, no tenant-isolation reconcile, no wipe. Pages that read ClickHouse -# answer 503 and the rest work. Use it where docker is not running. -# E2E_ENV_FILE and E2E_WORKSPACE can be set on the command line or in the -# environment (`make e2e-server E2E_ENV_FILE=/path/env`), -# so a driver script can keep its env file and workspace outside the checkout. +# Host-run API for dfe-ui's Playwright suite, wiped on every start: docs/local-dev.md. E2E_ENV_FILE ?= ./.env.e2e E2E_WORKSPACE ?= $(CURDIR)/tmp/e2e E2E_CH_PREFIX := dfe_e2e @@ -66,10 +36,7 @@ E2E_CH_HUNTS_DB := $(E2E_CH_PREFIX)_hunts E2E_CH_AUDIT_DB := $(E2E_CH_PREFIX)_audit E2E_CH_OTEL_DB := $(E2E_CH_PREFIX)_otel -# Drop every ClickHouse database under the e2e prefix. Enumerating the server -# beats a fixed list: a database added by a future bootstrap path is swept the -# day it appears. The prefix cannot match the shared dfe / dfe_hunts / dfe_audit -# / dfe_meta databases, so a dev stack on the same server is untouched. +# Drops every database under the e2e prefix, which no shared dfe database matches. define e2e-ch-wipe ch_host="$${DFE_ENGINE_CLICKHOUSE_HOST:-localhost}"; \ ch_port="$${DFE_ENGINE_CLICKHOUSE_PORT:-8123}"; \ @@ -98,12 +65,7 @@ define e2e-ch-wipe done endef -# Refuse to start behind another listener on the API port. This server binds -# 0.0.0.0, so a loopback-specific bind -- the dfe-engine container publishing -# 127.0.0.1:8003 -- shadows it for localhost traffic: both binds succeed, no -# error is raised, and the browser, dfe-ui and Playwright all reach the OTHER -# process and its persistent volume. A fresh workspace here cannot show through -# that, so the collision has to be fatal rather than silent. +# A loopback bind on the API port would silently take this 0.0.0.0 server's traffic. define e2e-port-guard if command -v lsof >/dev/null 2>&1 && lsof -nP -iTCP:"$$DFE_ENGINE_API_PORT" -sTCP:LISTEN >/dev/null 2>&1; then \ echo "error: something already listens on port $$DFE_ENGINE_API_PORT:"; \ @@ -114,8 +76,7 @@ define e2e-port-guard fi endef -# Teardown on demand: same wipe the next `make e2e-server` would do, for when a -# run is finished with and should leave nothing behind. +# The wipe the next e2e-server would do. e2e-clean: @test -f "$(E2E_ENV_FILE)" || { echo "Create $(E2E_ENV_FILE) first (copy .env.e2e.example and set DFE_ENGINE_ENV=test)"; exit 1; } @echo "Removing $(E2E_WORKSPACE)" diff --git a/README.md b/README.md index 3099e6d1a..bcc2ba4f4 100644 --- a/README.md +++ b/README.md @@ -7,14 +7,15 @@ The DFE control plane: an API and CLI that turn operator changes into git commit ## What it does -- Sources are defined once. The receiver's and loader's routing is compiled from them, so no app carries hand-written routing. -- Every change is a commit in the deployment's own git repo, and Argo CD applies it. The audit log is the git history. +- Sources are defined once. The receiver's and loader's routing is compiled from them, so no app has hand-written routing. +- Every config change is a commit in the deployment's git repo, and Argo CD applies it, so the audit log is the git history. The engine applies ClickHouse DDL and keeps its auth state and secrets itself. - The dfe-schemas manifest is applied at boot: the core ClickHouse objects, the Kafka bootstrap topics, and each source's table when that source is deployed. +- Sampling of a source's ClickHouse table or Kafka topic, where the default `smart` mode and the `anomaly` mode run logreducer for a representative view. - A hunt runner spreads scheduled hunts across their interval and coordinates through ClickHouse alone. - OIDC login, SCIM 2.0 provisioning for accounts and groups, and scoped RBAC. -- The `dfe` CLI, generated from the API's own OpenAPI spec. +- The `dfe` CLI, generated from the API's OpenAPI spec. -Records never flow through the engine. They go receiver, then Kafka or gRPC, then loader, then ClickHouse, and the engine decides what those apps are configured to do. DFE lands data first and lets you add columns later, one field at a time, with no reindex. [docs/data-evolution.md](docs/data-evolution.md) explains the three stages and the `promote` workflow. +Records never flow through the engine. They go receiver, then Kafka or gRPC, then loader, then ClickHouse, and the engine decides what those apps are configured to do. DFE lands data first and lets you add columns later, one field at a time, with no reindex. [docs/data-evolution.md](docs/data-evolution.md) explains the stages and the `promote` workflow. DFE is a product that other organisations deploy. Nothing in this repo is specific to any one deployment, and configuration is injected at deploy time. @@ -23,10 +24,10 @@ DFE is a product that other organisations deploy. Nothing in this repo is specif ```bash git clone https://github.com/hyperi-io/dfe-engine.git cd dfe-engine -uv sync # pulls dfe-schemas, which carries the schema and DDL trees +uv sync # pulls dfe-schemas, which holds the schema and DDL trees ``` -Run the API server. A clone has no `.env`, so set `DFE_ENGINE_ENV=dev`, without which the engine refuses to start on the placeholder JWT secret, and `DFE_CONFIG_DIR=./config`, without which it tries to create `/app/config`. The API answers on port 8000 and `/livez` and `/readyz` on the observability listener, port 9090: +Run the API server. A clone has no `.env`, so set `DFE_ENGINE_ENV=dev`, without which the engine won't start on the placeholder JWT secret, and `DFE_CONFIG_DIR=./config`, without which it tries to create `/app/config`. The API answers on port 8000 and `/livez` and `/readyz` on the observability listener, port 9090: ```bash env DFE_ENGINE_ENV=dev DFE_CONFIG_DIR=./config uv run dfe-engine run @@ -44,14 +45,14 @@ The entry points are `dfe-engine` (the API server), `dfe` (the CLI), `dfe-hunt-r ## Configuration -Every setting is an environment variable with the `DFE_ENGINE_` prefix, apart from `DFE_CONFIG_DIR`, which other suite apps read too. `src/dfe_engine/settings.py` is the reference. The two directory roots: +Every setting is an environment variable with the `DFE_ENGINE_` prefix, apart from `DFE_CONFIG_DIR`, which the other suite apps share. `src/dfe_engine/settings.py` is the reference. The two directory roots: | Variable | Image default | Local-dev default | What it holds | |----------|---------------|-------------------|---------------| | `DFE_CONFIG_DIR` | `/app/config` | `./config` with `make dev`, else `/app/config` | The deployment's config: a mounted volume or ConfigMap on Kubernetes, a checkout of the deployment's config repo for local dev. The engine finds `sources/`, `source-builds/`, `source-plans/`, `source-deploys/`, `fieldmaps/`, `services/`, `hunts/`, `rules/` and `alert-destinations/` under it. | | `DFE_ENGINE_SCHEMAS_DIR` | `/app/schemas` | unset | The schema tree root. Unset, the engine reads the trees from the installed `dfe-schemas` package. | -The image ships the dfe-schemas trees at `/app/schemas-seed`, copies them into `/app/schemas` on the first run of each engine version, and has an empty `/app/config`, so neither directory variable needs setting in a container. The deployment supplies the config by mount and the secrets and connection settings by env, and a container with none of those refuses to start on the placeholder JWT secret. A per-registry variable such as `DFE_ENGINE_SOURCES_DIR` wins over the subdirectory derived from `DFE_CONFIG_DIR`. The names are exact: a bare `CONFIG_DIR` is not read. ClickHouse and Kafka connection settings follow the same prefix. +The image ships the dfe-schemas trees at `/app/schemas-seed`, copies them into `/app/schemas` on the first run of each engine version, and has an empty `/app/config`, so neither directory variable needs setting in a container. The deployment supplies the config by mount and the secrets and connection settings by env, and a container with none of those won't start on the placeholder JWT secret. A per-registry variable such as `DFE_ENGINE_SOURCES_DIR` wins over the subdirectory derived from `DFE_CONFIG_DIR`. The names are exact: a bare `CONFIG_DIR` is not read. ClickHouse and Kafka connection settings follow the same prefix. ## Documentation @@ -65,7 +66,7 @@ The suite's published docs are at , this repository' ## Help and contributing -How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in a new issue. +How to work in this repo, the commands that check a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in an issue. Open bugs: [this repo](https://github.com/hyperi-io/dfe-engine/issues?q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor) | [whole DFE suite](https://github.com/search?type=issues&q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor+repo%3Ahyperi-io%2Fdfe-infra+repo%3Ahyperi-io%2Fdfe-docker+repo%3Ahyperi-io%2Fdfe-deploy+repo%3Ahyperi-io%2Fdfe-engine+repo%3Ahyperi-io%2Fdfe-loader+repo%3Ahyperi-io%2Fdfe-receiver+repo%3Ahyperi-io%2Fdfe-fetcher+repo%3Ahyperi-io%2Fdfe-archiver+repo%3Ahyperi-io%2Fdfe-transform-vrl+repo%3Ahyperi-io%2Fdfe-transform-vector+repo%3Ahyperi-io%2Fdfe-transform-elastic+repo%3Ahyperi-io%2Fdfe-ui+repo%3Ahyperi-io%2Fdfe-hyperdx+repo%3Ahyperi-io%2Fscalo-rs+repo%3Ahyperi-io%2Fscalo-py+repo%3Ahyperi-io%2Flogreducer+repo%3Ahyperi-io%2Fdfe-schemas+repo%3Ahyperi-io%2Fvector-vrl+repo%3Ahyperi-io%2Fculvert+repo%3Ahyperi-io%2Fdfe-clickhouse-rs+repo%3Ahyperi-io%2Ffactbook-rs) diff --git a/docker-compose.dev.yaml b/docker-compose.dev.yaml index fea9a56ec..bbe4470e1 100644 --- a/docker-compose.dev.yaml +++ b/docker-compose.dev.yaml @@ -1,35 +1,20 @@ -# Development services for DFE Engine API -# -# Prism mock server -- frontend devs can develop against the API contract -# without running the real backend: -# -# docker compose -f docker-compose.dev.yaml up prism -# -# Then point dfe-ui at http://localhost:4010 instead of :8000 -# -# TypeScript type generation (run from host): -# npx openapi-typescript openapi/openapi.json -o types.ts +# Local dev services: docs/local-dev.md. services: prism: - # Digest-pinned (#106) so the floating major tag cannot be repointed. image: stoplight/prism:5@sha256:3f6d29e31bfe0b99587f0f6f79c423858dd6a2ea7e1e4658273ed930a78a9acf command: mock /spec/openapi.json --host 0.0.0.0 ports: - "4010:4010" volumes: - ./openapi:/spec:ro - # Bounded so a runaway mock cannot take the dev host down (#106). deploy: resources: limits: cpus: "0.5" memory: 256M - # A local ClickHouse for `make e2e-server`: plain HTTP on 127.0.0.1:8123 with - # the password .env.e2e.example carries. One node with its own Keeper, which the - # replicated tables need. The multi-arch digest of the 26.3 LTS tag dfe-infra - # pins, so it runs natively on Apple silicon. + # For `make e2e-server`. single-node.xml adds the Keeper the replicated tables need. clickhouse: image: clickhouse/clickhouse-server:26.3.17.56@sha256:422be85ae7344058369cdd366ac0efea9daa8428b55c9cf50258e83a7d12fcb3 environment: diff --git a/docs/README.md b/docs/README.md index 354725f57..33d7c70cf 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,10 +1,14 @@ # dfe-engine documentation -The DFE engine's documentation: the architecture and data evolution, plus one directory each for the control plane, data plane and deployment, and the API route map. +The DFE engine's documentation: the architecture and data evolution, plus one directory each for the control plane, data plane, deployment and integrations, and the API route map. - [DFE architecture](architecture.md) +- [Data evolution](data-evolution.md) and its [acceptance path](data-evolution-acceptance.md) - [The DFE Control-Plane Layer](control-plane/README.md) -- [Data evolution](data-evolution.md) - [Data plane](data-plane/README.md) - [Deployment](deployment/README.md) +- [Integrations](integrations/README.md) +- [Auth at the edge](auth-envoy-topology.md) +- [Running dfe-engine locally](local-dev.md) - [API route map](reference/api-route-map.md) +- [Archive](archive/README.md) diff --git a/docs/architecture.md b/docs/architecture.md index aae765957..136f626fb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -5,182 +5,301 @@ scheduled detection, and exploration, all governed through one git-backed control plane. This page is the map - where each repo sits, the two principles that define the system, the invariants, and links into the depth docs. +The depth docs: + +| Area | Start at | Covers | +|------|----------|--------| +| Control-plane surface | [control-plane/README.md](control-plane/README.md) | governed ops, gitops commit standard, RBAC, OIDC, CLI, UI API, type-safety sync | +| Data plane | [data-plane/README.md](data-plane/README.md) | sources, schemas, field mapping, CEL, query API, hunts | +| Deployment | [deployment/README.md](deployment/README.md) | tiers + composition, backing services, KEDA, observability, supply-chain pinning, e2e acceptance | +| Integrations | [integrations/README.md](integrations/README.md) | dfe-ui guide, and where the dfe-hyperdx fork runbook lives | +| Archive | [archive/README.md](archive/README.md) | superseded research + snapshots, kept for the decision trail | + +Edge auth: [auth-envoy-topology.md](auth-envoy-topology.md). + ## Where the engine sits in the suite -dfe-engine is the control plane: a Python library + FastAPI server that turns -operator intent into governed git commits, which Argo CD reconciles into the -cluster. Everything else either feeds it (schemas, libraries) or consumes what -it writes (infra, runtime services, UIs). +dfe-engine is the control plane: a Python library and FastAPI server that turns +operator changes into git commits Argo CD syncs into the cluster, and applies the +ClickHouse DDL itself. Operators call the engine through dfe-ui and the CLI, and the engine commits to the deploy repo that Argo CD syncs. ```mermaid -flowchart TB - subgraph surface["Operator surface"] - UI[dfe-ui] - HDX[Observe
dfe-hyperdx fork] - CLI[dfe CLI] - end - subgraph control["Control plane"] - ENG[dfe-engine
config control plane + API] - SCH[dfe-schemas
DDL + meta-schema SSoT] - SPY[scalo-py
shared Python library] - end - subgraph deploy["Deployment vehicle"] - INF[dfe-infra
charts - appsets - bootstrap] - DREPO[(deploy repo)] - ARGO[Argo CD] - end - subgraph runtime["Runtime (Rust)"] - RSVC[dfe-receiver - loader - archiver
fetcher - transform-*] - SRS[scalo-rs
shared Rust library] - CH[(ClickHouse)] +flowchart LR + accTitle: L2 Services -- DFE + accDescr: Operators call dfe-engine through dfe-ui and the dfe CLI. The engine commits config to the deploy repo and applies DDL to ClickHouse. Argo CD syncs the deploy repo into the runtime services. + + subgraph clients["Clients"] + ui["dfe-ui"] + cli["dfe CLI"] end - UI --> ENG - HDX --> CH - CLI --> ENG - SCH --> ENG - SPY --> ENG - ENG -->|compiled values + DDL| DREPO - INF --> ARGO - DREPO --> ARGO - ARGO -->|helm| RSVC - SRS --> RSVC - RSVC --> CH + + engine[["dfe-engine"]] + repo[("Deploy repo")] + ch[("ClickHouse")] + argo["Argo CD"] + runtime["Runtime
services"] + + clients -->|calls| engine + engine -->|commits| repo + engine -->|DDL| ch + repo --> argo + argo -->|helm| runtime + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class repo,ch store + class clients group ``` +*Figure 1. L2 Services -- DFE.* Up: top of dfe-engine. Down: [L3 Components -- dfe-engine](#inside-the-engine). + +Argo CD also reads dfe-infra's charts. The runtime services write to ClickHouse, and dfe-hyperdx, embedded in dfe-ui, reads it. + | Repo | Role | Relationship to the engine | |------|------|---------------------------| | dfe-engine | config control plane + API server | the engine itself | | dfe-infra | charts, ApplicationSets, bootstrap | consumes engine-authored overlays via the deploy repo | -| dfe-ui | web UI | consumes the engine API only | +| dfe-ui | web UI | calls the engine API and nothing else | | dfe-hyperdx | the Observe view: our extended fork of the upstream HyperDX project, explore UI + telemetry sink | embedded in dfe-ui, reads ClickHouse | -| dfe-schemas | every table, view, role and bootstrap topic, plus the apply manifest | the only place they are defined. The engine applies the manifest at boot ([schema bootstrap](data-plane/schema-bootstrap.md)) | +| dfe-schemas | every table, view, role and bootstrap topic, plus the apply manifest | where they are defined, and nowhere else. The engine applies the manifest at boot ([schema bootstrap](data-plane/schema-bootstrap.md)) | | scalo-py / scalo-rs | shared libraries (neutral-identity OSS) | engine imports scalo-py, Rust services import scalo-rs | | dfe-receiver/loader/archiver/fetcher/transform-* | Rust runtime services | read YAML config the engine writes | | dfe-deploy | example deploy/overlay repo | the GitOps hand-off surface | -## The two principles that define DFE +An event goes from the receiver through Kafka to the loader and the archiver. -### 1. Orchestration over infra: helm through Argo, never hands on the cluster +```mermaid +flowchart LR + accTitle: L4 Detail -- Event path + accDescr: Log shippers send events to dfe-receiver, which routes them to Kafka topics. dfe-loader writes them to ClickHouse and dfe-archiver writes them to object storage. + + shippers["Log shippers"] + receiver["dfe-receiver"] + topics[("Kafka topics")] + + subgraph readers["Topic readers"] + loader["dfe-loader"] + archiver["dfe-archiver"] + end + + subgraph stores["Stores"] + ch[("ClickHouse")] + bucket[("Object storage")] + end + + shippers -->|events| receiver + receiver -->|routed| topics + topics --> readers + readers --> stores + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef async fill:#F6E6EF,stroke:#A8517F,stroke-width:1px,color:#3D1529 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class shippers ext + class topics async + class ch,bucket store + class readers,stores group +``` + +*Figure 6. L4 Detail -- Event path. The dashed box is external.* Up: [L2 Services -- DFE](#where-the-engine-sits-in-the-suite). -The engine performs every operational change as a governed git commit - -compiled Helm overlay values, ClickHouse DDL, rendered OIDC config - and Argo -CD reconciles the cluster to match. The engine never deploys backing services, -never authors Argo Applications, and never claims a commit is live (an -org-specific approval gate may sit between commit and sync). +The loader writes to ClickHouse and the archiver to object storage. With no broker, the receiver pushes to them over gRPC instead. The hunt runner, dfe-hyperdx and the query API read ClickHouse. + +Matched rows are written at full fidelity up to `hunts.max_detections_per_run` +per rule, and past it the runner writes one summary row. When a hunt names an alert +destination, the runner sends one message per fire that matched. There is no +alert grouping or cooldown. + +## The two principles that define DFE + +The first is orchestration over infra: helm through Argo, never hands on the +cluster. The engine makes every operational change as a governed git commit of +compiled Helm overlay values, and Argo CD reconciles the cluster to match. The +one thing it applies itself is ClickHouse DDL. The engine never deploys backing +services, never authors Argo Applications, and never claims a commit is live +(an org-specific approval gate may sit between commit and sync). One committed values file becomes one Argo Application and its pods. ```mermaid flowchart LR - OP[Operator via UI/CLI/API] --> ENG[dfe-engine] - ENG -->|governed commit| DREPO[(deploy repo
values/ ddl/ envoy/)] - DREPO --> APPSET[ApplicationSets
in dfe-infra] - APPSET -->|base chart + overlay| APPS[Argo Applications] - APPS -->|helm| PODS[Rust service pods] + accTitle: L4 Detail -- Instance deploy path + accDescr: dfe-engine commits an instance values file to the deploy repo. dfe-infra's ApplicationSet makes one Argo Application per values file, which renders the app's chart into service pods. + + engine[["dfe-engine"]] + values[("Instance
values file")] + appset["ApplicationSet"] + app["Argo Application"] + pods["Service pods"] + + engine -->|commit| values + values -->|matched| appset + appset --> app + app -->|helm| pods + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + class values core ``` +*Figure 2. L4 Detail -- Instance deploy path.* Up: [L2 Services -- DFE](#where-the-engine-sits-in-the-suite). + Adding a service instance is a git write of one -`values/--values.yaml` file - the git-files ApplicationSet -generator fans out one Argo Application per values file. Removing the file -removes the app. The engine declares instances. The appset deploys them. +`values/--values.yaml` file. dfe-infra's git-files +ApplicationSet (`argocd/appsets/layer2-apps.yaml`) makes one Argo Application per +values file. Removing the file removes the app. The engine declares instances. +The appset deploys them. An instance removed while its source still exists leaves the files authored into it in `values/--parked.yaml`, which the glob does not match, and its next deploy restores them. Depth: [deployment/README.md](deployment/README.md) and [control-plane/governed-ops-design.md](control-plane/governed-ops-design.md). -### 2. Close-coupled source - ingest - schema - transform (not separate ETL) - -Traditional pipelines split ingest, schema management, and transformation +The second is close coupling of source, ingest, schema and transform, where +other stacks run separate ETL. Traditional pipelines split ingest, schema management, and transformation across separate tools with separate config stores that drift apart. DFE keeps -them one artifact: a source definition in the config repo carries the routing -key, the schema binding, and the field-map bindings together, and every -runtime stage reads the same definition. +them one artifact: a source definition carries the routing rule, the schema +binding, and the field-map bindings together, and every runtime stage reads +what the engine compiles from it. One definition compiles to the receiver's rules, the loader's table map, the table and its Sigma view. ```mermaid flowchart LR - SRC[source YAML
routing + schema + field maps] --> RCV[receiver
routes by _source] - SRC --> LDR[loader
schema-aware insert] - SRC --> MAPS[standard views
sigma_* ecs_* cim_*] - RCV --> LDR --> CH[(ClickHouse)] - CH --> MAPS - MAPS --> HUNTS[hunts] & QAPI[query API] & XPLR[explore UI] + accTitle: L4 Detail -- One source definition, compiled + accDescr: A source definition goes to dfe-engine, which compiles the receiver's rules, the loader's table map, the source's table and its Sigma view from it. + + src["Source definition"] + engine[["dfe-engine"]] + + subgraph out["Compiled from it"] + direction LR + rules["Receiver
rules"] + tablemap["Loader
table map"] + table[("Source
table")] + view[("Sigma
view")] + rules ~~~ tablemap ~~~ table ~~~ view + end + + src -->|definition| engine + engine --> out + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class src core + class table,view store + class out group ``` -One definition, read by every stage: the receiver routes an event by its -`_source`, the loader inserts into the topology-correct table for that -source's schema, and the field-mapping layer projects the same columns back -out as standard vocabularies (Sigma, ECS, CIM) for detection and query. There -is no separate ETL config to fall out of sync. Depth: -[data-plane/README.md](data-plane/README.md). +*Figure 3. L4 Detail -- One source definition, compiled.* Up: [L3 Components -- dfe-engine](#inside-the-engine). + +One definition, read by every stage: the receiver routes an event by the +source's match rule, the loader writes it to that source's table, and the +`_sigma` view projects the same columns under Sigma field names for +detection (`services/source_routing.py`, `schema/schema_ddl.py`). Routing that +disagrees with the sources is reported as drift and re-synced +(`appmgmt/routing.py`), so there is no separate ETL config to fall out of step. +Depth: [data-plane/README.md](data-plane/README.md). ## The control pyramid -Four layers, each a narrower, more guarded window onto the one below. The API -is the single control plane. The CLI and UI only ever consume it. The API -never reaches past git to the cluster. +The control path has four layers, each a narrower window onto the one below. The API is the single +control plane: dfe-ui and the generated CLI commands call it, and the CLI's `local` +and `schema` groups are break-glass paths that work with no engine. The API never +reaches past git to the cluster. Both clients reach the engine through its API, and the API reaches the cluster through git. ```mermaid -flowchart TB - UI2[UI - the 80/20 high-value functions] --> CLI2[CLI - as much of the API as practical] - CLI2 --> API2[API - dfe-engine - ALL control-plane functions] - API2 --> BASE[Infrastructure + config files
helm - argo - k8s - YAML in git] +flowchart LR + accTitle: L4 Detail -- Control layers + accDescr: dfe-ui covers the common tasks and the dfe CLI every listed operation, both through the engine API. The API covers every control function and works through the charts and YAML in git. + + subgraph clients["Clients"] + ui["dfe-ui
common tasks"] + cli["dfe CLI
every operation"] + end + + api["Engine API
every control function"] + git[("Charts and
YAML in git")] + + clients -->|API calls| api + api -->|commits| git + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class api core + class git store + class clients group ``` -## Component map +*Figure 4. L4 Detail -- Control layers.* Up: [L2 Services -- DFE](#where-the-engine-sits-in-the-suite). + +## Inside the engine + +Inside the engine, the API server drives the registries and compilers, which commit through the gitops writer or apply DDL through the provisioners. ```mermaid -flowchart TB - subgraph clients["Clients"] - CUI[dfe-ui] - CCLI[CLI / CI] - GITENG[GitOps engineer
direct YAML edit] - end - subgraph engine["dfe-engine (Python: library + API server)"] - API[FastAPI server] - REG[Config registries
git-native YAML CRUD] - COMP[Compilers
Helm values - DDL - Sigma - field maps] - BRIDGE[GitOps bridge
writes deploy repo] - PROV[Provisioners
ClickHouse - Observe - orgs] - end - subgraph repos["Git repos (SSoT)"] - CONFIG[(config repo)] - DEPLOY[(deploy repo)] - INFRA[(dfe-infra)] - end - subgraph cluster["Customer cluster"] - ARGO2[Argo CD - argocd, installed or adopted] - APPSETS[ApplicationSets] - RUST[Rust services] - SUB[(Backing services
ClickHouse - Kafka - DocumentDB - Observe)] +flowchart LR + accTitle: L3 Components -- dfe-engine + accDescr: Clients call the API server. It reads and writes the config registries and runs the compilers. The gitops writer commits registry changes and compiled overlays to the deploy repo, and the provisioners apply compiled DDL to ClickHouse. + + clients["Clients"] + + subgraph engine["dfe-engine"] + api["API server"] + registries["Config registries"] + compilers["Compilers"] + writer["GitOps writer"] + provisioners["Provisioners"] end - CUI --> API - CCLI --> API - GITENG --> CONFIG - API --> REG --> CONFIG - API --> COMP --> BRIDGE --> DEPLOY - API --> PROV --> SUB - INFRA --> ARGO2 --> APPSETS - DEPLOY --> APPSETS - APPSETS --> RUST - RUST --> SUB + + repo[("Deploy repo")] + ch[("ClickHouse")] + + clients -->|calls| api + api --> registries + api --> compilers + registries --> writer + compilers --> writer + compilers --> provisioners + writer -->|commits| repo + provisioners -->|DDL| ch + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class repo,ch store + class engine group ``` -- The config repo is the engine's working data: service configs, deployment - configs, field maps, rules/hunts, OIDC providers, accounts/groups. - Engine-written AND directly human-editable. Writes are read-before-write - with git blob-SHA ETags: `If-Match` mismatch returns `409 Conflict` with a - diff rather than clobbering a human or CI edit. -- The deploy repo is the GitOps hand-off. Engine writes compiled artifacts. - Argo reads them. With gitops enabled it also holds the source definitions - SSoT (`config/sources/`) and the hunt definitions and detection rules - (`config/hunts/`, `config/rules/`), one attributed commit per mutation via - gitcrud. Standalone deployments fall back to plain directories in the config - repo. The hunt runner git-syncs those last two directories, so on Kubernetes - they are how a hunt reaches the pod that runs it. Provider-agnostic: - GitHub / GitLab external git, or the in-cluster Forgejo fallback (see - [deployment/README.md](deployment/README.md)). +*Figure 5. L3 Components -- dfe-engine.* Up: [L2 Services -- DFE](#where-the-engine-sits-in-the-suite). + +| Part | Code | What it does | +|---|---|---| +| API server | `api/` | FastAPI routers, the four auth paths and RBAC | +| Config registries | `source/registry.py`, `hunts/`, `fieldmap/registry.py`, `gitcrud/registry.py` | Read and write sources, hunts, rules, field maps, governance and app overlays | +| Compilers | `services/source_routing.py`, `appmgmt/routing.py`, `schema/schema_ddl.py`, `sigma/` | Receiver rules and loader table maps, table DDL and Sigma views, Sigma rules to SQL | +| GitOps writer | `gitcrud/`, `gitops/` | One attributed commit per change, with an `If-Match` revision check | +| Provisioners | `schema/phase.py`, `kafka/topics.py`, `hyperdx/`, `governance/ch/` | The dfe-schemas manifest and source tables, Kafka topics, dfe-hyperdx sources, ClickHouse roles and row policies | + +- The config directory (`DFE_CONFIG_DIR`) is the deployment's own config: a + mounted volume, or a checkout of its config repo. The engine keeps its + registries and its auth store (`auth/`) there. +- The deploy repo is the GitOps hand-off. The engine commits app overlays to + `values/`, backing-service overlays to `infra/` and governance to + `governance/` (`gitcrud/registry.py`), and Argo CD reads them. With gitops + enabled it also holds the source definitions (`config/sources/`) and the hunt + definitions and detection rules (`config/hunts/`, `config/rules/`), one + attributed commit per change. A write that names the revision it read + (`If-Match`) answers `409 Conflict` with the current document when that + revision is stale, rather than overwrite a human or CI edit. A write with no + `If-Match` is not checked, so a client that needs the guard sends it. Standalone deployments fall back to + plain directories under the config directory. The hunt runner git-syncs the + hunt and rule directories, so on Kubernetes they are how a hunt reaches the pod + that runs it. Provider-agnostic: GitHub / GitLab external git, or the + in-cluster Forgejo fallback (see [deployment/README.md](deployment/README.md)). - dfe-infra is the pinned deployment vehicle. Never written by the engine. -In small deployments the config repo and deploy repo can be one repo with -different subtrees (`config/` and `values/` + `ddl/`). - ## First principles (invariants) 1. Generic product, not an internal tool. Ships into a customer cluster @@ -193,54 +312,34 @@ different subtrees (`config/` and `values/` + `ddl/`). YAML 1.1, where `off`/`yes`/`no` become booleans). 3. Engine writes config, never installs infrastructure. The hard engine<->infra boundary: dfe-infra deploys backing services and owns - ApplicationSets. The engine writes config + DDL + OIDC values into the - deploy repo and turns dials through overlays. If the engine is about to - install ClickHouse or author an Argo Application, the work belongs in - dfe-infra. -4. Assume only a bare cluster. Layer 0 detect-or-install brings + ApplicationSets. The engine commits config and overlays to the deploy + repo, applies DDL to ClickHouse itself, and turns dials through overlays. + If the engine is about to install ClickHouse or author an Argo Application, + the work belongs in dfe-infra. +4. Assume a bare cluster. Layer 0 detect-or-install brings everything else: a default StorageClass, cert-manager, External Secrets, and Argo CD, installed into `argocd` or, where one already runs, adopted. 5. ClickHouse is the only operational store. All non-gitops state - events, results, watermarks, leases - lives in CH via - `effective_data_database`. The DocumentDB store is Observe's own, never + `effective_data_database`. The DocumentDB store is dfe-hyperdx's, never DFE state. 6. A commit is not a deployment. The engine reports what it committed. Argo (and any org approval gate) decides when it is live. 7. Fully operable from the repos alone. Engine and UI down = running system unaffected, manageable by editing the repos directly. -## Layered deployment model (summary) - -Layer 0 platform baseline (Argo CD + detect-or-install operators) -> -Layer 1 backing services (ClickHouse, optional Kafka, DocumentDB, Observe) -> -Layer 2 apps + config (digest-pinned thin charts built from each app's -deployment contract + engine-authored overlays). Four tiers select defaults: -**dfe-docker** (Compose), **slim** (gRPC, no Kafka), **single** (+ -single-broker Kafka) and **scale** (everything clustered). A fifth, **mesh** -(scale's replicas and ClickHouse cluster over gRPC, no broker), is in the -charts, but dfe-infra's bootstrap refuses to select it. Observe ships in EVERY tier - it is the explore UI over all DFE -data and the self-monitoring sink, not an add-on. Full detail including the -per-tier composition table, deploy-repo providers, backing-service modes, and -the network model: [deployment/README.md](deployment/README.md). - -## Runtime data flow - -```mermaid -flowchart LR - AGENTS[Log agents
Filebeat - syslog - API] -->|HTTP/gRPC| RCV[receiver] - RCV -->|route by _source| KFK[Kafka topics
optional - gRPC direct otherwise] - KFK --> LDR[loader] - RCV -. raw .-> ARC[archiver -> S3] - LDR -->|schema-aware insert| CH[(ClickHouse)] - CH --> VIEWS[standard views
sigma_ - ecs_ - cim_] - VIEWS --> HUNT[hunt runner] - VIEWS --> DASH[Observe] - VIEWS --> QAPI[query API] -``` - -All matched rows are written at full fidelity. When a hunt names an alert -destination, the runner sends one message per fire that matched. There is no -alert grouping or cooldown. +The deployment is layered. Layer 0 platform baseline (Argo CD + detect-or-install operators) -> +Layer 1 backing services (ClickHouse, optional Kafka, DocumentDB) -> +Layer 2 apps, dfe-hyperdx among them, and config (digest-pinned thin charts +built from each app's deployment contract + engine-authored overlays). Four +tiers select defaults: **dfe-docker** (Compose), **slim** (gRPC, no Kafka), +**single** (+ single-broker Kafka) and **scale** (everything clustered). A +fifth, **mesh** (scale's replicas and ClickHouse cluster over gRPC, no broker), +is in the charts, but dfe-infra's bootstrap refuses to select it. dfe-hyperdx +ships in EVERY tier - it is the explore UI over all DFE data and the +self-monitoring sink, not an add-on. Full detail including the per-tier +composition table, deploy-repo providers, backing-service modes, and the +network model: [deployment/README.md](deployment/README.md). ## Technology decisions @@ -249,26 +348,14 @@ alert grouping or cooldown. | Config storage | YAML + git (no DB) | human-readable, versioned, Rust reads directly | | API host | dfe-engine FastAPI | one artifact | | Auth | local JWT by default, plus OIDC, API key and disabled | OIDC through the engine's own relying party (`/api/v1/auth/oidc/{provider}/login`) | -| Conflict handling | git blob-SHA ETag + 409 | git-native, standard HTTP semantics | +| Conflict handling | optional `If-Match` revision + 409 | git-native, standard HTTP semantics | | Deployment | Argo CD + Helm, thin chart + dfe-extras + overlay, multi-source | engine turns dials, appsets author apps | -| Platform baseline | Layer 0 detect-or-install | assume only a bare cluster | +| Platform baseline | Layer 0 detect-or-install | assume a bare cluster | | Deploy repo | provider-agnostic (GitHub/GitLab/Forgejo) | no hard git-server dependency | | Backing services | DFE-owned, mode-driven (single/cluster/external) | never piggyback shared infra | | Secrets | External Secrets Operator for deployment secrets, `DFE_ENGINE_SECRETS_PROVIDER` for secrets the engine mints | ESO: customer Vault, or in-cluster generators on bare. Engine: file by default, OpenBao/Vault or AWS Secrets Manager | -| Observability | OTel -> ClickHouse -> Observe | single storage backend | +| Observability | OTel -> ClickHouse -> dfe-hyperdx | single storage backend | | Ingress / OIDC | Envoy Gateway (k8s) / oauth2-proxy (docker) | native OIDC at the edge | | Expression language | CEL (Python + Rust), DFE subset | one standard, tiered for performance | | Field mapping | YAML, two-tier (default + source) | human-editable, git-tracked, UI-editable | | Stage transport | one per source, `bus` or `direct`, default from the profile | a flow is never mixed. The bus provider (Kafka) is a deployment fact, so a second provider joins without a model change | - -## The docs tree - -| Area | Start at | Covers | -|------|----------|--------| -| Control-plane surface | [control-plane/README.md](control-plane/README.md) | governed ops, gitops commit standard, RBAC, OIDC, CLI, UI API, type-safety sync | -| Data plane | [data-plane/README.md](data-plane/README.md) | sources, schemas, field mapping, CEL, query API, hunts | -| Deployment | [deployment/README.md](deployment/README.md) | tiers + composition, backing services, KEDA, observability, supply-chain pinning, e2e acceptance | -| Integrations | [integrations/README.md](integrations/README.md) | dfe-ui guide, and where the dfe-hyperdx fork runbook lives | -| Archive | [archive/README.md](archive/README.md) | superseded research + snapshots, kept for the decision trail | - -Edge auth topology notes live in [auth-envoy-topology.md](auth-envoy-topology.md). diff --git a/docs/archive/dfe-infra-snapshot.md b/docs/archive/dfe-infra-snapshot.md index 743a779d1..24865b82e 100644 --- a/docs/archive/dfe-infra-snapshot.md +++ b/docs/archive/dfe-infra-snapshot.md @@ -1,10 +1,10 @@ # DFE Infrastructure (dfe-infra) - research summary (ARCHIVED) > **ARCHIVED SNAPSHOT.** Last synced from `hyperi-io/dfe-infra` on -> 2026-03-31; component versions and structures have moved on. Kept for +> 2026-03-31, and component versions and structures have moved on. Kept for > the 2.1 -> 2.2 decision trail. Current material lives in > [deployment/README.md](../deployment/README.md) and the dfe-infra repo's -> own docs. The KEDA scaling patterns that used to live here moved to +> docs. KEDA scaling is in > [deployment/keda-scaling.md](../deployment/keda-scaling.md). ## What dfe-infra Is @@ -40,18 +40,16 @@ Identical deployment on ANY K8s. Cloud-agnostic. | FerretDB | 1.24.0 | MongoDB wire protocol over PG17 | | OTel Collector | 0.114.0 | Telemetry (DaemonSet + Gateway) | | HyperDX | 1.7.0 | Observability UI | -| KEDA | 2.20.1 | Event-driven autoscaling (CPU default; scalo ScalingPressure opt-in via `dfe-keda-shim`) | +| KEDA | 2.20.1 | Event-driven autoscaling (CPU default, scalo ScalingPressure opt-in via `dfe-keda-shim`) | | DFE services | 2.2.0 | engine, UI, Rust pipeline apps | ### Deployment Flow -```text -Terraform (Layer 1) → bootstrap.sh → ArgoCD takes over - Wave 2: operators (Envoy, KEDA, metrics-server) - Wave 3: DNS/TLS addons - Wave 4: data platform (Kafka, ClickHouse, PG, OTel) - Wave 5: DFE services (engine, UI, receivers, loaders, transforms) -``` +1. Terraform (Layer 1), then `bootstrap.sh`, then ArgoCD takes over. +2. Wave 2: operators (Envoy, KEDA, metrics-server). +3. Wave 3: DNS/TLS addons. +4. Wave 4: data platform (Kafka, ClickHouse, PG, OTel). +5. Wave 5: DFE services (engine, UI, receivers, loaders, transforms). ## DFE Services Deployed @@ -60,7 +58,7 @@ Terraform (Layer 1) → bootstrap.sh → ArgoCD takes over | dfe-engine | Python | Control plane, API, config SSOT, RBAC | | dfe-ui | TypeScript | Next.js web UI + HyperDX integration | | dfe-receiver | Rust | High-throughput ingest | -| dfe-loader | Rust | Kafka → ClickHouse writer | +| dfe-loader | Rust | Kafka -> ClickHouse writer | | dfe-archiver | Rust | Long-term S3/MinIO storage | | dfe-fetcher | Rust | API data fetcher | | dfe-transform-wasm | Rust | WASM transformations | @@ -69,27 +67,19 @@ Terraform (Layer 1) → bootstrap.sh → ArgoCD takes over | dfe-transform-elastic | Rust | Elastic-compatible transforms | | dfe-transform-splack | Rust | Splunk-compatible transforms | -**Data flow:** receiver/fetcher → optional transform-* → loader/archiver → ClickHouse +**Data flow:** receiver/fetcher -> optional transform-* -> loader/archiver -> ClickHouse ## Auth Architecture -```text -HTTPS → Envoy Gateway OIDC SecurityPolicy - ├─ Authorization code flow - ├─ Session cookie - └─ Forward headers: X-Oidc-Subject, X-Oidc-Groups - ↓ - dfe-engine - ├─ OIDC header auth (production, Envoy-fronted) - ├─ JWT Bearer auth (API clients, standalone) - └─ LocalAuthProvider (dev/test fallback, bcrypt) - ↓ - RBAC: group_role_mapping - └─ OIDC groups → DFE roles (admin/operator/viewer) - ↓ - ArgoCD RBAC (generated by dfe-engine) - └─ generate_rbac_csv() → Helm values → ArgoCD ConfigMap -``` +1. HTTPS reaches the Envoy Gateway OIDC SecurityPolicy: authorization code + flow, session cookie, and forwarded `X-Oidc-Subject` and `X-Oidc-Groups`. +2. dfe-engine authenticates by OIDC header (production, Envoy-fronted), JWT + Bearer (API clients, standalone) or LocalAuthProvider (dev/test fallback, + bcrypt). +3. RBAC `group_role_mapping` maps OIDC groups to DFE roles (admin, operator, + viewer). +4. dfe-engine generates the ArgoCD RBAC: `generate_rbac_csv()`, then Helm + values, then the ArgoCD ConfigMap. **Auth modes:** @@ -99,7 +89,7 @@ HTTPS → Envoy Gateway OIDC SecurityPolicy ## Observability -**Stack:** OTel → ClickHouse → HyperDX (replaces Prometheus + Grafana) +**Stack:** OTel -> ClickHouse -> HyperDX (replaces Prometheus + Grafana) Two-tier OTel Collector: @@ -120,15 +110,15 @@ KEDA scaling patterns moved to ## Helm Values Inheritance -```text -chart/values.yaml → chart defaults -argocd/values/common.yaml → global overrides (all clouds) -argocd/values/{cloud}.yaml → cloud-specific overrides -``` +| File | Holds | +|---|---| +| `chart/values.yaml` | chart defaults | +| `argocd/values/common.yaml` | global overrides (all clouds) | +| `argocd/values/{cloud}.yaml` | cloud-specific overrides | -Cluster annotations bridge Terraform outputs → ArgoCD ApplicationSet templates. +Cluster annotations bridge Terraform outputs -> ArgoCD ApplicationSet templates. -## Relationship: dfe-infra ↔ dfe-engine +## Relationship: dfe-infra <-> dfe-engine | Concern | dfe-infra | dfe-engine | |---------|-----------|-----------| @@ -141,7 +131,7 @@ Cluster annotations bridge Terraform outputs → ArgoCD ApplicationSet templates | Scaling | KEDA ScaledObjects (CPU default) | `dfe-keda-shim` (opt-in pressure/backlog metrics, fail-safe) | **dfe-engine is deployed BY dfe-infra, not part of it.** dfe-engine manages -the DFE platform config and API; dfe-infra manages the infrastructure that +the DFE platform config and API, and dfe-infra manages the infrastructure that hosts it. ## Key Decisions (2.2 vs 2.1) @@ -153,8 +143,8 @@ hosts it. | Auth | Cognito + oauth2-proxy + Dex | Envoy OIDC SecurityPolicy | | Cache | Redis | Valkey | | Secrets | AWS SM | OpenBao / cloud SM + ESO | -| Observability | Prometheus + Grafana | OTel → ClickHouse → HyperDX | -| KEDA metrics | Prometheus scrape | CPU default; fail-safe `dfe-keda-shim` (opt-in pressure/backlog via OTel CH) | +| Observability | Prometheus + Grafana | OTel -> ClickHouse -> HyperDX | +| KEDA metrics | Prometheus scrape | CPU default, fail-safe `dfe-keda-shim` (opt-in pressure/backlog via OTel CH) | | Kafka | AWS MSK | Strimzi (operator, multi-cloud) | ## dfe-infra Docs Corpus diff --git a/docs/archive/monorepo-migration.md b/docs/archive/monorepo-migration.md index b9fdb3827..74b6991ba 100644 --- a/docs/archive/monorepo-migration.md +++ b/docs/archive/monorepo-migration.md @@ -24,7 +24,7 @@ This means: - **Monorepo (future):** All plugins are registered from the single `pyproject.toml`. Built-in plugins already live in `dfe_engine.services.plugins_builtin`. -No code changes are needed for the migration — only `pyproject.toml` entry points. +No code changes are needed for the migration -- only `pyproject.toml` entry points. ### Entry Points (pyproject.toml) @@ -61,28 +61,28 @@ The plugin function must return a `ServicePlugin` instance. ## Service Type Architecture -### Type 1 — Vanilla Services +### Type 1 -- Vanilla Services Receiver, loader, archiver. Single flat config per deployment instance. Standard Helm/ArgoCD env cascade. -### Type 2 — Multi-Source Services +### Type 2 -- Multi-Source Services Transform-wasm, fetcher. Vanilla base config plus CRUD-managed source deployments. Each source has: -- **ENV overrides** — per-source environment variables -- **Associated files** — CSV, MMDB, JSON, WASM modules (via `SourceFileConfig`) -- **Enabled state** — can be toggled without deletion +- **ENV overrides** -- per-source environment variables +- **Associated files** -- CSV, MMDB, JSON, WASM modules (via `SourceFileConfig`) +- **Enabled state** -- can be toggled without deletion Sources are managed via the engine API (called by the control plane REST API). -### Type 2+ — Vector Transform +### Type 2+ -- Vector Transform Transform-vector is Type 2 plus an instance-specific config tree: - Each source has a vector.dev YAML config file -- Sources can reference a parent (one YAML → multiple child transforms) +- Sources can reference a parent (one YAML -> multiple child transforms) - The tree structure is specific to vector transforms (not reused elsewhere) ## Config Directory Layout @@ -109,17 +109,17 @@ May or may not be git-managed. ```text dfe/ -├── engine/ # Python — dfe-engine +├── engine/ # Python -- dfe-engine │ ├── src/dfe_engine/ │ ├── tests/ │ └── pyproject.toml ├── services/ -│ ├── receiver/ # Rust — dfe-receiver -│ ├── loader/ # Rust — dfe-loader -│ ├── archiver/ # Rust — dfe-archiver -│ ├── transform-vector/# Rust — dfe-transform-vector -│ ├── transform-wasm/ # Rust — dfe-transform-wasm -│ └── fetcher/ # Rust — dfe-fetcher +│ ├── receiver/ # Rust -- dfe-receiver +│ ├── loader/ # Rust -- dfe-loader +│ ├── archiver/ # Rust -- dfe-archiver +│ ├── transform-vector/# Rust -- dfe-transform-vector +│ ├── transform-wasm/ # Rust -- dfe-transform-wasm +│ └── fetcher/ # Rust -- dfe-fetcher ├── helm/ │ ├── dfe-receiver/ │ ├── dfe-loader/ diff --git a/docs/auth-envoy-topology.md b/docs/auth-envoy-topology.md index 0186a4b4b..b7048ae66 100644 --- a/docs/auth-envoy-topology.md +++ b/docs/auth-envoy-topology.md @@ -12,38 +12,64 @@ Copyright: HyperI / DFE contributors The engine is the only issuer of the token the apps trust: an ES384 JWT, published through the engine's JWKS and carried in the `dfe_token` cookie or an `Authorization: Bearer` header. The engine and dfe-hyperdx verify that token themselves. The edge proxy routes requests, and only on Kubernetes can it also check a token, an opt-in for an external IdP's. -## One issuer, verified by each app +The edge proxy routes each request by path, and the engine and dfe-hyperdx check the token against the engine's JWKS. ```mermaid flowchart LR - B([browser]) --> P[edge proxy
Envoy] - P -->|/api/v1/*| EN[dfe-engine
issuer + verifier] - P -->|/| UI[dfe-ui] - P -->|own host or listener| HX[dfe-hyperdx] - EN -->|/.well-known/jwks.json| J[(JWKS)] - HX -.->|verifies dfe_token| J + accTitle: L4 Detail -- Token check at the edge + accDescr: The browser calls the edge proxy, which routes API paths to dfe-engine, the rest to dfe-ui, and the embed to dfe-hyperdx. The engine publishes its JWKS, and dfe-hyperdx verifies the dfe_token cookie against it. + + browser["Browser"] + proxy["Edge proxy"] + + subgraph apps["Apps"] + engine["dfe-engine"] + ui["dfe-ui"] + hyperdx["dfe-hyperdx"] + end + + jwks[("JWKS")] + + browser -->|requests| proxy + proxy -->|api paths| engine + proxy -->|other paths| ui + proxy -->|embed| hyperdx + engine -->|publishes| jwks + hyperdx -.->|verifies| jwks + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class browser ext + class engine core + class jwks store + class apps group ``` +*Figure 1. L4 Detail -- Token check at the edge. The dotted arrow is a read.* Up: [DFE architecture](architecture.md). + - Local login: `POST /api/v1/auth/login`, and the engine signs the JWT (`src/dfe_engine/api/v1/auth.py`). - External OIDC: the engine is the relying party. `/api/v1/auth/oidc/{provider}/login` redirects to the IdP, and the callback validates the IdP's token and mints an engine token (`src/dfe_engine/api/v1/oidc_login.py`). The IdP's token stays on the engine-to-IdP leg. -- dfe-hyperdx authenticates from the `dfe_token` cookie dfe-ui shares across the domain, so the fork is its own enforcement point. +- dfe-hyperdx authenticates from the `dfe_token` cookie dfe-ui shares across the domain, so the fork checks the token itself. - The engine believes proxy-set `X-Oidc-*` identity headers only with `DFE_ENGINE_AUTH_TRUST_PROXY_AUTH_HEADERS=true` (default off, `settings.py` `auth.trust_proxy_auth_headers`). Turn it on only where a proxy strips those headers from the client and the engine is reachable through that proxy alone. +## Signing key and claims + The engine signs with ES384 (ECDSA P-384 with SHA-384), `api.jwt_algorithm`. Ed25519 is not used: at 128-bit security it is below the CNSA 2.0 target. The `kid` is the RFC 7638 JWK thumbprint and the algorithm is a setting, so a post-quantum signature such as ML-DSA can replace ES384 once JOSE and Envoy can verify it (`src/dfe_engine/auth/jwt_authority.py`). The engine mints the key on first boot and stores it create-only, as a PKCS#8 PEM, in the scalo.secrets backend at `api.jwt_key_path`. The engine does not encrypt it, so at-rest protection is the backend's. -## What the token carries - -Every token carries `sub`. A login token adds `email` and `groups`, and the local login also carries `roles`, `org_id` and `org_ids`. The engine re-resolves roles from the group files on every request, so what a token asserts about them never decides an engine answer. +Every token has `sub`. A login token adds `email` and `groups`, and the local login adds `roles`, `org_id` and `org_ids`. The engine re-resolves roles from the group files on every request, so what a token asserts about them never decides an engine answer. One claim exists for a peer rather than for the engine: | Claim | Value | Read by | |---|---|---| -| `role` | `admin` for `admin` or `infra_admin` at system scope and for the engine's own `svc:dfe-engine` token, `member` for an account whose roles allow `query:write`, `viewer` for any other | dfe-hyperdx | +| `role` | `admin` for `admin` or `infra_admin` at system scope and for the engine's `svc:dfe-engine` token, `member` for an account with roles that allow `query:write`, `viewer` for any other | dfe-hyperdx | -The fork reads the same value from `GET /api/v1/auth/me` (`hyperdx_role`), resolved from the account's groups at that request, and falls back to the token's claim only when that answer carries none. `admin` (or `owner`, which the engine never mints) may change what the whole team sees, `member` may save searches and dashboards, and `viewer` is refused every write (`src/dfe_engine/auth/hyperdx_role.py`). +The fork reads the same value from `GET /api/v1/auth/me` (`hyperdx_role`), resolved from the account's groups at that request, and falls back to the token's claim when that answer has none. `admin` (or `owner`, which the engine never mints) may change what the whole team sees, `member` may save searches and dashboards, and `viewer` is refused every write (`src/dfe_engine/auth/hyperdx_role.py`). ## Kubernetes: Envoy Gateway @@ -51,24 +77,24 @@ dfe-infra's `helm/edge/gateway` chart configures Envoy Gateway. | Concern | What the chart does | |---|---| -| Routing | dfe-ui and the engine share one hostname, the engine on its `/api/v1` path families. dfe-hyperdx has its own hostname. | -| Infra routes | Argo CD and the other infra-class routes carry an OIDC `SecurityPolicy` that requires one of the admin groups. Kafbat is the exception (`edgePolicy: false`, `ownLogin: true`): it runs its own OIDC login. | -| Product routes | With `oidc.enabled: false`, the default, dfe-ui, the engine and dfe-hyperdx carry no edge OIDC policy and the apps check the engine token. Turned on, the engine route takes a `SecurityPolicy`, because the default `oidc.targetRoutes` names `dfe-engine`. dfe-hyperdx is left out on purpose: it is framed inside dfe-ui, where an interactive redirect cannot complete. | +| Routing | dfe-ui and the engine share one hostname, the engine on its `/api/v1` path families. dfe-hyperdx has a separate hostname. | +| Infra routes | Argo CD and the other infra-class routes take an OIDC `SecurityPolicy` that needs one of the admin groups. Kafbat is the exception (`edgePolicy: false`, `ownLogin: true`): it runs an OIDC login itself. | +| Product routes | With `oidc.enabled: false`, the default, dfe-ui, the engine and dfe-hyperdx take no edge OIDC policy and the apps check the engine token. Turned on, the engine route takes a `SecurityPolicy`, because the default `oidc.targetRoutes` names `dfe-engine`. dfe-hyperdx is left out on purpose: it is framed inside dfe-ui, where an interactive redirect cannot complete. | | Edge JWT (opt-in) | `jwtAuthn.enabled` (default `false`) adds Envoy's `jwt_authn` for an external OIDC issuer's Bearer token. It strips client `X-Oidc-*` headers first, then maps the token's claims into them. | | TLS | One `ClientTrafficPolicy`: TLS 1.2 minimum by default (1.3 for a high-security overlay), AES-256-GCM, P-384 ECDHE, and the hybrid ML-KEM group offered. | | Reachability | dfe-infra's `network-policies` chart (`templates/dfe-namespace.yaml`) admits ingress to the DFE namespaces only from the gateway namespace, the other DFE namespaces, KEDA on the shim port and the OTel collector. | ## Docker: one static Envoy config -dfe-docker's `config/proxy/envoy.yaml` runs the router filter only, with no token check: +dfe-docker's `config/proxy/envoy.yaml` runs the router filter and no token check: - One listener sends `/api/v1/` to `dfe-engine:8000` and everything else to `dfe-ui:3000`. - A second listener serves the dfe-hyperdx embed from `dfe-hyperdx:8090`. - The auth profile adds three oauth2-proxy gates (`compose/oauth2-proxy.yml`). -The engine API is also published on the host's loopback, port `DFE_ENGINE_PORT` (default 8003), for local tools. +For local tools, the engine API is published on the host's loopback, port `DFE_ENGINE_PORT` (default 8003). -## See also +Related: - The embedded HyperDX fork: the dfe-hyperdx repo, `docs/fork/`. - RBAC and sign-in: [control-plane/rbac-sign-in.md](control-plane/rbac-sign-in.md). diff --git a/docs/control-plane/README.md b/docs/control-plane/README.md index be45c0625..31580eea2 100644 --- a/docs/control-plane/README.md +++ b/docs/control-plane/README.md @@ -1,148 +1,149 @@ -# The DFE Control-Plane Layer +# The DFE control-plane layer -The **control-plane layer** is the small set of repos that own the DFE control -plane - configuration, RBAC, schema, the operator UI, and the deploy target. -When a doc says "the control-plane layer" it means exactly these repos. Anything -that interacts directly, or one level removed, with the dfe-engine API or what -it does is reasoning about this layer. +The control-plane layer is the small set of repos that own the DFE control plane: configuration, RBAC, schema, the operator UI and the deploy target. A doc that says "the control-plane layer" means these repos. The data-plane repos (receiver, loader, transforms, fetcher, archiver, hunt runner) are services the control plane configures, outside the layer. -## The layer +The API is the dfe-engine API, never "the dfe-api". The daemon is `dfe-engine` (`dfe-engine run`), the human CLI is `dfe`, and the engine's API settings read as `DFE_ENGINE_API_*`. + +dfe-ui calls the engine, which reads dfe-schemas and commits to the deploy repo that Argo CD syncs with dfe-infra's ApplicationSets. ```mermaid -flowchart TB - subgraph CP["Control-plane layer"] - engine["dfe-engine
config control plane + sole authz PDP
the FastAPI API + the dfe CLI"] - schemas["dfe-schemas
schema SSoT (event header, DDL, profiles)"] - ui["dfe-ui
operator console (window over the API)"] - infra["dfe-infra
GitOps SSoT + deploy vehicle (Argo CD)"] - deploy[("dfe-deploy
per-deployment RW gitops repo
we own only the INITIAL state")] - end - - ui -->|HTTP, engine-minted JWT| engine - engine -->|reads| schemas - engine -->|commits config| deploy - infra -->|deploys + syncs| deploy - - classDef cp fill:#cfe2ff,stroke:#084298; - classDef ext fill:#e2e3e5,stroke:#41464b; - class engine,schemas,ui,infra cp; - class deploy ext; +flowchart LR + accTitle: L4 Detail -- Control-plane repos + accDescr: dfe-ui calls the engine API and dfe-schemas ships the schema trees the engine reads. The engine commits config to the deploy repo, which Argo CD syncs from dfe-infra's ApplicationSets. + + ui["dfe-ui"] + schemas["dfe-schemas"] + + engine["dfe-engine"] + + deploy[("Deploy repo
dfe-deploy")] + + infra["dfe-infra
Argo CD"] + + ui -->|API calls| engine + schemas -->|schema trees| engine + engine -->|commits| deploy + deploy --> infra + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + class engine core + class deploy store ``` +*Figure 1. L4 Detail -- Control-plane repos.* Up: [DFE architecture](../architecture.md). + | Repo | Role in the layer | |---|---| -| **dfe-engine** | The config control plane + the sole authorisation PDP. Hosts the FastAPI API and the generated `dfe` CLI. Everything else in the layer keys off its API. | -| **dfe-schemas** | The schema SSoT (common event header, hunt-results DDL, profiles). The engine reads it (the version-pinned `dfe-schemas` wheel) and auto-creates from it. | -| **dfe-ui** | The operator console. A window over the engine API that holds no authority of its own. It carries the engine-minted JWT and reads its expiry, and the engine verifies it. | -| **dfe-infra** | The GitOps SSoT and deploy vehicle (Argo CD). Deploys the suite. Never a config authority. | -| **dfe-deploy** | The per-deployment read-write gitops repo the running engine commits to. We own only its **initial** state. After handover the deployment owns it. | +| dfe-engine | The config control plane and the sole authorisation decision point. Hosts the FastAPI API and the generated `dfe` CLI. Everything else in the layer keys off its API. | +| dfe-schemas | The schema source of truth: the common event header, the hunt-results DDL, the profiles. The engine reads the version-pinned `dfe-schemas` wheel and creates the ClickHouse objects from it. | +| dfe-ui | The operator console, a window over the engine API that decides nothing. It holds the engine-minted JWT and reads its expiry, and the engine verifies it. | +| dfe-infra | The charts, ApplicationSets and bootstrap that deploy the suite through Argo CD. Never a config authority. | +| dfe-deploy | The per-deployment read-write gitops repo the running engine commits to. dfe-deploy ships its starting content, and each deployment owns its copy. | -## The controlling SSoTs +## Each source of truth has one writer -The layer is organised around a handful of **single sources of truth**. The SSoTs -are how everything works: each holds one kind of authoritative state, every other -component reads from it (or from a generated artefact of it), and each has exactly -ONE write path - so a change is auditable and consumers can never diverge. +Each holds one kind of authoritative state, every other component reads it or an artefact generated from it, and each has exactly one write path, so a change is auditable and consumers cannot diverge. -| SSoT | Lives in | Consumed by | Only written by | +| Source of truth | Lives in | Consumed by | Written by | |---|---|---|---| -| **API surface** - `openapi.json` | dfe-engine `openapi/` | dfe CLI, dfe-ui types, contract mock, reference docs | the FastAPI routes via `scripts/generate_openapi.py` (never hand-edited) | -| **Schema** - event header, hunt-results DDL, profiles | dfe-schemas (the `dfe-schemas` wheel, pinned in `uv.lock`) | the engine (auto-creates the CH objects at startup), the CLI, dfe-ui | dfe-schemas commits | -| **Deployment definition** - Helm charts + values | each app's thin chart, built from its deployment contract, plus dfe-infra values + the deploy-repo overlays | Argo CD, the cluster | the thin charts by each app's release, the values by dfe-infra, the overlays by the engine, one `values/--values.yaml` per app instance, each write a gitops commit. Where no chart runs (Compose), `src/dfe_engine/appmgmt/appconfig.py` merges an overlay's `config:` over the deployment's base config into the app's config file | -| **Deployed state** - the gitops YAML | dfe-deploy (per deployment) | Argo CD (syncs to the cluster), the engine (reads current state) | the engine API - every mutation is a git commit (governed ops), and `dfe local` break-glass when the daemon is down | -| **AuthZ** - RBAC roles + scopes | dfe-engine `auth/` (YAML) | the API (`require_action`), the CLI | the engine (governed writes) | +| API surface, `openapi.json` | dfe-engine `openapi/` | dfe-ui types, the Prism contract mock, the route map | `scripts/generate_openapi.py` from the FastAPI routes, never by hand | +| Schema: event header, hunt-results DDL, profiles | dfe-schemas, the `dfe-schemas` wheel pinned in `uv.lock` | the engine, which creates the ClickHouse objects at startup, and `dfe schema` | dfe-schemas commits | +| Deployment definition: Helm charts and values | each app's thin chart, built from its deployment contract, plus dfe-infra values and the deploy-repo overlays | Argo CD | each app's release for its chart, dfe-infra for its values, the engine for the overlays (one `values/--values.yaml` per app instance, each write a gitops commit). Where no chart runs (Compose), `src/dfe_engine/appmgmt/appconfig.py` merges an overlay's `config:` over the base config into the app's config file | +| Deployed state, the gitops YAML | dfe-deploy, one per deployment | Argo CD, and the engine for current state | the engine API, every mutation a git commit, and `dfe local` break-glass when the daemon is down | +| AuthZ: RBAC roles and scopes | roles in `rbac/roles.yaml` under `DFE_CONFIG_DIR`, seeded from `src/dfe_engine/auth/resources/roles.yaml`. Scopes in `src/dfe_engine/auth/rbac_scopes/scope_constants.py` | the API (`require_action`), the CLI | the engine's role API for roles, a code change for scopes | + +The engine reads dfe-schemas and the RBAC roles, and Argo CD syncs the deploy repo with the charts and dfe-infra values. ```mermaid flowchart LR - subgraph SSoT["The SSoTs"] - api[("openapi.json")] - schema[("dfe-schemas")] - helm[("Helm charts + values")] - state[("gitops state")] - rbac[("RBAC roles + scopes")] - end - - routes["API routes"] -->|generate| api - api --> consumers["dfe CLI / dfe-ui / docs"] - schema --> engine["dfe-engine
(auto-creates CH objects)"] - engine -->|compiles + commits| helm - engine -->|every mutation = a commit| state - helm --> argo["Argo CD -> cluster"] - state --> argo - rbac --> engine - - classDef ssot fill:#fff3cd,stroke:#664d03,stroke-width:2px; - class api,schema,helm,state,rbac ssot; -``` + accTitle: L4 Detail -- Control-plane sources of truth + accDescr: The engine reads the dfe-schemas trees and the RBAC roles, and commits overlays and config to the deploy repo. Argo CD syncs the deploy repo together with the charts and dfe-infra values. -To change any of them you go through its one writer: edit the API routes and -regenerate the spec, commit to dfe-schemas, turn the engine's dials (which writes -Helm values + gitops state as commits), or edit the RBAC YAML. Never hand-edit a -generated artefact (openapi.json, compiled Helm values) - change the source and -let it regenerate. The next section walks the `openapi.json` case in full as the -worked example. + schemas[("dfe-schemas")] + rbac[("RBAC roles")] -## dfe-engine, never "dfe-api" + engine["dfe-engine"] -The API is **the dfe-engine API**. `dfe-api` was the old server entry-point name -and is retired: the daemon binary is now `dfe-engine` (`dfe-engine run`), the -human CLI is `dfe`. Anything talking to the API is talking to the dfe-engine API -- -do not call it "the dfe-api". Its own settings read as `DFE_ENGINE_API_*`. + deploy[("Deploy repo")] + charts[("Charts and
dfe-infra values")] -## openapi.json is the API SSoT + argo["Argo CD"] -`openapi/openapi.json` is the **single source of truth for the API surface**. -It is generated from the FastAPI app (`uv run python scripts/generate_openapi.py`) -and committed. The FastAPI routes feed it, and every consumer keys off it, never -off a hand-kept copy: + schemas -->|schema trees| engine + rbac -->|grants| engine + engine -->|commits| deploy + engine ~~~ charts + deploy --> argo + charts --> argo -```mermaid -flowchart TB - routes["FastAPI routes
(the live API definition)"] - routes -->|"scripts/generate_openapi.py"| spec[("openapi.json
THE API SSoT")] - - spec -->|"command tree at runtime
(x-scalo.cli.hidden opts out)"| cli["dfe CLI"] - spec -->|"openapi-typescript
-> @repo/dfe-engine-types"| ui["dfe-ui"] - spec -->|"Prism"| mock["contract mock
docker-compose.dev.yaml"] - spec -->|"Swagger UI"| refdocs["API reference docs"] - - classDef ssot fill:#fff3cd,stroke:#664d03,stroke-width:3px; - classDef src fill:#e2e3e5,stroke:#41464b; - class spec ssot; - class routes src; + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + class engine core + class schemas,rbac,deploy,charts store + class argo ext ``` -- The **`dfe` CLI** generates its whole command tree from the spec at runtime - - add an endpoint and the CLI grows a command, with no CLI code. An endpoint - opts out via `x-scalo.cli.hidden`. -- **dfe-ui** generates its TypeScript types from the same spec. +*Figure 2. L4 Detail -- Control-plane sources of truth. The dashed box is outside DFE's code.* Up: [DFE architecture](../architecture.md). + +To change one, go through its writer: edit the routes and regenerate the spec, commit to dfe-schemas, turn the engine's dials, or edit roles through the API. Never hand-edit a generated artefact such as `openapi.json` or a compiled overlay. + +`openapi/openapi.json`, the API source of truth, is generated from the FastAPI app (`uv run python scripts/generate_openapi.py`) and committed. The routes feed it, and every consumer keys off the routes or the file, never off a hand-kept copy. + +The dfe CLI and Swagger UI read the live routes, and dfe-ui and the Prism mock read the committed `openapi.json`. + +```mermaid +flowchart LR + accTitle: L4 Detail -- openapi.json and its consumers + accDescr: The FastAPI routes generate openapi.json and feed the dfe CLI and Swagger UI directly. dfe-ui types and the Prism mock read the committed file. + + routes["FastAPI routes"] + + spec[("openapi.json")] + cli["dfe CLI"] + swagger["Swagger UI"] + + ui["dfe-ui types"] + mock["Prism mock"] + + routes -->|generate| spec + routes -->|app factory| cli + routes -->|served| swagger + spec -->|sync PR| ui + spec -->|mounted| mock + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + class spec core +``` -So when the API changes: change the routes, regenerate `openapi.json`, and every -consumer follows. Never edit `openapi.json` by hand or let a consumer drift from -it. +*Figure 3. L4 Detail -- openapi.json and its consumers.* Up: [DFE architecture](../architecture.md). -## What is NOT in this layer +- The `dfe` CLI builds its whole command tree at runtime from the installed engine's app factory, so an endpoint becomes a command with no CLI code. An endpoint opts out with `x-scalo.cli.hidden`. +- dfe-ui copies both specs into its `@repo/dfe-engine-types` package through a PR the [sync workflow](type-safety-sync.md) opens, and generates its types from them. +- `docker-compose.dev.yaml` mounts `./openapi` into Prism. Swagger UI is served at `/docs` when `DFE_ENGINE_API_DOCS_ENABLED` allows it. -- The data-plane repos (receivers, loaders, transforms, fetchers, archiver, - hunt-runner) are downstream services the control plane configures - not part - of it. +When the API changes, change the routes, regenerate `openapi.json`, and every consumer follows. -## Documents in this area +The pages in this area: | Doc | Covers | |---|---| -| [governed-ops-design.md](governed-ops-design.md) | GitCrud + governance layer + curated actions | +| [governed-ops-design.md](governed-ops-design.md) | GitCrud, the governance layer and curated actions | | [gitops-commit-standard.md](gitops-commit-standard.md) | machine-commit conventions for gitops repos | -| [oidc-rbac-architecture.md](oidc-rbac-architecture.md) | OIDC + RBAC SSoT - auth model, the two RBAC planes, provider registry + adapters, management API, infra requirements | -| [rbac.md](rbac.md) | auth paths, RBAC model, multi-tenant ClickHouse, stores | +| [oidc-rbac-architecture.md](oidc-rbac-architecture.md) | OIDC and RBAC: the auth model, the two RBAC planes, the provider registry and adapters, the management API, infra requirements | +| [rbac.md](rbac.md) | auth paths, the RBAC model, multi-tenant ClickHouse, stores | | [rbac-sign-in.md](rbac-sign-in.md) | identity resolution, the sign-in allowlist, first login and password changes | | [rbac-groups.md](rbac-groups.md) | IdP group sync, linking a group to an IdP group, the checks on a link | -| [rbac-engine-reads.md](rbac-engine-reads.md) | the engine's own ClickHouse reads and how they bind to a caller's orgs | -| [idp-setup/](idp-setup/README.md) -- [google.md](idp-setup/google.md), [entra.md](idp-setup/entra.md), [okta.md](idp-setup/okta.md) | one-time setup for a customer's IdP admin -- sign-in client, groups, the values to send the operator | -| [rbac-vocabulary.md](rbac-vocabulary.md) | SSoT for role + group names and terms across engine, ui, infra, deploy, schemas and test envs | -| [synthetic-data.md](synthetic-data.md) | synthetic data generation - reference packs, lookalike scrub, standing demo streams | -| [ui-api-guide.md](ui-api-guide.md) | frontend integration guide (endpoint reference) | -| [type-safety-sync.md](type-safety-sync.md) | OpenAPI + RBAC-scope typegen sync to dfe-ui | -| [dfe-cli-design.md](dfe-cli-design.md) | the `dfe` client CLI design (AWS CLI v2 model) | +| [rbac-engine-reads.md](rbac-engine-reads.md) | the engine's ClickHouse reads and how they bind to a caller's orgs | +| [idp-setup/](idp-setup/README.md): [google.md](idp-setup/google.md), [entra.md](idp-setup/entra.md), [okta.md](idp-setup/okta.md) | one-time setup for a customer's IdP admin: sign-in client, groups, the values to send the operator | +| [rbac-vocabulary.md](rbac-vocabulary.md) | role and group names and terms across engine, ui, infra, deploy, schemas and test envs | +| [synthetic-data.md](synthetic-data.md) | synthetic data: reference packs, lookalike scrub, standing demo streams | +| [ui-api-guide.md](ui-api-guide.md) | frontend integration guide and endpoint reference | +| [type-safety-sync.md](type-safety-sync.md) | OpenAPI and RBAC-scope type sync to dfe-ui | +| [dfe-cli-design.md](dfe-cli-design.md) | how the `dfe` client CLI is built and resolves profiles | The system map is [../architecture.md](../architecture.md). diff --git a/docs/control-plane/dfe-cli-design.md b/docs/control-plane/dfe-cli-design.md index c97ecc926..7e8229243 100644 --- a/docs/control-plane/dfe-cli-design.md +++ b/docs/control-plane/dfe-cli-design.md @@ -12,12 +12,21 @@ `dfe-engine` is the server. The two are separate entry points: `dfe` talks to an engine over HTTP, and its `local` group works on the deploy repo when no engine is running. -## The command tree +The command tree: - One group per API tag, one command per operation: `dfe hunts list | get | create | apply | delete | run | status`. - Path parameters are positional arguments, in path order. Body fields become flags, and `--cli-input-json`, `--cli-input-yaml` and `--generate-cli-skeleton` take or print the whole body. -- The tree is read from the installed engine package's own app factory (`load_live_spec`), so it matches the engine version installed beside it. An endpoint opts out with `x-scalo.cli.hidden`. -- Commands with no operation behind them are mounted on top: `login`, `logout`, `config`, `auth list`, `local`, `schema` and `kafka client-config`. +- The tree is read from the installed engine package's app factory (`load_live_spec`), so it matches the engine version installed beside it. An endpoint opts out with `x-scalo.cli.hidden`. +- Commands with no operation behind them are mounted on top: `login`, `logout`, `config`, `auth list`, `auth print-access-token`, `local`, `schema` and `kafka client-config`. + +Global options: + +- `--output json|yaml|table|text` (`DFE_ENGINE_OUTPUT`). The default is `json` off a terminal, else `yaml` for a get and `table` for a list. +- `--query ` projects the answer before it is printed. +- A list follows `next_page_token` to the end. `--no-paginate` fetches one page. +- `--verbose`, `--quiet`, `--debug`, `--yes` (skip a delete's confirmation) and `--no-input` (never prompt). + +`dfe local` reads and writes a clone of the deploy repo through GitCrud with no daemon: `classes`, `ls`, `grep`, `get`, `log`, `versions`, `status`, `set`, `unset`, `rm`, `revert`, `restore` and `push`, plus `ch-cloud` for a ClickHouse Cloud service. `dfe local log` flags the commits made this way. ## Profiles and credentials @@ -31,17 +40,4 @@ Profiles live under `~/.config/dfe` (`DFE_ENGINE_CONFIG_HOME` moves the root): a `dfe login` stores an API key (`--api-key`), or exchanges `--username` and `--password` for a JWT through `POST /api/v1/auth/login`. `dfe config set | get | list | configurations` reads and writes profiles. -## Global options - -- `--output json|yaml|table|text` (`DFE_ENGINE_OUTPUT`). The default is `json` off a terminal, else `yaml` for a get and `table` for a list. -- `--query ` projects the answer before it is printed. -- A list follows `next_page_token` to the end. `--no-paginate` fetches one page. -- `--verbose`, `--quiet`, `--debug`, `--yes` (skip a delete's confirmation) and `--no-input` (never prompt). - -## Break-glass: `dfe local` - -`dfe local` reads and writes a clone of the deploy repo through GitCrud with no daemon: `classes`, `ls`, `grep`, `get`, `log`, `versions`, `status`, `set`, `unset`, `rm`, `revert`, `restore` and `push`, plus `ch-cloud` for a ClickHouse Cloud service. `dfe local log` flags the commits made this way. - -## Not built - A frozen single binary, a Homebrew tap, a container image and a browser OIDC login are not built. diff --git a/docs/control-plane/gitops-commit-standard.md b/docs/control-plane/gitops-commit-standard.md index 7def3cfc2..6976dc337 100644 --- a/docs/control-plane/gitops-commit-standard.md +++ b/docs/control-plane/gitops-commit-standard.md @@ -1,128 +1,71 @@ # DFE GitOps Commit Standard -The governed-ops engine enforces this standard in code (`src/dfe_engine/gitcrud/commit_policy.py`). -It follows the OpenGitOps principles and Argo CD's best practice, amended for DFE's -rule that the repo alone can drive a deployment. +The governed-ops engine enforces this standard in code (`src/dfe_engine/gitcrud/commit_policy.py`). It follows the OpenGitOps principles and Argo CD's best practice, amended for DFE's rule that the repo alone can drive a deployment. -## Scope +It governs the machine commits dfe-engine makes to a deployment's gitops repo, through the governed-ops API and the gitops writer. Human source-code commits to dfe-* repos follow the hyperi-ai CI standard (Conventional Commits, semantic-release). Deploy-repo commits are operational state changes and never trigger a release. -This standard governs **machine commits made by dfe-engine** (the governed-ops API -and gitops bridge) to the **gitops deploy/config repos**. It does NOT govern -human source-code commits to dfe-* repos - those follow the hyperi-ai CI standard -(Conventional Commits, `fix:`/`feat:` + semantic-release). Deploy-repo commits are -OPERATIONAL state changes. They never trigger semantic-release. - -## 1. Foundational principles (non-negotiable) +## Principles and layout The OpenGitOps four, plus our survivability rule: -1. Declarative - desired state is YAML in git, never imperative cluster calls. -2. Versioned + immutable - git history is the SoT. Pin every image/chart by - tag or SHA. NO `latest`, NO floating refs (immutability principle). -3. Pulled automatically - Argo pulls. The engine never `kubectl apply`s. -4. Continuously reconciled - auto-sync + self-heal on (drift is reverted). -5. Survivable - kill engine + ui and the repo still fully drives + manages DFE. - Every engine mutation is a git commit, full stop. - -## 2. Repository + environment layout +1. Declarative: desired state is YAML in git, never imperative cluster calls. +2. Versioned and immutable: git history is the source of truth. Pin every image and chart by tag or SHA, with no `latest` and no floating refs. +3. Pulled automatically: Argo CD pulls, and the engine never runs `kubectl apply`. +4. Continuously reconciled: auto-sync and self-heal are on, so drift is reverted. +5. Survivable: with the engine and the UI dead, the repo still drives and manages DFE, because every config change the engine makes is a git commit. -- The deploy repo is separate from the source repos, so access is separate and - there is no CI->commit->CI loop. -- One deploy repo per deployment. The engine writes `values/` (app-instance - overlays), `infra/` (cluster and platform overlays), `config/` (sources, schemas, - hunts, rules, the library) and `governance/` (`src/dfe_engine/gitcrud/registry.py`). -- The tracked branch is `main`. No environment branches: a second environment is a - second deployment with its own repo. +The repository layout: -## 3. Commit conventions (Conventional Commits format, gitops-typed) +- The deploy repo is separate from the source repos, so access is separate and there is no CI to commit to CI loop. +- One deploy repo per deployment. The engine writes `values/` (app-instance overlays), `infra/` (cluster and platform overlays), `config/` (sources, schemas, hunts, rules, the library) and `governance/` (`src/dfe_engine/gitcrud/registry.py`). +- The tracked branch is `main` (`DFE_ENGINE_GITOPS_BRANCH`). There are no environment branches: a second environment is a second deployment with a separate repo. -Subject: `type(scope): summary` - ASCII only, <= 50 chars, imperative. +## Commit conventions -- **types** (gitops-operational, NOT release types): `cfg` (config/helm-var change), - `hunt`, `rbac`, `action` (a defined-action invocation), `ops` (restart/scale/ - rollback), `schema`, `seed`, `lifecycle` (a service set running, paused or stopped). -- **scope** = the resource class + instance, e.g. `cfg(receiver-default)`, - `hunt(bruteforce)`, `rbac(group:soc-ro)`, `action(scale-receiver)`. -- Body wrap 72. Explains WHY when non-obvious. -- `[skip ci]` on every deploy-repo commit (deploy repo has no semantic-release). -- **Atomic:** one logical change = one commit. A defined-action that touches N vars - is ONE commit. +The subject is `type(scope): summary`, ASCII, 50 characters at most, imperative. The type is a gitops-operational type, never a release type: `cfg` (a config or helm-var change), `hunt`, `rbac`, `action` (a defined-action invocation), `ops` (restart, scale, rollback), `schema`, `seed` or `lifecycle` (a service set running, paused or stopped). The scope is the resource, for example `cfg(receiver-default-values)`, `hunt(bruteforce)` or `action(receiver-surge)`. When the subject would overrun, the summary is cut first, and the scope is cut too when the type and scope alone pass 50 characters. -Identity + audit trailers (machine-parseable, tie commit <-> SOC2 audit record): +The engine writes no prose body. The audit trailers follow the subject, and `[skip ci]` closes every message, because the deploy repo runs no release. One logical change is one commit, so a defined action that touches N vars is one commit. A helm-var write reads: ```text -cfg(receiver-default): set replicaCount=3 - -Scale receiver for ingest backlog. +cfg(receiver-default-values): set maxReplicaCount DFE-Actor: alice@example.com DFE-Role: helmvars:write -DFE-Action: scale-receiver # if via a defined action -DFE-Request-Id: 4f3c... # correlation id, also in the audit log -DFE-Base-Revision: 9c1d... # optimistic-concurrency base SHA -DFE-Audit-Id: aud_... # SOC2 audit record id +DFE-Base-Revision: 9c1d... + +[skip ci] ``` -- Author and committer are both the engine's configured git identity - (`src/dfe_engine/gitops/repo.py` `_commit`). The acting principal is the `DFE-Actor` trailer. - -## 4. Branch + PR policy (direct vs PR is POLICY-DRIVEN) - -Argo best practice = PRs for prod-controlling branches, enforced at the git level. -We honour it via the RBAC + protected-var policy, not by blocking the engine: - -- **Direct-commit mode** (dev / low-risk / non-protected classes): engine commits - straight to `main` -> Argo auto-syncs. The RBAC check + audit IS the gate. -- **PR mode** (prod env / protected vars / `governance` (RBAC) class / any class - flagged `require_pr`): engine creates a short-lived branch - `dfe///`, commits, and OPENS A PR via the git - PROVIDER API (Forgejo/GitHub/GitLab - provider-agnostic. REST via - AsyncHttpClient, NOT git CLI). A required approval (human or policy bot) merges - to `main`. Protected-branch rules + approvals are configured at the GIT level. -- Mode is resolved per change from: environment x class x protected-var-policy x a - `require_pr` flag on the defined action. Recorded in the audit trail. -- Sources, derived schemas, hunts, rules and app overlays all take this path. With - no forge configured, a production+team write still goes to the branch, never to - `main`, and takes effect once the branch is merged. -- Short-lived branches are deleted on merge. NO permanent branches besides `main`. - -## 5. Signing + verification (supply-chain) - -The engine does not sign its commits, and neither the engine nor dfe-infra -configures Argo CD's `sourceIntegrity` verification. Provenance rests on the -`DFE-*` trailers, the audit log and the forge's branch protection. - -## 6. Concurrency, rollback, no rewriting - -- **Optimistic concurrency** via the commit SHA as the version token (ETag / - `If-Match`). Stale base -> 409 carrying the current doc and revision, so the - caller can build a 3-way diff (`src/dfe_engine/gitcrud/engine.py` `ConcurrencyConflictError`). -- **Roll-forward only.** Rollback = `git revert` (or pin the prior tag), NEVER Argo - rollback (breaks under auto-sync) and NEVER `--force` / history rewrite on `main` - (Argo would thrash). Reverts are normal commits under this standard. -- Non-fast-forward push (another engine/user pushed first) -> pull + re-evaluate -> - retry or surface the conflict. Git is the only shared coordination point. - -## 7. Self-heal safety (avoid reconciliation loops) - -- Do NOT write fields owned by in-cluster controllers under self-heal. KEDA owns `replicas` -> "scale" sets `keda.minReplicaCount`/`keda.maxReplicaCount`, NOT `replicaCount`. Declare `ignoreDifferences` for any dynamic field. Build the exclusion list BEFORE enabling self-heal, not after. -- The exception is a document that explicitly sets `keda.enabled: false`. Nothing owns the count there, and `replicaCount` is the only way to set it. An unset flag is the chart default, which the engine reads off the app manifest (`appmgmt.scaling.keda_by_default`): an overlay in `values/` of a `scale_deployed` app, or one the catalogue cannot place, is read as KEDA-driven, while a platform chart in `infra/` is not unless the document enables KEDA. -- `validate_write` and `validate_revert` (`src/dfe_engine/gitcrud/commit_policy.py`) check the document a write leaves behind and refuse only the violations the write adds, so a document already breaking a rule still takes the write that repairs it. -- dfe-infra's ApplicationSets run auto-sync with self-heal on. Prune is on for the - layer-2 app, data, edge and platform sets and off for the add-ons and scale sets - (`argocd/appsets/*.yaml`). - -## 8. Enforcement in code (this is a code standard, not just prose) - -Implemented in the gitcrud engine: - -- `CommitPolicy` builds every message + trailers from the request context (actor, - role, class, action, request-id, base-SHA, audit-id) - callers cannot hand-write - messages. -- Mode resolver (direct vs PR) from env x class x policy. PR via the provider API. -- Pre-commit validators: ASCII-only, subject <= 50, type in the allowed set, image/ - chart refs pinned (reject `latest`/floating), no controller-owned fields. +`DFE-Action`, `DFE-Request-Id` and `DFE-Audit-Id` are added when the write has them. Author and committer are both the engine's configured git identity (`src/dfe_engine/gitops/repo.py` `_commit`), and the acting principal is the `DFE-Actor` trailer. + +The gitcrud engine enforces the conventions: + +- `build_message` writes every subject and its trailers from the request context (actor, role, action, request id, base SHA, audit id), so callers cannot hand-write a message. +- `resolve_mode` picks direct or PR from the environment, the class, the protected-var policy and the auto-merge setting. +- `validate_subject` rejects a non-ASCII or multi-line subject, one over 50 characters, and a type outside the allowed set. `validate_write` rejects an unpinned image ref (`latest`, or no tag and no `@sha256` digest) and a `replicaCount` KEDA owns. + +## Direct commit or PR is a policy decision + +Argo CD's best practice is PRs for production-controlling branches, enforced at the git level. We honour it through the environment and the RBAC class rather than by blocking the engine. + +- Direct-commit mode (a dev `DFE_ENGINE_ENV`, unprotected vars, any class but `governance`): the engine commits to `main` and Argo CD auto-syncs. The RBAC check and the audit trail are the gate. +- PR mode (a production environment, a protected var, or the `governance` class): the engine commits to a short-lived branch `dfe///` and opens a PR through the git provider's REST API, Forgejo, GitHub or GitLab, over scalo's `HttpClient` (`src/dfe_engine/gitcrud/forge.py`). A required approval merges it, and branch protection lives at the git level. +- `resolve_mode` in `commit_policy.py` decides per change. The auto-merge setting (`governance/settings/gitops.yaml`), allowed only on a dev environment or in solo mode, sends everything direct. +- Sources, derived schemas, hunts, rules and app overlays take this path. With no forge configured, a production team-mode write still goes to the branch, never to `main`, and takes effect once the branch is merged. +- Short-lived branches are deleted on merge, and `main` is the one permanent branch. + +## Concurrency, rollback and self-heal + +- The commit SHA is the version token (`If-Match`). A stale base answers 409 with the current doc and revision, so the caller can build a 3-way diff (`ConcurrencyConflictError` in `src/dfe_engine/gitcrud/engine.py`). +- Roll forward only. A rollback is `git revert` or the prior tag pinned, never an Argo CD rollback, which auto-sync undoes, and never `--force` or a history rewrite on `main`, which makes Argo CD thrash. A revert is a normal commit under this standard. +- A non-fast-forward push (another writer pushed first) pulls, re-evaluates, then retries or surfaces the conflict. Git is the one shared coordination point. +- The engine does not sign its commits, and neither the engine nor dfe-infra configures Argo CD's `sourceIntegrity` verification. Provenance rests on the `DFE-*` trailers, the audit log and the forge's branch protection. + +- Never write a field an in-cluster controller owns under self-heal. KEDA owns `replicas`, so a scale sets `keda.minReplicaCount` and `keda.maxReplicaCount`, never `replicaCount`. Declare `ignoreDifferences` for any dynamic field before self-heal is turned on. +- The exception is a document that sets `keda.enabled: false`. Nothing owns the count there, and `replicaCount` is the one way to set it. An unset flag is the chart default, which the engine reads off the app manifest (`keda_by_default` in `src/dfe_engine/appmgmt/scaling.py`): an overlay in `values/` of a `scale_deployed` app, or one the catalogue cannot place, is read as KEDA-driven, while a platform chart in `infra/` is not unless the document enables KEDA. +- `validate_write` and `validate_revert` (`src/dfe_engine/gitcrud/commit_policy.py`) check the document a write leaves behind and reject only the violations the write adds, so a document that breaks a rule still takes the write that repairs it. +- dfe-infra's ApplicationSets run auto-sync with self-heal on. Prune is on for the layer-2 app, data, edge, platform and deploy-repo sets and off for the add-ons and scale sets (`argocd/appsets/*.yaml` in dfe-infra). ## References @@ -130,5 +73,5 @@ Implemented in the gitcrud engine: - Argo CD Best Practices (env folders not branches, config and source repos split): - Argo CD Auto-Sync (self-heal, prune, roll-forward): - Argo CD anti-patterns (PRs over direct commits, pin bases): -- Argo CD signature verification (GnuPG deprecated -> sourceIntegrity): +- Argo CD signature verification (GnuPG deprecated, sourceIntegrity): - Conventional Commits: diff --git a/docs/control-plane/governed-ops-design.md b/docs/control-plane/governed-ops-design.md index 437cf1d33..3c18d3f13 100644 --- a/docs/control-plane/governed-ops-design.md +++ b/docs/control-plane/governed-ops-design.md @@ -1,345 +1,238 @@ # Governed Ops and the Hunt Runner -How DFE is operated, and how detection runs. Two subsystems, one idea: the GitOps -repo is the source of truth and the authority, and dfe-engine is a governed window -onto it. Companion to [architecture.md](../architecture.md), the -[commit standard](gitops-commit-standard.md), and the -[observability standard](../deployment/observability-standard.md). +How DFE is operated, and how detection runs. Everything you can change in DFE (deployment dials, detection hunts, access control) is YAML in a git repo, and dfe-engine is a governed window onto it. The engine holds no config database and never touches the live cluster: it reads and writes the YAML through one git-native CRUD engine, `GitCrud`, applies RBAC, and commits. Argo CD reconciles the commit. Kill the engine and the UI and DFE keeps running on hand edits to the repo. -## The one idea +## Every write is one GitCrud commit -Everything you can change in DFE -- deployment dials, detection hunts, access -control -- is YAML in a git repo. The engine does not hold a config database and -does not touch the live cluster. It reads and writes those YAML files through one -generic, git-native CRUD engine called `GitCrud`, applies RBAC, and commits. -Argo reconciles the commit. Kill the engine and the UI and DFE keeps running -- -you can still drive it by editing the repo by hand. That survivability is the -whole point, and it is the acceptance test for every piece below. - -## Layering +Every caller writes through GitCrud, which commits under the commit policy for Argo CD to reconcile. ```mermaid -graph TD - CLI["CLI - dfe local"] - HelmRouter["Tier-1 router - api/v1/helm"] - GovRouter["Tier-2 router - api/v1/governance"] - ConfigRouter["Config router - api/v1/config"] - Actions["ActionStore + PolicyStore"] - GitCrud["GitCrud engine - generic YAML CRUD"] - CommitPolicy["CommitPolicy - message + validators + mode"] - Repo["GitopsRepo - dulwich, no git CLI"] - Argo["Argo CD reconciles the commit"] - - CLI --> GitCrud - HelmRouter --> GitCrud - GovRouter --> Actions - Actions --> GitCrud - GitCrud --> CommitPolicy - GitCrud --> Repo - Repo --> Argo -``` +flowchart LR + accTitle: L4 Detail -- Governed write path + accDescr: dfe local, the Helm vars router, the backing services router and governance actions all write through GitCrud. GitCrud builds the message under the commit policy and commits through GitopsRepo, and Argo CD reconciles the commit. + + subgraph callers["Callers"] + local["dfe local"] + helm["Helm vars router"] + backing["Backing services
router"] + actions["Governance actions"] + end -Two tiers sit on the one engine: - -- **Tier-1** is raw, generic var CRUD over the overlay files. Powerful, so it is - admin-grade. -- **Tier-2** is curated *actions* -- named bundles of changes, each behind its own - permission. This is the safe surface an operator gets. They can run - "scale-receiver" without holding the keys to edit any var. - -RBAC binds at the high level only -- a resource class, a named action, an operation --- never per field. One permission check per request. That keeps the security -surface small and the policy readable. - -## GitCrud -- the generic engine - -`src/dfe_engine/gitcrud/`. One engine handles every resource class the same way, -because every resource is git-backed config. A `ResourceClass` says where a class -lives (a directory), how one resource is laid out, and which RBAC prefix governs -it. The default registry (`src/dfe_engine/gitcrud/registry.py`) covers the deploy repo: -`helmvars` (the app-instance overlays under `values/`), `infravars` (the cluster -and platform overlays under `infra/`), the datamodel classes `sources`, -`compat_profiles` and `derived_schemas` (under `config/sources/`, -`config/compat-profiles/` and `config/schemas/derived/`), `hunts` and `rules` -(`config/hunts/`, `config/rules/`), `library` (the versioned artefact library -under `config/library/`) and the governance classes (`accounts`, `groups`, -`roles`, `actions`, `policies`, `ch_tiers`, `ch_service_roles` and -`gov_settings` under `governance/`). - -`infravars` shares the `helmvars` RBAC prefix but needs its own directory: Argo -hands the app generator's `values/*-values.yaml` glob to git as a pathspec, where -`*` matches `/`, so anything under `values/` becomes an Argo application. Its -surface is `api/v1/backing-services`, and its reads are DECLARED, never observed --- the engine runs no Kubernetes client, by design. - -### Layouts -- one document, or a directory - -Most classes are `FILE`: one YAML document, one file. The fields ARE the resource, -so one doc keeps them consistent and gives one conflict domain per write. -`helmvars` cannot be anything else -- Argo and Helm read the overlay as one values -file. - -A class storing authored CONTENT declares `BUNDLE`: a directory per resource, -holding a `manifest.yaml` plus payload files. `library` is the first consumer, and -its layout is the general one: + gitcrud["GitCrud"] -```text -config/library//manifest.yaml -config/library//versions/0001.vrl -config/library//current.vrl + policy["Commit policy"] + repo["GitopsRepo
dulwich"] + + argo["Argo CD"] + + callers -->|writes| gitcrud + gitcrud -->|message| policy + gitcrud -->|commit| repo + repo --> argo + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class gitcrud core + class argo ext + class callers group ``` -Content in files rather than in the manifest means a reviewer diffs the language -itself, a validator runs against a real path, reading one version does not parse -the whole history, and no YAML emitter touches the bytes. `put_bundle` writes the -manifest and its files in ONE commit, so a manifest never names a version whose -content has not landed. Payload paths are refused if they escape the bundle or -collide with the manifest. +*Figure 1. L4 Detail -- Governed write path. The dashed box is outside the engine.* Up: [L3 Components -- dfe-engine](../architecture.md). -Nothing else moves to `BUNDLE` by default: the other versioned classes hold small -structured payloads that already diff well, so migrating them would cost a data -migration in every deployed gitops repo for no gain. +Tier 1 is raw var CRUD over the overlay files (`/api/v1/helm`), admin-grade because it can set anything. Tier 2 is curated actions (`/api/v1/governance`), each behind a permission of its name, so an operator can run `receiver-surge` without the keys to edit any var. RBAC binds to a resource class, an action or an operation, never to a field. -## One commit per mutation +A `ResourceClass` names the directory a class lives in, how one resource is laid out, and its RBAC prefix. The default registry (`src/dfe_engine/gitcrud/registry.py`) covers the deploy repo: `helmvars` (app-instance overlays under `values/`), `infravars` (cluster and platform overlays under `infra/`), `sources`, `compat_profiles` and `derived_schemas` (under `config/sources/`, `config/compat-profiles/` and `config/schemas/derived/`), `hunts` and `rules` (`config/hunts/`, `config/rules/`), `library` (`config/library/`), and the governance classes `accounts`, `groups`, `roles`, `actions`, `policies`, `ch_tiers`, `ch_service_roles` and `gov_settings` under `governance/`. -The engine is small on purpose. Read, flatten to dot-paths, set or delete a path, -write a whole doc, delete a resource -- every mutation ends in one commit via -`GitopsRepo` (which uses dulwich, so there is no shell-out to `git`). Three extras -earn their keep: +`infravars` shares the `helmvars` RBAC prefix in a separate directory: Argo hands the app generator's `values/*-values.yaml` glob to git as a pathspec, where `*` matches `/`, so anything under `values/` becomes an Argo application. Its surface is `/api/v1/backing-services`, and its reads are declared, never observed, because the engine runs no Kubernetes client. -- `put_many` writes several resources in ONE commit, so an action that touches N - files is atomic. -- `put_bundle` does the same for one bundle's manifest plus its content files. -- The HEAD commit SHA is the optimistic-concurrency token. A read returns it, a - write may require it, and a stale one raises `ConcurrencyConflictError` carrying - the current doc -- so two editors never silently clobber each other. +Most classes are `FILE`, one YAML document per resource and one conflict domain per write. A class storing authored content is `BUNDLE`, a directory per resource with a `manifest.yaml` and payload files, so a reviewer diffs the language itself: -`commit_policy.py` enforces the [commit standard](gitops-commit-standard.md) in -code: it builds every message and its audit trailers (callers cannot hand-write -them), rejects `latest`/unpinned image refs and controller-owned fields like -`replicaCount`, and resolves direct-commit vs PR from the environment and class. +```text +config/library//manifest.yaml +config/library//versions/0001.vrl +config/library//current.vrl +``` + +Every mutation ends in one commit through `GitopsRepo`, which uses dulwich rather than the `git` CLI. `put_many` writes several resources in one commit, so an action touching N files is atomic, and `put_bundle` does the same for a manifest and its content files. The HEAD commit SHA is the concurrency token, and a stale one raises `ConcurrencyConflictError` carrying the current doc. `src/dfe_engine/gitcrud/commit_policy.py` enforces the [commit standard](gitops-commit-standard.md): it builds every message and its audit trailers, rejects `latest` and unpinned image refs and a `replicaCount` KEDA owns, and resolves direct commit or PR from the environment and the class. ## Writing a helm var +A Tier 1 var write is checked for the grant and the protected-var policy, then committed, against its `If-Match` base when the caller sends one. + ```mermaid sequenceDiagram - participant User - participant Helm as Tier-1 router - participant Policy as PolicyStore - participant Crud as GitCrud - participant Repo as GitopsRepo - participant Argo - - User->>Helm: PUT /helm/files/receiver-default/vars/keda.maxReplicaCount (If-Match SHA) - Helm->>Helm: require_action helmvars:write - Helm->>Helm: validate_write (no new latest, no new replicaCount) - Helm->>Policy: protected? (override held?) - Helm->>Crud: set_key (base_revision = If-Match) - Crud->>Crud: stale base? -> 409 with current - Crud->>Repo: publish (one commit, audited) - Repo-->>Argo: commit on the tracked branch - Argo-->>User: reconciles to the cluster + accTitle: L4 Detail -- Writing a helm var + accDescr: The Helm vars router checks the caller's grant, validates the write and the protected-var policy, then GitCrud sets the key against the If-Match base and GitopsRepo commits. Argo CD reconciles the commit. + autonumber + participant user as Caller + participant helm as Helm router + participant policy as PolicyStore + participant crud as GitCrud + participant repo as GitopsRepo + participant argo as Argo CD + user->>helm: PUT var, If-Match + helm->>helm: Grant, validate_write + helm->>policy: Protected? + helm->>crud: set_key + alt stale base + crud-->>user: 409, current doc + end + crud->>repo: Commit + repo-->>argo: New commit ``` -The router is thin. The work -- validation, the protected-var check, the commit -- -lives in the engine and the policy store, so the CLI and the API take the identical -path. +*Figure 2. L4 Detail -- Writing a helm var. Solid arrows are requests, dashed arrows are responses.* Up: [L3 Components -- dfe-engine](../architecture.md). + +The full path is `PUT /api/v1/helm/files/{name}/vars/{path}`, for example `receiver-default-values` and `keda.maxReplicaCount`. A production `DFE_ENGINE_ENV` in team mode routes the write to a review branch instead of `main`. Validation, the protected-var check and the commit live in the engine, so the CLI and the API take the same path. + +## Actions, protected vars and rejected writes -## Defined actions and protected vars +An `ActionDef` (`src/dfe_engine/governance/`) is a name, the permission needed to run it (`required_action`) and a list of `VarChange`s. `ActionStore.invoke` applies them all in one commit and honours the protected-var policy, so one violation aborts the whole action, and `dry_run` returns the diff without writing. -`src/dfe_engine/governance/`. An `ActionDef` is a name, the permission needed to run -it (`required_action`), and a list of `VarChange`s. `ActionStore.invoke` applies them -all in one commit, honours the protected-var policy (a single violation aborts the -whole action -- it is atomic), and `dry_run` returns the diff without writing. +An action invoke checks the caller's grant and every change, then writes them all in one commit. ```mermaid sequenceDiagram - participant Op as Operator - participant Gov as Tier-2 router - participant Store as ActionStore - participant Pol as PolicyStore - participant Crud as GitCrud - - Op->>Gov: POST /governance/actions/scale-receiver/invoke - Gov->>Store: load the action - Gov->>Gov: require the action's OWN required_action - Gov->>Store: invoke (actor, dry_run) + accTitle: L4 Detail -- Invoking an action + accDescr: The governance router loads the action and checks the caller holds that action's required_action. ActionStore checks every change against the protected-var policy and writes them all in one commit through GitCrud. + autonumber + participant op as Operator + participant gov as Governance router + participant store as ActionStore + participant pol as PolicyStore + participant crud as GitCrud + op->>gov: POST actions/receiver-surge/invoke + gov->>store: Load the action + gov->>gov: Require its required_action + gov->>store: invoke, actor and dry_run loop each change - Store->>Pol: enforce - protected var aborts the lot + store->>pol: Enforce the protected policy end - Store->>Crud: put_many - ONE atomic commit - Crud-->>Op: diff + commit SHA + store->>crud: put_many, one commit + crud-->>op: Diff and commit SHA ``` -## Protected vars and refused writes +*Figure 3. L4 Detail -- Invoking an action. Solid arrows are requests, dashed arrows are responses.* Up: [L3 Components -- dfe-engine](../architecture.md). -A `ProtectedPolicy` is the one thing that reaches below a class -- a list of locked `cls:name:path` globs that even Tier-1 must respect unless the caller holds the override grant. It is a policy object, not a per-var ACL. A set or revert at a parent, or a set below a locked scalar, is refused like a write to the var itself, because either one changes it. A whole-document write is gated only on the leaves it adds, changes or removes. +A `ProtectedPolicy` is a list of locked `cls:name:path` globs that Tier 1 respects too, unless the caller holds the override grant. A set or revert at a parent, or a set below a locked scalar, is rejected like a write to the var itself, and a whole-document write is gated on the leaves it adds, changes or removes. Three policies ship: `baseline` locks image references, `storage-layout` locks the data-layer modes, the storage model, the `objectStore`, `tieredBlock` and `tieredObject` dial blocks and the disk size and class, and `sizing-locks` locks the Kafka sizing inputs, the KRaft controller layout and the cloud. -Three policies ship and seed into every deploy repo. `baseline` locks image references. `storage-layout` locks the data-layer modes, the storage model, the `objectStore` / `tieredBlock` / `tieredObject` dial blocks, and the disk size and class. `sizing-locks` locks the Kafka sizing inputs, the KRaft controller layout and the cloud. Seeding never rewrites an existing file, so a repo carrying the earlier `storage-model` policy keeps it and gains `storage-layout` beside it -- hence the pre-vocabulary key spellings in the shipped lock. +No grant lifts three rejections. A lower node or broker count is rejected with a 400 naming the reason, by the backing-services router and by any action leaving one in the infra overlay, because the hazard is data loss. A `replicaCount` is rejected while KEDA owns the count (`keda.enabled`, else the chart default), because it would render nothing. A write or revert of a `read_only` value on a backing service DFE does not run (`managed_by` `provider` or `external` in `src/dfe_engine/backing/providers.yaml`) answers 409 `managed_by_provider`, because the chart deploys no store there. -## Refused writes no grant lifts +## Action params and the closed sets -Not everything unsafe is a protected var. A lower node or broker count is refused -by the backing-services router itself, and by an action that leaves one in the infra overlay, with a 400 that names the reason, and the -override grant does NOT get past it: the hazard is data loss rather than -governance, and there is no correct way to grant your way through it. The same -distinction applies to `replicaCount`, which the commit standard refuses unless -the document being written explicitly disables KEDA -- with a controller owning -the count, writing one renders nothing. +An action may declare `params`, each closed: an enum lists its values, a numeric both bounds, and there is no free-string type. A `VarChange.value` references a param as `{"$param": "level"}`, whole-value only, or with a `map` turning an enum value into a per-var literal. Substitution never reaches `cls`, `name` or `path` and there is no string interpolation, so params add no injection surface. A dangling reference or a map that misses an enum value is rejected at define time, and only the caller's values can fail an invoke (422 `invalid_params`). A value of `{"$unset": true}` removes the path, handing the var back to the layer beneath, and an emptied parent map goes with it. `required_action` left empty derives `action:invoke:`, which the shipped `dfe_operator` and `infra_admin` roles grant as `action:invoke:*`. -Ahead of the count guard and the policy locks, the router, and an action writing the infra overlay, refuse a write or revert of a `read_only` value on a backing service DFE does not run (`managed_by` `provider` or `external`, per `src/dfe_engine/backing/providers.yaml`) with 409 `managed_by_provider`. The override grant does not lift it: the chart deploys no store there, so the commit would change nothing. +The contract enumerates every closed set: `GET /gitops/classes` (with `action_writable` exporting the no-governance-class guard), `.../classes/{cls}/resources` and `.../resources/{name}/vars`, each gated by that class's `:read` grant, and `POST /governance/admin/actions/validate` returns every violation and the would-be diff without committing. Schema fields hold `x-dfe-enum-source` annotations naming those endpoints ([ui-api-guide.md](ui-api-guide.md)). -## Constrained params -- a dial with detents, or a bounded knob +The engine packages a standard library of dials in `src/dfe_engine/governance/resources/` and seeds it into every deploy repo, never rewriting an existing file: `receiver-surge` and `receiver-normal` (KEDA ceiling detents), `hunts-pause`, `hunts-resume` and `hunts-throttle`. dfe-deploy's template holds the same files, and its CI validates them (`tools/validate_governance.py` in dfe-deploy). -An action may declare `params`, and every param carries a CLOSED constraint: -an enum carries its full value list, a numeric carries both bounds. There is -deliberately no free-string param type -- an unconstrained param would reopen -the hole curation closed. A `VarChange.value` references a param as -`{"$param": "level"}` (whole-value substitution) or with a `map` that turns an -enum value into a per-var literal -- the admin still curates what each detent -means. Substitution never reaches `cls`/`name`/`path`, and there is no string -interpolation, so params add zero injection surface. Wiring errors (dangling -reference, map not covering the enum) are rejected at DEFINE time. Only the -caller's supplied values can fail an invoke (422 `invalid_params`). +## ClickHouse data RBAC: a tier and a tenant per identity -A `value` of `{"$unset": true}`, alone or as a map entry, removes the path instead of setting it, which returns the var to the layer beneath the overlay (a chart default or a tier profile). Removing an absent path is a no-op that creates no file, an emptied parent map is removed with it, and the diff shows `new` as `{"$unset": true}`. A marker with any other content is rejected at define time. +`src/dfe_engine/governance/ch/`. dfe-hyperdx's app layer cannot contain a bad query, but ClickHouse quotas and limits can, so every ClickHouse identity composes two axes. -`required_action` is optional: left empty it derives `action:invoke:`, -the convention the shipped roles grant on (`dfe_operator` and `infra_admin` -carry `action:invoke:*`). +Each RBAC group binds to one ClickHouse user holding a tier role, the tenant role and, for an org, a pinned tenant setting. -## The contract exposes the closed sets - -Every stringly field on an action is a closed set, so the contract enumerates -it rather than trusting hand-typed values: `GET /gitops/classes` (the registry, -with `action_writable` exporting the no-governance-class guard), -`.../classes/{cls}/resources` and `.../resources/{name}/vars` (each gated by -that class's own `:read` grant), and `POST /governance/admin/actions/validate` -returns EVERY violation plus the would-be diff without committing. The schema -fields carry `x-dfe-enum-source` annotations naming these endpoints, so -generated clients render selects, not text boxes (the full convention: -[ui-api-guide.md](ui-api-guide.md)). +```mermaid +flowchart LR + accTitle: L4 Detail -- ClickHouse identity + accDescr: An RBAC group binds to one ClickHouse user. The user holds a tier role, the tenant role and, for an org-tied user, a read-only SQL_current_tenant_id setting that the tenant row policies read. -## The shipped action library + group["RBAC group"] -The deploy-repo template (dfe-deploy `governance/`) ships a standard library -of dials -- `receiver-surge`/`receiver-normal` (KEDA ceiling detents), -`hunts-pause`/`hunts-resume`, `hunts-throttle` (bounded concurrent-run cap) -- -plus a `baseline` protected-var policy locking `image.*` (images move through -stack pins, not dials). Every shipped action's structure is CI-validated in -that repo (`tools/validate_governance.py`). The chart vars they point at are -owned by dfe-infra's chart validation. An action's `description` is its -runbook line -- what it does, and which action reverts it. + user["ClickHouse user"] -## ClickHouse data RBAC -- a tier and a tenant per identity + tier["Tier role
grants, profile, quota"] + tenant["Tenant role
row policies"] + setting["SQL_current_tenant_id
read-only"] -`src/dfe_engine/governance/ch/`. Access control in dfe-hyperdx's app layer cannot -contain a bad query. Quotas and resource limits can, and those live in ClickHouse. -So every ClickHouse identity composes two axes: + group -->|binds| user + user -->|tier| tier + user -->|tenant| tenant + user -->|pinned| setting -```mermaid -graph LR - Group["RBAC group"] - User["ClickHouse user
bound to the group"] - Tier["Tier role - grants,
settings profile, quota"] - Tenant["Tenant role - one row policy
per _org_id table"] - Setting["SQL_current_tenant_id
pinned read-only"] - - Group --> User - User --> Tier - User --> Tenant - Tenant --> Setting + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + class user core ``` -A group binding ties an RBAC group's ClickHouse user to one tier and at most one -org. A tier is grants, a settings profile and a quota (`dfe__role`, -`_profile`, `_quota`). The tenant axis is one shared role plus a restrictive row -policy on each `_org_id` table, reading the caller's `SQL_current_tenant_id`, which -an org-tied user carries pinned read-only, so a `SETTINGS` override in a query is -refused. The catalogue of tiers and service roles is data in dfe-schemas -(`roles/clickhouse.yaml`), editable through the `ch_tiers` and `ch_service_roles` -classes, and `ChRbacReconciler` renders whatever exists into idempotent DDL and -runs it. - -The hunt runner connects as the `dfe_hunt_runner` service role, which carries -grants and no quota (`roles/clickhouse.yaml`, `src/dfe_engine/hunts/connect.py`). The catalogue's -`hunt_tier_*` tiers carry per-query memory and time limits and an hourly quota for a -group bound to one. +*Figure 4. L4 Detail -- ClickHouse identity.* Up: [L3 Components -- dfe-engine](../architecture.md). + +A group binding ties an RBAC group's ClickHouse user to one tier and at most one org. A tier is grants, a settings profile and a quota (`dfe__role`, `_profile`, `_quota`). The tenant axis is one shared role plus a restrictive row policy on each `_org_id` table, reading the caller's `SQL_current_tenant_id`, which an org-tied user holds read-only, so a `SETTINGS` override in a query is rejected. The catalogue of tiers and service roles is data in dfe-schemas (`roles/clickhouse.yaml`), editable through the `ch_tiers` and `ch_service_roles` classes, and `ChRbacReconciler` renders it into idempotent DDL and runs it. The hunt runner connects as the `dfe_hunt_runner` service role, with grants and no quota (`src/dfe_engine/hunts/connect.py`), and the catalogue's `hunt_tier_*` tiers set per-query memory and time limits and an hourly quota for a group bound to one. ## The hunt runner -`src/dfe_engine/hunts/`. A hunt is a rule query on a schedule that writes -matched rows to a table. The runner is a pool, and hunts are never *assigned* to -pods. Workers pull due runs from a -ClickHouse lease table. Add a pod and it just starts pulling. Kill one and its leases -expire and get reclaimed. There is no shard map to rebalance and nothing to go -split-brain over. +`src/dfe_engine/hunts/`. A hunt is a rule query on a schedule that writes matched rows to a table. Hunts are never assigned to pods: workers pull due runs from a ClickHouse lease table, and a dead pod's leases expire and are reclaimed. The ClickHouse the hunts query holds the lease, watermark and state, and with no row locks a claim is optimistic: insert a lease, then resolve the winner. -ClickHouse is the ONLY operational store -- no Postgres. Making hunts (the -mission-critical path) depend on a second database purely for coordination is not -worth the operational surface, so the same CH the hunts already query holds the -lease, watermark, and state. CH has no row locks, so the claim is optimistic: -insert-a-lease-then-resolve, not `SELECT ... FOR UPDATE`. +One worker wins each fire's claim, runs the window and advances the watermark. ```mermaid -graph TD - Scheduler["Scheduler - compute due fires"] - Claim["ClickHouse hunt_lease - insert + resolve winner"] - W1["Worker"] - W2["Worker"] - CH["ClickHouse - run query, write results"] - WM["Watermark on _timestamp_load"] - - W1 -->|claim| Claim - W2 -->|claim| Claim - W1 --> CH - W2 --> CH - CH --> WM - WM -->|resume incrementally| W1 +flowchart LR + accTitle: L4 Detail -- Hunt claim and run + accDescr: When a fire is due, runner workers race to claim it in hunt_lease. The winner runs the window query, writes matched rows to the results table and advances hunt_watermark, which sets the next window. + + due(["Fire due"]) + + subgraph workers["Runner workers"] + w1["Worker"] + w2["Worker"] + end + + lease[("hunt_lease")] + + run["Run the
window query"] + + results[("Results table")] + watermark[("hunt_watermark")] + + due --> workers + workers -->|claim| lease + lease --> run + run --> results + run --> watermark + watermark -.->|next window| workers + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class lease,results,watermark store + class workers group ``` -The smarts are deterministic, not random. `schedule.phase_offset` gives each hunt a -stable offset inside its interval from a hash of its id, so hunts on the same -schedule fan out evenly and never stampede ClickHouse at the boundary. `latest_fire` -names the most recent fire at or before now, and it stays owed until its watermark -is written. A global cap hard-limits how much hits CH at once. +*Figure 5. L4 Detail -- Hunt claim and run. The dotted arrow is the next run reading the watermark.* Up: [The DFE control-plane layer](README.md). + +`phase_offset` in `src/dfe_engine/hunts/schedule.py` gives each hunt a stable offset inside its interval from a hash of its id, so hunts on one schedule spread out. `latest_fire` names the most recent fire at or before now, and it stays owed until its watermark is written. A global cap limits concurrent runs. -## Claims, windows and coordination tables +To claim, a worker reads the hunt free, inserts a lease row and a marker for itself, waits a settle, and keeps the claim only if its row landed within the settle and is still the latest. Lease writes reach a majority of replicas and lease reads go through `FINAL`, so every worker names the same winner (`src/dfe_engine/hunts/ch_coordinator.py`). An overrun defers rather than starting a second copy, and flags `too_aggressive` so the UI can say the schedule is too tight. -A run never doubles up. To claim, a worker reads the hunt free, INSERTs a lease row -and a marker of its own, waits a settle, and keeps the claim only if its row landed -within that settle and is still the latest. Lease writes reach a majority of -replicas and lease reads go through `FINAL`, so every worker names the same winner -(`src/dfe_engine/hunts/ch_coordinator.py`). Exclusion has to hold, because a window run twice -writes its rows twice. Per-hunt state means an overrun defers -rather than starting a second copy -- and flags `too_aggressive` so the UI can tell -the user their schedule is too tight. +A fire goes from due to running to done, and an expired lease sends it back to due. ```mermaid stateDiagram-v2 - [*] --> due: phase_offset + latest_fire - due --> running: worker claims (insert+resolve, under cap) - running --> done: query committed, watermark advanced - running --> due: lease expired (worker died) -> reclaimed - done --> [*] - due --> deferred: previous run still running - deferred --> due: previous run finished + direction LR + accTitle: L4 Detail -- Hunt fire lifecycle + accDescr: A fire is due at its offset, deferred while the previous run is still going, running once a worker claims it, and done once its query commits and the watermark advances. A run with an expired lease goes back to due. + [*] --> Due: offset reached + Due --> Deferred: still running + Deferred --> Due: run finished + Due --> Running: claimed + Running --> Due: lease expired + Running --> Done: committed + Done --> [*] ``` -Each run is incremental and crash-safe. The query carries a `{window}` placeholder. -The worker substitutes the window predicate on the hunt's timestamp field, runs it, -and advances the watermark ONLY after the query commits. A pod killed mid-run re-runs -the same window from the last committed watermark -- no gap, no loss. The field -defaults to `_timestamp_load`, the common-header column that is always present, and a -hunt's `timestamp_field` overrides it (`src/dfe_engine/hunts/spec_loader.py`). +*Figure 6. L4 Detail -- Hunt fire lifecycle.* Up: [The DFE control-plane layer](README.md). -The three coordination tables, all ReplacingMergeTree in the effective data database. -`hunt_run` (one row per fire) and `hunt_runner_heartbeat` (one row per runner) sit -beside them: +Each run is incremental and crash-safe. Every statement holds a `{window}` placeholder, which the worker fills with the window predicate on the hunt's timestamp field (`src/dfe_engine/hunts/worker.py`), and the watermark advances only after the query commits, so a pod killed mid-run re-runs the same window. The field is `_timestamp_load` unless a hunt's `timestamp_field` overrides it (`src/dfe_engine/hunts/spec_loader.py`). `HuntRunner.tick(now)` is one cycle: find due hunts that are not already active, then claim up to the cap and run each at its window. + +Three ReplacingMergeTree tables in the effective data database, keyed by `hunt_id`, coordinate the runners. `hunt_run` (one row per fire) and `hunt_runner_heartbeat` (one row per runner) sit beside them. ```mermaid erDiagram + accTitle: L4 Detail -- Hunt coordination tables + accDescr: hunt_lease holds the owner and lease expiry of each hunt's current fire, hunt_watermark how far each hunt has run, and hunt_state its overrun count and too_aggressive flag. All three are keyed by hunt_id. hunt_lease { string hunt_id string owner @@ -360,37 +253,37 @@ erDiagram } ``` -`HuntRunner.tick(now)` is one cycle -- find due hunts that are not already active, -then claim up to the cap and execute each at its scheduled window. The daemon is -`while: tick(); sleep`. +*Figure 7. L4 Detail -- Hunt coordination tables.* Up: [The DFE control-plane layer](README.md). ## The smaller pieces -- `GET /api/v1/config/client` (authenticated, any role) hands the UI its API base, - Observe URL, auth mode and feature flags at runtime, so one UI image runs in every - environment. -- `src/dfe_engine/alerting/` is a scaffold: a rule model (threshold over a window, - severity to a channel) and a dispatcher whose only sender logs. A hunt's own alert - destinations are delivered by `src/dfe_engine/hunts/delivery.py`. -- `dfe local` (`ls`, `get`, `set`, `unset`, `rm`, `revert`, `restore`, `push`) wraps - GitCrud on a clone of the deploy repo, for break-glass when the engine is down. It - has no action invoke ([dfe-cli-design.md](dfe-cli-design.md)). +- `GET /api/v1/config/client` (any authenticated caller) hands the UI its API base, the Observe URL, the auth mode and feature flags at runtime, so one UI image runs in every environment. +- `src/dfe_engine/alerting/` is a scaffold: a rule model and a dispatcher with one sender, which logs. A hunt's alert destinations are delivered by `src/dfe_engine/hunts/delivery.py`. +- `dfe local` wraps GitCrud on a clone of the deploy repo for break-glass when the engine is down, with no action invoke ([dfe-cli-design.md](dfe-cli-design.md)). -## What holds it together +The repo as source of truth underpins RBAC, the ClickHouse safety boundary and no double-run. ```mermaid -graph TD - Surv["GitOps survivability - repo is SoT + authority"] - RBAC["RBAC at the abstraction - class / action / operation"] - CHsafe["ClickHouse is the safety boundary - quotas + limits"] - NoDouble["No double-run, ever - lease + per-hunt state"] - Crash["Crash-safe - watermark advances only after commit"] - - Surv --> RBAC - Surv --> CHsafe - Surv --> NoDouble - NoDouble --> Crash +flowchart LR + accTitle: L4 Detail -- What governed ops rests on + accDescr: The repo as source of truth and authority underpins RBAC at the class, action and operation level, ClickHouse as the safety boundary, and no double-run. A worker dying between its writes and the watermark repeats the window, and rule detections dedupe it. + + surv["Repo is the
source of truth"] + + rbac["RBAC per class,
action, operation"] + chsafe["ClickHouse quotas
and limits"] + nodouble["No double-run"] + + crash["Watermark after
the writes"] + + surv --> rbac + surv --> chsafe + surv --> nodouble + nodouble --> crash + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A ``` -If a change cannot be made by committing YAML to the repo, it does not belong in the -engine. That rule is what keeps DFE operable without us. +*Figure 8. L4 Detail -- What governed ops rests on.* Up: [The DFE control-plane layer](README.md). + +If a change cannot be made by committing YAML to the repo, it does not belong in the engine. diff --git a/docs/control-plane/idp-setup/README.md b/docs/control-plane/idp-setup/README.md index eda70f0aa..312e141e1 100644 --- a/docs/control-plane/idp-setup/README.md +++ b/docs/control-plane/idp-setup/README.md @@ -6,7 +6,7 @@ One page per identity provider, written for the customer's IdP admin. The DFE op |---|---|---|---| | Google Workspace | [google.md](google.md) | `google` | The Cloud Identity group id, the part after `groups/` | | Microsoft Entra ID | [entra.md](entra.md) | `entra_id` | The group's object id (a GUID) | -| Okta | [okta.md](okta.md) | `okta` | The group name the `groups` claim carries | +| Okta | [okta.md](okta.md) | `okta` | The group name the `groups` claim sends | Every page uses the same redirect URI, built from those two values: diff --git a/docs/control-plane/idp-setup/entra.md b/docs/control-plane/idp-setup/entra.md index 2ff200cd1..0c9723d5a 100644 --- a/docs/control-plane/idp-setup/entra.md +++ b/docs/control-plane/idp-setup/entra.md @@ -12,10 +12,14 @@ You need at least the Cloud Application Administrator role in the Microsoft Entr ## How DFE reads groups -Once you turn on the groups claim, Entra puts each user's group object ids in the ID token, and DFE reads them at sign-in. A user in more than 200 groups, nested groups included, gets a pointer instead of the list. DFE then reads that user's groups from Microsoft Graph `/me/transitiveMemberOf` with the user's own sign-in token. The User.Read permission every new app registration gets is enough for that call. +Once you turn on the groups claim, Entra puts each user's group object ids in the ID token, and DFE reads them at sign-in. A user in more than 200 groups, nested groups included, gets a pointer instead of the list. DFE then reads that user's groups from Microsoft Graph `/me/transitiveMemberOf` with the user's sign-in token. The User.Read permission an app registration starts with covers that call. + +DFE signs in with Entra ID by code exchange, and reads the groups of a user in more than 200 from Microsoft Graph. ```mermaid sequenceDiagram + accTitle: L4 Detail -- Entra sign-in + accDescr: The browser opens DFE sign-in and DFE redirects to Entra ID. Entra returns a code, which DFE exchanges for an ID token with the groups claim. For a user in more than 200 groups DFE reads the groups from Microsoft Graph with the user's token. participant U as User's browser participant D as DFE participant E as Entra ID @@ -32,19 +36,21 @@ sequenceDiagram D-->>U: Signed in, roles from the linked groups ``` +*Figure 1. L4 Detail -- Entra sign-in. Solid arrows are requests, dashed arrows are responses.* Up: [Identity provider setup](README.md). + ## Set it up 1. Go to **Entra ID > App registrations** and select **New registration**. Name it (for example `DFE`). Under **Supported account types** pick **Single tenant only - **. Select **Register**. 2. On the **Overview** page, copy the **Application (client) ID** and the **Directory (tenant) ID**. 3. Under **Manage**, select **Authentication**. Select **Add Redirect URI**, pick the **Web** tile, paste the redirect URI and select **Configure**. Leave the **ID tokens (used for implicit and hybrid flows)** box unticked: DFE uses the authorisation code flow. -4. Select **Certificates & secrets > Client secrets > New client secret**. Add a description, pick an expiry of 24 months or less and select **Add**. Copy the **Value** column now, not the **Secret ID**. Entra never shows the value again. DFE authenticates with a client secret, not a certificate. Diary the expiry: sign-in stops the day the secret lapses. +4. Select **Certificates & secrets > Client secrets > New client secret**. Add a description, pick an expiry of 24 months or less and select **Add**. Copy the **Value** column at this step, not the **Secret ID**: Entra never shows the value again. DFE authenticates with a client secret, not a certificate. Diary the expiry: sign-in stops the day the secret lapses. 5. Under **Manage**, select **Token configuration**, then **Add groups claim**. Tick **Security groups** and select **Save**. Leave each token's value at the default, the group ID. Switching it to sAMAccountName, cloud display names or role claims breaks DFE's group links. 6. Do not add a `groups` permission or scope. Entra adds the claim from step 5, and asking for a `groups` scope fails the sign-in. -Two optional extras, only if the operator asks: +Two optional extras, for when the operator asks: - Group names on the Graph path, which otherwise returns each group's id with a blank name: **API permissions > Add a permission > Microsoft Graph > Delegated permissions > GroupMember.Read.All**, then **Grant admin consent for **. DFE links on the id, so sign-in works without it. -- An app-only fallback: the Microsoft Graph application permission **GroupMember.Read.All** with admin consent. DFE then reads a user's groups with the app's own secret when Graph refuses the user's token, and the operator can run a group sync. +- An app-only fallback: the Microsoft Graph application permission **GroupMember.Read.All** with admin consent. DFE then reads a user's groups with the app's secret when Graph turns down the user's token, and the operator can run a group sync. ## Send these to the operator @@ -53,11 +59,9 @@ Two optional extras, only if the operator asks: | Application (client) ID | Step 2 | | Client secret | Step 4, the **Value**. Send it over a channel fit for a password, never plain email | | Directory (tenant) ID | Step 2. The issuer is `https://login.microsoftonline.com//v2.0` | -| Group object ids | One per group that should hold a DFE role, with the role it should get. See the next section | - -## Find the group ids +| Group object ids | One per group that should hold a DFE role, with the role it should get. See below | -DFE links an Entra group on its object id, a GUID. Renaming the group does not break the link. Go to **Entra ID > Groups > All groups**, open the group and copy its **Object ID** from the **Overview** or **Properties** page. Once one of the user's groups is linked, the check below also lists the ids of every group they are in. +DFE links an Entra group on its object id, a GUID, so renaming the group does not break the link. Go to **Entra ID > Groups > All groups**, open the group and copy its **Object ID** from the **Overview** or **Properties** page. Once one of the user's groups is linked, the check below lists the ids of every group they are in as well. ## Check it worked @@ -67,7 +71,7 @@ Once the operator confirms the provider is in place, open this URL in a private https:///api/v1/auth/oidc//login ``` -The page answers with JSON. `email` is the user, and `groups` lists their group object ids. The JSON also holds an `access_token`: it is a live DFE session, so do not paste it anywhere. A user in no linked group is refused, and DFE keeps no account for them. +The page answers with JSON. `email` is the user, and `groups` lists their group object ids. The JSON holds an `access_token` as well: it is a live DFE session, so do not paste it anywhere. A user in no linked group is turned away, and DFE keeps no account for them. ## Common failures @@ -76,9 +80,9 @@ The page answers with JSON. `email` is the user, and `groups` lists their group | Microsoft page: `AADSTS50011` | The redirect URI on the app differs from the one DFE sent | Add the operator's URI exactly under the **Web** platform | | Microsoft page: `AADSTS700016` | DFE holds the wrong client ID, or the issuer names the wrong tenant | Resend the client ID and tenant ID from step 2 | | DFE page: `OIDC login failed: invalid_client: AADSTS7000215 ...` | DFE holds the **Secret ID**, or a mistyped value | Send the secret's **Value** | -| DFE page: `OIDC login failed: invalid_client: AADSTS7000222 ...` | The client secret expired | Create a new secret and send its value | -| Microsoft page: `AADSTS50105` | The enterprise application requires assignment and this user is not assigned | Assign the user, or a group they are in, to the application | -| DFE page: `OIDC login refused: no DFE group with a role is linked to your identity provider groups.` | None of the user's groups is linked yet, or no groups arrived: no groups claim, or the user is in no security group | Send the operator the ids from "Find the group ids". If they are linked, redo step 5 and check the user's memberships | +| DFE page: `OIDC login failed: invalid_client: AADSTS7000222 ...` | The client secret expired | Create another secret and send its value | +| Microsoft page: `AADSTS50105` | The enterprise application needs assignment and this user is not assigned | Assign the user, or a group they are in, to the application | +| DFE page: `OIDC login refused: no DFE group with a role is linked to your identity provider groups.` | None of the user's groups is linked yet, or no groups arrived: no groups claim, or the user is in no security group | Send the operator the group object ids (see "Send these to the operator"). If they are linked, redo step 5 and check the user's memberships | ## For the DFE operator @@ -96,7 +100,7 @@ Create the provider with `POST /api/v1/auth/oidc-providers`. The `name` becomes } ``` -`token_claim` reads the `groups` claim, and the overage lookup runs on its own. `groups.tenant_id` is needed only for the app-only fallback, which uses the login client secret unless `groups.client_secret` is set. Leave `scopes` unset: the default is `openid email profile`. The client secret goes to the secret store and the provider keeps only its path. +`token_claim` reads the `groups` claim, and the overage lookup runs with no setting. The app-only fallback is the one thing that needs `groups.tenant_id`, and it uses the login client secret unless `groups.client_secret` is set. Leave `scopes` unset: the default is `openid email profile`. The client secret goes to the secret store and the provider keeps its path, never the value. Link each DFE group by setting two fields in its group file: `source_provider` to ``, the name the provider was created with, and `source_id` to an object id the admin sent. A group file with a `source_id` and no `source_provider` loads unlinked, with a warning naming it in the engine log, and no sign-in reaches it through that id: see [rbac-groups.md, section 4.4](../rbac-groups.md#44-group-sync-process). @@ -107,7 +111,7 @@ source_provider: "" source_id: "" ``` -Behind a TLS-terminating proxy, set `api.forwarded_allow_ips` (`DFE_ENGINE_API_FORWARDED_ALLOW_IPS`), or the engine builds the redirect URI with `http://` and Entra refuses it. +Behind a TLS-terminating proxy, set `api.forwarded_allow_ips` (`DFE_ENGINE_API_FORWARDED_ALLOW_IPS`), or the engine builds the redirect URI with `http://` and Entra turns it down. ## Microsoft references diff --git a/docs/control-plane/idp-setup/google.md b/docs/control-plane/idp-setup/google.md index 22357d33f..4d413799d 100644 --- a/docs/control-plane/idp-setup/google.md +++ b/docs/control-plane/idp-setup/google.md @@ -12,16 +12,20 @@ You need a Google Cloud project inside your Workspace organisation, where you ca ## How DFE reads groups -Google puts no groups in its sign-in token. So DFE asks for a read-only groups scope at sign-in, then asks the Cloud Identity API for that user's groups with the user's own token. There is no service account, no domain-wide delegation and no admin role to grant. +Google puts no groups in its sign-in token. So DFE asks for a read-only groups scope at sign-in, then asks the Cloud Identity API for that user's groups with the user's token. There is no service account, no domain-wide delegation and no admin role to grant. + +DFE signs in with Google by code exchange, then reads the user's groups from the Cloud Identity API with the user's token. ```mermaid sequenceDiagram + accTitle: L4 Detail -- Google sign-in + accDescr: The browser opens DFE sign-in and DFE redirects to Google with the Cloud Identity groups scope. DFE exchanges the code for an ID token and the user's access token, then searches the user's groups in the Cloud Identity API with that token. participant U as User's browser participant D as DFE participant G as Google sign-in participant C as Cloud Identity API U->>D: Open DFE sign-in - D->>G: Redirect with scopes openid email profile cloud-identity.groups.readonly + D->>G: Redirect with the Cloud Identity groups scope G-->>D: Authorisation code, via the redirect URI D->>G: Exchange the code with the client ID and secret G-->>D: ID token and the user's access token @@ -30,7 +34,9 @@ sequenceDiagram D-->>U: Signed in, roles from the linked groups ``` -DFE asks for nested (transitive) membership first. Google serves that only on Workspace Enterprise Standard, Enterprise Plus, Enterprise for Education and Cloud Identity Premium. On other editions DFE falls back to direct membership, so map the groups users are directly in. +*Figure 1. L4 Detail -- Google sign-in. Solid arrows are requests, dashed arrows are responses.* Up: [Identity provider setup](README.md). + +DFE asks for nested (transitive) membership first. Google serves that on Workspace Enterprise Standard, Enterprise Plus, Enterprise for Education and Cloud Identity Premium alone. On other editions DFE falls back to direct membership, so map the groups users are directly in. Google documents the Cloud Identity Groups API for apps acting for users who are not administrators. The check at the end confirms it works in your organisation: run it as a user who is not an admin. @@ -40,7 +46,7 @@ Google documents the Cloud Identity Groups API for apps acting for users who are 2. Go to **Menu > Google Auth Platform > Branding**. If the project is not configured yet, click **Get Started**. Enter an **App name** (for example `DFE`) and a **User support email**. Under **Audience** pick **Internal**, which limits sign-in to your organisation's accounts. Add a contact email, accept the User Data Policy and click **Create**. 3. Open **Data Access** and click **Add or Remove Scopes**. Add `https://www.googleapis.com/auth/cloud-identity.groups.readonly`, pasting it in as a manual scope if it is not listed. Click **Save**. 4. Open **Clients** and click **Create client**. Set **Application type** to **Web application** and give it a name. Under **Authorized redirect URIs**, add the redirect URI from the operator. Leave **Authorized JavaScript origins** empty. Click **Create**. -5. Copy the client ID and the client secret now. Google shows the secret only at creation. Afterwards it shows the last four characters, and a lost secret means rotating it. +5. Copy the client ID and the client secret at this step. Google shows the secret at creation and never again. Afterwards it shows the last four characters, and a lost secret means rotating it. 6. If your organisation restricts apps under **Admin console > Security > Access and data control > API controls**, trust this client there. **Trust internal apps** covers it. ## Send these to the operator @@ -60,9 +66,9 @@ DFE links a Google group on its Cloud Identity id: the part after `groups/` in t gcloud identity groups describe ``` -The `name` field reads `groups/`. Send the `` part. Once one of the user's groups is linked, the check below also lists the ids of every group they are in. +The `name` field reads `groups/`. Send the `` part. Once one of the user's groups is linked, the check below lists the ids of every group they are in as well. -Google leaves a group out of the answer when the group's **Who can view members** setting does not include its own members. For each group that maps to a DFE role, open **Admin console > Directory > Groups**, pick the group, open **Access Settings** and let group members view members. +Google leaves a group out of the answer when the group's **Who can view members** setting does not include the group's members. For each group that maps to a DFE role, open **Admin console > Directory > Groups**, pick the group, open **Access Settings** and let group members view members. ## Check it worked @@ -72,7 +78,7 @@ Once the operator confirms the provider is in place, open this URL in a private https:///api/v1/auth/oidc//login ``` -The page answers with JSON. `email` is the user, and `groups` lists their group ids. The JSON also holds an `access_token`: it is a live DFE session, so do not paste it anywhere. A user in no linked group is refused, and DFE keeps no account for them. +The page answers with JSON. `email` is the user, and `groups` lists their group ids. The JSON holds an `access_token` as well: it is a live DFE session, so do not paste it anywhere. A user in no linked group is turned away, and DFE keeps no account for them. ## Common failures @@ -80,7 +86,7 @@ The page answers with JSON. `email` is the user, and `groups` lists their group |---|---|---| | Google page: `Error 400: redirect_uri_mismatch` | The redirect URI on the client differs from the one DFE sent | Paste the operator's URI exactly: scheme, host, case and trailing slash all count | | Google page: `Error 403: org_internal` | The account is outside your organisation and the audience is Internal | Sign in with an account in your organisation | -| DFE page: `OIDC login failed: invalid_client: ...` | DFE holds the wrong client secret | Rotate the secret and send the new one | +| DFE page: `OIDC login failed: invalid_client: ...` | DFE holds the wrong client secret | Rotate the secret and send its value | | DFE page: `OIDC login refused: no DFE group with a role is linked to your identity provider groups.` | None of the user's groups is linked yet, or none was found: the Cloud Identity API is off in the client's project, the groups scope was not granted, API controls block the client, or the user's groups hide their members | Send the operator the ids from "Find the group ids". If they are linked, recheck steps 1, 3 and 6 and the group's **Who can view members**. The operator's log names Google's reason: `Google Cloud Identity refused the user-token group lookup (403)` | ## For the DFE operator @@ -99,7 +105,7 @@ Create the provider with `POST /api/v1/auth/oidc-providers`. The `name` becomes } ``` -A `google` provider accepts only `api` mode, and DFE turns on `enrich_on_login` for it. Leave `scopes` unset: the default carries the Cloud Identity scope. The client secret goes to the secret store and the provider keeps only its path. A service account (`groups.service_account_json`) holding a groups admin role is optional: it runs the group sync and answers a sign-in the user's token could not. Without one the group sync has no directory to list, so the provider's `last_sync_status` reads `user_token` and `sync_error` stays empty. +A `google` provider accepts `api` mode alone, and DFE turns on `enrich_on_login` for it. Leave `scopes` unset: the default holds the Cloud Identity scope. The client secret goes to the secret store and the provider keeps its path, never the value. A service account (`groups.service_account_json`) holding a groups admin role is optional: it runs the group sync and answers a sign-in the user's token could not. Without one the group sync has no directory to list, so the provider's `last_sync_status` reads `user_token` and `sync_error` stays empty. Link each DFE group by setting two fields in its group file: `source_provider` to ``, the name the provider was created with, and `source_id` to an id the admin sent. A group file with a `source_id` and no `source_provider` loads unlinked, with a warning naming it in the engine log, and no sign-in reaches it through that id: see [rbac-groups.md, section 4.4](../rbac-groups.md#44-group-sync-process). @@ -110,7 +116,7 @@ source_provider: "" source_id: "" ``` -Behind a TLS-terminating proxy, set `api.forwarded_allow_ips` (`DFE_ENGINE_API_FORWARDED_ALLOW_IPS`), or the engine builds the redirect URI with `http://` and Google refuses it. +Behind a TLS-terminating proxy, set `api.forwarded_allow_ips` (`DFE_ENGINE_API_FORWARDED_ALLOW_IPS`), or the engine builds the redirect URI with `http://` and Google turns it down. ## Google references diff --git a/docs/control-plane/idp-setup/okta.md b/docs/control-plane/idp-setup/okta.md index d10232167..c0a1dde61 100644 --- a/docs/control-plane/idp-setup/okta.md +++ b/docs/control-plane/idp-setup/okta.md @@ -12,16 +12,16 @@ You need an Okta administrator who can create app integrations and edit authoriz ## How DFE reads groups -Okta puts no groups in its tokens until you add a `groups` claim. Once you do, the ID token carries the names of the user's groups that match your filter, and DFE reads them at sign-in. DFE does not call the Okta API for this. DFE reads groups only from the ID token, never from the userinfo endpoint, so the claim must always be in the ID token. +Okta puts no groups in its tokens until you add a `groups` claim. Once you do, the ID token holds the names of the user's groups that match your filter, and DFE reads them at sign-in. DFE does not call the Okta API for this. DFE reads groups from the ID token and never from the userinfo endpoint, so the claim must always be in the ID token. -Put the claim on Okta's default custom authorization server, named `default`, with the issuer `https:///oauth2/default`. The org authorization server (issuer `https://`, no `/oauth2/...`) sends no groups claim in the ID token, so DFE sees no groups through it. +Put the claim on Okta's default custom authorization server, `default`, with the issuer `https:///oauth2/default`. The org authorization server (issuer `https://`, no `/oauth2/...`) sends no groups claim in the ID token, so DFE sees no groups through it. -A production Okta org has custom authorization servers only with API Access Management. Integrator Free Plan orgs have it. Without it, DFE's only other route is reading groups from the Okta API with an API token (see "Find the group names"), which this page does not set up. +A production Okta org gets custom authorization servers with API Access Management, and Integrator Free Plan orgs have it. Without it, DFE's one other route is reading groups from the Okta API with an API token (see "Find the group names"), which this page does not set up. ## Set it up 1. In the Admin Console go to **Applications and Resources > Applications** (**Applications > Applications** in older consoles) and select **Create App Integration**. If asked, choose the classic experience. Pick **OIDC - OpenID Connect** as the sign-in method and **Web Application** as the type. Select **Next**. -2. Name the integration (for example `DFE`). Keep the **Authorization Code** grant type. Under **Sign-in redirect URIs**, put only the redirect URI from the operator. Under **Assignments**, pick **Limit access to selected groups** and choose the groups that use DFE, or **Allow everyone in your organization to access**. Select **Save**. +2. Name the integration (for example `DFE`). Keep the **Authorization Code** grant type. Under **Sign-in redirect URIs**, put the redirect URI from the operator and nothing else. Under **Assignments**, pick **Limit access to selected groups** and choose the groups that use DFE, or **Allow everyone in your organization to access**. Select **Save**. 3. Copy the **Client ID** from the app's settings, and the **Client secret** from its **Client Credentials** section. 4. Add the groups claim to the `default` authorization server, as below. @@ -29,7 +29,7 @@ A production Okta org has custom authorization servers only with API Access Mana Go to **Security > API**, open the **Authorization Servers** tab and select `default`. -1. On the **Scopes** tab, look for a scope named `groups`. If there is none, select **Add Scope**, name it `groups`, leave user consent off and create it. DFE asks for this scope, and Okta fails a sign-in that asks for a scope the server does not define. +1. On the **Scopes** tab, look for the `groups` scope. If there is none, select **Add Scope**, name it `groups`, leave user consent off and create it. DFE asks for this scope, and Okta fails a sign-in that asks for a scope the server does not define. 2. On the **Claims** tab, select **Add Claim**. Set **Name** to `groups`. Set **Include in token type** to **ID Token** and **Always**. Set **Value type** to **Groups**, and **Filter** to **Matches regex** with a pattern for the groups DFE should see, such as `dfe-.*`. `.*` sends every group the user is in. Set **Include in** to **Any scope** and create the claim. With no scope condition the claim arrives whether or not the sign-in asks for `groups`. 3. On the **Access Policies** tab, check that an active policy applies to the DFE app, with an active rule that allows the **Authorization Code** grant for any scopes. If there is none, select **Add Policy**, assign it to the DFE app (or all clients), then **Add Rule** with that grant. The Integrator Free Plan's `default` server ships with no policy. @@ -44,9 +44,9 @@ Go to **Security > API**, open the **Authorization Servers** tab and select `def ## Find the group names -DFE links an Okta group on its name, exactly as the claim carries it. **Directory > Groups** lists them. Because the link is the name, renaming a group in Okta breaks its link, and a new group given a linked name picks up its DFE roles. Keep the claim filter narrow and limit who can create or rename groups. Once one of the user's groups is linked, the check below also lists the names the claim sends for them. +DFE links an Okta group on its name, exactly as the claim sends it. **Directory > Groups** lists them. Because the link is the name, renaming a group in Okta breaks its link, and a group created later under a linked name picks up its DFE roles. Keep the claim filter narrow and limit who can create or rename groups. Once one of the user's groups is linked, the check below lists the names the claim sends for them as well. -DFE can instead read groups from the Okta API with an API token. Groups link on their names that way too. The claim above is the normal setup: use the API only if the operator asks for it. +DFE can instead read groups from the Okta API with an API token, and groups link on their names that way as well. The claim above is the normal setup: use the API when the operator asks for it. ## Check it worked @@ -56,7 +56,7 @@ Once the operator confirms the provider is in place, open this URL in a private https:///api/v1/auth/oidc//login ``` -The page answers with JSON. `email` is the user, and `groups` lists the group names the claim sent. The JSON also holds an `access_token`: it is a live DFE session, so do not paste it anywhere. A user in no linked group is refused, and DFE keeps no account for them. +The page answers with JSON. `email` is the user, and `groups` lists the group names the claim sent. The JSON holds an `access_token` as well: it is a live DFE session, so do not paste it anywhere. A user in no linked group is turned away, and DFE keeps no account for them. ## Common failures @@ -83,7 +83,7 @@ Create the provider with `POST /api/v1/auth/oidc-providers`. The `name` becomes } ``` -The default `token_claim` mode reads the `groups` claim, and the default scopes are `openid email profile groups`. To drop the `groups` scope, send `"scopes": ["openid", "email", "profile"]`: a claim set to **Always** and **Any scope** still arrives. The API alternative is `groups.mode: api` with `groups.okta_domain` and `groups.api_token`. The sync and `groups.enrich_on_login: true` both read group names, the same identifiers the claim carries. The client secret goes to the secret store and the provider keeps only its path. +The default `token_claim` mode reads the `groups` claim, and the default scopes are `openid email profile groups`. To drop the `groups` scope, send `"scopes": ["openid", "email", "profile"]`: a claim set to **Always** and **Any scope** still arrives. The API alternative is `groups.mode: api` with `groups.okta_domain` and `groups.api_token`. The sync and `groups.enrich_on_login: true` both read group names, the same identifiers the claim sends. The client secret goes to the secret store and the provider keeps its path, never the value. Link each DFE group by setting two fields in its group file: `source_provider` to ``, the name the provider was created with, and `source_id` to a group name the admin sent. The provider is what stops another provider's sign-in asserting the same name, and a group file with a `source_id` and no `source_provider` loads unlinked, with a warning naming it in the engine log, so no sign-in reaches it through that id: see [rbac-groups.md, section 4.4](../rbac-groups.md#44-group-sync-process). @@ -94,7 +94,7 @@ source_provider: "" source_id: "" ``` -A linked group admits its members only once it carries a role, and until one does no Okta user can sign in, so link the first group as the local admin or the break-glass account. Behind a TLS-terminating proxy, set `api.forwarded_allow_ips` (`DFE_ENGINE_API_FORWARDED_ALLOW_IPS`), or the engine builds the redirect URI with `http://` and Okta refuses it. +A linked group admits its members once it holds a role, and until one does no Okta user can sign in, so link the first group as the local admin or the break-glass account. Behind a TLS-terminating proxy, set `api.forwarded_allow_ips` (`DFE_ENGINE_API_FORWARDED_ALLOW_IPS`), or the engine builds the redirect URI with `http://` and Okta turns it down. ## Okta references diff --git a/docs/control-plane/oidc-rbac-architecture.md b/docs/control-plane/oidc-rbac-architecture.md index 7b969acd3..53d445ae1 100644 --- a/docs/control-plane/oidc-rbac-architecture.md +++ b/docs/control-plane/oidc-rbac-architecture.md @@ -1,238 +1,217 @@ # OIDC + RBAC architecture and design -The single source of truth for how dfe-engine handles authentication, -external OIDC, group->role mapping, the two RBAC planes, the management API -and the infrastructure it needs. The RBAC/multi-tenant ClickHouse detail lives in -[rbac.md](rbac.md), and this doc references it rather than repeating it. - -Exhaustive reference (exact request/response bodies, tofu outputs, secret -keys) lives with its source of truth - `openapi.json`, the code -(`src/dfe_engine/auth/oidc/`), and the dfe-infra tofu modules - not -transcribed here where it would drift. - -## Contents - -1. The auth model - local always, OIDC additive -2. The layered pipeline -3. Layer 2 - the provider registry and adapters -4. The two RBAC planes -5. The management API and CLI -6. Infrastructure requirements -7. Provider-agnostic onboarding recipe +How dfe-engine handles authentication, external OIDC, group to role mapping, the +two RBAC planes, the management API and the infrastructure they need. The RBAC and +multi-tenant ClickHouse detail is in [rbac.md](rbac.md), and this page links it +rather than repeating it. + +Request and response bodies, tofu outputs and secret keys stay with their +sources: `openapi.json`, the code (`src/dfe_engine/auth/oidc/`) and the +dfe-infra tofu modules. ## 1. The auth model - local always, OIDC additive -The engine is the RP and the **single token issuer**. An external IdP does the -interactive login. The engine validates the IdP id_token (signature/nonce/aud/ -exp against the IdP JWKS, via Authlib) and re-mints its own ES384 token. -Downstream (dfe-ui, dfe-hyperdx) only ever sees the engine token - the IdP's -token stays on the engine<->IdP leg. Code: `src/dfe_engine/auth/oidc/rp.py`, -`src/dfe_engine/api/v1/oidc_login.py`. - -By design several auth paths run at once and merge. OIDC never replaces local -auth, it augments it: - -- Local accounts (username/password -> JWT), API keys and the - disabled/dev path are always available - the fallback that keeps an - operator in even if the IdP is down or misconfigured. -- External OIDC is added when provided (recommended for real deployments). - When present it takes precedence, but the local and key paths stay live. +The engine is the relying party and the **single token issuer**. An external IdP +does the interactive login. The engine validates the IdP's id_token (signature, +nonce, audience and expiry against the IdP's JWKS, through Authlib) and re-mints +an ES384 token of its making. dfe-ui and dfe-hyperdx see the engine token and +nothing else: the IdP's token stays on the engine-to-IdP leg. Code: +`src/dfe_engine/auth/oidc/rp.py`, `src/dfe_engine/api/v1/oidc_login.py`. + +Several auth paths run at once and merge, so OIDC adds to local auth and never +replaces it: + +- Local accounts (username and password to JWT), API keys and the disabled dev + path are always available, which keeps an operator in when the IdP is down or + misconfigured. +- External OIDC is added when provided, and is the path for a production + deployment. When present it takes precedence, but the local and key paths stay live. - Groups from both sources unify. An IdP group takes a DFE group's roles - only through that group file's `source_provider` + `source_id` link. A - matching name grants nothing (rbac-sign-in.md section 2.1). One role model, - two merged identity sources. + through that group file's `source_provider` + `source_id` link and no other + way. A matching name grants nothing (rbac-sign-in.md section 2.1). -The four auth paths, precedence, and the header-trust security precondition are -detailed in rbac.md section 1. +The four auth paths, their precedence and the header-trust precondition are in +rbac.md section 1. ## 2. The layered pipeline -The whole design confines provider chaos to Layer 2 and reuses the existing -engine machinery above it: - -```text - [0] Trust / issuer external IdP authenticates -> engine validates the - IdP id_token (JWKS) -> engine RE-MINTS its own ES384 - token. Downstream sees ONLY the engine token. - [1] Identity NormalizedIdentity { subject(sub), email, raw_groups } - [2] Group resolution <-- ALL provider-specific mess lives HERE, behind an - (per-provider adapter. Raw token refs (names | ids | nothing) - adapter) -> NormalizedGroup { id, name, source }. - token_claim mode: refs in token. - api mode: refs absent/id-only -> call the - provider directory API. - engine group file (config/auth/groups/*.yaml) - -> roles + org_ids + scope - | - +-------------------------+---------------------------+ - [3a] CONTROL-PLANE axis [3b] DATA-PLANE axis - group -> static roles group -> org_ids - (admin, data_analyst, (DYNAMIC: presence of - infra_admin, ...) org_ids confers scoped - [4a] dfe-engine authorization access - no named per-org - roles -> permissions -> role) - what you can DO (manage [4b] Observe data browsing - sources/hunts/config/deploy) (Kibana-style) - org_ids -> dfe-hyperdx team + - the org's pinned CH user - -> row-filtered DATA VIEW - ONLY (read, no control) +Provider-specific code stops at one layer, the group adapter, and the engine +machinery above it is the same for every IdP. A sign-in goes from the IdP through +the engine's relying party and the group files to both RBAC planes. + +```mermaid +flowchart LR + accTitle: L4 Detail -- OIDC sign-in layers + accDescr: The external IdP authenticates the user and hands the engine an id_token. The engine's relying party validates it and re-mints an engine token. The provider adapter turns the raw group references into group ids, the group files map them to roles and org ids, and those feed the control plane and the data plane. + + idp["External IdP"] + rp["Relying party"] + adapt["Provider adapter"] + files[("Group files")] + ctrl["Control plane
roles to actions"] + data["Data plane
orgs to rows"] + + idp -->|id_token| rp + rp -->|groups| adapt + adapt -->|ids| files + files --> ctrl + files --> data + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + class idp ext + class adapt core + class files store ``` -Two principles: +*Figure 1. L4 Detail -- OIDC sign-in layers. The blue box is the one layer that knows the provider.* Up: [rbac.md](rbac.md). -1. **Provider-specific code stops at Layer 2.** Below it is generic OIDC - plumbing. Above it, generic machinery that does not care which IdP the user - came from. Adding a provider = one adapter at Layer 2, nothing else changes. -2. **Control-plane and data-plane are orthogonal.** The same group membership - feeds both, but control-plane roles decide what you can DO while org_ids - decide what data you can SEE. A pure customer gets ONLY the data-plane axis. +| Layer | What it does | +|---|---| +| Relying party | Validates the IdP's id_token against its JWKS and re-mints an ES384 engine token. `NormalizedIdentity` holds the subject, email, name and raw group references | +| Provider adapter | Turns raw references (names, ids or nothing) into group ids. In `token_claim` mode the references are in the token. In `api` mode the adapter calls the provider's directory API | +| Group files | `config/auth/groups/*.yaml` map a linked group id to roles, `org_ids` and a scope | +| Control plane | Static roles (`admin`, `data_analyst`, `infra_admin`, ...) to permissions: what you can DO with sources, hunts, config and deploys | +| Data plane | `org_ids` to the org's dfe-hyperdx team and pinned ClickHouse user: a row-filtered view of data, read-only, with no control | -## 3. Layer 2 - the provider registry and adapters +Two principles follow. Adding a provider is one adapter, and nothing else changes. +The control plane and data plane are orthogonal: the same group membership feeds +both, roles decide what you can DO and `org_ids` decide what data you can SEE, and +a pure customer gets the data plane alone. -### Provider config +## 3. The provider registry Providers are YAML files (one per provider, filename stem = name) managed by -`OIDCProviderRegistry`. The model (`src/dfe_engine/auth/oidc/models.py`, the SSoT) carries: -`type` (generic|google|entra_id|okta), `enabled`, `issuer`, `client_id`, +`OIDCProviderRegistry`. The model (`src/dfe_engine/auth/oidc/models.py`) holds +`type` (generic, google, entra_id or okta), `enabled`, `issuer`, `client_id`, `client_id_env`, `client_secret_path`, `client_secret_env`, `scopes`, and a -`groups` block (mode + claim_name + per-provider directory fields). +`groups` block (mode, claim name and the per-provider directory fields). -A provider file that does not load, whether its YAML is unreadable or its fields fail validation, is left out of the provider list, the relying party and the sync scheduler. The engine logs it once and counts it on `auth_oidc_providers_skipped_total{reason}` (`unreadable`, `invalid`), and every other provider carries on. +A provider file that does not load, with unreadable YAML or fields that fail validation, is left out of the provider list, the relying party and the sync scheduler. The engine logs it once and counts it on `auth_oidc_providers_skipped_total{reason}` (`unreadable`, `invalid`), and every other provider keeps working. -A secret is never stored in config: the client id is not secret and is held in -the clear, every other credential is a PATH into the `DfeSecrets` store or the +A secret is never stored in config. The client id is not secret and is held in +the clear. Every other credential is a PATH into the `DfeSecrets` store or the NAME of an env var. Resolution reads the store before the environment, so a -secret sent to the API works immediately while an ESO-mounted env var keeps -serving a provider configured that way. - -### Group resolution modes - -- `token_claim` (the default) - groups arrive as strings in the id_token. Direct lookup. -- `api` - the token lacks group names (or carries only ids). An adapter calls - the provider directory API to resolve names and enumerate membership on a - schedule. Each provider syncs every `groups.sync_interval` seconds (60 at least) from a background loop in every engine replica. `DFE_ENGINE_AUTH_OIDC_GROUP_SYNC_ENABLED=false` leaves only `POST /oidc-providers/{name}/sync`. -- `manual` - membership mapped by hand in the engine. The token's groups are ignored at login and nothing syncs. - -Which fields each provider type accepts and requires in each mode is enforced by the provider admin API, from the rules in `src/dfe_engine/auth/oidc/field_rules.py`. A `generic` provider has no `api` mode and a `google` provider has only `api` mode, with `enrich_on_login` on. - -## 3.1 The normalized group - -Every provider's groups normalise to `NormalizedGroup { id, name, source }`. -`GroupInfo { id, name, email, description }` (models.py) already provides this -for api-mode. The token_claim/login path (`extract_identity`, rp.py) emits a flat -`list[str]`, and the matching layer resolves it: each group file is linked by `source_provider` and that provider's -`source_id`, and a login takes the groups whose `source_id` it asserts, for the -provider it signed in through (`membership.linked_groups`). A group's name links -nothing. So an Entra login carrying object GUIDs - which match no group filename - -resolves against `source_id` and lands the right roles. The sync, SCIM's `externalId` and an admin through the groups API or dfe-ui's group form each set the link, so a `token_claim` login carrying names (Okta, dex, Keycloak) lands its roles once a group's `source_id` is the name the IdP sends. -The adapter is the ONLY place the remaining quirks live: - -| Quirk | Real cause | Adapter's job | Status | -|-------|-----------|---------------|--------| -| refs are names (dex/okta) vs ids (entra) vs absent (google) | provider design | normalise to {id,name}. Resolve GUIDs by source_id, call the directory API when the token lacks the name | id->name resolution by source_id match | -| id stable, name mutable | directories rename groups | map roles on `id`, display `name` | source_id is the stable key | -| overage (entra >200 -> a Graph pointer) | token size limits | detect the marker, fall back to the directory API | `rp.py` detects `_claim_names` and fetches `transitiveMemberOf` with the user's own login token (`/me`: User.Read returns each group's id, which is all role resolution needs. `GroupMember.Read.All` adds names), falling back to app credentials (tenant id + client secret) when Graph refuses that token | -| nested / transitive groups | AD/entra/google nesting | take the closure the provider already emits, or walk it when it does not | entra emits the full closure in-token. Honour it (per-provider fact, not a global toggle) | -| name collisions across tenants | bare names aren't unique | namespace the key (source+domain) | a name held by a group not linked to that IdP group is skipped (`name_taken`). Namespacing not yet built | -| non-role groups (mailing lists) | not all groups grant access | filter by group type | not yet built | - -## 3.2 Adapters - -`src/dfe_engine/auth/oidc/adapters/`: `generic` (token_claim/manual, no API), `google` (Cloud -Identity Groups API with the user's own login token. The Admin SDK Directory API -as an optional service account), `entra` (Microsoft Graph), `okta` (Groups API). Contract: -`list_all_groups`, `resolve_user_groups`, `test_connection`. The two a login can -reach never raise: `resolve_user_groups` reports an outage or refusal. An outage -keeps the token's groups, else the account's. Anything else replaces them. -`list_all_groups` serves the sync only and raises when a page fails, so the sync +secret sent to the API works at once, and an ESO-mounted env var keeps serving a +provider configured that way. + +Three group resolution modes: + +- `token_claim`, the default: groups arrive as strings in the id_token and are looked up directly. +- `api`: the token lacks group names or holds ids alone, so an adapter calls the + provider's directory API to resolve names and list membership on a + schedule. Each provider syncs every `groups.sync_interval` seconds (60 at least) from a background loop in the engine. `DFE_ENGINE_AUTH_OIDC_GROUP_SYNC_ENABLED=false` leaves `POST /oidc-providers/{name}/sync` as the one way to sync. +- `manual`: membership is mapped by hand in the engine. The token's groups are ignored at login and nothing syncs. + +The provider admin API enforces which fields each provider type accepts and needs in each mode, from `src/dfe_engine/auth/oidc/field_rules.py`. A `generic` provider has no `api` mode, and a `google` provider has `api` mode alone, with `enrich_on_login` on. + +## 4. Group identifiers, adapters and sync + +The api-mode adapters return `GroupInfo { id, name, email, description }` +(models.py), with `id` the identifier the provider's sign-ins assert. The +token_claim path (`extract_identity`, rp.py) emits a flat `list[str]` of +identifiers. Either way a group file is linked by `source_provider` and that +provider's `source_id`, and a login takes the groups with a `source_id` it asserts, +for the provider it signed in through (`membership.linked_groups`). A group's name +links nothing, so an Entra login with object GUIDs resolves against `source_id` +and lands the right roles. The sync, SCIM's `externalId` and an admin through the +groups API or dfe-ui's group form each set the link. + +`src/dfe_engine/auth/oidc/adapters/` holds `generic` (token_claim or manual, no API), `google` (the Cloud +Identity Groups API with the user's login token, and the Admin SDK Directory API +for an optional service account), `entra` (Microsoft Graph) and `okta` (Groups API). Each implements +`list_all_groups`, `resolve_user_groups` and `test_connection`. `resolve_user_groups`, the one a login reaches, never raises: it reports an outage or a refusal. An outage +keeps the token's groups, else the account's, and anything else replaces them. +`list_all_groups` serves the sync and raises when a page fails, so the sync never reads a short listing as the whole directory. -## 3.3 Sync and control-plane independence +`src/dfe_engine/auth/oidc/sync.py` lists a provider's groups in api mode. It creates a group file for each unseen group, linked by `source_provider` and `source_id`, and updates the files linked to that IdP group, keeping their roles ([rbac-groups.md](rbac-groups.md#44-group-sync-process) has the skips). Detaching a provider does NOT delete its groups: they orphan with `source_provider` set, and the delete answer lists them. Sign-in needs the engine, the relying party and one token issuer, so with the engine down nobody can sign in and provider CRUD and group sync stop as well. Running pipelines keep running, because the data path does not call the engine. -`src/dfe_engine/auth/oidc/sync.py` enumerates a provider's groups (api mode). It creates a group file for each new one, linked by `source_provider` and `source_id`, and updates only files already linked to that IdP group (their `source_id` is its id and their `source_provider` this provider, or one `auth.source_provider_bindings` joins to it), preserving their roles. A file of the same name with no link is skipped as `name_taken`: a display name is no identity, so linking one is an admin's act through the groups API or dfe-ui's group form (rbac-groups.md section 4.4). A new group whose name is a stored group's `source_id` or whose id is a stored group's name is skipped as `name_taken` too, since an API key resolves an identifier by name, then by `source_id`. A file carrying the id under any other provider, whatever its name, is skipped as `provider_mismatch`, so a sync never moves a group between providers or creates a second group carrying the id. Detaching a provider does NOT delete its groups - they orphan with `source_provider` set, reported on delete. +The adapter is where the provider quirks live: -**Sign-in needs the engine.** It is the relying party and the only token -issuer, so with the engine down nobody can sign in, and provider CRUD and group -sync stop too. Running pipelines carry on: the data path does not call the -engine. +| Quirk | Cause | Adapter's job | Status | +|-------|-----------|---------------|--------| +| refs are names (dex, Okta) vs ids (Entra) vs absent (Google) | provider design | resolve ids by `source_id`, and call the directory API when the token lacks the name | built: id resolution by `source_id` match | +| id stable, name mutable | directories rename groups | map roles on `id`, display `name` | built: `source_id` is the stable key | +| overage (Entra over 200 groups gives a Graph pointer) | token size limits | detect the marker, fall back to the directory API | built: `rp.py` detects `_claim_names` and fetches `transitiveMemberOf` with the user's login token (`/me`: User.Read returns each group's id, which is all role resolution needs, and `GroupMember.Read.All` adds names), falling back to app credentials (tenant id + client secret) when Graph turns that token down | +| nested groups | AD, Entra and Google nesting | take the closure the provider emits, or walk it when it does not | built: Entra emits the full closure in the token, and it is honoured per provider, not by a global toggle | +| name collisions across tenants | bare names are not unique | namespace the key (source + domain) | partial: a name held by a group not linked to that IdP group is skipped (`name_taken`). Namespacing is not built | +| non-role groups (mailing lists) | not all groups grant access | filter by group type | not built | -## 4. The two RBAC planes +## 5. The two RBAC planes -One group resolution (`deps.py _resolve_group_grants` + the `Group` model) feeds -both. A group file carries `roles`, `org_ids`, and `scope` -(`system`|`org:`). A user's roles and org_ids are the union across groups. +One group resolution (`deps.py _resolve_group_grants` and the `Group` model) feeds +both planes. A group file holds `roles`, `org_ids` and `scope` +(`system` or `org:`). A user's roles and `org_ids` are the union across groups. -- Control plane (3a/4a): `roles.yaml` + wildcard `permission_matches` gate - the dfe-engine API (manage sources/hunts/config/deploy). Static named roles. -- Data plane (3b/4b): org_ids drive dynamic, data-only access. A caller fenced to one org (an `org_viewer`, or any role bound at that org's scope) browses Observe as the org's pinned ClickHouse user, whose READONLY tenant setting and the shared RESTRICTIVE row policies filter every query to that org (rbac.md section 5). Read-only. No control-plane power. +- Control plane: `roles.yaml` and the wildcard `permission_matches` gate + the dfe-engine API (manage sources, hunts, config and deploys). Static roles. +- Data plane: `org_ids` drive data-only access. A caller fenced to one org (an `org_viewer`, or any role bound at that org's scope) browses Observe as the org's pinned ClickHouse user. That user's READONLY tenant setting and the shared RESTRICTIVE row policies filter every query to that org (rbac.md section 5). Read-only, with no control-plane power. -Both axes resolve from the group files. The re-minted token carries +Both planes resolve from the group files. The re-minted token holds `{sub, email, groups}` plus the `role` claim dfe-hyperdx reads (see -auth-envoy-topology.md). Roles AND org_ids still derive engine-side from the -group files, not the token. Full detail: rbac.md sections 2, 5, 6, 7. - -## 5. The management API and CLI - -Admin CRUD lives at `/api/v1/auth/oidc-providers` (`src/dfe_engine/api/v1/oidc_providers.py`): -create / list / get / update / delete (delete reports orphaned groups) / -`{name}/sync` / `{name}/test` / `{name}/verify-login`. RBAC scopes -`oidc_read|write|delete`, audit on every mutation, pagination on list. The `dfe` -CLI gets these for free - generated from `openapi.json` - so UI and CLI share one -CRUD surface. Create/update take the credential VALUES (`client_secret`, the -Okta API token, the Entra client secret, the Google SA JSON), write them to the -secret store and record only the path, then re-register the provider with the -relying party - so a provider configured purely through the API completes a -login without an env edit or a restart. A `*_env` field takes an environment -variable NAME. A credential pasted there is a 422 naming the field. Detach -deletes the provider's stored secrets. - -Verification for a thorough onboard/offboard workflow: +[auth-envoy-topology.md](../auth-envoy-topology.md)). Roles and `org_ids` derive +engine-side from the group files, not the token. Full detail: rbac.md sections 2, 5, 6 and 7. + +## 6. The management API and CLI + +Admin CRUD is at `/api/v1/auth/oidc-providers` (`src/dfe_engine/api/v1/oidc_providers.py`): +create, list, get, update, delete (which lists orphaned groups), `{name}/sync`, +`{name}/test` and `{name}/verify-login`, under the RBAC scopes +`oidc_read`, `oidc_write` and `oidc_delete`, with an audit event on every mutation and pagination on list. The `dfe` +CLI is generated from `openapi.json`, so UI and CLI share one CRUD surface. +Create and update take the credential VALUES (`client_secret`, the Okta API +token, the Entra client secret, the Google service account JSON), write them to +the secret store and record the path, then re-register the provider with the +relying party. A provider configured through the API alone completes a login with +no env edit and no restart. A `*_env` field takes an environment variable NAME, +and a credential pasted there is a 422 naming the field. Detach deletes the +provider's stored secrets. + +Two checks help an onboard or offboard: - `GET /{name}/verify-login` reports whether the client_id and client_secret - resolve (value never returned) and whether the issuer's discovery document is - reachable and well-formed. `{name}/test` checks the group-directory credentials - only, and is a no-op for generic providers. -- No single endpoint returns raw claims, normalized groups, resolved roles and - org_ids for a login. Assemble them from the callback JSON - (`{subject,email,groups}`) and `GET /auth/me` (roles and org_ids). + resolve (the value is never returned) and whether the issuer's discovery + document is reachable and well-formed. `{name}/test` checks the group-directory + credentials, and does nothing for a generic provider. +- No single endpoint returns raw claims, groups, resolved roles and + `org_ids` for a login. Assemble them from the callback JSON + (`{subject,email,groups}`) and `GET /auth/me` (roles and `org_ids`). -## 6. Infrastructure requirements +## 7. Infrastructure requirements -What a deployment must provide (dfe-infra owns these and the engine consumes them). +A deployment provides these, dfe-infra owns them and the engine consumes them: - OIDC client credentials per provider, as env vars from a K8s Secret - (materialised by ESO from OpenBao / cloud SM). Env-var NAMES are declared in - the provider config. The chart mounts the values. Per-provider secret keys - (client-id, client-secret, and provider extras - Google SA JSON, Entra - tenant-id, Okta api-token) are defined by the tofu modules, not here. -- Cloud IAM provisioning (tofu): `tf-oidc-google` (OAuth client is - console-only - the module does the group-resolver SA), `tf-oidc-entra` (app - registration + admin-consented `GroupMember.Read.All`), `tf-oidc-okta` (OIDC - app + groups claim + API token). Modules live in dfe-infra. This engine just - reads the resulting creds. -- DNS + TLS for the callback host. The native RP callback is - `https://{dfe_host}/api/v1/auth/oidc/{provider}/callback` (+ localhost for dev). -- Network egress to IdP directory APIs for api-mode group resolution - (`cloudidentity.googleapis.com`, and the Admin SDK at `www.googleapis.com` when a - service account is set, `graph.microsoft.com`, `{org}.okta.com`). -- The edge header path, a fallback only. dfe-infra's gateway can verify an + (materialised by ESO from OpenBao or a cloud secrets manager), or sent to the API. The provider config gives the env var NAMES, and the chart mounts the values. The tofu modules define the per-provider secret keys (client-id, client-secret, and extras such as the Google service account JSON, the Entra tenant-id and the Okta api-token). +- Cloud IAM provisioning in tofu: `tf-oidc-google` (the OAuth client is + console-only, so the module creates the group-resolver service account), `tf-oidc-entra` (app + registration and admin-consented `GroupMember.Read.All`) and `tf-oidc-okta` (OIDC + app, groups claim and API token), under dfe-infra `terraform/modules/`. The engine reads the resulting credentials. +- DNS and TLS for the callback host. The relying-party callback is + `https://{dfe_host}/api/v1/auth/oidc/{provider}/callback` (plus localhost for dev). +- Network egress to the IdP directory APIs for api-mode group resolution: + `cloudidentity.googleapis.com`, the Admin SDK at `www.googleapis.com` when a + service account is set, `graph.microsoft.com` and `{org}.okta.com`. +- The edge header path, a fallback. dfe-infra's gateway can verify an IdP token and inject `X-Oidc-Subject` (`jwtAuthn`, off by default), and the - engine trusts it only with `auth.trust_proxy_auth_headers`. Envoy Gateway cannot + engine trusts it with `auth.trust_proxy_auth_headers` on and not otherwise. Envoy Gateway cannot map an array claim to a header, so `X-Oidc-Groups` never arrives and such a login - holds no roles (dfe-infra `helm/edge/gateway/values.yaml`, dfe-infra#222). The - native RP is the login path. When the header path is used, the engine MUST be - reachable only via Envoy (NetworkPolicy), or the headers are forgeable. + holds no roles (dfe-infra `helm/edge/gateway/values.yaml`). The + relying party is the login path. When the header path is used, Envoy must be + the one way to reach the engine (NetworkPolicy), or the headers can be forged. -## 7. Provider-agnostic onboarding recipe +## 8. Provider-agnostic onboarding recipe -Because the mess is quarantined at Layer 2, onboarding any real external OIDC is -fixed: +Because the provider code stays in the adapter, onboarding an external OIDC IdP +takes four steps: -1. Register a provider config (type + issuer + client env-var names + groups mode/claim). -2. The IdP emits whatever it emits, and the Layer-2 adapter normalises it. -3. Author group files mapping the normalised group id -> roles + org_ids. -4. Layers 3-4 do the rest unchanged. +1. Register a provider config: type, issuer, client credentials and the groups mode. +2. The IdP emits whatever it emits, and the adapter turns it into group ids. +3. Link group files to those ids, with their roles and `org_ids`. +4. The group files, the control plane and the data plane do the rest unchanged. -Canonical role + group names are in [rbac-vocabulary.md](rbac-vocabulary.md) - -the SSoT every consumer (engine, ui, infra, deploy, schemas, test envs) aligns to. +The role and group names are in [rbac-vocabulary.md](rbac-vocabulary.md), which +every consumer (engine, ui, infra, deploy, schemas, test environments) aligns to. diff --git a/docs/control-plane/rbac-engine-reads.md b/docs/control-plane/rbac-engine-reads.md index acda35bd4..f40c7e52f 100644 --- a/docs/control-plane/rbac-engine-reads.md +++ b/docs/control-plane/rbac-engine-reads.md @@ -1,32 +1,38 @@ -# RBAC - the engine's own reads +# RBAC - the reads the engine makes for a caller -Section 5.3 of [rbac.md](rbac.md), which keeps its number so section references resolve. +This page is section 5.3 of [rbac.md](rbac.md), kept under that number so section references resolve. -## 5.3 The Engine's Own Reads +## 5.3 Row reads for a caller The query API does not lean on a ClickHouse user for tenancy: the executor injects the caller's `org_id` as a reserved bind parameter that a client cannot override (`src/dfe_engine/query/executor.py`, `RESERVED_PARAMS`). -The row reads the engine makes for a caller with no platform grant -- the sampler, the transform dry run, and the schema routes `sample-rows`, `json-paths` and `promote-field` -- bind `_org_id` to the tenant ids of the registered orgs a grant holding the read's action reaches (`api/deps.py::held_orgs`), the same ids each org's pinned user carries (`src/dfe_engine/orgs/tenant_scope.py`). A grant bound at an org's scope reaches that org. A tenant role bound system-wide (`org_viewer`, or a `scoped` role) reaches only the orgs its own group names, never one that only another of the caller's groups names, as its group's ClickHouse user does. The sampler, `/operations`, `sample-rows` and `json-paths` accept that action at system scope or at one of the caller's orgs. `promote-field` changes a source every org shares, so it still needs `schema:write` at system scope. Each org marker the caller carries resolves the same way as a group's (`resolve_orgs`): by name, or failing that by the one org declaring it as a tenant id. A marker no org declares, or a tenant id two orgs share, binds nothing, and a caller left with no tenant id, or a table with no `_org_id`, is refused before any row is read. `/operations` shows such a caller only the operations held to its orgs. +The row reads the engine makes for a caller with no platform grant -- the sampler, the transform dry run, and the schema routes `sample-rows`, `json-paths` and `promote-field` -- bind `_org_id` to the tenant ids of the registered orgs a grant holding the read's action reaches (`api/deps.py::held_orgs`). Those are the ids each org's pinned user holds (`src/dfe_engine/orgs/tenant_scope.py`). A grant bound at an org's scope reaches that org. A tenant role bound system-wide (`org_viewer`, or a `scoped` role) reaches the orgs its group lists and none another of the caller's groups lists, as its group's ClickHouse user does. + +The sampler, `/operations`, `sample-rows` and `json-paths` accept that action at system scope or at one of the caller's orgs. `promote-field` changes a source every org shares, so it still needs `schema:write` at system scope. Each org marker the caller holds resolves the way a group's does (`resolve_orgs`): by name, or failing that by the one org declaring it as a tenant id. A marker no org declares, or a tenant id two orgs share, binds nothing. A caller left with no tenant id, or a table with no `_org_id`, is turned away before any row is read, and `/operations` shows such a caller the operations held to its orgs and no others. ## Source reads -`GET /api/v1/sources` and every other source read (`/sources/{name}`, `/columns`, `/flow`, `/signals`, `/export`, `/versions/{version}`, `/catalogue`, `/catalogue-tags`, `/engines`) accept `source:read` at system scope or at one of the caller's orgs. A source definition is shared by every org, so every such caller lists those. A fetcher source belongs to the org it names (the default org when it names none), so a caller without a platform grant lists the fetcher sources of its held orgs only. A named read of a fetcher source bound to an org the caller does not hold gets a 403 `org_not_granted`. A caller with no `source:read` at system scope or in any org gets the 403. The same rule covers `GET /schemas/{source}/columns`, `GET /kafka/topics` (a held caller sees the topics of the sources it may read) and `GET /ingest/rules` (a held caller sees the blocks of those sources). `GET /ingest/compat` and `/ingest/compat-profiles` are the deployment's own bindings and profiles, shared by every source, so any `source:read` holder reads them. `GET /hyperdx/sources` lists every org's dfe-hyperdx team, so it needs `source:read` at system scope and refuses a caller held to its orgs. +`GET /api/v1/sources` and every other source read (`/sources/{name}`, `/columns`, `/flow`, `/signals`, `/export`, `/versions/{version}`, `/catalogue`, `/catalogue-tags`, `/engines`) accept `source:read` at system scope or at one of the caller's orgs. A source definition is shared by every org, so every such caller lists those. A fetcher source belongs to the org it names (the default org when it names none), so a caller without a platform grant lists the fetcher sources of its held orgs and no others. A read by name of a fetcher source bound to an org the caller does not hold gets a 403 `org_not_granted`, and a caller with no `source:read` at system scope or in any org gets the 403. + +The same rule covers `GET /schemas/{source}/columns`, `GET /kafka/topics` (a held caller sees the topics of the sources it may read) and `GET /ingest/rules` (a held caller sees the blocks of those sources). `GET /ingest/compat` and `/ingest/compat-profiles` are the deployment's bindings and profiles, shared by every source, so any `source:read` holder reads them. `GET /hyperdx/sources` lists every org's dfe-hyperdx team, so it needs `source:read` at system scope and turns away a caller held to its orgs. ## Hunt reads -The hunt runner's ClickHouse user reads every org, so every statement it compiles carries `_org_id IN ()` as its own conjunct, and a rule condition may read no other table except one in `hunts.lookup_databases`. The alert-volume preview a rule write runs carries the same conjunct over the customers of each hunt that runs or names the rule, cut to the orgs the author may read with `rule:write`. A rule no hunt runs is counted over the author's orgs alone, every org for a platform author, and a fence left with no org leaves the rule unmeasured. A listed database holds shared reference data and no org's rows. The runner refuses to start, or stops at its next reload, while one of its tables has an `_org_id` column or is not a plain store of its own rows, as a view is. The engine refuses to start on one too when ClickHouse answers at its start, and starts with the check unrun when it does not ([source-detections.md](../data-plane/source-detections.md#lookup-databases)). +The hunt runner's ClickHouse user reads every org, so every statement it compiles holds `_org_id IN ()` as a separate conjunct, and a rule condition may read no other table except one in `hunts.lookup_databases`. The alert-volume preview a rule write runs adds the same conjunct over the customers of each hunt that runs or lists the rule, cut to the orgs the author may read with `rule:write`. A rule no hunt runs is counted over the author's orgs alone, every org for a platform author, and a fence left with no org leaves the rule unmeasured. + +A listed database holds shared reference data and no org's rows. The runner will not start, or stops at its next reload, when one of its tables has an `_org_id` column or is not a plain store of rows, as a view is not. The engine will not start on one either when ClickHouse answers at its start, and starts with the check unrun when it does not ([source-detections.md](../data-plane/source-detections.md#lookup-databases)). ## Usage reads -`GET /api/v1/usage` and `GET /api/v1/usage/overview` read the usage rollup (`usage_daily` and `usage_minute`) and need `usage:read` at system scope or at the scope of one of the caller's orgs ([rbac.md](rbac.md#23-permission-taxonomy)). The deciding step is the same one the other reads use: whether the caller's grant reads every org for `usage:read` (`api/deps.py::reads_every_org`). +`GET /api/v1/usage` and `GET /api/v1/usage/overview` read the usage rollup (`usage_daily` and `usage_minute`) and need `usage:read` at system scope or at the scope of one of the caller's orgs ([rbac.md](rbac.md#23-permission-taxonomy)). The deciding step is the one the other reads use: whether the caller's grant reads every org for `usage:read` (`api/deps.py::reads_every_org`). -A caller whose grant reads every org sees the deployment-wide rows, and may name any registered org with `org`. It can read all four metrics. An unregistered org answers `422 unknown_org`, and an org named for `cores_deployed` or `node_cores` answers `422 not_per_org`, because those two are measured for the deployment. +A caller with a grant that reads every org sees the deployment-wide rows, and may name any registered org with `org`. It can read all four metrics. An unregistered org answers `422 unknown_org`, and an org given for `cores_deployed` or `node_cores` answers `422 not_per_org`, because those two are counted for the deployment. -Any other caller is held to the tenant ids of the registered orgs its groups name (`resolve_orgs`) and grant `usage:read` at. An org where its groups grant only some other role is never read. It reads: +Any other caller is held to the tenant ids of the registered orgs its groups name (`resolve_orgs`) and grant `usage:read` at. An org where its groups grant some other role and not `usage:read` is never read. It reads: - `ingested_bytes` and `events_per_second` for those orgs added together, or for the one it names with `org`. -- The two core counts as `not_permitted` with no series. Naming one of its own orgs changes nothing for them, and the response is still `200`. +- The two core counts as `not_permitted` with no series. Naming one of its orgs changes nothing for them, and the response is still `200`. -It cannot read the deployment-wide row, the one with an empty `_org_id`, and it cannot read an org it does not hold. Naming one answers `403 org_not_granted`, registered or not. A caller whose groups name no registered org answers `403 no_org`. +It cannot read the deployment-wide row, the one with an empty `_org_id`, and it cannot read an org it does not hold. Naming one answers `403 org_not_granted`, registered or not. A caller with groups that name no registered org answers `403 no_org`. Events per second over more than one tenant id is the busiest minute of their sum, taken from `usage_minute`, which keeps 7 days. A held caller with several orgs therefore gets no events-per-second days older than 7. A range with no row reads as `not_measured`, and a day with no row is left out of the series, never 0. diff --git a/docs/control-plane/rbac-groups.md b/docs/control-plane/rbac-groups.md index 7796a3bfc..0f42a707f 100644 --- a/docs/control-plane/rbac-groups.md +++ b/docs/control-plane/rbac-groups.md @@ -1,88 +1,105 @@ # RBAC - IdP group sync and links -Section 4.4 of [rbac.md](rbac.md), which keeps its number so section references resolve. +This page is section 4.4 of [rbac.md](rbac.md), kept under that number so section references resolve. ## 4.4 Group Sync Process +A sync run for an api-mode provider lists every remote group and updates, creates or skips each one (`src/dfe_engine/auth/oidc/sync.py`). + ```mermaid sequenceDiagram - participant Sync as sync_provider() - participant Reg as OIDCProviderRegistry - participant Adapter as OIDC Adapter + accTitle: L4 Detail -- OIDC group sync + accDescr: The sync reads the provider from the registry, lists every group through the provider's adapter, and reads the stored groups. Each remote group then updates, creates or is skipped, and the sync writes the run's time and status back to the registry. + participant Sync as sync_provider + participant Reg as Provider registry + participant Adapter as Provider adapter participant GS as GroupStore - - Sync->>Reg: get(provider_name) - Reg-->>Sync: OIDCProvider config - Note over Sync: Skip if disabled or mode != api - - Sync->>Adapter: list_all_groups() - Adapter-->>Sync: list[GroupInfo] - - Sync->>GS: list() - GS-->>Sync: stored groups, by their link - + Sync->>Reg: Get the provider + Reg-->>Sync: Provider config + Sync->>Adapter: list_all_groups + Adapter-->>Sync: Every group + Sync->>GS: list + GS-->>Sync: Stored groups and their links loop Each remote group - alt Group linked to this IdP group, through this provider - Sync->>GS: update(description only,
preserve existing roles) - else Group of any name carrying its id, linked to another provider - Note over Sync: skip, count provider_mismatch - else No group of that name - Sync->>GS: create(empty roles,
source_provider + source_id) - else Group not linked to it - Note over Sync: skip, count name_taken - end + Sync->>GS: Update, create or skip, as Figure 2 decides end + Sync->>Reg: Update last_sync_at and last_sync_status +``` + +*Figure 1. L4 Detail -- OIDC group sync. Solid arrows are requests, dashed arrows are responses.* Up: [rbac.md](rbac.md). - Sync->>Reg: update(last_sync_at,
last_sync_status) +A remote group linked through this provider updates its group, and a clash with another provider or a taken name is skipped. + +```mermaid +flowchart LR + accTitle: L4 Detail -- Group sync decision + accDescr: A remote group linked to a stored group through this provider updates that group. One with an id linked to another provider is skipped as provider_mismatch. Otherwise a free name creates a linked group, and a name already held is skipped as name_taken. + + linked{"Linked here?"} + other{"Id linked elsewhere?"} + update(["Update description"]) + free{"Name free?"} + mismatch(["Skip provider_mismatch"]) + create(["Create, linked"]) + taken(["Skip name_taken"]) + + linked -->|no| other + linked -->|yes| update + other -->|no| free + other -.->|yes| mismatch + free -->|yes| create + free -.->|no| taken + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class linked,other,free decision + class mismatch,taken error ``` -The sync updates a stored group only when it is already linked to that IdP group: its `source_id` is the IdP group's id and its `source_provider` is this provider, or one `auth.source_provider_bindings` joins to it. The directory assigns that id, so a user cannot choose it the way they choose a display name. Okta is the exception: its groups claim carries names, so an Okta group's id here is its name ([idp-setup/okta.md](idp-setup/okta.md)). A group the sync creates is linked in the same write. A linked group keeps its roles, and the sync refreshes only its description. Admins assign roles to synced groups by hand. +*Figure 2. L4 Detail -- Group sync decision. Dotted arrows end in a skip.* Up: [rbac.md](rbac.md). + +A provider that is disabled, or not in `api` mode, is skipped before the listing. The sync updates a stored group when it is linked to that IdP group: its `source_id` is the IdP group's id and its `source_provider` is this provider, or one `auth.source_provider_bindings` joins to it. The directory assigns that id, so a user cannot choose it the way they choose a display name. Okta is the exception: its groups claim holds names, so an Okta group's id here is its name ([idp-setup/okta.md](idp-setup/okta.md)). -A name match links nothing. Directory display names are not unique, and in a tenant where users can create groups anyone can pick one, so an IdP group named `DFE Admins` would otherwise take the seeded `dfe-admins` and its `admin` role. A stored group of the same name with no `source_id`, or carrying another IdP group's id, is left untouched and counted as `name_taken`. So is a new group whose name is a stored group's `source_id` or whose id is a stored group's name. +A group the sync creates is linked in the same write. A linked group keeps its roles, and the sync refreshes its description and nothing else. Admins assign roles to synced groups by hand. -A sync never moves a group between providers. Two providers can assign one id, and Okta's ids are group names, so an Okta group named `SOC Analysts` carries the same id as a Keycloak group of that name. A stored group carrying the IdP group's id but linked to another provider, whatever it is called, is left untouched and counted as `provider_mismatch`, so the sync never creates a second group carrying the id, and the engine log names the syncing provider, the provider the group is linked to and the reason. `auth.source_provider_bindings` declares two names one identity source, either way round, so with `{"scim": "entra"}` the `entra` sync refreshes a group SCIM stamped `scim` and leaves the stamp as it is. +A name match links nothing. Directory display names are not unique, and in a tenant where users can create groups anyone can pick one, so an IdP group called `DFE Admins` would otherwise take the seeded `dfe-admins` and its `admin` role. A stored group of the same name with no `source_id`, or holding another IdP group's id, is left untouched and counted as `name_taken`. So is a group the sync would create when its name is a stored group's `source_id`, or its id is a stored group's name. ---- +A sync never moves a group between providers. Two providers can assign one id, and Okta's ids are group names, so an Okta group called `SOC Analysts` has the same id as a Keycloak group of that name. A stored group holding the IdP group's id but linked to another provider, whatever it is called, is left untouched and counted as `provider_mismatch`. The sync never creates a second group with that id, and the engine log names the syncing provider, the provider the group is linked to and the reason. `auth.source_provider_bindings` declares two names one identity source, either way round, so with `{"scim": "entra"}` the `entra` sync refreshes a group SCIM stamped `scim` and leaves the stamp as it is. ## Linking an IdP group -Linking an IdP group to a stored group is an admin's act. Set both `source_provider`, the provider's name in the registry, and `source_id`, the IdP group's id, through the groups API (`POST` or `PUT /api/v1/auth/groups`), dfe-ui's group form, the group's file in the live store or the chart's `authConfig.groupsConfigMap`. `PUT /api/v1/scim/v2/Groups/{name}` sets `source_id` from `externalId`. On a group not yet linked it also sets `source_provider` to `scim`, so an OIDC sign-in reaches the group only through the provider `auth.source_provider_bindings` binds to `scim`. A group already linked keeps its provider. +Linking an IdP group to a stored group is an admin's act. Set both `source_provider`, the provider's name in the registry, and `source_id`, the IdP group's id, through the groups API (`POST` or `PUT /api/v1/auth/groups`), dfe-ui's group form, the group's file in the live store or dfe-infra's `authConfig.groupsConfigMap` value. `PUT /api/v1/scim/v2/Groups/{name}` sets `source_id` from `externalId`. On a group not yet linked it sets `source_provider` to `scim` as well, so an OIDC sign-in reaches the group through the provider `auth.source_provider_bindings` binds to `scim` and no other. A group linked before keeps its provider. -The groups API refuses a link no login could use to reach the group alone: +The groups API turns down a link no login could use to reach the group alone: -- `409 conflict`: another group carries the `source_id`, under any provider (SCIM answers `uniqueness`). -- `409 identifier_taken`: a new group's name is another group's `source_id` or a new or changed `source_id` is another group's name (SCIM answers `uniqueness` for a `displayName` or `externalId`). An API key resolves each group identifier by name, then by `source_id`, so an identifier in both namespaces would let a key minted for one group take the other's roles once the first is deleted. A group whose name is its own `source_id` is one group and is accepted. A pair already stored is logged as a warning at startup, naming both groups. +- `409 conflict`: another group holds the `source_id`, under any provider (SCIM answers `uniqueness`). +- `409 identifier_taken`: a group's name is another group's `source_id`, or a changed `source_id` is another group's name (SCIM answers `uniqueness` for a `displayName` or `externalId`). An API key resolves each group identifier by name, then by `source_id`, so an identifier in both namespaces would let a key minted for one group take the other's roles once the first is deleted. A group with its name as its `source_id` is one group and is accepted. A pair stored before this check is logged as a warning at startup, naming both groups. - A caller that cannot see every group gets one answer to either collision, `409 conflict` with `Group name '' is not available`. A group outside the caller's scope answers `404` on get, update, member changes and delete, exactly as a missing one does. - `422 invalid_source_id`: the `source_id` is over 512 characters, starts or ends with whitespace or holds a control character. -- `422 invalid_source_provider`: the `source_provider` is over 512 characters or is not an OIDC provider, a binding, `scim` or (while `auth.trust_proxy_auth_headers` is on) `auth.proxy_provider`. -- `422 missing_source_provider`: a new or changed link has a `source_id` but no `source_provider`. +- `422 invalid_source_provider`: the `source_provider` is over 512 characters or is not an OIDC provider, a binding, `scim` or, when `auth.trust_proxy_auth_headers` is on, `auth.proxy_provider`. +- `422 missing_source_provider`: a changed link has a `source_id` but no `source_provider`. -Only a changed value is checked, so an edit to other fields of a group whose stored link these rules now refuse still succeeds. Setting, changing or clearing a link needs `group:write` at system scope, checked before any other group is read, and the role rights adding a member does, since each changes who holds the group's roles. Deleting a linked group needs them too, through the groups API and SCIM alike. - ---- +A changed value is the one checked, so an edit to other fields of a group with a stored link these rules would turn down still succeeds. Setting, changing or clearing a link needs `group:write` at system scope, checked before any other group is read, and the role rights adding a member does, since each changes who holds the group's roles. Deleting a linked group needs them as well, through the groups API and SCIM alike. ## Account and group writes -An account write is checked against the groups the account holds before and after it. An IdP-owned account's own list is also its assertion, so a name in that list that is another group's `source_id` is checked as that group as well. - -A group write, through the groups API or SCIM, changes only the group's member list. It updates a local account's own `groups` list to match but never an IdP-owned account's (one with `source_provider` set): that list is what the IdP asserts, so a group name written there would be read as an assertion. The member still holds the group through the member list, which SCIM `/Users` reports. +An account write is checked against the groups the account holds before and after it. An IdP-owned account's group list is its assertion as well, so a name in that list that is another group's `source_id` is checked as that group as well. ---- +A group write, through the groups API or SCIM, changes the group's member list and nothing else. It updates a local account's `groups` list to match but never an IdP-owned account's (one with `source_provider` set): that list is what the IdP asserts, so a group name written there would be read as an assertion. The member still holds the group through the member list, which SCIM `/Users` reports. ## Failed and partial syncs -One bad group never aborts the sync. A provider group that carries no id, whose email, name or id makes no valid group name, whose name is held by a stored group that does not load, whose name is held by a stored group not linked to it, or whose stored group is linked to another provider, is skipped and counted on `auth_oidc_sync_groups_skipped_total{reason}` (`no_id`, `invalid_name`, `stored_unloadable`, `name_taken`, `provider_mismatch`), and the provider's `last_sync_status` reads `partial` with each reason. A group with no id can never be linked, from any provider, so the engine log counts those in one line per run and names none of them. - -A listing that fails on any page, the first or the last, stops the sync before it writes a group. So does a page without the listing's format, and a listing still handing out pages after 1,000. `last_sync_status` reads `error`, and `sync_error` says what failed: the provider's HTTP status, a page it could not read, or the page limit. A 429 reads as throttling, not as a refusal. The engine log carries the provider's error code, and the next run lists again from the start. +One bad group never aborts the sync. A provider group with no id, a name or id that makes no valid group name, a name held by a stored group that does not load or is not linked to it, or a stored group linked to another provider is skipped and counted on `auth_oidc_sync_groups_skipped_total{reason}` (`no_id`, `invalid_name`, `stored_unloadable`, `name_taken`, `provider_mismatch`). The provider's `last_sync_status` then reads `partial` with each reason. A group with no id can never be linked, from any provider, so the engine log counts those in one line per run and names none of them. -A credential that is set but gives no access fails the same way. Entra issuing no token for the client secret, a Google service account JSON that does not build a client, or Google's token endpoint refusing the service account (`invalid_grant`), reads `error`, and `sync_error` says the directory API could not use the configured credential. The engine log has the reason. An Entra that cannot be reached, throttles the token request (429) or fails it (5xx) says that instead, and never blames the credential. `GET /oidc-providers/{name}/test` reports an Entra credential the same way, and keeps "not configured" for one that was never set. +A listing that fails on any page, the first or the last, stops the sync before it writes a group. So does a page in the wrong format, and a listing still handing out pages after 1,000. `last_sync_status` reads `error`, and `sync_error` says what failed: the provider's HTTP status, a page it could not read, or the page limit. A 429 reads as throttling, not as a refusal. The engine log has the provider's error code, and the next run lists again from the start. ---- +A credential that is set but gives no access fails the same way. Entra issuing no token for the client secret, a Google service account JSON that does not build a client, or Google's token endpoint turning the service account down (`invalid_grant`) reads `error`, and `sync_error` says the directory API could not use the configured credential. The engine log has the reason. An Entra that cannot be reached, throttles the token request (429) or fails it (5xx) says that instead, and never blames the credential. `GET /oidc-providers/{name}/test` reports an Entra credential the same way, and keeps "not configured" for one that was never set. ## Sync outcomes -An api-mode Entra or Okta provider with no directory credential, or a Google provider whose configured service account resolves to nothing, has nothing to list with. Its `last_sync_status` reads `not_configured`, `sync_error` says no credential is configured, and `POST /oidc-providers/{name}/sync` answers with the same text in `skipped`. A Google provider with no service account reads `user_token` with an empty `sync_error`, because each sign-in reads that user's groups with their own token. A credential whose secret store does not answer reads `error`, and `sync_error` names the secret store. One whose secret store refuses the engine access (a 403 on the path, or a refused AppRole login) also reads `error`, and `sync_error` says the store refused access, so the fix is the engine's secrets policy rather than the store's reachability. +An api-mode Entra or Okta provider with no directory credential, or a Google provider with a configured service account that resolves to nothing, has nothing to list with. Its `last_sync_status` reads `not_configured`, `sync_error` says no credential is configured, and `POST /oidc-providers/{name}/sync` answers with the same text in `skipped`. A Google provider with no service account reads `user_token` with an empty `sync_error`, because each sign-in reads that user's groups with the user's token. -A `generic` provider has no directory API, so one set to api mode reads `error`, and `sync_error` says to set `groups.mode` to `token_claim`. +A credential behind a secret store that does not answer reads `error`, and `sync_error` names the secret store. One behind a secret store that turns the engine away (a 403 on the path, or a failed AppRole login) reads `error` as well, and `sync_error` opens "The secret store refused access", so the fix is the engine's secrets policy rather than the store's reachability. A `generic` provider has no directory API, so one set to api mode reads `error`, and `sync_error` says to set `groups.mode` to `token_claim`. -The sync answer's `status` field carries the outcome: `ok`, `partial`, `error`, `not_configured`, `user_token` or `skipped`. The scheduled sync counts each run on `auth_oidc_sync_scheduled_runs_total{outcome}`. A provider that is not configured is tried again after its `sync_interval`, so adding the credential is picked up without a restart. +The sync answer's `status` field holds the outcome: `ok`, `partial`, `error`, `not_configured`, `user_token` or `skipped`. The scheduled sync counts each run on `auth_oidc_sync_scheduled_runs_total{outcome}`. A provider that is not configured is tried again after its `sync_interval`, so adding the credential is picked up without a restart. diff --git a/docs/control-plane/rbac-sign-in.md b/docs/control-plane/rbac-sign-in.md index 0411d9e3e..3a4a75e34 100644 --- a/docs/control-plane/rbac-sign-in.md +++ b/docs/control-plane/rbac-sign-in.md @@ -1,88 +1,90 @@ # RBAC - identity resolution and first login -Sections 2.1 and 3.5 of [rbac.md](rbac.md), which keep their numbers so section references resolve. +This page is sections 2.1 and 3.5 of [rbac.md](rbac.md), kept under those numbers so section references resolve. ## 2.1 Identity Resolution Chain +Every request takes one credential to an account or a key, the account or key to its groups, and the groups to roles, before `authorize()` decides the action. + ```mermaid -flowchart TD - subgraph "Authentication (who are you?)" - OIDC_AUTH["OIDC: X-Oidc-Subject → user_id"] - KEY_AUTH["API Key: X-API-Key → APIKeyStore.verify() → key name"] - JWT_AUTH["JWT: Bearer -> verify -> sub binds an account"] - LOCAL_AUTH["Login: POST /auth/login → AccountStore.verify_password()"] - end - - subgraph "Group Resolution (what groups?)" - OIDC_GRP["OIDC: X-Oidc-Groups, recorded on the
account the sign-in binds"] - ACCT_GRP["Account: the group files listing it as a member,
plus its IdP-asserted groups when an IdP owns it"] - KEY_GRP["API Key: key.groups field"] - end - - subgraph "Role Resolution (what roles?)" - ROLE_RES["Member groups by name, IdP identifiers by linked source_id
Union of roles, scoped grants and org_ids"] - end - - subgraph "Permission Check (can you do this?)" - PERM["authorize(auth, action)
permission_matches() with wildcards"] - end - - OIDC_AUTH --> OIDC_GRP - KEY_AUTH --> KEY_GRP - JWT_AUTH --> ACCT_GRP - LOCAL_AUTH --> ACCT_GRP - - OIDC_GRP --> ACCT_GRP - ACCT_GRP --> ROLE_RES - KEY_GRP --> ROLE_RES - - ROLE_RES --> PERM - - PERM -->|allowed| ALLOW[AuthzResult: allowed=true] - PERM -->|denied| DENY[AuthzResult: allowed=false] - - style ALLOW fill:#4a4,color:#fff - style DENY fill:#f44,color:#fff +flowchart LR + accTitle: L4 Detail -- Identity resolution chain + accDescr: OIDC headers, a password login and a bearer JWT each bind an account, and the account's groups resolve. An API key resolves the groups the key lists. The groups give the roles, grants and orgs, and the authorisation check allows or denies the action. + + hdr["OIDC headers"] + login["Password login"] + jwt["Bearer JWT"] + key["API key"] + + acct["Account groups"] + keyg["Key groups"] + + res["Roles, grants
and orgs"] + + authz{"Action allowed?"} + + allow(["Allowed"]) + deny(["Denied"]) + + hdr --> acct + login --> acct + jwt --> acct + key --> keyg + acct --> res + keyg --> res + res --> authz + authz -->|yes| allow + authz -.->|no| deny + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class authz decision + class deny error ``` -An identifier an IdP asserts, in a token's groups claim, `X-Oidc-Groups` or a SCIM record, takes a group only when that group's `source_id` is the identifier and its `source_provider` is the provider the login came through (or one `auth.source_provider_bindings` joins to it). A group's name links nothing, so an IdP group that happens to be called `dfe-admins` gets no roles until an admin links it. Logins through `X-Oidc-Groups` count as the provider named in `auth.proxy_provider` (default `oidc`). +*Figure 1. L4 Detail -- Identity resolution chain. The dotted arrow is the refusal.* Up: [rbac.md](rbac.md). + +| Credential | Identity | Groups | +|---|---|---| +| OIDC headers | `X-Oidc-Subject`, read when `auth.trust_proxy_auth_headers` is on | `X-Oidc-Groups`, recorded on the account the sign-in binds | +| Password login | `POST /auth/login`, `AccountStore.verify_password()` | The account's | +| Bearer JWT | The verified `sub` binds an account | The account's: the group files listing it as a member, plus the groups an IdP asserts when an IdP owns it | +| API key | `X-API-Key`, `APIKeyStore.verify()` | The key's `groups` field | + +The roles step takes member groups by name and IdP identifiers by linked `source_id`, and unions their roles, scoped grants and `org_ids`. `authorize(auth, action)` then matches the action against the roles' permissions with `permission_matches()`, wildcards included ([rbac.md, section 2.5](rbac.md#25-authorization-engine)). -A link names its provider. A group with a `source_id` and no `source_provider` would answer every provider that asserts the id, so the group stores refuse to write one. A group file or document already carrying one loads unlinked: the id is dropped and links no provider, its members keep the group's roles, and the engine logs a warning naming the group and counts it on `auth_groups_unlinked_total{reason="no_provider"}`. The next write to that group saves it without the id, so link it again by setting both fields. +An identifier an IdP asserts, in a token's groups claim, `X-Oidc-Groups` or a SCIM record, takes a group when that group's `source_id` is the identifier and its `source_provider` is the provider the login came through (or one `auth.source_provider_bindings` joins to it). A group's name links nothing, so an IdP group that happens to be called `dfe-admins` gets no roles until an admin links it. Logins through `X-Oidc-Groups` count as the provider `auth.proxy_provider` gives (default `oidc`). -`auth.source_provider_bindings` joins each stamp straight to one OIDC provider, and settings refuse a chain such as `{"legacy": "scim", "scim": "entra"}`, where a target is itself a stamp. An IdP sign-in is admitted on the providers its account's own stamp answers for, the set its roles resolve through, so a SCIM account stamped `scim` is admitted on `scim` and `entra` but not on another stamp bound to `entra`. +A link names its provider. A group with a `source_id` and no `source_provider` would answer every provider that asserts the id, so the group stores will not write one. A group file or document that has one loads unlinked: the id is dropped and links no provider, its members keep the group's roles, and the engine logs a warning naming the group and counts it on `auth_groups_unlinked_total{reason="no_provider"}`. The next write to that group saves it without the id, so link it again by setting both fields. -A sign-in through `X-Oidc-Groups` resolves its roles the way a token does, from the account the sign-in binds: the groups its asserted identifiers link to, plus every group whose members list the account. A gateway that forwards no groups claim, Envoy among them, reaches roles only through that membership, so on those deployments adding an account to a group by hand is how its roles arrive. +`auth.source_provider_bindings` joins each stamp straight to one OIDC provider, and settings turn down a chain such as `{"legacy": "scim", "scim": "entra"}`, where a target is itself a stamp. An IdP sign-in is admitted on the providers its account's stamp answers for, the set its roles resolve through, so a SCIM account stamped `scim` is admitted on `scim` and `entra` but not on another stamp bound to `entra`. ---- +A sign-in through `X-Oidc-Groups` resolves its roles the way a token does, from the account the sign-in binds: the groups its asserted identifiers link to, plus every group with the account in its member list. A gateway that forwards no groups claim, Envoy among them, reaches roles through that membership and no other way, so on those deployments adding an account to a group by hand is how its roles arrive. ## The sign-in allowlist -The linked groups are also the sign-in allowlist. An IdP sign-in, through the OIDC callback or `X-Oidc-Groups`, is admitted only when the account it binds to would hold a role, from a group its identifiers link to or a group whose members name the account, so a linked group with no role (every group the sync creates, until an admin gives it one) admits nobody. Anyone else gets `401 no_role_granted` (a sign-in the console started, with `return_to`, goes back to the console with that code in the URL fragment instead) and an `auth.jit.login_refused` audit event, a first sign-in leaves no account, and an existing account records the groups its IdP now asserts and has every session ended, as a logout ends them, so an issued token answers `401 session_ended` on any route, refresh included. The local admin or the break-glass account (section 3.5) links the first group, and adding an account to a group by hand admits someone the IdP groups do not cover, which is the only way in through a `manual`-mode provider. +The linked groups are the sign-in allowlist as well. An IdP sign-in, through the OIDC callback or `X-Oidc-Groups`, is admitted when the account it binds to would hold a role, from a group its identifiers link to or a group with the account in its member list. A linked group with no role, which is every group the sync creates until an admin gives it one, admits nobody. -An admin sets `source_id` and `source_provider` together through the groups API or dfe-ui's group form ([section 4.4](rbac-groups.md#44-group-sync-process)). `source_provider` names the provider whose logins the link answers. Where a `token_claim` provider's claim carries names (Okta's default, dex, Keycloak), `source_id` is the name the IdP sends. For Entra it is the object ID. +Anyone else gets `401 no_role_granted` (a sign-in the console started, with `return_to`, goes back to the console with that code in the URL fragment instead) and an `auth.jit.login_refused` audit event. A first sign-in leaves no account. An existing account records the groups its IdP asserts at that sign-in and has every session ended, as a logout ends them, so an issued token answers `401 session_ended` on any route, refresh included. The local admin or the break-glass account (section 3.5) links the first group, and adding an account to a group by hand admits someone the IdP groups do not cover, which is the one way in through a `manual`-mode provider. -A groups claim sent as one string is split on commas, never on whitespace (`src/dfe_engine/auth/oidc/rp.py`), and `X-Oidc-Groups` is split on commas. So a string claim of `SOC Analysts` is the one group `SOC Analysts`. A `source_id` containing a comma matches only an array claim on the relying-party path. +An admin sets `source_id` and `source_provider` together through the groups API or dfe-ui's group form ([section 4.4](rbac-groups.md#44-group-sync-process)). `source_provider` names the provider with the logins the link answers. Where a `token_claim` provider's claim holds names (Okta's default, dex, Keycloak), `source_id` is the name the IdP sends. For Entra it is the object ID. ---- +A groups claim sent as one string is split on commas, never on whitespace (`src/dfe_engine/auth/oidc/rp.py`), and `X-Oidc-Groups` is split on commas. So a string claim of `SOC Analysts` is the one group `SOC Analysts`. A `source_id` containing a comma matches an array claim on the relying-party path and nothing else. ## Local accounts and role changes -For a local account the group files are the authority. The account's own `groups` list is kept in step by the API routes, but it grants nothing. A member removed from a group file in the live store loses that group's roles and orgs at its next engine request, whether the API removed it, someone edited the file in the engine's auth directory, or the Helm chart's `authConfig.groupsConfigMap` copied a new file in at pod start. The deploy repo's `governance/rbac` classes are neither read nor written at runtime, so an edit there changes nothing. An account an IdP owns (JIT or SCIM) also holds the groups its record says the IdP asserts, so a group an operator adds it to by hand sits beside those. +For a local account the group files are the authority. The API routes keep the account's `groups` list in step, but that list grants nothing. A member removed from a group file in the live store loses that group's roles and orgs at its next engine request, whether the API removed it, someone edited the file in the engine's auth directory, or dfe-infra's `authConfig.groupsConfigMap` value seeded a changed file in at pod start. The engine's auth never reads the deploy repo's `governance/rbac/groups` or `governance/rbac/roles`, so an edit there changes nothing. An account an IdP owns (JIT or SCIM) holds the groups its record says the IdP asserts as well, so a group an operator adds it to by hand sits beside those. dfe-hyperdx decides on the `role` claim of the token it holds, so there a role taken away lasts until that token expires or is refreshed (`api.jwt_expire_minutes`). ---- - ## 3.5 First Login: the two seeded accounts -The deploy mints the credentials, the engine refuses defaults, the UI never holds -a password. +The deploy mints the credentials, the engine turns down defaults, and the UI never holds a password. -**`admin`** takes its password from `DFE_ENGINE_AUTH_LOCAL_ADMIN_PASSWORD`, filled by the deployment's own secret store, issued with a forced change at first login. Every boot reasserts it until the admin replaces it, and after that only on a rotation. +`admin` takes its password from `DFE_ENGINE_AUTH_LOCAL_ADMIN_PASSWORD`, filled by the deployment's secret store and issued with a forced change at first login. Every boot reasserts it until the admin replaces it, and after that on a rotation alone. -**`breakglass`** is the recovery admin, verified against a bcrypt hash committed -at `governance/settings/auth.yaml` in the deploy repo, so it survives losing the -engine, the UI and the secret store: +`breakglass` is the recovery admin, verified against a bcrypt hash committed at `governance/settings/auth.yaml` in the deploy repo, so it survives losing the engine, the UI and the secret store: ```yaml breakglass: @@ -90,41 +92,25 @@ breakglass: enabled: true # false locks the account out ``` -`DFE_ENGINE_AUTH_BREAKGLASS_PASSWORD` mints that hash on the first boot with none -committed and is ignored afterwards. Gitops off means no hash and no break-glass -account. Every attempt is audit-logged (`auth.breakglass.login`). With -`enabled: false` the login returns 403 naming the setting. - -**No `changeme` outside dev.** The engine refuses to start on an unset or default -admin password unless `DFE_ENGINE_ENV` is a dev environment -- the predicate gitops -auto-merge gates on (`settings.is_dev_posture`), and the error names the variable -and the fix. A dev environment running the default reports `default_credentials: true` on setup-status, login and refresh, for the UI to banner and force a change. Each reads the admin account at the request, so the forced change clears it with no restart. +`DFE_ENGINE_AUTH_BREAKGLASS_PASSWORD` mints that hash on the first boot with none committed and is ignored afterwards. Gitops off means no hash and no break-glass account. Every attempt is audit-logged (`auth.breakglass.login`). With `enabled: false` the login answers `403 breakglass_disabled`, giving the setting and the file that turn the account back on. -**The predicate other repos copy** is `auth.bootstrap.default_credentials_in_use`: -strip surrounding whitespace, then treat empty and `changeme` as the default. An -exact comparison reads the `"changeme\n"` a Secret or `.env` line delivers as a -minted password. +The engine will not start on an unset or default admin password unless `DFE_ENGINE_ENV` is a dev environment, the predicate gitops auto-merge gates on (`settings.is_dev_posture`), and the error gives the variable and the fix. A dev environment running the default reports `default_credentials: true` on setup-status, login and refresh, for the UI to banner and force a change. Each reads the admin account at the request, so the forced change clears it with no restart. -**Reading the minted password.** setup-status carries `deploy_kind` (`docker` | -`kubernetes` | `local`, from `DFE_ENGINE_DEPLOYMENT_TARGET` else scalo's runtime -detection) and `credential_fetch_command`: `make creds` for docker, `kubectl -n - get secret -o jsonpath='{.data.}' | base64 -d` for kubernetes, -the environment file for local. Both survive setup completion -- that is -when an operator has lost the password. +Other repos copy the predicate `auth.bootstrap.default_credentials_in_use`: strip surrounding whitespace, then treat empty and `changeme` as the default. An exact comparison reads the `"changeme\n"` a Secret or `.env` line delivers as a minted password. ---- +setup-status reports `deploy_kind` (`docker`, `kubernetes` or `local`, from `DFE_ENGINE_DEPLOYMENT_TARGET`, else scalo's runtime detection) and `credential_fetch_command`, the way to read the minted password: `make creds` for docker, `kubectl -n get secret -o jsonpath='{.data.}' | base64 -d` for kubernetes, the environment file for local. Both survive setup completion, because that is when an operator has lost the password. ## Changing and rotating passwords -**Changing either password.** `POST /auth/accounts/{username}/reset-password` works for both accounts on every deployment, and is the console's one password action for each. The new password works at the next sign-in, and the engine keeps it where that account's password lives so a restart does not undo it: +`POST /auth/accounts/{username}/reset-password` changes either password on every deployment, and is the console's one password action for each. The changed password works at the next sign-in, and the engine keeps it where that account's password lives so a restart does not undo it: | Account | Kept in | Gitops off | |---|---|---| -| `admin` | the live account store, and its whole account doc at `governance/rbac/accounts/.yaml` | the live account store only | +| `admin` | the live account store, and its whole account doc at `governance/rbac/accounts/.yaml` | the live account store alone | | `breakglass` | its bcrypt hash at `governance/settings/auth.yaml`, which every boot seeds the account from | no `breakglass` account | -A deployment that commits straight to main (dev, solo, auto-merge) is done at once. Production with `DFE_ENGINE_GITOPS_MODE=team` opens a review instead, and the response's `git.pending` carries the PR URL, or the merge command when no forge is configured. A restart before the review merges brings the old password back. `GET /auth/accounts/{username}/git-status` reports when it has merged. +A deployment that commits straight to main (dev, solo, auto-merge) is done at once. Production with `DFE_ENGINE_GITOPS_MODE=team` opens a review instead, and the response's `git.pending` holds the PR URL, or the merge command when no forge is configured. A restart before the review merges brings the old password back. `GET /auth/accounts/{username}/git-status` reports when it has merged. -**Rotation.** `POST /auth/accounts/{admin}/rotate-password` changes the value the deployment injects, not the live password: it writes the deployment's secret store when `DFE_ENGINE_AUTH_LOCAL_ADMIN_PASSWORD_SECRET_PATH` is set, else returns 501 with the command that changes it where the deployment keeps it (`store_command`). The next boot issues the rotated value with a forced change. The engine never writes the password into its own YAML store. The new password faces the two rules that gate startup -- 12 characters minimum, never the default -- so a rotation cannot lock the deployment out of its own next boot. Either is a 422 naming the field. Any other account answers 400 `not_store_backed` and is reset instead. +`POST /auth/accounts/{admin}/rotate-password` changes the value the deployment injects, not the live password. It writes the deployment's secret store when `DFE_ENGINE_AUTH_LOCAL_ADMIN_PASSWORD_SECRET_PATH` is set, else answers 501 with the command that changes it where the deployment keeps it (`store_command`). The next boot issues the rotated value with a forced change. The engine never writes the password into its YAML store. The rotated password faces the two rules that gate startup, 12 characters minimum and never the default, so a rotation cannot lock the deployment out of its next boot. Either is a 422 naming the field. Any other account answers 400 `not_store_backed` and is reset instead. -**The admin is kept.** The console never retires `admin`: its first login replaces the minted password, and from then on it is reset like any other account. `POST /api/v1/auth/setup/retire-admin` remains for an operator whose policy is to disable it once the deployment has an admin of its own. +The console never retires `admin`: its first login replaces the minted password, and from then on it is reset like any other account. `POST /api/v1/auth/setup/retire-admin` remains for an operator with a policy to disable it once the deployment has another admin. diff --git a/docs/control-plane/rbac-vocabulary.md b/docs/control-plane/rbac-vocabulary.md index d1fc5d7b2..95a99b249 100644 --- a/docs/control-plane/rbac-vocabulary.md +++ b/docs/control-plane/rbac-vocabulary.md @@ -16,28 +16,22 @@ Related: [oidc-rbac-architecture.md](oidc-rbac-architecture.md) for how external identities reach these roles, and [rbac.md](rbac.md) for enforcement and the ClickHouse tenant model. -## The two planes +Roles split across two planes, and conflating them is the most common modelling +mistake. The control plane is what you can DO: static roles granting API actions, +such as configuring sources, running hunts, deploying services and managing +groups. The data plane is what you can SEE: driven by `org_ids` and limited to +data browsing in Observe, so a user can be a powerful analyst on the control +plane and see one org's rows. `org_viewer` is not a junior `admin`. It is a +different axis. -Roles split across two orthogonal planes, and conflating them is the most common -modelling mistake: +## Roles and default groups -- Control plane - what you can DO. Static roles granting API actions: - configure sources, run hunts, deploy services, manage groups. -- Data plane - what you can SEE. Dynamic, driven by `org_ids`, and - restricted to data browsing (Observe, Kibana-style). A user can be a powerful - analyst on the control plane and see exactly one org's rows. - -A role is not "more senior" than another across planes. `org_viewer` is not -a junior `admin`. It is a different axis. - -## Canonical roles - -Defined in `roles.yaml`. `scoped: true` means the role never reads across orgs: its holders are held to their own orgs wherever it is bound, system-wide included. +Roles are defined in `roles.yaml`. `scoped: true` means the role never reads across orgs: its holders are held to their orgs wherever it is bound, system-wide included. | Role | Plane | For | |---|---|---| | `admin` | control | Full access to all resources and all orgs | -| `infra_admin` | control | Service configs, deployments, Helm, ArgoCD, orgs, groups, Governed Ops | +| `infra_admin` | control | Service configs, deployments, Helm, Argo CD, orgs, groups, Governed Ops | | `infra_viewer` | control | Infrastructure read-only | | `data_analyst` | control | Hunt, query, source, fieldmap, alert, rule, transform CRUD | | `data_analyst_viewer` | control | Analyst scope, read-only | @@ -45,10 +39,10 @@ Defined in `roles.yaml`. `scoped: true` means the role never reads across orgs: | `dfe_operator` | control | Curated operational dials - invoke shipped actions, read governed state | | `org_viewer` | data | Org-restricted data viewer (`scoped: true`) | -## Canonical default groups +`org_viewer` is the one role hard-tied to the org CRUD: a user or group holding it is associated with one or more `org_ids`, and that association IS the grant. When such a user browses data through the Observe iframe inside dfe-ui, dfe-hyperdx connects as that org's pinned ClickHouse user. That user's READONLY tenant setting and the shared RESTRICTIVE row policies filter every query to the org's `_org_id` values. There is no unfiltered path: the scoping lives in the ClickHouse user, not in anything the UI or the user sends ([rbac.md](rbac.md) sections 5 and 6). -The engine SEEDS the first four on bootstrap when the groups directory is empty. -They are not suggestions - a fresh install has them, so every consumer can rely +The engine SEEDS the first four default groups on bootstrap when the group store is empty. +They are not suggestions: a fresh install has them, so every consumer can rely on them existing. | Group | Roles | Seeded | For | @@ -62,44 +56,41 @@ on them existing. | `dfe-operators` | `dfe_operator` | no | Curated ops dials | | `dfe--viewers` | `org_viewer` | no | Per-org data plane, `scope: org:` | -Note `dfe-infra`, not `dfe-infra-admins`. The engine seeded that name first and -it stands, because a parallel name would mean two groups granting one role. - -## org_viewer -- the org-tied exception role - -`org_viewer` is the one role hard-tied to the org_id CRUD: a user or group carrying it is associated with one or more `org_ids`, and that association IS the grant. When such a user browses data through the Observe iframe inside dfe-ui, dfe-hyperdx connects as that org's pinned ClickHouse user, whose READONLY tenant setting and the shared RESTRICTIVE row policies filter every query to the org's `_org_id` values, always. There is no unfiltered path: the scoping lives in the ClickHouse user, not in anything the UI or the user sends. See [rbac.md](rbac.md) sections 5-6 for the mechanism. +The admin group is `dfe-infra`, not `dfe-infra-admins`. The engine seeded that +name first and it stands, because a parallel name would mean two groups granting +one role. -## Naming conventions +## Naming, rules and consumers - Roles: `snake_case`, singular, describing the person - `data_analyst`, not `DataAnalystRole` or `ANALYST_USER`. - Groups: `kebab-case`, plural, `dfe-` prefixed - `dfe-analysts`. The prefix - matters because these names land in customers' own IdPs alongside their groups. + matters because these names land in customers' IdPs beside their groups. - Per-org groups: `dfe--viewers`, paired with `scope: org:`. - Permissions: `resource:action`, colon-separated, `*` wildcards - `hunt:read`, `service:*:config:*`. -- Orgs and tenants: the term is **org** everywhere in the product surface (`org_id`, `org_ids`, `scope: org:`). "Tenant" appears only inside the ClickHouse isolation model (the tenant role, the tenant setting and its row policies). One concept, and the boundary between the two words is exactly that model. +- Orgs and tenants: the term is **org** everywhere in the product surface (`org_id`, `org_ids`, `scope: org:`). "Tenant" appears inside the ClickHouse isolation model alone (the tenant role, the tenant setting and its row policies), and that model is the boundary between the two words. -## Who consumes what +Four rules follow: -| Repo | Consumes | How | -|---|---|---| -| dfe-engine | roles + groups | Defines them, enforces via `_resolve_group_grants` and `RoleConfig` | -| dfe-ui | roles | Shows/hides UI by resolved role from `GET /auth/me`, and must not hardcode its own names | -| dfe-infra | groups | ArgoCD RBAC group mappings, ESO secret wiring for providers | -| dfe-deploy | nothing | Ships no group files. The engine seeds the default groups at boot | -| dfe-schemas | orgs | `org_id` column semantics for tenant row policies | -| Test environments | both | The OIDC test fixture renders these names into dex, Entra and Okta | - -## Rules - -1. Add roles here first. A role that exists in a consumer but not in - `roles.yaml` grants nothing - unknown groups resolve to zero roles, which - presents as a broken IdP rather than a config error. +1. Add roles here first. A role a consumer uses that is not in `roles.yaml` + grants nothing: unknown groups resolve to zero roles, which presents as a + broken IdP rather than a config error. 2. Never rename in a consumer. Renaming propagates from this document and the two engine files outward, in one change. 3. Group names are external contract. They are created in customers' Entra, - Okta and Workspace directories. Renaming one is a customer-facing migration, - not a refactor. + Okta and Workspace directories, so renaming one is a customer-facing + migration, not a refactor. 4. Map roles on stable ids where the provider has them, and display names. Entra emits group GUIDs, not names. See the OIDC architecture doc. + +Each repo consumes the vocabulary this way: + +| Repo | Consumes | How | +|---|---|---| +| dfe-engine | roles + groups | Defines them, enforces via `_resolve_group_grants` and `RoleConfig` | +| dfe-ui | roles | Shows/hides UI by resolved role from `GET /auth/me`, and must not hardcode names of its making | +| dfe-infra | groups | `adminGroups` in `argocd/values/common.yaml` (`dfe-admins`, `dfe-infra`) and the Argo CD `policy.csv` that `bootstrap/argocd_login.py` writes | +| dfe-deploy | nothing | Ships no group files. The engine seeds the default groups at boot | +| dfe-schemas | orgs | `org_id` column semantics for tenant row policies | +| Test environments | groups | dfe-infra's `dfe-ops idp` fixture renders these group names for its dex test IdP (`bootstrap/fixtures/tester-idp-groups.toml`) | diff --git a/docs/control-plane/rbac.md b/docs/control-plane/rbac.md index 39d5bf620..527635c61 100644 --- a/docs/control-plane/rbac.md +++ b/docs/control-plane/rbac.md @@ -8,49 +8,55 @@ ### 1.1 Four Authentication Paths +`get_current_user()` (`src/dfe_engine/api/deps.py`) tries the credentials in a fixed order and stops at the first one present. + ```mermaid -flowchart TD - REQ[Incoming Request] --> OIDC{X-Oidc-Subject header
and auth.trust_proxy_auth_headers?} - OIDC -->|Yes| EXTRACT_OIDC[Extract user_id + groups
from OIDC headers] - EXTRACT_OIDC --> RESOLVE_OIDC[GroupStore resolves
groups -> roles] - RESOLVE_OIDC --> CTX[Build AuthContext] - - OIDC -->|No| APIKEY{X-API-Key
header?} - APIKEY -->|Yes| VERIFY_KEY[APIKeyStore.verify
Parse short+long token
SHA-384 compare] - VERIFY_KEY --> RESOLVE_KEY[GroupStore resolves
key groups -> roles] - RESOLVE_KEY --> CTX - - APIKEY -->|No| JWT{Authorization:
Bearer?} - JWT -->|Yes| DECODE["Verify signature ES384 by default,
issuer and expiry
Take sub"] - DECODE --> BIND["Bind sub to its account
disabled or blocked -> 401"] - BIND --> RESOLVE_JWT["GroupStore resolves the account's
groups -> roles + org_ids
token claims ignored"] - RESOLVE_JWT --> CTX - - JWT -->|No| DISABLED{auth.enabled = false
and a dev DFE_ENGINE_ENV?} - DISABLED -->|Yes| ROOT[Root context
roles=admin
user_id=dev] - ROOT --> CTX - - DISABLED -->|No| REJECT[401 Unauthorized] - - CTX --> AUTH_DONE[AuthContext ready] - - style REJECT fill:#f44,color:#fff - style AUTH_DONE fill:#4a4,color:#fff - style ROOT fill:#fa0,color:#fff +flowchart LR + accTitle: L4 Detail -- Four authentication paths + accDescr: A request is checked for trusted OIDC headers, then an API key, then a bearer token. Each resolves its groups to roles in the AuthContext. A request with none gets the root context when auth is off in a dev environment, and a 401 otherwise. + + which{"Which credential?"} + + hdr["Header identity"] + key["Key verify
SHA-384"] + tok["Token verify
bind account"] + root(["Root AuthContext
admin"]) + reject(["401"]) + + resolve(["AuthContext
groups to roles"]) + + which -->|trusted headers| hdr + which -->|API key| key + which -->|bearer| tok + which -->|auth off| root + which -.->|none| reject + hdr --> resolve + key --> resolve + tok --> resolve + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class which decision + class reject error ``` +*Figure 1. L4 Detail -- Four authentication paths. The dotted arrow is the refusal, and `auth off` needs a dev `DFE_ENGINE_ENV`.* Up: [Control plane](README.md). + +A bearer token is verified for its signature (ES384 by default), issuer and expiry, and its `sub` binds an account. A disabled or blocked account answers 401, and the token's claims are ignored. A key or a header identity resolves its groups through the `GroupStore` the same way. + | Path | Use case | Credential storage | Token lifetime | |------|----------|-------------------|----------------| -| OIDC headers | Fallback, off by default: an engine reachable only through an OIDC proxy | IdP (Entra, Google, etc.) | Session cookie (proxy managed) | +| OIDC headers | Fallback, off by default: an engine an OIDC proxy fronts, with no other way in | IdP (Entra, Google, etc.) | Session cookie (proxy managed) | | API key | CI/CD, Terraform, scripts | `config/auth/api-keys/*.yaml` (SHA-384 hash) | `expires_at` if set, else `auth.api_key_default_ttl_days` from creation (default 90, `0` for no expiry). Revoke by deleting the file | | JWT Bearer | Console sessions, standalone, dev | Issued by `POST /api/v1/auth/login`, the OIDC login callback and `POST /api/v1/auth/refresh` | `api.jwt_expire_minutes` (`DFE_ENGINE_API_JWT_EXPIRE_MINUTES`, default 60) | | Disabled | Dev/test | N/A | N/A | -A JWT carries identity, not authority. Its `org_id`, `roles`, `groups` and `org_ids` claims grant nothing: every request re-resolves them from the account the `sub` binds to, so a role or org taken away is gone from the next request. +A JWT holds identity, not authority. Its `org_id`, `roles`, `groups` and `org_ids` claims grant nothing: every request re-resolves them from the account the `sub` binds to, so a role or org taken away is gone from the next request. -A session's `org_id` is its account's `org`, set by `POST` or `PUT /api/v1/auth/accounts`, or its API key's `org`, set by `POST /api/v1/auth/api-keys` (`422 unknown_org` for an unregistered org), and empty for one bound to none. A view taking `org_id` runs only when the session's groups grant that org or it reads every org, else `403 no_org` or `403 org_not_granted`. A `sub` that binds no account holds nothing -- an API-key subject, a deleted account -- and with auth enabled `POST /api/v1/auth/refresh` answers 401 for it, since each refresh issues a new expiry. +A session's `org_id` is its account's `org`, set by `POST` or `PUT /api/v1/auth/accounts`, or its API key's `org`, set by `POST /api/v1/auth/api-keys` (`422 unknown_org` for an unregistered org), and empty for one bound to none. A view taking `org_id` runs when the session's groups grant that org or it reads every org, else `403 no_org` or `403 org_not_granted`. A `sub` that binds no account holds nothing -- an API-key subject, a deleted account -- and with auth enabled `POST /api/v1/auth/refresh` answers 401 for it, since each refresh issues a later expiry. -OIDC headers a proxy injects, read only when `auth.trust_proxy_auth_headers` (`DFE_ENGINE_AUTH_TRUST_PROXY_AUTH_HEADERS`) is true. dfe-infra's gateway sets them with its opt-in `jwtAuthn`, and Envoy Gateway cannot map the array `groups` claim, so `X-Oidc-Groups` does not arrive from it: +OIDC headers a proxy injects, read when `auth.trust_proxy_auth_headers` (`DFE_ENGINE_AUTH_TRUST_PROXY_AUTH_HEADERS`) is true. dfe-infra's gateway sets them with its opt-in `jwtAuthn`, and Envoy Gateway cannot map the array `groups` claim, so `X-Oidc-Groups` does not arrive from it: | Header | Content | |--------|---------| @@ -67,44 +73,53 @@ the headers and falls through to API key and JWT. | Mode | Setting | What's active | Use case | |------|---------|--------------|----------| -| **Dev/test** | `auth.enabled=false` with a dev `DFE_ENGINE_ENV` | Requests with no credential get root admin context. Any other `DFE_ENGINE_ENV` refuses to start | Local development | +| **Dev/test** | `auth.enabled=false` with a dev `DFE_ENGINE_ENV` | Requests with no credential get root admin context. Any other `DFE_ENGINE_ENV` will not start | Local development | | **Standalone** | `auth.enabled=true` (the default) | JWT Bearer + API keys + local accounts | Docker, no external IdP | | **Production** | `auth.enabled=true` + an OIDC provider | The engine is the OIDC relying party (`/api/v1/auth/oidc/{provider}/login`) and mints the session JWT, plus API keys and local accounts | Kubernetes | -| **Proxy headers (fallback)** | `auth.enabled=true` + `auth.trust_proxy_auth_headers=true` | OIDC headers (precedence) + JWT + API keys | An engine reachable only through an OIDC proxy | +| **Proxy headers (fallback)** | `auth.enabled=true` + `auth.trust_proxy_auth_headers=true` | OIDC headers (precedence) + JWT + API keys | An engine an OIDC proxy fronts, with no other way in | ### 1.3 OIDC Header Trust Model **Security precondition:** When `auth.trust_proxy_auth_headers` is on, -dfe-engine MUST only be accessible via the proxy that sets the headers. Direct pod access MUST be +the proxy that sets the headers MUST be the one way to reach dfe-engine. Direct pod access MUST be blocked by K8s NetworkPolicy. Without this, any client with pod access can forge OIDC headers and impersonate any user. ## 1.4 Credential Storage Architecture +The auth files hold accounts, groups, keys and roles, and the secrets backend holds the signing key and ClickHouse passwords. + ```mermaid flowchart LR - subgraph "YAML Config (safe to commit)" - ACC["accounts/*.yaml
username, bcrypt hash,
groups an IdP asserts"] - GRP["groups/*.yaml
group name, roles,
member list"] - KEY["api-keys/*.yaml
short token, SHA-384 hash,
group memberships"] - ROLES["roles.yaml
role definitions,
permission patterns"] + accTitle: L4 Detail -- Credential storage + accDescr: The auth files hold accounts, groups, API keys and the role definitions, and the engine reads them to verify passwords and keys and to resolve roles. The secrets backend holds the JWT signing key and the ClickHouse passwords. + + subgraph files["Auth files"] + direction TB + acc[("accounts/")] + grp[("groups/")] + keys[("api-keys/")] + roles[("rbac/roles.yaml")] end - subgraph "Secrets Backend (never committed)" - ENV[".env / K8s Secret /
OpenBao / Cloud SM"] - end + sec[("Secrets backend")] - subgraph "Runtime" - API["dfe-engine API"] - end + api["dfe-engine API"] + + files -->|hashes, roles| api + sec -->|keys, passwords| api - ACC -->|bcrypt verify| API - GRP -->|membership +
role lookup| API - KEY -->|SHA-384 verify| API - ROLES -->|permission check| API - ENV -->|JWT signing key,
CH passwords| API + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class acc,grp,keys,roles,sec store + class files group ``` +*Figure 2. L4 Detail -- Credential storage.* Up: [Control plane](README.md). + +The auth files are YAML and hold one-way hashes, so they are safe to commit. The secrets backend (a `.env`, a Kubernetes Secret, OpenBao or a cloud secrets manager) is never committed. + What goes where: | Data | Where stored | Format | Safe to commit? | @@ -140,63 +155,31 @@ key name. The filename IS the identity -- never stored inside the YAML body (avoids DirectoryConfigStore YAML 1.1 boolean coercion on values like `off`, `yes`, `no`). -## 1.5 Password Verification Flow +## 1.5 Password Verification -```python -# AccountStore.verify_password() -account = self.get(username) -if account is None: - bcrypt.checkpw(password.encode(), _DUMMY_HASH) # Timing-safe rejection - return False -return bcrypt.checkpw(password.encode(), account.password_hash.encode()) -``` - -Constant-time rejection for unknown usernames prevents enumeration attacks. +`AccountStore.verify_account_password()` (`src/dfe_engine/auth/accounts.py`) runs one bcrypt check on every path. A missing account, an empty password and an account with no usable hash are checked against a dummy hash and turned down, so the time an answer takes does not reveal whether a username exists. ## 1.6 API Key Format -```text -dfe_ak_8a3f2c91_7f3b2c4d8e9a1b5f6c7d8e9f0a1b2c3d -| | | -prefix short long token (shown once, stored as SHA-384 hash) - token - (lookup index, shown in UI) -``` +A key reads `dfe_ak__`, for example `dfe_ak_8a3f2c91_7f3b2c4d8e9a1b5f6c7d8e9f0a1b2c3d`. -- Prefix (`dfe_ak_`): enables secret scanning by GitHub, GitGuardian -- Short token (8 hex chars): plaintext in YAML, used for lookup and display -- Long token (32 hex chars): SHA-384 hashed. Not bcrypt -- keys are - high-entropy random, bcrypt's slowness adds no security value -- Full key shown **once** at creation, never retrievable again -- Expiry (`expires_at`, optional ISO-8601 stored as UTC): enforced on every +| Part | Form | Stored as | +|---|---|---| +| Prefix `dfe_ak_` | Fixed, so GitHub and GitGuardian secret scanning can match it | Not stored | +| Short token | 8 hex characters | Plaintext in the key file, the lookup index the UI shows | +| Long token | 32 hex characters | A SHA-384 hash. Not bcrypt: the token is high-entropy random, so bcrypt's slowness adds no security | + +- The full key is shown **once** at creation and can never be read back. +- Expiry (`expires_at`, optional ISO-8601 stored as UTC) is enforced on every verify, so a lapsed key stops authenticating with no sweeper running. The file stays until revoked, and `list` reports `expired: true`. A key created through the API without one expires after `auth.api_key_default_ttl_days` - (default 90, `0` for no expiry) -- Revocation: delete the key's YAML file or call the revoke API + (default 90, `0` for no expiry). +- Revocation: delete the key's YAML file or call the revoke API. -## 1.7 API Key Verification Flow +## 1.7 API Key Verification -```python -# APIKeyStore.verify() -# Input: "dfe_ak_8a3f2c91_7f3b2c4d8e9a1b5f..." -prefix, short_token, long_token = parse_api_key(submitted_key) - -# 1. Scan for matching short_token across key files -key_meta = find_by_short_token(short_token) - -# 2. Hash verify, timing-safe (SHA-384; legacy sha256: keys still verify) -actual_hash = "sha384:" + hashlib.sha384(long_token.encode()).hexdigest() -if not hmac.compare_digest(key_meta.key_hash, actual_hash): - raise AuthenticationError("Invalid API key") - -# 3. Status checks run only AFTER possession is proven, so the specific -# reason cannot confirm a key the caller does not already hold -if not key_meta.enabled: - raise AuthenticationError("API key disabled") -if key_meta.is_expired(): - raise AuthenticationError("API key expired") -``` +`APIKeyStore.verify_detailed()` (`src/dfe_engine/auth/api_keys.py`) parses the key, finds the key file by its short token, hashes the long token with the key's stored algorithm (`sha384`, or `sha256` for a key made before SHA-384) and compares the two with `hmac.compare_digest`. The disabled and expired checks run after that comparison, so the reason a key is turned down cannot confirm a key the caller does not hold. The reasons are `malformed_key`, `unknown_key`, `invalid_key`, `disabled_key` and `expired_key`. --- @@ -213,33 +196,45 @@ boot the engine copies that file to `rbac/roles.yaml` beside the auth directory and loads roles from the copy. Custom roles are added to it through `/api/v1/auth/roles`. -The API refuses edits to a built-in role, and every boot rewrites each built-in role in the copy to the package's definition. A built-in grant changed in a later engine release therefore reaches an existing deployment with its next restart. Custom roles are left as they are, and a hand edit to a built-in role does not survive a boot. +The API turns down edits to a built-in role, and every boot rewrites each built-in role in the copy to the package's definition. A built-in grant changed in a later engine release therefore reaches an existing deployment with its next restart. Custom roles are left as they are, and a hand edit to a built-in role does not survive a boot. -8 built-in roles, flat hierarchy (no inheritance): +The 8 built-in roles form a flat set with no inheritance, split across the two planes of [rbac-vocabulary.md](rbac-vocabulary.md). ```mermaid -graph LR - subgraph "Global Roles" - ADMIN["admin
permissions: *"] - DA["data_analyst
hunt, query, source,
fieldmap, alert, rule,
schema, transform"] - DAV["data_analyst_viewer
read-only subset of
data_analyst"] - DV["data_viewer
query:execute,
source:read,
dashboard:read"] - IA["infra_admin
config, service,
helm, deployment,
argo"] - IV["infra_viewer
read-only subset of
infra_admin"] - OP["dfe_operator
invoke shipped actions,
read governed state"] +flowchart LR + accTitle: L4 Detail -- Built-in roles + accDescr: Six built-in roles grant control-plane actions: admin, infra_admin, infra_viewer, data_analyst, data_analyst_viewer and dfe_operator. Two grant data-plane reads: data_viewer and the scoped org_viewer. + + subgraph ctrl["Control plane"] + direction TB + admin["admin"] + infra_admin["infra_admin"] + infra_viewer["infra_viewer"] + data_analyst["data_analyst"] + data_analyst_viewer["data_analyst_viewer"] + dfe_operator["dfe_operator"] end - subgraph "Scoped Roles" - CV["org_viewer
org-restricted
data viewer
(scoped: true)"] + subgraph dataplane["Data plane"] + direction TB + data_viewer["data_viewer"] + org_viewer["org_viewer"] end - style ADMIN fill:#c33,color:#fff - style CV fill:#36c,color:#fff + ctrl ~~~ dataplane + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class ctrl,dataplane group ``` -A role write grants only what its caller already holds. Creating a role, or changing what one grants, needs every permission it lists through the caller's system-scope grants, where a pattern covers another when it matches everything that one does (`config:*` covers `config:read`). An unscoped role reads every org once bound at system scope, so it needs those permissions through roles that themselves read every org. Handing a role to a group, an account or an API key follows the same rule at the group's scope, `role:write` adds nothing to it, and a role `roles.yaml` does not declare needs `*`. +*Figure 3. L4 Detail -- Built-in roles. `org_viewer` is the one scoped role.* Up: [Control plane](README.md). + +[rbac-vocabulary.md](rbac-vocabulary.md) says what each role is for, and `src/dfe_engine/auth/resources/roles.yaml` lists the permissions each grants. + +A role write grants what its caller holds and nothing more. Creating a role, or changing what one grants, needs every permission it lists through the caller's system-scope grants, where a pattern covers another when it matches everything that one does (`config:*` covers `config:read`). An unscoped role reads every org once bound at system scope, so it needs those permissions through roles that themselves read every org. Handing a role to a group, an account or an API key follows the same rule at the group's scope, `role:write` adds nothing to it, and a role `roles.yaml` does not declare needs `*`. -A `scoped` role grants only `...:read` patterns and `query:execute`, as at system scope it passes every org-scope check. Anything else fails the roles file's load, and the roles API answers 422. +A `scoped` role grants `...:read` patterns and `query:execute` and nothing else, as at system scope it passes every org-scope check. Anything else fails the roles file's load, and the roles API answers 422. ## 2.3 Permission Taxonomy @@ -265,9 +260,9 @@ The routes enforce more domains than this table lists (accounts, groups, API key The usage scope: -`usage:read` opens `GET /api/v1/usage` and `GET /api/v1/usage/overview` ([ui-api-guide.md](ui-api-guide.md#usage-apiv1usage)). Four built-in roles carry it: `infra_admin`, `infra_viewer`, `dfe_operator` and `org_viewer`. `admin` holds it through `*`. A changed built-in grant reaches an existing deployment at its next restart, see [2.2](#22-role-definitions). +`usage:read` opens `GET /api/v1/usage` and `GET /api/v1/usage/overview` ([ui-api-guide.md](ui-api-guide.md#usage-apiv1usage)). Four built-in roles hold it: `infra_admin`, `infra_viewer`, `dfe_operator` and `org_viewer`. `admin` holds it through `*`. A changed built-in grant reaches an existing deployment at its next restart, see [2.2](#22-role-definitions). -A grant at system scope or at an org's scope opens both routes. A holder whose grant reads every org sees the whole deployment and may name any org. Any other holder is held to its own orgs that grant `usage:read`. `org_viewer` is scoped, so it is always held. The reads are described in [rbac-engine-reads.md](rbac-engine-reads.md#usage-reads). +A grant at system scope or at an org's scope opens both routes. A holder with a grant that reads every org sees the whole deployment and may name any org. Any other holder is held to the orgs it holds that grant `usage:read`. `org_viewer` is scoped, so it is always held. The reads are described in [rbac-engine-reads.md](rbac-engine-reads.md#usage-reads). ## 2.4 Wildcard Permission Matching @@ -328,47 +323,37 @@ which the error handler converts to HTTP 403. ### 3.1 Store Architecture +Three YAML-backed stores hold accounts, groups and API keys, and the local login provider reads the first two. + ```mermaid classDiagram + accTitle: L4 Detail -- Auth stores + accDescr: AccountStore, GroupStore and APIKeyStore are YAML-backed stores with one file per entity. LocalAuthProvider verifies a password through AccountStore and resolves roles through GroupStore. class AccountStore { - +create(username, password, groups) Account - +get(username) Account | None - +list() list~Account~ - +update(username, **fields) Account + +create(username, password, groups) + +verify_password(username, password) +reset_password(username, new_password) - +delete(username) - +verify_password(username, password) bool } - class GroupStore { - +create(name, roles, description) Group - +get(name) Group | None - +list() list~Group~ - +update(name, **fields) Group - +delete(name) + +create(name, roles, description) +add_member(group_name, username) - +remove_member(group_name, username) - +resolve_roles_for_member(username) list~str~ + +resolve_roles_for_member(username) } - class APIKeyStore { - +create(name, groups, description) tuple~APIKey, str~ - +get(name) APIKey | None - +list() list~APIKey~ - +verify(submitted_key) APIKey | None + +create(name, groups, description) + +verify(submitted_key) +revoke(short_token) } - class LocalAuthProvider { - +authenticate(username, password, ...) AuthContext + +authenticate(username, password) AuthContext } - LocalAuthProvider --> AccountStore : verify password LocalAuthProvider --> GroupStore : resolve roles - AccountStore ..> GroupStore : groups field references ``` -All stores are YAML-backed (one file per entity, filename = identity). +*Figure 4. L4 Detail -- Auth stores. Arrows are calls.* Up: [Control plane](README.md). + +An account's `groups` field names groups in `GroupStore`. `AccountStore` and `GroupStore` have `get`, `list`, `update` and `delete` as well, `GroupStore` has `remove_member`, and `APIKeyStore` has `get` and `list`. All stores are YAML-backed (one file per entity, filename = identity). ### 3.2 File Formats @@ -422,16 +407,14 @@ lists every route. ### 3.4 Bootstrap Defaults -On first startup (empty `config/auth/` directory), `bootstrap_auth()` seeds: - -**Groups:** +`bootstrap_auth()` seeds these groups when the group store is empty: - `dfe-admins` -> roles: `[admin]` - `dfe-analysts` -> roles: `[data_analyst]` - `dfe-viewers` -> roles: `[data_viewer]` - `dfe-infra` -> roles: `[infra_admin]` -**Accounts:** `admin` and `breakglass`, both in `dfe-admins`. See 3.5. +The two seeded accounts, `admin` and `breakglass`, are both in `dfe-admins` (section 3.5). --- @@ -445,38 +428,51 @@ Moved to [rbac-sign-in.md](rbac-sign-in.md#35-first-login-the-two-seeded-account ### 4.1 Architecture +A group sync reads the provider from the registry, lists its groups through the adapter and writes them to the group store. + ```mermaid -flowchart TD - subgraph "OIDC Providers" - GOOGLE["Google Workspace
Cloud Identity as the user at login
Admin SDK groups.list, optional"] - ENTRA["Microsoft Entra ID
groups claim, Graph on overage
Graph API /groups for sync"] - OKTA["Okta
groups claim, or
Groups API in api mode"] - GENERIC["Generic OIDC
(no admin API)"] - end +flowchart LR + accTitle: L4 Detail -- OIDC provider sync + accDescr: The group sync reads a provider from the provider registry, calls the provider's adapter, which lists groups from Google, Entra ID or Okta, and writes groups to the GroupStore. It records the sync's time and status back on the provider. - subgraph "dfe-engine" - REG["OIDCProviderRegistry
(YAML-backed CRUD)"] - SYNC["sync_provider()
Async group enumeration"] - ADAPT["Adapter Factory
get_adapter(provider)"] - GS["GroupStore
Create/update groups
with source_provider metadata"] - end + reg[("Provider registry")] - REG --> ADAPT - ADAPT --> GOOGLE - ADAPT --> ENTRA - ADAPT --> OKTA - ADAPT --> GENERIC - SYNC --> ADAPT - SYNC --> GS - SYNC --> REG - - subgraph "Group Resolution Modes" - MANUAL["manual
Membership managed
in dfe-engine only"] - CLAIM["token_claim
Groups from OIDC
token claim at login"] - API["api
Groups from the provider API:
scheduled sync, and at login
with enrich_on_login"] + sync["Group sync"] + + adapt["Provider adapter"] + gstore[("GroupStore")] + + subgraph idps["Directory APIs"] + google["Google"] + entra["Entra ID"] + okta["Okta"] end + + reg --> sync + sync --> adapt + sync --> gstore + adapt -->|list groups| idps + sync -.->|status| reg + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class reg,gstore store + class google,entra,okta ext + class idps group ``` +*Figure 5. L4 Detail -- OIDC provider sync. Dashed boxes are external, and the dotted arrow writes the sync's status back.* Up: [Control plane](README.md). + +`get_adapter()` picks the adapter by provider type. A `generic` provider has no directory API, so it never syncs. Each provider resolves groups in one of three modes: + +| Mode | Groups come from | +|---|---| +| `manual` | Membership managed in dfe-engine alone | +| `token_claim` | The OIDC token's groups claim at login | +| `api` | The provider's API: a scheduled sync, and at login with `enrich_on_login` | + ### 4.2 Provider Configuration ```yaml @@ -496,7 +492,7 @@ groups: domain: "example.com" ``` -A Google login reads the user's groups from Cloud Identity with that login's own token (scope `cloud-identity.groups.readonly`, Cloud Identity API enabled in the OAuth client's project). A group file links on `source_id`, the id after `groups/` in the group's resource name. +A Google login reads the user's groups from Cloud Identity with that login's token (scope `cloud-identity.groups.readonly`, Cloud Identity API enabled in the OAuth client's project). A group file links on `source_id`, the id after `groups/` in the group's resource name. The customer's IdP admin sets up their side from [idp-setup/google.md](idp-setup/google.md), [idp-setup/entra.md](idp-setup/entra.md) or [idp-setup/okta.md](idp-setup/okta.md), and sends back the client credentials, the issuer and the group identifiers each provider links on. @@ -514,21 +510,21 @@ Surrounding whitespace, such as the newline a mounted secret file ends with, is | `*_env` set to `""` | the name is cleared, the stored copy is kept | | a name, with no value for the secret, that differs from the one held or sits beside a stored copy | the env var must resolve non-empty in the engine: the name is saved and the stored copy is deleted once the update is saved | | the same, but the engine does not set that env var | `422`, the field error `env_var_not_set` on that `*_env` field. Nothing is saved or deleted | -| the name already held, with nothing stored | no change | +| the name the provider holds, with nothing stored | no change | | a value for the secret | the value is stored and the name saved as sent | -A create naming an env var the engine does not set, with no value for that secret, is refused the same way. A stored copy whose delete fails after the update is saved is logged by its store path. The update still answers 200. +A create naming an env var the engine does not set, with no value for that secret, is turned down the same way. A stored copy that fails to delete after the update is saved is logged by its store path. The update still answers 200. ### 4.3 Adapter Implementations | Adapter | Status | Admin API | Group sync | |---------|--------|-----------|------------| | Generic | Done | None | No (use `token_claim` mode) | -| Google | Done | Cloud Identity as the user at login. Admin SDK `groups().list()` as an optional service account | Only with a service account | -| Entra ID | Done | Graph `/me/transitiveMemberOf` with the user's token on a >200 group overage, falling back to app-only `/users/{oid}/transitiveMemberOf` when Graph refuses that token. Graph `/groups` for sync | Yes, api mode with app credentials -- `$top=999` pagination | -| Okta | Done | Groups API `/api/v1/groups` with an API token, api mode only. `token_claim` calls nothing | Yes, api mode -- `Link` header pagination | +| Google | Done | Cloud Identity as the user at login. Admin SDK `groups().list()` as an optional service account | With a service account, else no | +| Entra ID | Done | Graph `/me/transitiveMemberOf` with the user's token on a >200 group overage, falling back to app-only `/users/{oid}/transitiveMemberOf` when Graph turns that token down. Graph `/groups` for sync | Yes, api mode with app credentials -- `$top=999` pagination | +| Okta | Done | Groups API `/api/v1/groups` with an API token, in api mode. `token_claim` calls nothing | Yes, api mode -- `Link` header pagination | -A sign-in's group lookup fails safe. A credential or API failure, or a lookup still paging after 50 pages, returns no groups, so sign-in keeps working while group resolution degrades. The group sync's listing does the opposite and raises, so a failed listing never reads as a directory with no groups (section 4.4). +A directory outage, a lookup past its deadline included, keeps the groups the ID token carries, or the account's stored groups when it carries none, so an outage never locks a user out. A refusal (a 401 or 403, a missing scope or credential) is judged as no directory groups, and the login then needs a membership that grants a role (`auth/oidc/rp.py`, `_enrich_groups_from_directory`). The group sync's listing does the opposite and raises, so a failed listing never reads as a directory with no groups (section 4.4). ### 4.4 Group Sync Process @@ -540,10 +536,10 @@ Moved to [rbac-groups.md](rbac-groups.md#44-group-sync-process). ### 5.1 Pinned Org Users, One Shared Policy Set -Every registered org gets its own ClickHouse user, and that user is what fences it in. The reconciler in `src/dfe_engine/governance/ch/` renders the whole model from the org registry and the tier catalogue. The role, setting and user names come from the dfe-schemas catalogue, not from engine code. +Every registered org gets a ClickHouse user for that org alone, and that user is what fences it in. The reconciler in `src/dfe_engine/governance/ch/` renders the whole model from the org registry and the tier catalogue. The role, setting and user names come from the dfe-schemas catalogue, not from engine code. - One shared tenant role. Every org-pinned user holds it and nothing else does. -- One RESTRICTIVE row policy per table that carries `_org_id`, on that role. Its predicate is `has(splitByChar(',', getSetting('')), _org_id)`. +- One RESTRICTIVE row policy per table with an `_org_id` column, on that role. Its predicate is `has(splitByChar(',', getSetting('')), _org_id)`. - The tenant setting, pinned READONLY on each org user, listing that org's `_org_id` values. A `SETTINGS` clause in query text cannot override it: ClickHouse answers 452 (SETTING_CONSTRAINT_VIOLATION). A platform identity does not hold the tenant role, so no policy targets it and it reads every row. RESTRICTIVE-only is deliberate. A PERMISSIVE policy would flip the table to default-deny for everyone. A holder of the tenant role with no pinned setting makes `getSetting` throw, so a bad grant fails closed. @@ -554,26 +550,55 @@ Adding an org adds one pinned user. The policy set does not change. Each reconcile replaces a pinned or group user's whole role set, so a group that loses `admin` loses `otel_reader` with it, and one that turns platform-wide loses the tenant role. -It also replaces each tier, service and tenant role's privileges with the ones configured, so a grant dropped from a tier is revoked. Where ClickHouse refuses the list, the role is cleared and given each entry it accepts. +It replaces each tier, service and tenant role's privileges with the ones configured as well, so a grant dropped from a tier is revoked. Where ClickHouse turns the list down, the role is cleared and given each entry it accepts. ## 5.2 Who Reads As Whom +`GET /api/v1/hyperdx/connection` (`src/dfe_engine/api/v1/hyperdx.py`) picks the ClickHouse user a caller reads as. + ```mermaid -flowchart TD - CALLER["Caller's scoped grants"] --> PLAT{"platform_grants:
a SYSTEM-scope role,
declared, not scoped,
not org_viewer?"} - PLAT -->|"yes, with query:execute"| READER["Platform reader
no tenant role, reads every org"] - PLAT -->|no| ORG{"Resolves to exactly
one registered org?"} - ORG -->|yes| PINNED["That org's pinned user
tenant role + pinned setting"] - ORG -->|no| DENY["403 no_single_org
fails closed"] +flowchart LR + accTitle: L4 Detail -- Who reads as whom + accDescr: A caller without query execute is refused. A caller with a platform grant reads as the platform reader, which sees every org. Any other caller that resolves to exactly one registered org reads as that org's pinned user, and the rest are refused with no_single_org. + + exec{"query:execute?"} + + plat{"Platform grant?"} + forbid(["403 forbidden"]) + + one{"Exactly one org?"} + reader(["Platform reader"]) + + pinned(["Org's pinned user"]) + nosingle(["403 no_single_org"]) + + exec -->|yes| plat + exec -.->|no| forbid + plat -->|no| one + plat -->|yes| reader + one -->|yes| pinned + one -.->|no| nosingle + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class exec,plat,one decision + class reader,pinned store + class forbid,nosingle error ``` +*Figure 6. L4 Detail -- Who reads as whom. Green ends are ClickHouse users, orange are refusals.* Up: [Control plane](README.md). + +The platform reader holds no tenant role and reads every org. A pinned user holds the tenant role and its org's pinned setting. Both refusals fail closed. + `platform_grants` (`src/dfe_engine/auth/models.py`) is the one filter for "every org". The ClickHouse group bindings, the sampler, the dfe-hyperdx connection read, the fork's role claim and JIT team assignment all go through it. A role bound at an org's scope covers that org alone, a role marked `scoped` in `roles.yaml` is held to its holder's orgs wherever it is bound, a role `roles.yaml` does not declare unfences nothing, and `org_viewer` never unfences anyone. -Group bindings (`src/dfe_engine/governance/ch/bindings.py`) follow the same rule. An org-scoped group binds to its org's pinned user whatever roles it holds. A system group holding a platform role reads unrestricted, and an `admin` or `infra_admin` group also reads the otel database. A group's claimed org markers resolve to a registered org by name or, failing that, by the one org that declares the marker as a tenant id (`orgs/tenant_scope.py::resolve_orgs`). A marker no org declares, or a tenant id two orgs both declare, resolves to none, so the group gets no ClickHouse user at all -- the alternative is an unrestricted one. +Group bindings (`src/dfe_engine/governance/ch/bindings.py`) follow the same rule. An org-scoped group binds to its org's pinned user whatever roles it holds. A system group holding a platform role reads unrestricted, and an `admin` or `infra_admin` group reads the otel database as well. A group's claimed org markers resolve to a registered org by name or, failing that, by the one org that declares the marker as a tenant id (`orgs/tenant_scope.py::resolve_orgs`). A marker no org declares, or a tenant id two orgs both declare, resolves to none, so the group gets no ClickHouse user at all -- the alternative is an unrestricted one. -### 5.3 The Engine's Own Reads +### 5.3 Row reads for a caller -Moved to [rbac-engine-reads.md](rbac-engine-reads.md#53-the-engines-own-reads). +Moved to [rbac-engine-reads.md](rbac-engine-reads.md#53-row-reads-for-a-caller). --- @@ -581,42 +606,47 @@ Moved to [rbac-engine-reads.md](rbac-engine-reads.md#53-the-engines-own-reads). ### 6.1 Org Registry -YAML-backed CRUD via `OrgRegistry`. One file per org in `config/orgs/`. +YAML-backed CRUD via `OrgRegistry`, one file per org in `orgs/` under `DFE_CONFIG_DIR` (`src/dfe_engine/api/app.py`). ### 6.2 Org Creation Flow +`POST /api/v1/orgs` stores the org's tenant ids in its hunts, creates the org, then reconciles ClickHouse RBAC and asks dfe-hyperdx for a team (`src/dfe_engine/api/v1/orgs.py`, `src/dfe_engine/orgs/lifecycle.py`). + ```mermaid sequenceDiagram - participant API as REST API + accTitle: L4 Detail -- Org creation + accDescr: The orgs API stores the org's tenant ids in every hunt naming it, then creates the org in the registry. The registry's change requests a ClickHouse RBAC reconcile, and the API asks dfe-hyperdx for its team, where a failure is logged and the org stands. + participant API as Orgs API + participant Hunts as Hunt configs participant OR as OrgRegistry + participant CH as CH RBAC reconcile participant HDX as HyperDXClient + API->>Hunts: Store the tenant ids in each hunt naming the org + Hunts-->>API: Committed, or opened for review + API->>OR: create name, org_ids, display_name + OR-->>API: Org + OR->>CH: Request a reconcile + API->>HDX: get_team + HDX-->>API: Team id, or none on failure +``` - API->>OR: create(name, org_ids, display_name) - OR-->>API: Org YAML created - - Note over API: The org's pinned CH user arrives on the
next CH RBAC reconcile, not here - - API->>HDX: get_team() - HDX-->>API: team_id (or None on failure) - - Note over API: The team's ClickHouse connection is the
fork's to create, from GET /api/v1/hyperdx/connection +*Figure 7. L4 Detail -- Org creation. Solid arrows are requests, dashed arrows are responses.* Up: [Control plane](README.md). - Note over HDX: dfe-hyperdx failures are non-fatal
Background retry reconciliation -``` +The team's ClickHouse connection is the fork's to create, from `GET /api/v1/hyperdx/connection`, and a dfe-hyperdx failure is logged and does not fail the request. -Creating an org does not create its ClickHouse user. The CH RBAC reconcile does, and it runs at engine startup, in the background after every org or group change, and on `POST /api/v1/governance/ch-rbac/reconcile` (the console's "Reconcile Clickhouse RBAC" drawer). Until one of those runs, the org's users get `503 org_unprovisioned` from `GET /api/v1/hyperdx/connection`. The row policies need no change for a new org. +Creating an org does not create its ClickHouse user. The CH RBAC reconcile does, and it runs at engine startup, in the background after every org or group change, and on `POST /api/v1/governance/ch-rbac/reconcile` (the console's "Reconcile Clickhouse RBAC" drawer). Until one of those runs, the org's users get `503 org_unprovisioned` from `GET /api/v1/hyperdx/connection`. The row policies need no change for an added org. -A reconcile that raises, ClickHouse unreachable being the usual cause, is logged with the cause and run again in the background on a jittered back-off from 5 seconds up to 5 minutes, until one gets through. That includes the startup run, so a ClickHouse that comes up after the engine still gets every service user the dfe-schemas role catalogue mints without a restart. With tenant isolation off only the service roles reconcile, and they retry the same way. A successful `POST` stands a waiting retry down. Each retry adds 1 to `ch_rbac_reconcile_retries_total` (`reconcile`: `rbac` or `service_roles`), and every run counts on `ch_rbac_reconciles_total` (`outcome`). None of it gates readiness. +A reconcile that raises, ClickHouse unreachable being the usual cause, is logged with the cause and run again in the background on a jittered back-off from 5 seconds up to 5 minutes, until one gets through. That includes the startup run, so a ClickHouse that comes up after the engine still gets every service user the dfe-schemas role catalogue mints without a restart. With tenant isolation off the service roles reconcile and nothing else, and they retry the same way. A successful `POST` stands a waiting retry down. Each retry adds 1 to `ch_rbac_reconcile_retries_total` (`reconcile`: `rbac` or `service_roles`), and every run counts on `ch_rbac_reconciles_total` (`outcome`). None of it gates readiness. -`create` and `update` both refuse a name or tenant id that collides with another org's name or tenant ids -- `409 conflict` naming the other org -- because a shared marker would let their pinned ClickHouse users read each other's rows. An org keeping its own name among its own tenant ids is not a collision. +`create` and `update` both turn down a name or tenant id that collides with another org's name or tenant ids -- `409 conflict` naming the other org -- because a shared marker would let their pinned ClickHouse users read each other's rows. An org keeping its name among its tenant ids is not a collision. -### 6.3 Hunts Follow Their Orgs +## 6.3 Hunts Follow Their Orgs A hunt stores the tenant ids each of its customer orgs resolved to, because the hunt runner reads the deploy repo and the org registry is not in it. `POST /api/v1/orgs`, and `PUT` and `DELETE` on `/api/v1/orgs/{name}`, store the org's tenant ids afresh in every hunt naming it, as one deploy-repo commit, before the org itself is written. A hunt write that fails leaves the org as it was, so the same request can be sent again, and a `PUT` with nothing changed repairs a hunt holding stale ids. - Delete: the hunt keeps the org's name and loses its ids, so it fails as `unknown_customers` and holds its watermark until someone edits it. - Rename: an org's name is its identity, so `PUT` cannot change it. -- Review: production with `DFE_ENGINE_GITOPS_MODE=team` puts the hunt change on one review branch, the org answer carries `X-DFE-Review-Required`, and the runner keeps the old ids until it is merged. +- Review: production with `DFE_ENGINE_GITOPS_MODE=team` puts the hunt change on one review branch, the org answer has the `X-DFE-Review-Required` header, and the runner keeps the old ids until it is merged. Each hunt rewritten emits `resource.hunt.updated` naming the org. @@ -626,21 +656,21 @@ Each hunt rewritten emits `resource.hunt.updated` naming the org. ### 7.1 Strategy -- Connections: the fork asks `GET /api/v1/hyperdx/connection` with the user's own token and seeds that one per-org credential on their team. The engine writes none itself +- Connections: the fork asks `GET /api/v1/hyperdx/connection` with the user's token and seeds that one per-org credential on their team. The engine writes none of them - Runtime: `HyperDXClient` calls the dfe-hyperdx internal API for the team and for per-DFE-source CRUD -- Failures: non-fatal. First failure sets `_connected=False`, subsequent calls logged as warnings +- Failures: non-fatal and logged as warnings. An unreachable fork (no answer, or a 5xx) opens the client's circuit breaker, and a 4xx fails that one call and nothing else ### 7.2 Teams and Connections - Team name: JIT picks it from the caller's grants (`src/dfe_engine/auth/jit.py`, `resolve_hyperdx_team`): `dfe-admin` for `admin` or `infra_admin` at system scope, `dfe-analysts` for `data_analyst` at system scope, else `customer-` for the caller's first org, else none. -- Connection: a team's ClickHouse connection is whatever `GET /api/v1/hyperdx/connection` returned to the user whose token seeded it (section 5.2): the platform reader, or that user's org's pinned user. -- Role: `GET /api/v1/auth/me` hands the fork `hyperdx_role` (`src/dfe_engine/auth/hyperdx_role.py`). `admin` goes to `admin` or `infra_admin` at system scope and may change what a whole team sees. `member` goes to anyone whose roles allow `query:write`, the dfe-hyperdx authoring permission, and may save searches and dashboards in the observe embed. Everyone else is `viewer`, and the fork refuses it every write. So `admin` and `data_analyst` author, and the other shipped roles read. +- Connection: a team's ClickHouse connection is whatever `GET /api/v1/hyperdx/connection` returned to the user with the token that seeded it (section 5.2): the platform reader, or that user's org's pinned user. +- Role: `GET /api/v1/auth/me` hands the fork `hyperdx_role` (`src/dfe_engine/auth/hyperdx_role.py`). `admin` goes to `admin` or `infra_admin` at system scope and may change what a whole team sees. `member` goes to anyone with roles that allow `query:write`, the dfe-hyperdx authoring permission, and may save searches and dashboards in the observe embed. Everyone else is `viewer`, and the fork turns it down on every write. So `admin` and `data_analyst` author, and the other shipped roles read. --- ## 8. Argo CD Permissions -Roles carry `argo:{resource}:{action}` permissions in `roles.yaml`, and the scope catalogue publishes the `argo:` prefix (`src/dfe_engine/auth/rbac_scopes`). The engine does not export them: it writes no `argocd-rbac-cm` policy and no AppProject roles. +Roles hold `argo:{resource}:{action}` permissions in `roles.yaml`, and the scope catalogue publishes the `argo:` prefix (`src/dfe_engine/auth/rbac_scopes`). The engine does not export them: it writes no `argocd-rbac-cm` policy and no AppProject roles. --- @@ -654,7 +684,7 @@ Logins, denied authorisation decisions and admin changes are logged for complian | Login denied | `auth.login.denied` | warning | | Break-glass login | `auth.breakglass.login` | warning | | Permission denied | `auth.permission.denied` | warning | -| Account change | `auth.account.{change}` -- `created`, `updated`, `deleted`, `password_reset`, `password_changed` (the owner's own), `attributes_updated`, `sensitive_attributes_updated`, a password rotation, ending sessions, retiring the bootstrap admin | info | +| Account change | `auth.account.{change}` -- `created`, `updated`, `deleted`, `password_reset`, `password_changed` (by the account's owner), `attributes_updated`, `sensitive_attributes_updated`, a password rotation, ending sessions, retiring the bootstrap admin | info | | Group change | `auth.group.{change}` -- `created`, `updated`, `deleted`, `member_added`, `member_removed`, `attributes_updated`, `sensitive_attributes_updated` | info | | API key change | `auth.api_key.{change}` -- `created`, `revoked` | info | | Role change | `auth.role.{change}` -- `created`, `updated`, `deleted` | info | @@ -662,9 +692,9 @@ Logins, denied authorisation decisions and admin changes are logged for complian | Org change | `org.{change}` | info | | Resource change | `resource.{type}.{change}` -- OIDC providers, sources, fieldmaps and other config resources | info | -A refused bearer token emits `auth.login.denied` with `login_method: jwt`, the verifier's error class as `reason`, and the `reference` its 401 answer names. The verifier's own text goes to a `bearer token refused` line at most once a minute, the next one naming how many it held back, and `api_bearer_tokens_refused_total{reason}` counts every refusal. +A bearer token the engine turns down emits `auth.login.denied` with `login_method: jwt`, the verifier's error class as `reason`, and the `reference` its 401 answer names. The verifier's text goes to a `bearer token refused` line at most once a minute, the next one naming how many it held back, and `api_bearer_tokens_refused_total{reason}` counts every refusal. -Every account, group, API key and role write through `/api/v1/auth` emits one event naming the actor (`admin_id`), the change and the target. Its `details` carry the roles, groups, members or permissions the write set, and the names of contact fields changed, never their values. No event carries a password, a hash, an attribute value or an API key's secret half. +Every account, group, API key and role write through `/api/v1/auth` emits one event naming the actor (`admin_id`), the change and the target. Its `details` hold the roles, groups, members or permissions the write set, and the names of contact fields changed, never their values. No event holds a password, a hash, an attribute value or an API key's secret half. Events flow through the OTel pipeline -> JSON -> ClickHouse -> Observe. No custom ClickHouse audit table -- standard OTel log ingestion is used. @@ -686,6 +716,7 @@ class AuthContext(BaseModel): request_id: str | None = None client_ip: str | None = None user_agent: str | None = None + session_marker: str | None = None # The account record a local login checked, carried by the token it mints ``` Permissions are resolved at authorisation time from roles -- never stored on @@ -708,6 +739,8 @@ class AuthSettings(BaseModel): proxy_provider: str = "oidc" # Provider name the header path stamps on accounts it provisions source_provider_bindings: dict[str, str] = {} # e.g. {"scim": "entra"} api_key_default_ttl_days: int = 90 # 0 = keys created without expires_at never expire + local: LocalAuthSettings # The seeded admin and break-glass accounts + login_throttle: LoginThrottleSettings # Failed sign-in throttling ``` Environment variables: `DFE_ENGINE_AUTH_ENABLED`, `DFE_ENGINE_AUTH_DIR`, @@ -715,80 +748,67 @@ Environment variables: `DFE_ENGINE_AUTH_ENABLED`, `DFE_ENGINE_AUTH_DIR`, `DFE_ENGINE_AUTH_SOURCE_PROVIDER_BINDINGS` (a JSON object), `DFE_ENGINE_AUTH_API_KEY_DEFAULT_TTL_DAYS`, `DFE_ENGINE_AUTH_OIDC_GROUP_SYNC_ENABLED`, `DFE_ENGINE_AUTH_OIDC_GROUP_SYNC_TICK_SECONDS`, -`DFE_ENGINE_AUTH_OIDC_LOGIN_DIRECTORY_DEADLINE_SECONDS`. OIDC provider files always live in the +`DFE_ENGINE_AUTH_OIDC_LOGIN_DIRECTORY_DEADLINE_SECONDS`, with `DFE_ENGINE_AUTH_LOCAL_*` and `DFE_ENGINE_AUTH_LOGIN_THROTTLE_*` for the two nested blocks (`src/dfe_engine/settings.py`). OIDC provider files always live in the auth directory's `oidc-providers/`. -A login whose directory lookup is still running at the deadline is handled as a directory that did not answer: it keeps the groups the token carries, else the account's stored groups, and logs the timeout once with the elapsed time. +A login with a directory lookup still running at the deadline is handled as a directory that did not answer: it keeps the groups the token holds, else the account's stored groups, and logs the timeout once with the elapsed time. --- ## 12. Module Map -```mermaid -graph TD - subgraph "auth/" - ENGINE["engine.py
authorize()"] - MODELS["models.py
AuthContext, AuthzResult"] - ROLES_MOD["roles.py
RoleConfig, permission_matches()"] - ACCOUNTS["accounts.py
AccountStore"] - GROUPS["groups.py
GroupStore"] - APIKEYS["api_keys.py
APIKeyStore"] - LOCAL["local_provider.py
LocalAuthProvider"] - BOOT["bootstrap.py
bootstrap_auth()"] - AUDIT["audit.py
audit_*() functions"] - - subgraph "oidc/" - OIDC_REG["registry.py
OIDCProviderRegistry"] - OIDC_SYNC["sync.py
sync_provider()"] - OIDC_MODELS["models.py
OIDCProvider, GroupInfo"] - subgraph "adapters/" - GENERIC_A["generic.py"] - GOOGLE_A["google.py"] - ENTRA_A["entra.py"] - OKTA_A["okta.py"] - end - end - - subgraph "resources/" - ROLES_YAML["roles.yaml
8 built-in roles"] - end - end +The auth package holds authorisation, roles and the stores, and its oidc package the provider registry, sync and adapters. - subgraph "api/" - DEPS["deps.py
get_current_user()
require_action()"] - AUTH_ROUTER["v1/auth.py
login, refresh, me"] - APP["app.py
lifespan bootstrap"] +```mermaid +flowchart LR + accTitle: L4 Detail -- Auth modules + accDescr: The auth package holds the authorisation engine, roles, and the account and group stores. Its oidc package holds the provider registry, the group sync and the adapters. The API dependencies, the ClickHouse RBAC reconciler and the org registry sit outside it. + + subgraph auth["auth/"] + engine["engine.py"] + roles_mod["roles.py"] + accounts["accounts.py"] + groups["groups.py"] end - subgraph "governance/ch/" - CH_RBAC["reconciler.py
tenant role, row policies,
pinned org users"] + subgraph oidc["auth/oidc/"] + oidc_reg["registry.py"] + oidc_sync["sync.py"] + adapters["adapters/"] end - subgraph "orgs/" - ORG_REG["registry.py
OrgRegistry"] + subgraph outside["Outside auth/"] + deps["api/deps.py"] + chrbac["governance/ch/"] + orgreg["orgs/registry.py"] end - subgraph "hyperdx/" - HDX["client.py
HyperDXClient"] - end + auth ~~~ oidc ~~~ outside - DEPS --> ENGINE - DEPS --> ACCOUNTS - DEPS --> GROUPS - DEPS --> APIKEYS - AUTH_ROUTER --> LOCAL - LOCAL --> ACCOUNTS - LOCAL --> GROUPS - ENGINE --> ROLES_MOD - ROLES_MOD --> ROLES_YAML - BOOT --> ACCOUNTS - BOOT --> GROUPS - OIDC_SYNC --> OIDC_REG - OIDC_SYNC --> GROUPS - CH_RBAC --> ORG_REG - CH_RBAC --> GROUPS + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class auth,oidc,outside group ``` +*Figure 8. L4 Detail -- Auth modules.* Up: [Control plane](README.md). + +| Module | Holds | +|---|---| +| `src/dfe_engine/auth/engine.py` | `authorize()` | +| `src/dfe_engine/auth/models.py` | `AuthContext`, `AuthzResult`, `ScopedGrant`, `platform_grants()` | +| `src/dfe_engine/auth/roles.py` | `RoleConfig`, `permission_matches()` | +| `src/dfe_engine/auth/accounts.py`, `groups.py`, `api_keys.py` | `AccountStore`, `GroupStore`, `APIKeyStore` | +| `src/dfe_engine/auth/local_provider.py` | `LocalAuthProvider` | +| `src/dfe_engine/auth/bootstrap.py` | `bootstrap_auth()`, the default groups and the seeded accounts | +| `src/dfe_engine/auth/audit.py` | The `audit_*()` event functions | +| `src/dfe_engine/auth/resources/roles.yaml` | The 8 built-in roles | +| `src/dfe_engine/auth/oidc/` | `OIDCProviderRegistry`, `sync_provider()`, `OIDCProvider`, `GroupInfo` and the `generic`, `google`, `entra` and `okta` adapters | +| `src/dfe_engine/api/deps.py` | `get_current_user()`, `require_action()` | +| `src/dfe_engine/api/v1/auth.py`, `src/dfe_engine/api/app.py` | Login, refresh and `me`, and the lifespan bootstrap | +| `src/dfe_engine/governance/ch/reconciler.py` | The tenant role, row policies and pinned org users | +| `src/dfe_engine/orgs/registry.py` | `OrgRegistry` | +| `src/dfe_engine/hyperdx/client.py` | `HyperDXClient` | + --- ## 13. Edge Cases @@ -797,10 +817,10 @@ graph TD |----------|-----------| | Multi-role user | A platform grant with `query:execute` reads as the platform user. Otherwise the caller must resolve to exactly one org, or gets `403 no_single_org` (section 5.2) | | dfe-hyperdx unavailable | Non-fatal. Org CRUD succeeds. Sync retried in background | -| ClickHouse unavailable | `/readyz` reports NotReady while its ping fails. Liveness is untouched, so the pod is not restarted. The CH RBAC reconcile retries in the background (section 6.2) | -| Unknown OIDC group | No linked group with a role, so the sign-in is refused with `401 no_role_granted` and no account is created (section 2.1) | +| ClickHouse unavailable | `/readyz` reports NotReady as long as its ping fails. Liveness is untouched, so the pod is not restarted. The CH RBAC reconcile retries in the background (section 6.2) | +| Unknown OIDC group | No linked group with a role, so the sign-in is turned down with `401 no_role_granted` and no account is created (section 2.1) | | OIDC adapter failure | Failsafe: returns empty results, auth continues with available info | -| Default password in use | Dev `DFE_ENGINE_ENV`: warning at startup and `default_credentials: true` on setup-status. Any other `DFE_ENGINE_ENV`: the engine refuses to start (section 3.5) | +| Default password in use | Dev `DFE_ENGINE_ENV`: warning at startup and `default_credentials: true` on setup-status. Any other `DFE_ENGINE_ENV`: the engine will not start (section 3.5) | --- diff --git a/docs/control-plane/synthetic-data.md b/docs/control-plane/synthetic-data.md index 4c109effb..607ac2dac 100644 --- a/docs/control-plane/synthetic-data.md +++ b/docs/control-plane/synthetic-data.md @@ -1,63 +1,54 @@ # Synthetic Data -Realistic synthetic events on demand, for two jobs: a continuous live-looking -inbound stream for sales demos, and repeatable data for testing. Never a load -generator - every knob is a ceiling, and the shipped defaults generate -nothing until an operator or caller asks. +Realistic synthetic events on demand, for two jobs: a continuous live-looking inbound stream for sales demos, and repeatable data for testing. It is never a load generator: every knob is a ceiling, and the shipped defaults generate nothing until an operator or caller asks. -Every synthetic event carries a marker (overridable per request), which the common header lands in `_tags` - synthetic data is always identifiable downstream. +Every synthetic event has a marker (overridable per request), which the common header lands in `_tags`, so synthetic data is always identifiable downstream. Where the pack's `tags` is an object, the event has `tags.synthetic: true`, found as `_tags.synthetic`. Where it is an ECS keyword list (`meta/elastic/ecs` and the beats packs that map ECS `tags`), the event has the keyword `synthetic` in that list, found with ``has(_tags.list.:`Array(Nullable(String))`, 'synthetic')``, and extra `tags` in the request are rejected because a list cannot hold keys (`src/dfe_engine/synthetic_data/tags.py`). -Where the pack's own `tags` is an object the event carries `tags.synthetic: true`, found as `_tags.synthetic`. - -Where it is an ECS keyword list (`meta/elastic/ecs` and the beats packs that map ECS `tags`) the event carries the keyword `synthetic` in that list, found with ``has(_tags.list.:`Array(Nullable(String))`, 'synthetic')``, and extra `tags` in the request are refused because a list cannot hold keys. +Schema packs and sampled rows feed two event factories, and a paced stream posts their events to dfe-receiver. ```mermaid flowchart LR - P[dfe-schemas meta packs
synthetic: hints + scenarios] --> G[SchemaEventFactory] - S[sampler rows / lines] --> L[SampleEventFactory
identity scrub] - G --> ST[Poisson-paced stream] - L --> ST - ST --> R[dfe-receiver /ingest] --> LD[dfe-loader] --> CH[(ClickHouse)] + accTitle: L4 Detail -- Synthetic data paths + accDescr: dfe-schemas meta packs feed the schema event factory, and sampler rows or lines feed the sample event factory, which scrubs identities. Both feed a Poisson-paced stream that posts events to dfe-receiver's ingest endpoint. + + packs[("dfe-schemas
meta packs")] + sample[("Sampler rows
or lines")] + + schemaf["SchemaEventFactory"] + samplef["SampleEventFactory
identity scrub"] + + stream["Poisson-paced
stream"] + + receiver["dfe-receiver
ingest"] + + packs --> schemaf + sample --> samplef + schemaf --> stream + samplef --> stream + stream --> receiver + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + class packs,sample store + class receiver ext ``` -## Modes +*Figure 1. L4 Detail -- Synthetic data paths. The dashed box is outside the engine.* Up: [The DFE control-plane layer](README.md). + +From the receiver, a synthetic event takes the same path as any other: the loader writes it to ClickHouse. | Mode | Input | Realism source | |------|-------|----------------| -| Reference packs (default) | A dfe-schemas meta schema (`meta/syslog`, `meta/otel/logs`, `meta/beats/filebeat`, the cyber sets) | `@source:` exprs give the event fields; name/type heuristics pick typed generators; schema-authored `synthetic:` hints and scenarios carry vocabularies and message templates | -| Lookalike | A sample's parsed `rows` or raw `lines` (the sampler's output) | The sample's own distributions - enums replay weighted, numerics fit the observed range, free text keeps structure with identity tokens substituted in place | - -Both draw from a seeded **entity pool**: a generated host keeps its -FQDN/IP/MAC and its users across the whole stream, and an identical seed -reproduces the identical stream. Lookalike mode never replays sample -identities - IPs, MACs, UUIDs, emails and name-identified user/host keys are -synthesised, so it doubles as a scrub. - -## Coherence: scenarios - -Correlated columns must not draw independently (an sshd message under -`appname: cron` reads as fake). A pack's version entry may declare -`synthetic.scenarios` - one weighted scenario is drawn per event and pins the -correlated columns together. Format reference: the dfe-schemas repo, at - in that tree. - -## API surface - -`GET /synthetic-data/packs`, `POST /synthetic-data/generate`, -`POST /synthetic-data/lookalike`, and the stream task endpoints -(`POST /synthetic-data/stream`, `GET /synthetic-data/streams`, -`GET /synthetic-data/streams/{operation}`, -`POST /synthetic-data/streams/{operation}/cancel`). -Reads need `synthetic-data:read`; generating or streaming needs -`synthetic-data:run` (write-grade - it injects data into the pipeline). -Full contract: `openapi/openapi.json`. - -## Ceilings and the standing demo stream - -`SyntheticDataSettings` (`DFE_ENGINE_SYNTHETIC_DATA_*`) caps inline counts, stream -rate and stream duration. The **autostart** knobs run a standing sales demo: -name pack refs and a receiver URL (engine environment variables -`DFE_ENGINE_SYNTHETIC_DATA_AUTOSTART_*`, set through the overlay's `extraEnv`) -and the engine keeps those streams -running for the pod's lifetime - bounded segments, retry while the receiver -is away, cancelled at shutdown. Default empty = off. +| Reference packs (default) | A dfe-schemas meta schema (`meta/syslog`, `meta/otel/logs`, `meta/beats/filebeat`, the cyber sets) | `@source:` expressions give the event fields, name and type heuristics pick typed generators, and schema-authored `synthetic:` hints and scenarios hold vocabularies and message templates | +| Lookalike | A sample's parsed `rows` or raw `lines` (the sampler's output) | The sample's distributions: enums replay weighted, numerics fit the observed range, free text keeps its structure with identity tokens substituted in place | + +Both draw from a seeded entity pool: a generated host keeps its FQDN, IP, MAC and users across the whole stream, and an identical seed reproduces an identical stream. Lookalike mode never replays sample identities. IPs, MACs, UUIDs, emails and name-identified user and host keys are synthesised, so it doubles as a scrub. + +Correlated columns must not draw independently (an sshd message under `appname: cron` reads as fake). A pack's version entry may declare `synthetic.scenarios`: one weighted scenario is drawn per event and pins the correlated columns together. The format is in dfe-schemas' [meta-schema.md](https://github.com/hyperi-io/dfe-schemas/blob/main/docs/meta-schema.md). + +## API and limits + +`GET /synthetic-data/packs`, `POST /synthetic-data/generate`, `POST /synthetic-data/lookalike`, and the stream task endpoints (`POST /synthetic-data/stream`, `GET /synthetic-data/streams`, `GET /synthetic-data/streams/{operation}`, `POST /synthetic-data/streams/{operation}/cancel`). Reads need `synthetic-data:read`. Generating or streaming needs `synthetic-data:run`, write-grade because it injects data into the pipeline. The full contract is `openapi/openapi.json`. + +`SyntheticDataSettings` (`DFE_ENGINE_SYNTHETIC_DATA_*`) caps inline counts, stream rate and stream duration. The autostart settings run a standing sales demo: name the pack refs and a receiver URL (`DFE_ENGINE_SYNTHETIC_DATA_AUTOSTART_*`, set through the overlay's `extraEnv`) and the engine keeps those streams running for the pod's lifetime, in bounded segments, retrying when the receiver is away, cancelled at shutdown. Empty, the default, is off. diff --git a/docs/control-plane/type-safety-sync.md b/docs/control-plane/type-safety-sync.md index d3cfab8f9..398090049 100644 --- a/docs/control-plane/type-safety-sync.md +++ b/docs/control-plane/type-safety-sync.md @@ -1,73 +1,29 @@ # Type safety sync (dfe-engine -> dfe-ui) -dfe-engine is the source of truth for the HTTP API contract and the RBAC scope catalog, and `openapi.json` carries both. It is synced into **dfe-ui** so the frontend can use strict TypeScript types and scope constants without hand-maintaining duplicates. +dfe-engine is the source of truth for the HTTP API contract and the RBAC scope catalogue, and `openapi.json` holds both. A workflow syncs it into dfe-ui, so the frontend uses strict TypeScript types and scope constants with no hand-kept duplicates. -## What it covers +- API types: routes, request bodies and responses defined in FastAPI and Pydantic become TypeScript types in dfe-ui. +- RBAC scopes: the permission strings in `src/dfe_engine/auth/rbac_scopes/scope_constants.py` become the `RbacScope` string enum in the spec's `components.schemas`, with their constant names in `x-enumNames` (`src/dfe_engine/api/app.py`), and so a generated constant in dfe-ui. +- Automation: a push to `main` or `master` that changes a spec opens a PR in dfe-ui, so frontend code is checked at compile time against the latest engine contract. -- API types: Routes, request bodies, and responses defined in FastAPI/Pydantic become TypeScript types in dfe-ui. -- RBAC scopes: Permission strings in `scope_constants.py` become the `RbacScope` enum in the spec, and so in dfe-ui. -- Automation: Merges to `main`/`master` that touch the spec open a PR in dfe-ui. Frontend code can rely on compile-time checks against the latest engine contract. - -## Generators in this repo - -### OpenAPI spec - -| Item | Location | -|------|----------| -| Script | `scripts/generate_openapi.py` | -| Output | `openapi/openapi.json` and `openapi/openapi.e2e.json` (committed) | - -Regenerate after API model or route changes: +`scripts/generate_openapi.py` loads the live FastAPI app and writes the OpenAPI 3 schema to `openapi/openapi.json` and `openapi/openapi.e2e.json`, both committed. Regenerate after any model or route change: ```bash uv run python scripts/generate_openapi.py ``` -The script loads the live FastAPI app and writes the OpenAPI 3 schema. That JSON is used for Prism mocks, contract checks, and (in dfe-ui) `openapi-typescript` code generation. - -See also [ui-api-guide.md](ui-api-guide.md) for local mock and typegen workflows. - -### RBAC scope constants - -`src/dfe_engine/auth/rbac_scopes/scope_constants.py` is the source of truth. The generator publishes every scope in `openapi.json` as the `RbacScope` string enum in `components.schemas`, its constant names in `x-enumNames`, so dfe-ui generates them with the rest of the API types. - -## CI sync to dfe-ui +The Prism mock, the contract test (`tests/unit/test_openapi_spec_committed.py`) and dfe-ui's `openapi-typescript` generation read the committed files. A change to the Python API alone does not start the sync: commit the regenerated specs with the code change. [ui-api-guide.md](ui-api-guide.md) covers the local mock. -Workflow: [`.github/workflows/sync-dfe-engine-types.yml`](../../.github/workflows/sync-dfe-engine-types.yml) +## The sync workflow -**Name:** `Sync dfe-engine-types` - -### Triggers - -- **Push** to `main` or `master` when any of these paths change: - - `openapi/openapi.json` - - `openapi/openapi.e2e.json` -- **Manual:** `workflow_dispatch` - -Note: Changing only Python API files without regenerating and committing the OpenAPI JSON will not trigger the workflow. Regenerate locally with `uv run python scripts/generate_openapi.py` and commit `openapi/openapi.json` and/or `openapi/openapi.e2e.json` when the contract changes. - -### What the job does +[`.github/workflows/sync-dfe-engine-types.yml`](../../.github/workflows/sync-dfe-engine-types.yml), `Sync dfe-engine-types`, runs on a push to `main` or `master` that changes `openapi/openapi.json` or `openapi/openapi.e2e.json`, and on `workflow_dispatch`. It: 1. Checks out dfe-engine. -2. Mints a token from the GitHub App and clones **dfe-ui** (repo from `DFE_UI_REPO`, default `hyperi-io/dfe-ui`). -3. Copies into dfe-ui's types package: - - `openapi/openapi.json` -> `packages/dfe-engine-types/specs/openapi.json` - - `openapi/openapi.e2e.json` -> `packages/dfe-engine-types/specs/openapi.e2e.json` -4. Runs `yarn install --immutable` and `yarn generate` in `packages/dfe-engine-types`, then generates `types/e2e.d.ts` from `openapi.e2e.json` and exports it as `@repo/dfe-engine-types/e2e`. -5. Opens a PR in dfe-ui (closes older open sync PRs on the same branch naming pattern so only one sync PR is active). - -### Secrets and configuration - -- `vars.GH_APP_CLIENT_ID` and `secrets.GH_APP_PRIVATE_KEY` (required): the GitHub App whose installation token, scoped to dfe-ui with contents and pull-requests write, pushes the branch and opens and closes the PRs. -- `vars.DFE_UI_REPO` (optional): `owner/name` for the UI repo (default `hyperi-io/dfe-ui`). - -## Developer workflow (dfe-engine) - -1. Change API or scopes in Python. -2. Regenerate the specs: `uv run python scripts/generate_openapi.py`. -3. Commit the updated `openapi/openapi.json` and `openapi/openapi.e2e.json` together with your code change. -4. After merge to `main`/`master`, wait for **Sync dfe-engine-types** to open the dfe-ui PR. Merge that PR to pick up types and scopes on the frontend. +2. Mints a token from the GitHub App and clones dfe-ui (`vars.DFE_UI_REPO`, default `hyperi-io/dfe-ui`). +3. Copies the two specs to `packages/dfe-engine-types/specs/` in dfe-ui. +4. Runs `yarn install --immutable` and `yarn generate` in `packages/dfe-engine-types`, which writes `types/index.d.ts`, `types/e2e.d.ts` (exported as `@repo/dfe-engine-types/e2e`), the scope constants and the limits. +5. Opens a PR in dfe-ui and closes older open sync PRs, so one sync PR is open at a time. -## Developer workflow (dfe-ui) +It needs `vars.GH_APP_CLIENT_ID` and `secrets.GH_APP_PRIVATE_KEY`: the GitHub App's installation token, scoped to dfe-ui with contents and pull-requests write, pushes the branch and opens and closes the PRs. -Consume `packages/dfe-engine-types` (generated API types, `RbacScope` among them). Do not copy scope strings or paths by hand. Import from the package so renames and new endpoints fail at compile time instead of at runtime. +In dfe-engine, change the API or the scopes in Python, regenerate the specs, commit them with the code, and merge the dfe-ui PR the workflow opens. In dfe-ui, import from `packages/dfe-engine-types` and never copy a scope string or path by hand, so a rename or a new endpoint fails at compile time instead of at runtime. diff --git a/docs/control-plane/ui-api-guide.md b/docs/control-plane/ui-api-guide.md index f9dc3fb21..e4c16a764 100644 --- a/docs/control-plane/ui-api-guide.md +++ b/docs/control-plane/ui-api-guide.md @@ -1,12 +1,11 @@ # DFE Engine API -- UI Integration Guide -This document covers everything a frontend developer needs to integrate -with the DFE Engine REST API. +How a frontend calls the DFE Engine REST API: auth, errors, paging, the routes a UI uses most, and the contract tooling. ## Quick Start ```bash -# Generate TypeScript types from OpenAPI spec +# Types for a client outside dfe-ui (dfe-ui imports @repo/dfe-engine-types) npx openapi-typescript openapi/openapi.json -o src/types/api.d.ts # Start Prism mock server for development @@ -26,7 +25,7 @@ Every `/api/v1/*` endpoint but login and the OIDC sign-in routes needs a credent |--------|--------|----------| | JWT Bearer | `Authorization: Bearer ` | UI login flow | | API Key | `X-API-Key: dfe_xxxx...` | Machine-to-machine | -| OIDC | The engine's own sign-in (`/api/v1/auth/oidc/{provider}/login`) mints the JWT above | Production SSO | +| OIDC | The engine's sign-in (`/api/v1/auth/oidc/{provider}/login`) mints the JWT above | Production SSO | | Proxy headers | `X-Oidc-Subject`, read only with `auth.trust_proxy_auth_headers` | An engine reachable only through an OIDC proxy | ### Login Flow @@ -75,8 +74,8 @@ interface FieldError { | 403 | `forbidden` | Insufficient permissions | | 404 | `not_found` | Resource doesn't exist | | 409 | `conflict` | Duplicate name or match conflict | -| 412 | `etag_mismatch` | `If-Match` is stale. `etag` carries the current one | -| 413 | `payload_too_large` | Request body over `api.max_request_body_bytes` (4 MiB, `DFE_ENGINE_API_MAX_REQUEST_BODY_BYTES`), or over a file upload's own limit | +| 412 | `etag_mismatch` | `If-Match` is stale. `etag` holds the current one | +| 413 | `payload_too_large` | Request body over `api.max_request_body_bytes` (4 MiB, `DFE_ENGINE_API_MAX_REQUEST_BODY_BYTES`), or over a file upload's limit | | 422 | `validation_error` | Request body validation failed | | 500 | `internal_error` | Unexpected server error | | 503 | `not_configured` | Required backend not available | @@ -122,7 +121,7 @@ The tables below cover the routes a UI uses most. `openapi/openapi.json` lists e | GET | `/auth/permissions` | Available permissions for current user | | CRUD | `/auth/accounts` | Account management | | CRUD | `/auth/groups` | Group + role management | -| CRUD | `/auth/roles` | Role definitions (`config/rbac/roles.yaml`) | +| CRUD | `/auth/roles` | Role definitions (`rbac/roles.yaml` under `DFE_CONFIG_DIR`) | | GET | `/auth/roles/scopes` | RBAC permission scopes for role editors | | CRUD | `/auth/api-keys` | API key management | | CRUD | `/auth/oidc-providers` | OIDC provider configuration | @@ -148,7 +147,7 @@ A group sync answers 200 with `created`, `updated`, `total`, `groups_skipped`, ` | PUT | `/sources/{name}` | Update source | | PATCH | `/sources/{name}` | Set lifecycle state (`state: active\|dormant\|disabled`, or compat `enabled`). No new version. `409 match_conflict` if activating duplicates another non-disabled source's match | | DELETE | `/sources/{name}` | Delete source | -| POST | `/sources/bulk` | Bulk action (`enable` / `disable` / `dormant` / `delete` / `deploy`) on named sources | +| POST | `/sources/bulk` | Bulk action (`enable` / `disable` / `dormant` / `delete` / `deploy`) on listed sources | | POST | `/sources/seed` | Seed defaults | A PUT that omits `header`, `schema`, `views`, `transform`, `transport` or `archive` keeps the stored value. Send `null`, `[]` or `false` to clear one. @@ -249,7 +248,7 @@ There is no progress stream: poll the operation, and read `metadata` for progres ## Governed Ops (curated actions - the operator dials) The Governance screen is INVOKE-FIRST: operators see the shipped dials and -pull them. Defining new dials is rare admin work (`governance:write`), done +pull them. Defining dials is rare admin work (`governance:write`), done through the guided flow below - never through free-text fields. ### Operator surface (list + invoke) @@ -262,8 +261,8 @@ through the guided flow below - never through free-text fields. | POST | `/governance/actions/{name}/invoke` | Apply (per-action `action:invoke:` grant) | Invoke body (only when the action declares params): -`{"params": {"level": "2x"}}`. Each param is CONSTRAINED - enum params carry -their full `values` list, numerics carry `min`/`max` - so render enum params +`{"params": {"level": "2x"}}`. Each param is CONSTRAINED - enum params list +their `values`, numerics set `min`/`max` - so render enum params as a select and numerics as a bounded input. Never a free text box. A violation returns 422 `invalid_params`. @@ -295,9 +294,9 @@ behind an advanced toggle. House rule: if the server will reject values outside a set, the contract exposes the set. Three mechanisms, in order of preference: -1. **Static set** - a schema enum (e.g. repository `scope`). Generated types - already give a union. Render a select. -2. **Dynamic set** - the schema field carries an `x-dfe-enum-source` +1. Static set: a schema enum (e.g. repository `scope`). Generated types + give a union. Render a select. +2. Dynamic set: the schema field holds an `x-dfe-enum-source` annotation naming the endpoint that enumerates its legal values: ```json @@ -310,7 +309,7 @@ exposes the set. Three mechanisms, in order of preference: endpoint returns objects. Absent means the items are plain strings. Build ONE generic hook that resolves an annotated field to a select fed by a TanStack query - every annotated field then renders correctly for free. -3. **Derived value** - the field is optional and the server fills a documented +3. Derived value: the field is optional and the server fills a documented default (e.g. `required_action`). Show the derived value, do not ask for it. A free-text input for any of these is a contract bug - report it against the @@ -346,7 +345,7 @@ What each metric counts: | `ingested_bytes` | bytes | The decoded payload bytes of records accepted at the first hop, summed over the day | | `events_per_second` | events/s | The busiest minute of the day | -Only `ingested_bytes` and `events_per_second` are kept per org. The two core counts are measured for the deployment. +Only `ingested_bytes` and `events_per_second` are kept per org. The two core counts are counted for the deployment as a whole. ### Response @@ -380,7 +379,7 @@ The series holds one entry per UTC day that has a row, oldest first, so a gap is } ``` -The real response lists all four metrics in the order `cores_deployed`, `node_cores`, `ingested_bytes`, `events_per_second`, and the series is cut to two entries here. `/usage` returns one metric's fields at the top level (`metric`, `label`, `unit`, `status`, `granularity`, `orgs`, `from`, `to`, `highest`, `latest`, `series`), where `highest` is the highest day in the whole range. +The full response lists all four metrics in the order `cores_deployed`, `node_cores`, `ingested_bytes`, `events_per_second`, and the series is cut to two entries here. `/usage` returns one metric's fields at the top level (`metric`, `label`, `unit`, `status`, `granularity`, `orgs`, `from`, `to`, `highest`, `latest`, `series`), where `highest` is the highest day in the whole range. ### Who sees what @@ -425,7 +424,7 @@ key) - plus the user-layer etag: ``` PATCH takes an RFC 7396 merge patch, writes ONLY the user layer, and -returns the same fields with the new effective doc. `null` deletes a +returns the same fields with the updated effective doc. `null` deletes a user-layer key so the value falls back to the inherited layer. The theme toggle is one call: `PATCH {"theme": "dark"}`. @@ -443,7 +442,7 @@ const patchPrefs = useMutation({ headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(patch), }).then(r => r.json()), - // Response IS the new effective doc - no refetch needed + // Response IS the updated effective doc - no refetch needed onSuccess: (data) => queryClient.setQueryData(['preferences'], data), }); @@ -468,14 +467,14 @@ For `system` the `scope_id` MUST be `-`. `namespace` and `key` match `^[a-zA-Z0-9][a-zA-Z0-9._-]{0,127}$` (422 otherwise). Access: any authenticated user reads the system layer. Members read their -own org/group layers and their own user layer. Everything else needs the +org and group layers and their user layer. Everything else needs the `repository:read` / `repository:write` RBAC actions at the matching scope (admins system-wide, org-wildcard admins within their org). Reads you cannot see return 404 (existence stays hidden). Forbidden writes return 403. ### ETag semantics -Every stored object carries an opaque etag (returned in the `ETag` +Every stored object has an opaque etag (returned in the `ETag` header and in metadata). Send `If-Match: ` on PUT/PATCH for optimistic concurrency - a mismatch returns `412 {"code": "precondition_failed"}` with the current etag in the @@ -491,27 +490,16 @@ optimistic concurrency - a mismatch returns | Dry run `limit` | 100 events (`DFE_ENGINE_TRANSFORM_DRY_RUN_MAX_EVENTS`) | `422 {"code": "validation_error"}` | | Scaling `replica_count`, `max_replicas` | 1000 (`DFE_ENGINE_DEPLOYMENT_MAX_REPLICAS_CEILING`) | `400 {"code": "invalid_dial"}` | -## TypeScript Type Generation +## Types, the mock and drift -```bash -# Install openapi-typescript -pnpm add -D openapi-typescript - -# Generate types (run after any API change) -npx openapi-typescript ../dfe-engine/openapi/openapi.json \ - -o src/types/dfe-engine-api.d.ts -``` - -Then use with fetch or a typed client: +dfe-ui imports the generated types from `@repo/dfe-engine-types`, which the [sync workflow](type-safety-sync.md) keeps in step with `openapi.json`. A type reads off `paths`: ```typescript -import type { paths } from './types/dfe-engine-api'; +import type { paths } from '@repo/dfe-engine-types'; type Source = paths['/api/v1/sources/{name}']['get']['responses']['200']['content']['application/json']; ``` -## Mock Server (Prism) - `docker-compose.dev.yaml` runs Prism, digest-pinned, over `./openapi`: ```bash @@ -521,8 +509,6 @@ docker compose -f docker-compose.dev.yaml up prism Prism generates mock responses from the OpenAPI spec. Point the UI at `http://localhost:4010` during development. -## Contract Drift Detection - `tests/unit/test_openapi_spec_committed.py` fails CI when the committed spec, or the route map built from it, differs from what the app generates. Regenerate and commit after a route or model change: diff --git a/docs/data-evolution-acceptance.md b/docs/data-evolution-acceptance.md index f9f0d6b57..7745e5b3b 100644 --- a/docs/data-evolution-acceptance.md +++ b/docs/data-evolution-acceptance.md @@ -5,11 +5,9 @@ stage. A claim in that document with no step here is marketing. -## The steps - A step marked `live` is one test in -[tests/e2e/test_data_evolution.py](../tests/e2e/test_data_evolution.py), named -for its number. +[tests/e2e/test_data_evolution.py](../tests/e2e/test_data_evolution.py), with +its number in the test name. | # | Stage | Prove | Test | |---|---|---|---| @@ -17,7 +15,7 @@ for its number. | 0.2 | 0 | A nested sub-field in `_json` answers a query (`fred.nerk.frog`) | live | | 1.1 | 1 | A pre-supplied read-only meta schema imports from dfe-schemas | live | | 1.2 | 1 | A meta schema exports, then re-imports into a clean deployment | live, part | -| 1.3 | 1 | A field seen in `_json` promotes to a real column | live | +| 1.3 | 1 | A field seen in `_json` promotes to a column | live | | 2.1 | 2 | A source declares a `key=value` routing condition | live | | 2.2 | 2 | Deploy renders the routing, the receiver reloads it in place, and records route by it | live, part | | 2.3 | 2 | Records land in the source's OWN table, typed columns filled by dfe-loader rather than left in `_json` | live | @@ -30,8 +28,6 @@ for its number. | 3.2 | 3 | The feed still lands with typed columns filled and `_json` empty | live | | 3.3 | 3 | Turning population back on refills `_json` for new records -- the decision is reversible | live | -## What the marks mean - `xfail` names the open issue the step fails on. The marker is strict, so the suite goes red the day that issue is fixed, which is how it gets removed. @@ -40,27 +36,26 @@ rendered and the table created. That the receiver reloads in place is the flow suite's routing check, and that records route by it is 2.3. 2.6 applies the ALTER directly rather than through a derived-schema edit, which is what 2.4 and 2.5 cover. 1.2 re-imports into the SAME deployment under a second path, which proves -the bundle carries what the import needs but not that it lands in a deployment +the bundle holds what the import needs but not that it lands in a deployment that has never seen the schema -- that needs a second stack. -2.8 drives the source that already has a transform instance, because Compose declares one service per transform app and binds it at deploy. So it proves a different engine on the same corpus, never a second binding. It reads that source's match rule off the API, since the source belongs to the deployment. +2.8 drives the source that has a transform instance, because Compose declares one service per transform app and binds it at deploy. So it proves a different engine on the same corpus, never a second binding. It reads that source's match rule off the API, since the source belongs to the deployment. A derived schema stops population with `capture_json: false` and `capture_raw: false`, which the engine compiles to dfe-loader's `extracted_only`. -## Three carry the weight +Three steps matter most. **0.2 proves the pitch.** Without it we have tested insertion, not usefulness. **1.3 is `promote`.** It answers "must I model this up front", so it is never optional coverage. -**2.3 is the sharp one.** A source can get its own table and schema and still -have every field reachable only through `_json`. Count rows and it passes. Read -the typed columns and it does not. +**2.3 checks the typed columns.** A source can get a table and schema to itself and +still have every field reachable only through `_json`, which a row count alone would pass. ## The fixture -The beats MODULE level, three concurrent sources -- which also proves the +The beats MODULE level, three concurrent sources -- which proves the receiver routing between several at once rather than one in isolation. The three are what our pre-supplied filebeat VRL actually handles: **Cisco IOS, @@ -77,11 +72,11 @@ Two constraints on that corpus: (`{message, tags, timestamp}`) with a raw syslog line in `message`. transform-elastic works on Elastic's ingest-pipeline format. If those differ, one corpus cannot feed all three unchanged. -- **Known divergence.** `scripts/filebeat/extract.py` strips geoip from the +- **Known divergence.** dfe-transform-vrl's `scripts/filebeat/extract.py` strips geoip from the ported VRL, so transform-elastic emits geo fields the other two never will. Assert around it rather than trip over it. -The VRL also needs the `timezones.csv` enrichment table, and optionally +The VRL needs the `timezones.csv` enrichment table, and optionally `._conf.tz_offset` and `._conf.tz_map`. 2.7 posts the Cisco IOS module alone, through whichever transform app diff --git a/docs/data-evolution.md b/docs/data-evolution.md index dc79564af..4d001e478 100644 --- a/docs/data-evolution.md +++ b/docs/data-evolution.md @@ -3,12 +3,10 @@ DFE is a data evolution platform by design. Data lands by default and it just works. You improve and evolve from there. -## Why it is built this way - There are two ways the industry handles incoming data, and both make you pay up front. -At one end, Elastic and Splunk. Decide the schema before the data arrives, index +At one end, the search and index engines. Decide the schema before the data arrives, index it all, query it fast. Change the schema later and you are reindexing. Send a field you did not plan for and it is either dropped, mapped wrong, or it breaks the mapping for everyone. @@ -22,36 +20,38 @@ reindex, no reingest. That is the best of both without the limitation of either. ## The path +A source moves from the main table to a meta schema to its own topic and table, one choice at a time. + ```mermaid flowchart LR - IN["dfe-receiver
or dfe-fetcher"] + accTitle: L4 Detail -- Data evolution stages + accDescr: In stage 0 dfe-receiver or dfe-fetcher lands data in the main table as _json. Promoting fields gives the source a meta schema in stage 1. In stage 2 the source gets a routing condition and its own topic and table, with an optional transform. - subgraph S0["Stage 0 -- it just works"] - MAIN[("main table")] - JSON["_json
sub-field queryable"] - MAIN --> JSON + subgraph s0["0 It lands"] + main[("main table
_json")] end - - subgraph S1["Stage 1 -- meta schema"] - META["names the fields you USE"] - COLS["real columns
_json still the fallback"] - META --> COLS + subgraph s1["1 Meta schema"] + meta["Meta schema"] end - - subgraph S2["Stage 2 -- DFE source"] - ROUTE["routing condition
key exists / key=value"] - TOPIC["own topic"] - XFORM["transform
(optional)"] - OWN[("own table")] - ROUTE --> TOPIC --> XFORM --> OWN + subgraph s2["2 DFE source"] + route["Routing
condition"] + own[("Own topic
and table")] end - IN --> MAIN - JSON -."promote a field".-> META - COLS --> ROUTE + main -.->|promote| meta + meta -.->|own source| route + route --> own + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class main,own store + class s0,s1,s2 group ``` -Nothing forces you rightwards. A source can sit in stage 0 forever. +*Figure 1. L4 Detail -- Data evolution stages. Dotted arrows are optional steps, and dfe-receiver or dfe-fetcher fills the main table.* Up: [DFE architecture](architecture.md). Down: [L4 Detail -- How a table gets built](#how-a-table-gets-built), [L4 Detail -- What a transform does](#what-a-transform-is-actually-for). + +Nothing forces you down the stages. A source can sit in stage 0 forever, and stage 3 below drops `_json` from a source you know fully. ## Stage 0 -- ground zero @@ -68,68 +68,72 @@ That is the whole of stage 0. Many sources never leave it. Fine. A meta schema names the fields you want out of `_json` and into real columns. -You write one, or you import one. DFE ships fully expansive meta schemas for -common sources, starting with Elastic in 2.2.0. +You write one, or you import one. dfe-schemas ships meta schemas for common +sources under `meta/`: Elastic, Beats, OTel, syslog, the clouds, M365 and more. -What makes it different from traditional ETL and from a data lake's schema -work: +What sets it apart from ETL and from a lake's schema work: - Cover what you USE, not everything you receive. You are never forced to model the full breadth, because `_json` is still there as the slower fallback for anything off the hot path. Partial is a valid answer. -- Written at the primitive level, in the types the engine can actually - store. +- Written in the primitive types the engine stores. - Written at the use-case level. A column declares the question it answers -- `dimension`, `range`, `word_search` -- and DFE picks the ClickHouse index. The label never names the engine primitive. -- Readable by the SME who owns the data. Whoever understands the source and - the workflow can write one. It does not require a data engineer, a ClickHouse - expert or a Kafka expert. +- Readable by the SME who owns the data, with no data engineer, ClickHouse + expert or Kafka expert needed. - Narrowable to ClickHouse specifics where you do want that control. A large meta schema is normally NOT deployed as it stands. It is the base a derived schema selects from. -A meta schema may carry its own opinionated indexes beyond the common header, +A meta schema may hold its own opinionated indexes beyond the common header, but that is unusual and worth a second look when you see it. ### How a table gets built +A deployed table is the common header plus the fields a derived schema selects from a meta schema. + ```mermaid -flowchart TB - HEADER["common header
_json, _timestamp, _timestamp_load, _tags
partitioning, TTL"] - META["meta schema
full breadth, rarely deployed as-is"] - DERIVED["derived schema
selection + one index type per field"] - TABLE[("deployed table")] - - META -->|"select the fields you deploy"| DERIVED - HEADER --> TABLE - DERIVED --> TABLE - META -.->|"small schema, deploy direct"| TABLE +flowchart LR + accTitle: L4 Detail -- How a table gets built + accDescr: A derived schema selects the deployed fields from a meta schema. The deployed table is the common header plus the derived schema. + + meta["Meta schema
full breadth"] + derived["Derived schema
fields and indexes"] + header["Common header
_json, _timestamp"] + table[("Deployed table")] + + meta -->|select fields| derived + derived --> table + header --> table + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class table core ``` -### Two kinds of table, and every table has a common header +*Figure 2. L4 Detail -- How a table gets built.* Up: [L4 Detail -- Data evolution stages](#the-path). -Every schema is built for one of two kinds of data: +A small meta schema deploys as it stands, with no derived schema. Every schema is built for one of two kinds of data: - **Time-series** -- logs, a large fact table, observability data. - **Time-stamped** -- a database or REST dump from dfe-fetcher, which is a point-in-time context or dimension table. -And every table is built on a **common header**. It is not boilerplate. It -carries the mandatory fields, indexes, partitioning and TTL that the rest of the -DFE machinery depends on: +Every table is built on a **common header**, which holds the mandatory fields, +indexes, partitioning and TTL the rest of DFE depends on: - `_json` is the fallback that makes stage 0 work and keeps partial meta schemas honest. - `_timestamp_load` drives partitioning and retention, and `_timestamp`, the event time read from the record's `timestamp`, every time-bounded query. -- `_tags` carries contextual divisions and data routing. +- `_tags` holds contextual divisions and data routing. Remove a header field and something concrete breaks: no `_timestamp_load` and the table cannot partition or expire, no `_json` and a partial meta schema silently -loses every field it did not model. That is why the header is enforced rather -than suggested. +loses every field it did not model. So the header is enforced. ## Stage 2 -- make it a DFE source @@ -149,10 +153,10 @@ because that is what defines its table. Derived schemas: this is where a large meta schema usually gets one. A derived schema is a sub-selection of fields from its parent meta schema, plus -which of those fields carry which index type -- one index type per field, added +which of those fields take which index type -- one index type per field, added and removed as a CRUD operation on both the definition and the deployed table. -Transforms, optional: a source carries at most one transform. DFE ships +Transforms, optional: a source has at most one transform. DFE ships transform-elastic, transform-vrl and transform-vector. ### What a transform is actually for @@ -162,7 +166,7 @@ meta schema: 1. **Parse.** Take a line of text encoded in `message`, `msg`, `description` or similar, and pull it out into dedicated performant columns. -2. **Enrich.** Add what the line itself does not carry -- geoip, IP reputation, +2. **Enrich.** Add what the line itself does not hold -- geoip, IP reputation, asset and identity lookups, a timezone resolved from an abbreviation, and so on. @@ -171,14 +175,31 @@ one opaque blob is queryable only by scanning it. The same line after a transform has `source_ip`, `action` and `rule_name` as real indexed columns, plus the fields the enrichment added. +A transform turns one blob in `_json` into typed, indexed columns. + ```mermaid flowchart LR - RAW["_json.message
'%ASA-6-302013: Built outbound TCP...'
one blob, scan to search"] - XF["transform
parse + enrich"] - COLS["source_ip | dest_ip | action | rule_name | source_country
typed, indexed, matches the schema"] - RAW --> XF --> COLS + accTitle: L4 Detail -- What a transform does + accDescr: A message held as one blob in _json goes through a transform that parses and enriches it into typed, indexed columns that match the schema. + + raw[("_json.message
one blob")] + xform["Transform
parse and enrich"] + cols[("Typed columns
source_ip, action")] + + raw -->|record| xform + xform -->|columns| cols + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + class xform core + class raw,cols store ``` +*Figure 3. L4 Detail -- What a transform does.* Up: [L4 Detail -- Data evolution stages](#the-path). + +The blob in figure 3 is a line such as `%ASA-6-302013: Built outbound TCP...`, and the typed columns are `source_ip`, `dest_ip`, `action`, `rule_name` and `source_country`. + The schema says which columns exist. The transform is what fills them. ## Stage 3 -- drop the safety net @@ -195,24 +216,21 @@ header is what the rest of DFE depends on. What changes is one instruction to the loader, so the blob is never written. That matters: the schema is unchanged, so the decision is reversible. Turn -population back on and new records carry `_json` again. What you do not get back +population back on and new records get `_json` again. What you do not get back is the period it was off -- a field you did not model, for rows loaded while the net was down, is gone rather than slow. It is a maturity statement. Never a default. Nothing pushes you here and most tables should never arrive. -The data plane already does this. dfe-loader's `metadata.capture_mode` +dfe-loader's `metadata.capture_mode` (`full`, `json_only`, `raw_only`, `extracted_only`) decides what it writes, with `metadata.table_capture_modes` for per-table control, and the column directive framework reaches the same end with `@skip` on `_json`. The control plane says it through a derived schema's `capture_json` and `capture_raw`, which the engine compiles into the loader's per-table `table_capture_modes`. -### Column comments carry loader instructions - -Worth knowing before writing a schema, because it explains what a column's -`expr` is really doing. dfe-loader reads directives out of ClickHouse column +A column's `expr` is a loader instruction. dfe-loader reads directives out of ClickHouse column COMMENTs, and every one has an exact config-cascade equivalent. Config wins over the DDL comment. @@ -226,7 +244,7 @@ the DDL comment. `@source` is the vocabulary dfe-schemas emits -- so a meta schema's `expr: "@source: host.name"` becomes a column comment, which is what tells the -loader where to read that field from. This is why a derived schema carries its +loader where to read that field from. So a derived schema keeps its base's `expr` unchanged and only overrides the index. CEL also appears at `@computed` on a column. Routing CEL is held to the Fast @@ -234,15 +252,12 @@ profile because it runs against every record. ## Promote: see it, then give it a column -Meta schema development does not have to start from a blank page. - Select real data in the UI, find a field inside `_json`, and promote it to a dedicated column. The field you just looked at becomes part of the schema. -Traditional schema management asks you to be theoretical and exhaustive about -data you have not seen yet, which is slow and usually wrong. Promotion inverts -it: you work on what is arriving, one field at a time, and the schema grows to -match reality. +Traditional schema work asks you to be exhaustive about data you have not seen, +which is slow and usually wrong. Promotion works on what is arriving, one field +at a time. Why promote at all? `_json` takes whatever arrives. Each value keeps the type it came with, row by row, so `port` can be `8080` in one record and `"http"` in the next. Inserts @@ -262,7 +277,7 @@ So a rule, a hunt or a saved search can pass the day it is written and fail the day a source sends one odd value. And no path inside `_json` has an index, so even the query that works reads every row. -That is the first reason to promote a field, and usually the most important. A +That is the main reason to promote a field. A promoted column has ONE type, so the query is the plain comparison you meant. Then give it a `use_case` so it gets the index that answers how you query it (`exact_match` for an id or an IP, `range` for a number, `word_search` for text). @@ -276,9 +291,7 @@ there for those rows -- it is in `_json`, where it always was, at `_json` speed. Promote a field you want a year of history on and you get the column plus a year of nulls, so the query that spans the change has to read both. -## Proving it - -Every claim above has a step that proves it: +Every claim on this page has a step that proves it: [data-evolution-acceptance.md](data-evolution-acceptance.md). A claim with no step is marketing. diff --git a/docs/data-plane/README.md b/docs/data-plane/README.md index d611a08dc..98989cde1 100644 --- a/docs/data-plane/README.md +++ b/docs/data-plane/README.md @@ -1,17 +1,45 @@ # Data plane -The close-coupled source - ingest - schema - transform path. One source -definition carries routing, schema binding, and field-map bindings together. -The receiver, loader, views, hunts, and query API all read the same -definition, so there is no separate ETL config to drift. The system map is +The close-coupled source, ingest, schema and transform path. One source +definition holds the routing, the schema binding and the field-map bindings +together, and the engine compiles every stage's config from it, so there is no +separate ETL config to drift. The system map is [../architecture.md](../architecture.md). +The engine compiles one source definition into receiver rules, loader table routing and the table with its views. + ```mermaid flowchart LR - SRC[source YAML] --> RCV[receiver] --> LDR[loader] --> CH[(ClickHouse)] - CH --> VIEWS[standard views] --> HUNTS[hunt runner] & QAPI[query API] + accTitle: L4 Detail -- What a source definition compiles into + accDescr: The engine compiles one source definition into the receiver's routing rules, the loader's table routing and the table with its standard views. The hunt runner and the query API read the table and its views. + + source["Source
definition"] + + engine["dfe-engine
compilers"] + + receiver["Receiver
rules"] + loader["Loader
table routing"] + views[("Table and
standard views")] + + hunts["Hunt runner"] + query["Query API"] + + source --> engine + engine --> receiver + engine --> loader + engine --> views + views --> hunts + views --> query + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + class source core + class views store ``` +*Figure 1. L4 Detail -- What a source definition compiles into.* Up: [L3 Components -- dfe-engine](../architecture.md). + ## Documents in this area | Doc | Covers | diff --git a/docs/data-plane/expressions-cel.md b/docs/data-plane/expressions-cel.md index b97488293..8de9a94de 100644 --- a/docs/data-plane/expressions-cel.md +++ b/docs/data-plane/expressions-cel.md @@ -2,8 +2,6 @@ DFE uses **CEL (Common Expression Language)** for conditions across the Python engine, the Rust services and the CLI. CEL is not Turing-complete: no loops, no recursion, no side effects, so every expression terminates. An expression is compiled once and evaluated against every row. -**Packages:** - | Language | Package | Notes | |----------|---------|-------| | Rust | `cel` | Native implementation, through `scalo::expression` | @@ -11,7 +9,7 @@ DFE uses **CEL (Common Expression Language)** for conditions across the Python e scalo-rs's `Cargo.toml` and scalo-py's `pyproject.toml` carry the versions. -**References:** +References: - [CEL Specification](https://github.com/google/cel-spec) - [CEL Overview](https://cel.dev/overview/cel-overview) @@ -20,7 +18,7 @@ scalo-rs's `Cargo.toml` and scalo-py's `pyproject.toml` carry the versions. --- -## What CEL Does NOT Replace +CEL does not replace these: | System | Reason | |--------|--------| @@ -31,11 +29,11 @@ scalo-rs's `Cargo.toml` and scalo-py's `pyproject.toml` carry the versions. --- -## DFE Expression Profile +## The DFE expression profile DFE uses a **subset** of CEL. Expressions outside this profile are rejected at validation time. -### Allowed Operations +Allowed operations: | Category | Operations | Examples | |----------|-----------|---------| @@ -45,7 +43,7 @@ DFE uses a **subset** of CEL. Expressions outside this profile are rejected at v | Arithmetic | `+`, `-`, `*`, `/`, `%` | `price * quantity` | | Ternary | `? :` | `is_admin ? 95 : 50` | -### Allowed Functions +Allowed functions: | Function | Purpose | Example | |----------|---------|---------| @@ -60,7 +58,7 @@ DFE uses a **subset** of CEL. Expressions outside this profile are rejected at v | `string()` | Cast to string | `string(code) == "200"` | | `bool()` | Cast to boolean | `bool(enabled)` | -### Excluded (Rejected at Validation) +Rejected at validation: | Feature | Reason | |---------|--------| @@ -68,9 +66,7 @@ DFE uses a **subset** of CEL. Expressions outside this profile are rejected at v | `timestamp()`, `duration()` | ClickHouse handles time natively -- no need to duplicate | | Macros | Complexity with no benefit for DFE use cases | ---- - -## Supported Types +Supported types: | CEL Type | Python | Rust | ClickHouse | |----------|--------|------|------------| @@ -84,9 +80,9 @@ DFE uses a **subset** of CEL. Expressions outside this profile are rejected at v --- -## Usage Examples +## Where DFE uses it -### Alert Triggers +An alert trigger: ```yaml alerts: @@ -94,9 +90,7 @@ alerts: - when: "result_count >= 10" ``` -A trigger sees only `result_count`, the rows the fire wrote. A trigger on a row field never fires. - -### Source Routing +A trigger sees `result_count`, the rows the fire wrote, and nothing else. A trigger on a row field never fires. A source selects its records with `when`, a CEL condition, in place of `match`. The receiver evaluates it against every inbound record under its CEL Fast profile, so regex, iteration and time functions are refused at save. See [source-definition.md](source-definition.md). @@ -104,7 +98,7 @@ A source selects its records with `when`, a CEL condition, in place of `match`. when: "data_stream.dataset == 'cisco_ios.log' && agent.type == 'filebeat'" ``` -### Computed Values (Transforms) +A computed column in the loader: ```yaml columns: @@ -112,10 +106,6 @@ columns: expr: '@computed: risk_score > 80 ? "high" : risk_score > 40 ? "medium" : "low"' ``` ---- - -## CEL-to-ClickHouse SQL Transpilation - CEL expressions can be mechanically converted to ClickHouse WHERE clauses for query pushdown: | CEL | ClickHouse SQL | @@ -133,23 +123,38 @@ This enables expressions written once in YAML to work both as in-memory evaluati --- -## Implementation Architecture +## Where it is implemented + +Two scalo wrappers enforce the DFE profile: scalo-py's for the engine and scalo-rs's for the loader and receiver. ```mermaid -flowchart TB - PROF["DFE expression profile
a subset of CEL"] - PY["scalo-py scalo.expression
wraps common-expression-language"] - RS["scalo-rs scalo::expression
wraps the cel crate"] - ENG["dfe-engine
hunts/alert.py, rule filters"] - LDR["dfe-loader
src/transform/computed.rs"] - PROF --> PY & RS - PY --> ENG - RS --> LDR +flowchart LR + accTitle: L4 Detail -- CEL implementations + accDescr: The DFE expression profile is enforced by two wrappers. scalo-py's scalo.expression serves dfe-engine. scalo-rs's scalo::expression serves dfe-loader and dfe-receiver. + + profile["DFE expression
profile"] + + py["scalo-py
scalo.expression"] + rs["scalo-rs
scalo::expression"] + + engine["dfe-engine"] + loader["dfe-loader"] + receiver["dfe-receiver"] + + profile --> py + profile --> rs + py --> engine + rs --> loader + rs --> receiver + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + class profile core ``` -Both wrappers enforce the same DFE profile: the same allowed operators, functions and types. +*Figure 1. L4 Detail -- CEL implementations.* Up: [Data plane](README.md). -### API surface +scalo-py wraps `common-expression-language` and scalo-rs the `cel` crate. Both enforce the same allowed operators, functions and types. In the engine the callers are `hunts/alert.py`, the rule filters and source routing. In dfe-loader it is `src/transform/computed.rs`, and in dfe-receiver the filter validation in `src/config/mod.rs`. scalo-py's names (`scalo.expression`): @@ -157,5 +162,5 @@ scalo-py's names (`scalo.expression`): compile_expression(expr) -> Program # Parse + validate + compile evaluate(expr, data) -> Any # One-shot convenience (compile + execute) validate(expr) -> list[str] # Syntax + profile check, returns errors for UI -transpile_to_clickhouse(expr) -> str # The SQL pushdown below +transpile_to_clickhouse(expr) -> str # The SQL pushdown above ``` diff --git a/docs/data-plane/external-pipelines.md b/docs/data-plane/external-pipelines.md index c503cfad5..f1f180e91 100644 --- a/docs/data-plane/external-pipelines.md +++ b/docs/data-plane/external-pipelines.md @@ -4,28 +4,37 @@ You already run Vector, Fluentd, Fluent Bit or Logstash, and you don't want to r There are two places it can plug in, and both are on the bus. A source on the direct (gRPC) transport refuses either one at save with a 422, because there is no topic for your pipeline to meet. -## Two places to plug in +Your pipeline meets a source at its topics, as the origin writing the landing topic or as the transform between the two. ```mermaid -flowchart LR - subgraph origin [External origin] - P1(Your pipeline) -->|produces to| L1[(acme-logs_land)] - L1 -->|read by| LD1[dfe-loader] +flowchart TB + accTitle: L4 Detail -- External pipeline plug-in points + accDescr: As an origin, your pipeline produces to the source's landing topic and the loader reads it. As a transform, the receiver produces to the landing topic, your pipeline reads it and produces to the transformed topic, and the loader reads that. + + subgraph origin["External origin"] + direction LR + pipe1["Your pipeline"] --> land1[("acme-logs_land")] --> loader1["dfe-loader"] end - subgraph transform [External transform] - R[dfe-receiver] -->|produces to| L2[(acme-logs_land)] - L2 -->|read in dfe-external-acme-logs by| P2(Your pipeline) - P2 -->|produces to| T2[(acme-logs_load)] - T2 -->|read by| LD2[dfe-loader] + + subgraph transform["External transform"] + direction LR + receiver["dfe-receiver"] --> land2[("acme-logs_land")] --> pipe2["Your pipeline"] --> load2[("acme-logs_load")] --> loader2["dfe-loader"] end - classDef store fill:#2EA4F6,stroke:#2EA4F6,stroke-width:2px,color:#000647 - classDef ext fill:#8CCDFA,stroke:#2EA4F6,stroke-width:2px,color:#000647 - class L1,L2,T2 store - class P1,P2 ext + origin ~~~ transform + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef async fill:#F6E6EF,stroke:#A8517F,stroke-width:1px,color:#3D1529 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class pipe1,pipe2 ext + class land1,land2,load2 async + class origin,transform group ``` -Rounded light-blue nodes are your pipeline. Cylinders are the topics DFE creates. +*Figure 1. L4 Detail -- External pipeline plug-in points. Dashed boxes are your pipeline.* Up: [Data plane](README.md). + +As a transform, your pipeline reads `acme-logs_land` in the consumer group `dfe-external-acme-logs`. **External origin.** Your pipeline is where the records come from. It produces straight to `_land`, past the receiver, so the source has no match rule and no fetcher. If the source names a DFE transform, that reads `_land` as usual. Otherwise the loader does. @@ -87,7 +96,7 @@ Your pipeline sets `_org_id` and `_source` itself, and the loader holds them on Nothing is refused. A replaced record still lands, and the loader counts it in `loader_external_org_id_replaced_total` or `loader_external_source_replaced_total`. A climbing counter is a pipeline stamping labels it shouldn't, and you'll want to go and look. -The allow-lists are for a provider whose pipeline fans in its own customers' records. List each customer's tenant id in `allowed_org_ids` and the pipeline can land each record in its customer's org: +The allow-lists are for a provider whose pipeline fans in its customers' records. List each customer's tenant id in `allowed_org_ids` and the pipeline can land each record in its customer's org: ```yaml source: msp-firewall @@ -105,8 +114,8 @@ Nothing makes your records carry the DFE common header. The loader fills in a mi | Field | Missing, the loader | Set, you get | |---|---|---| -| `_timestamp` | reads `timestamp`, and takes the load time when it is not there | the event's own time, so time-range queries and retention follow when it happened | -| `_org_id` | reads `_org_id`, and lands the record in the source's org when it is not there | the source's org or one `allowed_org_ids` lists, which is how a provider lands each customer in their own org | +| `_timestamp` | reads `timestamp`, and takes the load time when it is not there | the event's time, so time-range queries and retention follow when it happened | +| `_org_id` | reads `_org_id`, and lands the record in the source's org when it is not there | the source's org or one `allowed_org_ids` lists, which is how a provider lands each customer in that customer's org | | `_source` | reads `_source`, and takes the topic name, which is this source, when it is not there | this source or one `allowed_sources` lists, which picks the table | | `_tags` | reads `_tags`, then `tags`, then `meta`, then `metadata.tags`, and leaves it empty when none is there | tags a query or an alert can filter on without parsing the payload | | `_timestamp_received` | reads `_timestamp_received`, then `timestamp_received`, then `received_at`, and takes the load time when none is there | when your pipeline first saw the event, so you can measure the delay before DFE | diff --git a/docs/data-plane/field-mapping.md b/docs/data-plane/field-mapping.md index 9e27b1857..21c8d13b4 100644 --- a/docs/data-plane/field-mapping.md +++ b/docs/data-plane/field-mapping.md @@ -6,7 +6,7 @@ written in a standard's vocabulary run transparently against DFE tables through generated views - the Sigma converter is a CONSUMER of this layer, not the owner of mapping logic. -## Resolution order +The mapping for a column is the first layer that names it. | Priority | Layer | Description | |----------|-------|-------------| @@ -16,16 +16,15 @@ not the owner of mapping logic. | 4 (lowest) | passthrough | field name used as-is | A `field_map` pin replaces the source-name convention for the override -layer, so one named map can be shared across many sources; a view with only -inline mappings renders even without a registry. +layer, so several sources can share one map. A view with inline mappings and +no pin renders with no registry at all. -Maps are YAML in the config repo (engine-, UI-, or human-editable). The -resolver lives at `src/dfe_engine/fieldmap/resolver.py`; view generation at +Maps are YAML files under `fieldmaps/` in the config directory, editable by the +engine, the console or a person. The resolver is +`src/dfe_engine/fieldmap/resolver.py` and the view generator `src/dfe_engine/fieldmap/view_generator.py`. -## Generated views - -Each standard x source produces a ClickHouse VIEW named +Each standard and source pair produces a ClickHouse VIEW named `{table}_{standard}` (e.g. `windows_audit_sigma`) that aliases DFE columns back to the standard's field names: @@ -38,7 +37,7 @@ SELECT FROM {db}.windows_audit; ``` -A view carries only the mappings whose column the source's table holds. ClickHouse refuses a whole view when one of its columns is missing, and a default map names far more columns than any one table has, so the rest are dropped and logged at debug. A view left with no mapping is not created. The source build, plan, deploy and the sigma view preview all apply the same rule. +A view holds the mappings whose column the source's table has, and drops the rest. ClickHouse refuses a whole view when one of its columns is missing, and a default map names far more columns than any one table has, so each dropped one is logged at debug. A view left with no mapping is not created. The source build, plan, deploy and the sigma view preview all apply the same rule. Zero storage overhead - views compute at query time. Consumers: [hunt-runner-scaling.md](hunt-runner-scaling.md) (detections), diff --git a/docs/data-plane/hunt-match-delivery.md b/docs/data-plane/hunt-match-delivery.md index d9ad3bb6a..a3d757309 100644 --- a/docs/data-plane/hunt-match-delivery.md +++ b/docs/data-plane/hunt-match-delivery.md @@ -8,24 +8,17 @@ # How a hunt match reaches a person -A match is a row in a ClickHouse table. When the hunt names an alert destination, the runner also sends one message for each fire that matched. - -## Where a match goes +A match is a row in a ClickHouse table. When the hunt names an alert destination, the runner sends one message as well for each fire that matched. The hunt runner runs each rule of a hunt on the hunt's schedule. Every row the rule matches is written, in full, to the hunt's target table. The default is `detection` in the data database (`.detection`). A hunt or one of its rules can name another table. Each run is capped at `hunts.max_detections_per_run` rows per rule (`DFE_ENGINE_HUNTS_MAX_DETECTIONS_PER_RUN`), and a rule hitting its cap is logged and counted in the `detections_capped` metric. The runner is described in [hunt-runner-scaling.md](hunt-runner-scaling.md) and the rules in [source-detections.md](source-detections.md). -## Who sees it - -- Search the detection table in Observe, embedded in the console. -- Query it through the [query API](query-api.md). - -Observe's own alerting is switched off for DFE rules, so its webhooks and alert history stay empty. +A person finds a match by searching the detection table in Observe, embedded in the console, or through the [query API](query-api.md). Observe's alerting is switched off for DFE rules, so its webhooks and alert history stay empty. ## Alert destinations -A destination is a name and an [Apprise](https://github.com/caronc/apprise) URL, which covers Slack, email, PagerDuty, Teams and about 90 other services. The URL carries its own credential, so the API masks it on every read. +A destination is a name and an [Apprise](https://github.com/caronc/apprise) URL, which covers Slack, email, PagerDuty, Teams and about 90 other services. The URL holds its credential, so the API masks it on every read. Create destinations through `POST /api/v1/alerts/destinations`, then `PUT` and `DELETE` on `/api/v1/alerts/destinations/{name}`. A name starts with a letter or digit and holds letters, digits, `.`, `_` and `-`, at most 128 characters. @@ -48,13 +41,13 @@ With no trigger the default is `result_count > 0`. Creating a destination with a ## What the runner sends -After a fire commits its rows and its watermark, the runner checks the hunt's `alerts` block and sends one message through Apprise if a trigger holds. The message carries the hunt, the customers, the rules and `result_count`, the rows the fire wrote. +After a fire commits its rows and its watermark, the runner checks the hunt's `alerts` block and sends one message through Apprise if a trigger holds. The message has the hunt, the customers, the rules and `result_count`, the rows the fire wrote. - One message per fire, however many rows it wrote. 10,000 matches in a window is one message. -- Triggers see `result_count` only. The runner does not read the matched rows back, so a trigger over a row field such as `severity` never fires. +- Triggers see `result_count` and nothing else. The runner does not read the matched rows back, so a trigger over a row field such as `severity` never fires. - Destinations are read from the secrets backend on every fire, so a change through the API reaches the next fire with no restart. - A destination that is down, or a secrets backend that cannot be read, is logged and the fire stands. The rows are already written, and re-running the window would write them twice. -- The runner reads each destination by name from the secrets backend its own `settings.secrets` names, which has to be the engine's. On `openbao` and `aws` it needs read access to `alert-destinations/*` and nothing of the engine's volume. -- The `file` directory sits on the engine's own volume. On docker the runner mounts `/alert-destinations` read-only at the same path, and the engine creates that directory at startup so the mount finds it. On Kubernetes no runner pod can mount the engine's volume, so `hunts.alert_destinations_secret` (`DFE_ENGINE_HUNTS_ALERT_DESTINATIONS_SECRET`) names a Secret the engine also writes each destination into, one key per name. The engine needs `get` and `update` on that one Secret, and the runner mounts it read-only at `/alert-destinations` with the `file` provider. Every engine startup brings the Secret back into step, so one recreated empty is refilled. +- The runner reads each destination by name from the secrets backend its `settings.secrets` names, which has to be the engine's. On `openbao` and `aws` it needs read access to `alert-destinations/*` and nothing of the engine's volume. +- The `file` directory sits on the engine's volume. On docker the runner mounts `/alert-destinations` read-only at the same path, and the engine creates that directory at startup so the mount finds it. On Kubernetes no runner pod can mount the engine's volume, so `hunts.alert_destinations_secret` (`DFE_ENGINE_HUNTS_ALERT_DESTINATIONS_SECRET`) names a Secret the engine writes each destination into as well, one key per name. The engine needs `get` and `update` on that one Secret, and the runner mounts it read-only at `/alert-destinations` with the `file` provider. Every engine startup brings the Secret back into step, so one recreated empty is refilled. There is no cooldown: a hunt that matches on every fire sends a message on every fire. diff --git a/docs/data-plane/query-api-python.md b/docs/data-plane/query-api-python.md index 484d2f270..6efe32ccf 100644 --- a/docs/data-plane/query-api-python.md +++ b/docs/data-plane/query-api-python.md @@ -1,6 +1,6 @@ # DFE Query API - Python client -`dfe_engine.query.QueryClient` ships inside the dfe-engine image. dfe-engine is not published to PyPI, so code outside the image calls the [HTTP API](query-api.md) with its own HTTP client and a bearer token or API key. +`dfe_engine.query.QueryClient` ships inside the dfe-engine image. dfe-engine is not published to PyPI, so code outside the image calls the [HTTP API](query-api.md) with any HTTP client and a bearer token or API key. --- @@ -28,7 +28,7 @@ ### Direct Mode (In-Process) -Use inside the engine process. Direct mode runs the view through the engine's own `ViewExecutor` with an admin context: +Use inside the engine process. Direct mode runs the view through the engine's `ViewExecutor` with an admin context: ```python from dfe_engine.query import QueryClient @@ -72,7 +72,7 @@ df = result.to_pandas() ## Security Model -A view is a ClickHouse parameterised view deployed through the governed DDL path and discovered by the engine's `ViewCatalog`. There is no YAML query registry and no per-query role list. The executor passes on only the parameters the view declares, unvalidated, and injects `org_id` from the auth context. A missing or mistyped parameter fails in ClickHouse. +A view is a ClickHouse parameterised view deployed through the governed DDL path and discovered by the engine's `ViewCatalog`. There is no YAML query registry and no per-query role list. The executor passes on the parameters the view declares and drops the rest, unvalidated, and injects `org_id` from the auth context. A missing or mistyped parameter fails in ClickHouse. ```python result = client.query( @@ -103,7 +103,7 @@ class QueryClient: ### Methods -#### `query(query_label, params?, *, limit?, offset?, order_by?, order_dir?, after_key?, tiebreak_by?, after_tiebreak?, time_from?, time_to?, timeout_seconds?, store?, cache?) -> QueryResult` +#### `query(query_label, params?, *, limit?, offset?, order_by?, order_dir?, after_key?, tiebreak_by?, after_tiebreak?, time_from?, time_to?, timeout_seconds?, cache?) -> QueryResult` Execute a labeled query and return result. @@ -128,7 +128,7 @@ result = client.query( ) ``` -**Parameters:** +Parameters: | Parameter | Type | Description | |-----------|------|-------------| @@ -144,8 +144,7 @@ result = client.query( | `time_from` | `str` | Start time (ISO8601) for time-bounded queries | | `time_to` | `str` | End time (ISO8601), defaults to now | | `timeout_seconds` | `int` | Query timeout (clamped to query/global max) | -| `store` | `str` | Accepted and ignored: the view executor always runs in the data database | -| `cache` | `bool` | Allow cached results (default: `True`) | +| `cache` | `bool` | Allow cached results (default: `True`). No server-side result cache exists, so it changes nothing | To get a pandas DataFrame directly, call `.to_pandas()` on the `QueryResult`: @@ -334,7 +333,7 @@ while result.metadata.has_more: ) ``` -`after_key` is compared as the `order_by` column's type, so a timestamp goes as `"2024-01-15T12:00:00"` (with a fraction if the column has one), never a `datetime` or pandas `Timestamp` object. The key must be non-NULL, and without `tiebreak_by` it must be unique, or rows tying with the cursor are skipped. A view that declares its own `limit` parameter refuses keyset and offset paging. +`after_key` is compared as the `order_by` column's type, so a timestamp goes as `"2024-01-15T12:00:00"` (with a fraction if the column has one), never a `datetime` or pandas `Timestamp` object. The key must be non-NULL, and without `tiebreak_by` it must be unique, or rows tying with the cursor are skipped. A view that declares a `limit` parameter refuses keyset and offset paging. ### Pagination Recommendations diff --git a/docs/data-plane/query-api.md b/docs/data-plane/query-api.md index 7cf882c34..b560d84e1 100644 --- a/docs/data-plane/query-api.md +++ b/docs/data-plane/query-api.md @@ -9,7 +9,7 @@ A label-based query interface over ClickHouse parameterised views, with multi-te - No raw SQL from clients - a view is referenced by label, and its SQL is a ClickHouse parameterised view deployed server-side - Mandatory tenant isolation - `org_id` injected from the org the session's account or API key is bound to, cannot be overridden - Scope-based access control - `query:read` browses the views and `query:execute` runs them -- Declared parameters only - the executor passes on only the parameters the view declares, unvalidated, and ClickHouse checks their types +- Declared parameters - the executor passes on the parameters the view declares and drops the rest, unvalidated, and ClickHouse checks their types - Native JSON responses - dict rows via clickhouse-connect, no intermediate wire format ### Supported Datasources @@ -17,35 +17,39 @@ A label-based query interface over ClickHouse parameterised views, with multi-te | Datasource | Description | Wire Format | |------------|-------------|-------------| | `clickhouse` | ClickHouse: parameterised views, and raw read-only queries | JSON | -| `s3` | S3 bucket directory listing, raw route only | JSON | -| `minio` | MinIO (S3-compatible) listing, raw route only | JSON | -| `file` | Local filesystem directory listing, raw route only | JSON | +| `s3` | S3 bucket directory listing, on the raw route | JSON | +| `minio` | MinIO (S3-compatible) listing, on the raw route | JSON | +| `file` | Local filesystem directory listing, on the raw route | JSON | --- ## Security Model -### Example Security Flow +A view execution checks scope, keeps the declared parameters, injects `org_id` and runs the view on ClickHouse. ```mermaid sequenceDiagram - participant C as Client - participant API as dfe-engine API - participant CAT as ViewCatalog - participant EX as ViewExecutor - participant DB as ClickHouse - - C->>API: POST /api/v1/queries/views/hunts/active_threats/execute
params: {severities: ["critical"]} - API->>API: AuthN + RBAC scope query:execute - API->>CAT: Lookup view definition (TTL-cached, discovered from CH) - CAT-->>EX: View + parameter schema - EX->>EX: Keep only the view's declared parameters - EX->>EX: Inject org_id from the auth context (never client-settable) - EX->>DB: Execute the parameterized view - DB-->>C: JSON response (rows + columns + metadata) + accTitle: L4 Detail -- View execution + accDescr: The client posts to a view's execute route. The API checks the caller's scope. The executor looks the view up in the catalog, keeps the parameters the view declares, injects org_id from the auth context and runs the view on ClickHouse, and the rows return to the client as JSON. + participant client as Client + participant api as Engine API + participant catalog as ViewCatalog + participant executor as ViewExecutor + participant ch as ClickHouse + client->>api: Execute view + api->>api: Check query:execute + api->>catalog: Look up view + catalog-->>executor: View and parameters + executor->>executor: Keep declared parameters + executor->>executor: Inject org_id + executor->>ch: Run view + ch-->>api: Rows + api-->>client: JSON ``` -**Client Request** (the view label lives in the URL path): +*Figure 1. L4 Detail -- View execution. Solid arrows are requests, dashed arrows are answers.* Up: [Data plane](README.md). + +The client request names the view in the URL path: ```json { @@ -54,69 +58,63 @@ sequenceDiagram } ``` -**Server-side definition:** a ClickHouse parameterized VIEW, deployed +The server side is a ClickHouse parameterised VIEW, deployed through the governed DDL path (gitops) and discovered by the engine's `ViewCatalog` at runtime - there is no separate YAML query registry in the -engine. The executor passes on only the parameters the view's placeholders +engine. The executor passes on the parameters the view's placeholders declare, and `org_id` is injected from the auth context. It does not check that a required parameter is present or that a value fits its type, so ClickHouse refuses those and the request fails 500 `query_error`. ### Access control on the raw path -`POST /api/v1/queries/raw` executes against a registered datasource adapter (`clickhouse:...`, `s3:...`, `minio:...` or `file:...`). The SQL runs as the engine's own ClickHouse user, not a tenant's, so the route needs `raw_query:execute`. That action sits outside `query:*`, and no built-in role but `admin` holds it. +`POST /api/v1/queries/raw` executes against a registered datasource adapter (`clickhouse:...`, `s3:...`, `minio:...` or `file:...`). The SQL runs as the engine's ClickHouse user, not a tenant's, so the route needs `raw_query:execute`. That action sits outside `query:*`, and no built-in role but `admin` holds it. -ClickHouse runs the query with `readonly=1`. That refuses every write and DDL statement, every `INSERT INTO FUNCTION` (`url()`, `file()`, `s3()`), a read through `url()`, and any `SETTINGS` clause in the query. The request's timeout still applies. The `s3`, `minio` and `file` adapters only list objects and never write, so `readonly` does not apply to them. +ClickHouse runs the query with `readonly=1`. That refuses every write and DDL statement, every `INSERT INTO FUNCTION` (`url()`, `file()`, `s3()`), a read through `url()`, and any `SETTINGS` clause in the query. The request's timeout still applies. The `s3`, `minio` and `file` adapters list objects and never write, so `readonly` does not apply to them. ### The identity a view runs as -A view runs as `dfe_query_reader` (`query_views.restricted_user`), which holds `SELECT` on the data database and ClickHouse's introspection tables and no `SOURCES` grant, so `url()`, `s3()` and `remote()` are refused. Its password is `DFE_ENGINE_QUERY_VIEWS_RESTRICTED_PASSWORD` when set, else the minted user's own: `DFE_ENGINE_CLICKHOUSE_QUERY_READER_PASSWORD` when the deployment provides one, else `ch/service/query_reader` in the engine's secrets store. Its profile pins `readonly=2`, so the executor sends no `readonly` setting. Until the reader has a password, the view routes answer 503 `reader_unprovisioned`. +A view runs as `dfe_query_reader` (`query_views.restricted_user`), which holds `SELECT` on the data database and ClickHouse's introspection tables and no `SOURCES` grant, so `url()`, `s3()` and `remote()` are refused. Its password is `DFE_ENGINE_QUERY_VIEWS_RESTRICTED_PASSWORD` when set, else the minted user's password: `DFE_ENGINE_CLICKHOUSE_QUERY_READER_PASSWORD` when the deployment provides one, else `ch/service/query_reader` in the engine's secrets store. Its profile pins `readonly=2`, so the executor sends no `readonly` setting. Until the reader has a password, the view routes answer 503 `reader_unprovisioned`. --- ## Architecture +A view execution runs on ClickHouse, and a raw query goes through a datasource adapter to ClickHouse or object storage. + ```mermaid -flowchart TB - subgraph Clients["Client Layer"] - HTTP["Any HTTP client"] - end +flowchart LR + accTitle: L4 Detail -- Query routes + accDescr: A client calls the query routes. A view execution goes to the ViewExecutor, which runs the view on ClickHouse. A raw query goes to a datasource adapter, which reads ClickHouse or lists S3, MinIO or a local directory. - subgraph ControlPlane["dfe-engine API layer"] - Auth["AuthN (JWT / OIDC headers / API key)"] - API["POST /api/v1/queries/views/{label}/execute
POST /api/v1/queries/raw"] - end + client["HTTP client"] - subgraph QueryEngine["dfe-engine query module"] - Registry["ViewCatalog
discovered from ClickHouse
TTL-cached"] - Validator["ViewExecutor
declared parameters
auth context injection"] + api["Query routes"] - subgraph Adapters["Datasource Adapters"] - CH["ClickHouse
+ EXPLAIN"] - Storage["Storage
S3/MinIO/FS"] - end + executor["ViewExecutor"] + adapters["Datasource
adapters"] - JSONResp["JSON Response
rows + columns + metadata"] - end + ch[("ClickHouse")] + storage[("S3, MinIO
or files")] - subgraph Backends["Data Backends"] - CHServer[("ClickHouse")] - S3[("S3/MinIO")] - FS[("Filesystem")] - end + client --> api + api -->|view| executor + api -->|raw| adapters + executor --> ch + adapters --> ch + adapters --> storage - Clients -->|"view label + params"| ControlPlane - Auth --> API - API -->|"execute"| Registry - Registry --> Validator - Validator --> Adapters - CH --> CHServer - Storage --> S3 - Storage --> FS - Adapters --> JSONResp - JSONResp -->|"JSON body"| Clients + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + class client ext + class ch,storage store ``` +*Figure 2. L4 Detail -- Query routes. The dashed box is the caller.* Up: [L3 Components -- dfe-engine](../architecture.md). + +The executor looks each view up in the `ViewCatalog`, which discovers the views from ClickHouse and caches them for 60 seconds. Every route authenticates by JWT, OIDC headers or API key first. + --- ## API Reference @@ -129,7 +127,7 @@ All routes live under `/api/v1/queries`: | `GET /api/v1/queries/views/namespaces` | `query:read` | list view namespaces | | `GET /api/v1/queries/views/{label}` | `query:read` | one view definition with parameters | | `POST /api/v1/queries/views/{label}/execute` | `query:execute` | execute a parameterized view | -| `POST /api/v1/queries/raw` | `raw_query:execute` | ad-hoc read-only query against a datasource adapter, as the engine's own ClickHouse user | +| `POST /api/v1/queries/raw` | `raw_query:execute` | ad-hoc read-only query against a datasource adapter, as the engine's ClickHouse user | ### Execute a view @@ -170,7 +168,7 @@ Response: JSON body (metadata embedded alongside rows, no custom headers) ```typescript interface ViewExecuteRequest { - // View parameters (only those the view declares are passed on) + // View parameters (those the view declares are passed on) params?: Record; // Execution options @@ -238,25 +236,32 @@ interface QueryMetadata { ## Pagination -The API supports two pagination modes: +The API pages two ways. Offset paging sends a growing offset, and keyset paging sends the last `order_by` value as the next `after_key`. ```mermaid -flowchart LR - subgraph Offset["Offset-Based"] - O1["Page 1
offset=0"] - O2["Page 2
offset=100"] - O3["Page 3
offset=200"] - O1 --> O2 --> O3 +flowchart TB + accTitle: L4 Detail -- Offset and keyset paging + accDescr: Offset paging sends offset 0, then 100, then 200. Keyset paging sends no after_key for the first page, then the last order_by value of each page as the next page's after_key. + + subgraph offset["Offset"] + direction LR + o1["Page 1
offset 0"] --> o2["Page 2
offset 100"] --> o3["Page 3
offset 200"] end - subgraph Keyset["Keyset-Based"] - K1["Page 1
after_key=null"] - K2["Page 2
after_key=ts1"] - K3["Page 3
after_key=ts2"] - K1 -->|"last value"| K2 -->|"last value"| K3 + subgraph keyset["Keyset"] + direction LR + k1["Page 1
no after_key"] --> k2["Page 2
after_key ts1"] --> k3["Page 3
after_key ts2"] end + + offset ~~~ keyset + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class offset,keyset group ``` +*Figure 3. L4 Detail -- Offset and keyset paging. ts1 and ts2 are the last `order_by` values of pages 1 and 2.* Up: [Data plane](README.md). + ### 1. Offset-Based Pagination Simple pagination using `limit` and `offset`. Best for small datasets. @@ -270,7 +275,7 @@ Simple pagination using `limit` and `offset`. Best for small datasets. } ``` -**Response includes:** +The response has: - `has_more: true` if more rows available - `next_offset: 100` for next page @@ -296,13 +301,13 @@ The rules a keyset page follows: - The key must be non-NULL: a row whose key is NULL is never returned. - Without a tiebreak the key must be unique: a row tying with the cursor is skipped. For a key that is not unique, add `tiebreak_by` (a unique, non-NULL second column) and send `after_tiebreak` with `after_key`. - `after_key` and `offset` cannot be combined. -- A view that declares its own `limit` parameter caps its rows before any outer paging, so `order_by`, `after_key` and `offset` on it are refused with 400. Page it with `limit` only. +- A view that declares a `limit` parameter caps its rows before any outer paging, so `order_by`, `after_key` and `offset` on it are refused with 400. Page it with `limit` alone. --- ## Query Labels -A view is referenced by a label in `namespace/name` form, the label goes in the URL path, and the body carries only `params` and `options`. `GET /api/v1/queries/views` lists the labels a deployment has, and `GET /api/v1/queries/views/namespaces` its namespaces. +A view is referenced by a label in `namespace/name` form, the label goes in the URL path, and the body holds `params` and `options`. `GET /api/v1/queries/views` lists the labels a deployment has, and `GET /api/v1/queries/views/namespaces` its namespaces. ```text POST /api/v1/queries/views/analytics/user_activity/execute @@ -433,15 +438,33 @@ there is no per-query role list: | `query:execute` | execute views | | `raw_query:execute` | run raw datasource queries. Only `admin` holds it | +Every query route answers 403 without the route's scope, and otherwise injects `org_id` and runs the query. + ```mermaid -flowchart TD - Start["Incoming request"] --> AuthN["AuthN (JWT / OIDC headers / API key)"] - AuthN --> Scope{"has the route's
RBAC scope?"} - Scope -->|"No"| Deny403["403 Forbidden"] - Scope -->|"Yes"| Inject["Executor injects org_id
from the auth context"] - Inject --> Allow["Execute against ClickHouse"] +flowchart LR + accTitle: L4 Detail -- Query route authorisation + accDescr: An authenticated request is checked for the route's scope. Without it the answer is 403. With it the executor injects org_id from the auth context and runs the query on ClickHouse. + + request(["Authenticated
request"]) + scope{"Route
scope?"} + inject["Inject org_id"] + deny(["403"]) + run(["Run on ClickHouse"]) + + request --> scope + scope -->|yes| inject + scope -.->|no| deny + inject --> run + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class scope decision + class deny error ``` +*Figure 4. L4 Detail -- Query route authorisation. The dotted arrow is the refusal.* Up: [Data plane](README.md). + Tenant isolation is structural, not per-query config: the executor injects `org_id` from the authenticated context (never client-settable), and the ClickHouse user's GRANTs + row policies bound what any deployment exposes. @@ -450,22 +473,7 @@ ClickHouse user's GRANTs + row policies bound what any deployment exposes. ## Wire Format -### Native JSON - -All responses are a single native JSON body: - -```text -Content-Type: application/json -``` - -**Benefits:** - -- No client-side deserialization library required -- Human-readable, easy to debug (curl, browser devtools) -- Universal cross-language support -- Rows arrive as plain dicts/objects, ready to consume - -### Metadata Embedding +Every response is one JSON body (`Content-Type: application/json`), with each row a plain object. Query metadata is returned as top-level fields alongside `rows` and `columns` in the JSON response body - not in headers or a separate schema: diff --git a/docs/data-plane/schema-bootstrap.md b/docs/data-plane/schema-bootstrap.md index 13e2af0e3..1ef4afe76 100644 --- a/docs/data-plane/schema-bootstrap.md +++ b/docs/data-plane/schema-bootstrap.md @@ -1,7 +1,7 @@ -# Schema bootstrap -- the engine is the only controller +# Schema bootstrap -- the engine is the one controller dfe-engine applies every ClickHouse object and every bootstrap Kafka topic a DFE -deployment needs, at its own startup, from the pinned `dfe-schemas` wheel. No +deployment needs, at its startup, from the pinned `dfe-schemas` wheel. No other component creates, alters or drops a database, table, view, role or topic: not a Job, not an init container, not an app at runtime, not the OTel collector. @@ -15,12 +15,12 @@ release. The phase sits inside the API lifespan, ahead of readiness (`src/dfe_engine/schema/phase.py`): -1. **Connect**, retrying an unreachable ClickHouse for +1. Connect, retrying an unreachable ClickHouse for `DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS` (default 180). The datastore can be in the same deploy wave, so not-up-yet and broken are different answers. -2. **Resolve the schema tree**: the wheel inside the image, read-only, then the +2. Resolve the schema tree: the wheel inside the image, read-only, then the deployment's optional overlay directory (`DFE_ENGINE_SCHEMAS_OVERLAY_DIR`). -3. **Render the plan** for the ClickHouse platform dfe-schemas probes off the live +3. Render the plan for the ClickHouse platform dfe-schemas probes off the live server, unless `DFE_ENGINE_CLICKHOUSE_PLATFORM` pins it. Every table is an argumentless `Replicated*` engine with no `ON CLUSTER`, inside a Replicated database on self-hosted ClickHouse and the Shared database on ClickHouse Cloud. A self-hosted @@ -32,21 +32,21 @@ The phase sits inside the API lifespan, ahead of readiness throw_only_active`, so a down replica catches up from the database log rather than holding the DDL until it times out. A CREATE or DROP DATABASE ON CLUSTER replays from the DDL task queue instead, so a replica down longer than its - `task_max_lifetime` (7 days by default) misses it. Every rendered object carries a checksum taken + `task_max_lifetime` (7 days by default) misses it. Every rendered object has a checksum taken with the platform token removed, so one schema reads as one checksum on either platform. The pattern is at -4. **Take the lease** in `schema_lock`. A second replica waits for the holder to finish, then takes the lease and applies. One whose wait runs out first applies nothing, reports `observed` and stays NotReady, and the engine runs the pass again in the background. An expired lease is taken over. A source deploy takes the same lease around its DDL, waits for it up to `DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS`, and answers 503 `schema_locked` when the wait runs out. Claims wait for a majority of ClickHouse replicas, so with a majority down the lease cannot be taken and the pass reports `observed`. A two-replica cluster cannot take the lease while either replica is down, because a majority of 2 is 2. A claim ClickHouse acknowledges 2 seconds or more after its free read is not kept and is not released either, so it holds the lease until its expiry. -5. **Compare** against `schema_migrations` and against the live catalogue, and +4. Take the lease in `schema_lock`. A second replica waits for the holder to finish, then takes the lease and applies. One with a wait that runs out first applies nothing, reports `observed` and stays NotReady, and the engine runs the pass again in the background. An expired lease is taken over. A source deploy takes the same lease around its DDL, waits for it up to `DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS`, and answers 503 `schema_locked` when the wait runs out. Claims wait for a majority of ClickHouse replicas, so with a majority down the lease cannot be taken and the pass reports `observed`. A two-replica cluster cannot take the lease while either replica is down, because a majority of 2 is 2. A claim ClickHouse acknowledges 2 seconds or more after its free read is not kept: its marker row is ended, so it blocks nobody and the attempt claims again (`src/dfe_engine/schema/lock.py`, `src/dfe_engine/clickhouse/lease.py`). +5. Compare against `schema_migrations` and against the live catalogue, and apply what is additive. -6. **Record** every applied object: its kind, the dfe-schemas release, the engine +6. Record every applied object: its kind, the dfe-schemas release, the engine release, the checksum, the statement and the action. -7. **Create the topics**, create-only, skipping with a logged notice where the +7. Create the topics, create-only, skipping with a logged notice where the deployment has no bus. -8. **Release the lease** and report. +8. Release the lease and report. `schema_migrations` and `schema_lock` are manifest objects like any other and the manifest declares them first, so the same pass creates them before it records -anything into them. The engine carries no DDL even for its own bookkeeping. +anything into them. The engine holds no DDL even for its bookkeeping. ## What is applied, and what is refused @@ -56,8 +56,8 @@ Applied automatically: - a column the definition adds and the table lacks is added - a JSON table's settings are brought to the platform's: `MODIFY SETTING` where a self-hosted table lacks `write_marks_for_substreams_in_compact_parts = 0`, - `RESET SETTING` where a Cloud table carries it -- a plain view whose SELECT changed is replaced, because it holds no state + `RESET SETTING` where a Cloud table has it +- a plain view with a changed SELECT is replaced, because it holds no state - a role, settings profile, quota or grant is re-asserted, because every statement in that set converges @@ -70,7 +70,7 @@ Refused and named on the status route: A refusal is not a failed pass: the object is named, the rest of the manifest still converges, and readiness is still reported. An operator applies the change -deliberately with `dfe schema apply --allow-drift`, which carries out the two +deliberately with `dfe schema apply --allow-drift`, which makes the two changes ClickHouse has an operation for -- a column type and a TTL. An `ORDER BY` or `PARTITION BY` change stays refused either way, because the table has to be rebuilt. @@ -85,10 +85,10 @@ consulted -- the pod is not restarted out from under whoever is reading it. ## The overlay -A deployment's own schema tree is additive. It may declare objects the core does -not. An object of its own whose definition sits under a core directory -(`tables/core/`, `tables/internal/`, `tables/otel/`, `tables/meta/`, -`common-header/`, `registries/`, `roles/`, `topics/`, `views/`) is refused by name +A deployment's schema tree is additive. It may declare objects the core does +not. A deployment object defined under a core directory +(`tables/core/`, `tables/internal/`, `tables/meta/`, `tables/otel/`, +`tables/usage/`, `common-header/`, `registries/`, `roles/`, `topics/`, `views/`) is refused by name and reported, and the core apply is unaffected. The overlay is optional: an install with no deploy repo boots core-only. @@ -98,7 +98,7 @@ install with no deploy repo boots core-only. |---|---|---| | `DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_TABLES` | `true` | Apply the manifest at startup. Off reports the schema state as unknown and gates nothing. | | `DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS` | `180` | How long to keep retrying an unreachable ClickHouse before reporting the pass failed. | -| `DFE_ENGINE_CLICKHOUSE_DDL_MAX_QUERY_SIZE` | `16777216` | Ceiling in bytes on the `max_query_size` sent with schema DDL. A statement over ClickHouse's 256 KiB default, such as a wide Elastic meta schema's table, is sent with its own size plus 4 KiB, by the manifest apply and by a source deploy alike. One that would need more is refused before it is sent. A caller's own SQL never gets the setting. | +| `DFE_ENGINE_CLICKHOUSE_DDL_MAX_QUERY_SIZE` | `16777216` | Ceiling in bytes on the `max_query_size` sent with schema DDL. A statement over ClickHouse's 256 KiB default, such as a wide Elastic meta schema's table, is sent with its size plus 4 KiB, by the manifest apply and by a source deploy alike. One that would need more is refused before it is sent. A caller's SQL never gets the setting. | | `DFE_ENGINE_CLICKHOUSE_PLATFORM` | `auto` | The ClickHouse platform schema DDL is rendered for: `auto` probes the server, then reads the hostname. `cloud` or `self_hosted` pins it, for a server the probe reads wrongly. Anything else refuses to start. | | `DFE_ENGINE_KAFKA_BOOTSTRAP_TOPICS` | `true` | Create the declared topic set at startup, create-only. | | `DFE_ENGINE_KAFKA_TIERED_STORAGE` | `false` | The brokers tier to object storage, so the landing topic is created with `remote.storage.enable`. Off leaves the key unset rather than false, which is what a broker-level setting needs. | @@ -116,7 +116,7 @@ engine, no tables. - `GET /api/v1/system/schema` -- the last pass, object by object: state, versions, platform, per-object action and checksum, the refused list, the overlay refusals, the topics created -- `GET /api/v1/system/version` -- carries the dfe-schemas release as `schemas` +- `GET /api/v1/system/version` -- reports the dfe-schemas release as `schemas` - `dfe schema plan` -- what an apply would do, changing nothing - `dfe schema apply [--allow-drift]` -- the boot phase by hand, under the same lease @@ -130,13 +130,13 @@ engine, no tables. ## What the engine still renders itself -Per-SOURCE objects are defined at use time from an operator's own config, so they +Per-SOURCE objects are defined at use time from an operator's config, so they are not manifest data: - a source's table and its standard views, rendered by `src/dfe_engine/schema/schema_builder_v2.py` with the engine form and platform settings dfe-schemas supplies -- an operator's own quota tiers and the per-org and per-group ClickHouse users, +- an operator's quota tiers and the per-org and per-group ClickHouse users, reconciled by `src/dfe_engine/governance/ch/` Both go through `SchemaApplier`, so the one DDL gate cannot be bypassed. diff --git a/docs/data-plane/schema-classes.md b/docs/data-plane/schema-classes.md index b781cebcd..d41213138 100644 --- a/docs/data-plane/schema-classes.md +++ b/docs/data-plane/schema-classes.md @@ -24,18 +24,38 @@ overview: [schema.md](schema.md). ## Build pipeline +`SchemaBuilderV2.build` composes and validates the columns, then writes the CREATE TABLE and the standard views for one source version. + ```mermaid flowchart LR - SRCY[Source YAML] --> B[SchemaBuilderV2.build] - PROF[profile YAML] --> B - META[meta schema YAML] --> B - B --> V[validate vs TypeRegistry] - V --> DDL[DDLGenerator
CREATE TABLE + indexes] - V --> VIEWS[ViewGenerator
standard views] - DDL --> RES[SchemaBuildResult] - VIEWS --> RES + accTitle: L4 Detail -- Schema build + accDescr: SchemaBuilderV2 loads the header profile's columns and the source's schema columns, composes them, and validates them against the TypeRegistry. DDLGenerator writes the CREATE TABLE and ViewGenerator the standard views, and both go into the SchemaBuildResult. + + profile["Header profile"] + schema["Source schema
meta, derived, additional"] + + compose["SchemaLoader
compose"] + + validate["Validate against
TypeRegistry"] + + ddl["DDLGenerator
CREATE TABLE"] + views["ViewGenerator
standard views"] + + result["SchemaBuildResult"] + + profile --> compose + schema --> compose + compose --> validate + validate --> ddl + validate --> views + ddl --> result + views --> result + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A ``` +*Figure 1. L4 Detail -- Schema build. Validation errors go into the result and do not stop the build.* Up: [Data plane](README.md). + ## Key classes ### SchemaColumn (`src/dfe_engine/source/models.py`) diff --git a/docs/data-plane/schema-sync.md b/docs/data-plane/schema-sync.md index f1ba8d598..fd4512d65 100644 --- a/docs/data-plane/schema-sync.md +++ b/docs/data-plane/schema-sync.md @@ -6,29 +6,53 @@ The engine renders and applies the DDL. dfe-loader reads the deployed table back The dfe-schemas package holds the common-header profiles (`common-header/`), the source meta schemas (`meta/`), the hunt output schemas (`hunts/`) and the registries (`registries/`): table engines, the type system (primitive to ClickHouse type) and the default field maps. -The engine reads the version-pinned `dfe-schemas` wheel through `importlib.resources` and ships no schema files of its own. It looks in `DFE_ENGINE_SCHEMAS_DIR`, then the installed package, then the image's seed directory (copied out of that wheel at build), and when none carries what it needs, loading fails with `No dfe-schemas tree carries ...: set DFE_ENGINE_SCHEMAS_DIR or install dfe-schemas` (`src/dfe_engine/schema/schema_loader.py`). +The engine reads the version-pinned `dfe-schemas` wheel through `importlib.resources` and ships no schema files. It looks in `DFE_ENGINE_SCHEMAS_DIR`, then the installed package, then the image's seed directory (copied out of that wheel at build), and when none has what it needs, loading fails with `No dfe-schemas tree carries ...: set DFE_ENGINE_SCHEMAS_DIR or install dfe-schemas` (`src/dfe_engine/schema/schema_loader.py`). ## The deployed table is the runtime contract +The engine applies the DDL, and dfe-loader reads the table back from `system.columns`. + ```mermaid flowchart LR - S[dfe-schemas wheel] --> E[dfe-engine
renders + applies DDL] - E --> CH[(ClickHouse table)] - CH -->|system.columns,
column COMMENTs| L[dfe-loader] + accTitle: L4 Detail -- Engine and loader schema contract + accDescr: The dfe-schemas wheel and the source definition feed the engine. The engine renders the DDL and applies it to the ClickHouse table. dfe-loader reads the table's columns and comments back from system.columns. + + subgraph defs["Definitions"] + schemas["dfe-schemas
wheel"] + source["Source
definition"] + end + + engine["dfe-engine"] + + table[("ClickHouse
table")] + + loader["dfe-loader"] + + defs --> engine + engine -->|DDL| table + table -->|system.columns| loader + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class table store + class defs group ``` +*Figure 1. L4 Detail -- Engine and loader schema contract.* Up: [Data plane](README.md). + dfe-loader reads three things from the table, never a profile YAML or a source definition: 1. Column types, from `system.columns`. -2. Column COMMENTs, which carry the loader directives. -3. The table COMMENT, which carries `@profile` and `@profile_version` for a header-composed table (dfe-schemas `render.py`). +2. Column COMMENTs, which hold the loader directives. +3. The table COMMENT, which holds `@profile` and `@profile_version` for a header-composed table (dfe-schemas `render.py`). -The loader caches that schema for `schema.cache_ttl_secs` (300 by default, dfe-loader `docs/config-schema.yaml`), so an applied change reaches it on the next refresh, and until then it carries on with the schema it holds. +The loader caches that schema for `schema.cache_ttl_secs` (300 by default, dfe-loader `docs/config-schema.yaml`), so an applied change reaches it on the next refresh, and until then it keeps using the schema it holds. ## What has to agree -- Directive vocabulary. The loader parses `@skip`, `@default`, `@source`, `@renamed`, `@computed` and `@coerce` (dfe-loader `src/column_meta/mod.rs`). `tests/unit/test_services/test_loader_directive_contract.py` restates that list as a literal and fails when the engine writes a directive the loader does not carry. +- Directive vocabulary. The loader parses `@skip`, `@default`, `@source`, `@renamed`, `@computed` and `@coerce` (dfe-loader `src/column_meta/mod.rs`). `tests/unit/test_services/test_loader_directive_contract.py` restates that list as a literal and fails when the engine writes a directive the loader does not parse. - Types. The engine maps primitives through dfe-schemas `registries/types.yaml`. The loader parses whatever ClickHouse type `system.columns` reports. - Header columns. Both sides take them from the profile in `common-header/`. -No cross-repo CI job runs the loader's type parser over engine-generated DDL, and the loader has no profile migration of its own: the engine's schema apply adds the columns a new profile version brings. +No cross-repo CI job runs the loader's type parser over engine-generated DDL, and the loader has no profile migration: the engine's schema apply adds the columns a new profile version brings. diff --git a/docs/data-plane/schema.md b/docs/data-plane/schema.md index c070f7885..6e66d0a9b 100644 --- a/docs/data-plane/schema.md +++ b/docs/data-plane/schema.md @@ -5,17 +5,14 @@ wheel), registries included, and the engine keeps no copies. --- -## Audience - -This document is written for **data subject matter experts** -- people who +This document is written for data subject matter experts -- people who understand their data, their queries, and their retention requirements. You do NOT need to be a ClickHouse expert. The schema system picks the codecs, indexes, Nullable wrapping and type widths. You pick meaningful types and describe how your data will be queried. The engine does the rest. If you need a specific ClickHouse type that the primitives don't cover, -there's an escape hatch (`ch_override`). But you shouldn't need it for 95% -of columns. +there's an escape hatch (`ch_override`). Most columns never need it. --- @@ -35,14 +32,7 @@ The engine reads the Source YAML, generates DDL, and applies it to ClickHouse. The loader reads the deployed table's columns and comments from `system.columns` at runtime. There is nothing in between. -Schema changes flow in one direction: - -```text -Source YAML (intent) -> engine generates DDL -> ClickHouse (deployed truth) - ^ - dfe-loader reads it at runtime -``` - +Schema changes flow one way, drawn in [schema-sync.md](schema-sync.md). dfe-loader reads column types, comments and table metadata from `system.columns` at runtime, never from the YAML. It caches them for `schema.cache_ttl_secs`, so an applied change reaches it on the next refresh @@ -54,7 +44,7 @@ Schema definitions are **YAML**. YAML supports structured data (lists of attributes, nested overrides) and is consistent with the rest of the Source definition format. -**CSVs remain only for enrichment files** used by transform stages +CSV is left for the enrichment files transform stages use (e.g. GeoIP lookups, threat intelligence feeds, asset inventories). These are data files, not schema definitions. @@ -349,7 +339,7 @@ YAML -- multiple attributes can be combined: | Attribute | What It Does | |-----------|-------------| | `lowcardinality` | Retired -- the old spelling of `cardinality: low` | -| `nullable` | Allows NULL values (2x performance cost -- use only when NULL != empty) | +| `nullable` | Allows NULL values, at a storage and query cost -- use it when NULL means something other than empty | | `not_null` | Explicitly prevents NULL (overrides the primitive's default) | | `materialized` | Column computed on insert, not stored in source data | | `alias` | Virtual column computed at query time | @@ -361,7 +351,7 @@ enforces valid combinations: | Attribute | Valid Primitives | Why | |-----------|-----------------|-----| -| `lowcardinality` | `string`, `text`, `integer`, `float`, `date`, `ip` | Dictionary encoding only works on types with a finite value space. `json`, `geo_point` are not supported. | +| `lowcardinality` | `string`, `text` | The two the registry allows it on (dfe-schemas `registries/types.yaml`, `attributes`) | | `nullable` | all | Any type can be nullable | | `not_null` | all | Any type can be forced non-null | | `materialized` | all | Computed on insert -- any type | @@ -385,7 +375,7 @@ ClickHouse stores `Nullable(T)` as two columns -- the data column plus a UInt8 bitmap tracking which rows are null -- so it costs storage and query speed. Booleans and `timestamp` are non-null by default for this reason. -**Design decision: ORDER BY keys must not be Nullable.** ClickHouse +ORDER BY keys must not be Nullable. ClickHouse refuses a Nullable column in a sorting key unless the merge-tree setting `allow_nullable_key = 1` is enabled, and even with it on a null in the sort key cripples index effectiveness. Rather than silently flip @@ -429,9 +419,8 @@ keep it off your ORDER BY and high-filter columns. Use cases describe **how you query the column** -- not how it's stored. The engine translates use cases into ClickHouse indexes and codecs in the -generated DDL. You don't need to know what a `set(0)` or -`tokenbf_v1(8192, 4, 0)` index is -- just pick the use case that matches -your query pattern. +generated DDL. You don't need to know what a `set(0)` or a `text` index is -- +pick the use case that matches your query pattern. ### Use Cases @@ -496,7 +485,7 @@ vocabulary. | `similarity_search()` | `vector_similarity('hnsw', 'cosineDistance', )` | 100000000 | One index per part. `hnsw` is the only method the pinned ClickHouse implements | | *(empty)* | No index | -- | | -**Note on text indexes:** `word_search` and `substring_search` generate the GA +`word_search` and `substring_search` generate the GA text index instead of the older bloom-filter based `tokenbf_v1` and `ngrambf_v1`. The text index is deterministic (no false positives) and filters at row level rather than granule level. Every table gets the text index. @@ -507,7 +496,7 @@ a `TableSpec` from the source and calls it. An index whose answer changes is dropped and added again on the next apply. The new index covers parts written or merged after that, and existing parts gain it on merge: there is no `MATERIALIZE INDEX`. -**Note on word_search vs substring_search:** Both use the text index but with +`word_search` and `substring_search` both use the text index but with different tokenizers. `word_search` uses word-level tokenization (`splitByNonAlpha`) -- good for searching whole words in log messages. `substring_search` uses character n-grams -- good for fragments inside longer @@ -646,7 +635,7 @@ The engine rebuilds it after every source deploy or delete. At startup it rebuil Each rebuild applies the tenant row policy ([rbac.md](../control-plane/rbac.md)) to `all_sources` itself. Member tables keep their own policies, which also fence a read through it. -The engine registers it on every team as the Observe source `all_sources`, timed on `_timestamp_load` because `_timestamp` reads as epoch 0 for a passthrough member. Registration retries with backoff from 60 s to 15 min until dfe-hyperdx accepts it. +The engine registers it on every team as the Observe source `all_sources`, timed on `_timestamp_load` because `_timestamp` reads as epoch 0 for a passthrough member. Registration retries with a growing backoff until dfe-hyperdx accepts it (`src/dfe_engine/hyperdx/engine_sources.py`). ## The `dlq` Table @@ -654,54 +643,14 @@ The engine registers it on every team as the Observe source `all_sources`, timed --- -## Sigma Column Mappings - -Sigma rules reference fields by standardised names (e.g. `SourceIP`, -`CommandLine`, `EventID`). These must map to actual ClickHouse column names. - -### Approach: ClickHouse View per Sigma Taxonomy - -Rather than embedding Sigma field names in the base schema, create -**ClickHouse views** that expose Sigma-compatible column aliases: - -```sql --- Auto-generated from Source schema + Sigma field mapping -CREATE VIEW {db}.{source}_sigma AS -SELECT - source_ip AS SourceIP, - dest_ip AS DestinationIP, - user_name AS User, - process_name AS Image, - command_line AS CommandLine, - event_id AS EventID, - * -FROM {db}.{source} -``` - -**Benefits:** - -- Base schema stays clean (ClickHouse-native names) -- Sigma rules query the view (standard Sigma field names) -- Field mapping is per-source (different sources map differently) -- Zero storage overhead (view, not materialised table) -- Mappings resolved by `src/dfe_engine/sigma/source_mapper.py` from the source's field map and sigma view +## Sigma views -### Sigma Mapping as Source Metadata - -The Source definition declares its naming-standard views in the generic -`views` list. The sigma entry carries the logsource binding and overrides: - -```yaml -# sources/windows_audit.yaml -views: - - standard: sigma - taxonomy: windows # Sigma logsource product binding - custom_mappings: # Per-source overrides (win over field_map) - CommandLine: command_line - ParentCommandLine: parent_cmd -``` - -The engine generates the Sigma view from source schema + the sigma view entry. +Sigma rules use standard field names (`SourceIP`, `CommandLine`, `EventID`). A +source declares a `sigma` entry in its `views`, and the engine generates a view +that aliases its columns to those names, resolved by +`src/dfe_engine/sigma/source_mapper.py` from the source's field map. The view and +its YAML are in [source-detections.md](source-detections.md), and the mapping +layer in [field-mapping.md](field-mapping.md). --- @@ -712,23 +661,32 @@ The Elastic/OpenSearch index template converter meta schema from a supplied Elastic template JSON, as YAML in the Source schema format: -```text -Elastic Template JSON - │ - ▼ -Template Converter (dfe-engine) - │ Maps Elastic types -> primitives - │ Maps Elastic analyzers -> use_cases - │ Preserves field hierarchy as flat columns - │ - ▼ -Source schema YAML (new format) - │ name, type, attribute, use_case, comment - │ - ▼ -Attached to a Source definition +The template converter turns an Elastic index template into a source meta schema in YAML. + +```mermaid +flowchart LR + accTitle: L4 Detail -- Elastic template conversion + accDescr: An Elastic index template goes to the template converter, which maps Elastic types to primitives and analyzers to use cases. It writes a source meta schema in YAML, which a source definition then names. + + template["Elastic index
template"] + converter["Template
converter"] + schema["Source meta
schema YAML"] + source["Source
definition"] + + template --> converter + converter --> schema + schema --> source + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + class template ext ``` +*Figure 1. L4 Detail -- Elastic template conversion. The dashed box comes from outside DFE.* Up: [Data plane](README.md). + +The converter flattens the field hierarchy into columns and writes `name`, +`type`, `attribute`, `use_case` and a `comment` naming the Elastic type. + ### Elastic -> Primitive Type Mapping | Elastic Type | Primitive | Use Case | Notes | @@ -757,11 +715,8 @@ See [source-definition.md](source-definition.md) for the full Source definition. ### Rules and hunts -A **Rule** is a detection condition (a SQL WHERE fragment or a CEL filter) on a -source's table, so it sees exactly the columns the source's schema defines. A -**Hunt** groups rules, names the customers whose rows they read, and carries the -interval schedule. Matches land in `.detection` unless the hunt or -rule names another table. Current rule and hunt files: +A rule is a detection condition on a source's table, and a hunt groups rules and +holds the schedule. Their files and fields are in [source-detections.md](source-detections.md). ### Sigma Rules (converted) diff --git a/docs/data-plane/source-definition.md b/docs/data-plane/source-definition.md index c3d571c44..88fbd4cb5 100644 --- a/docs/data-plane/source-definition.md +++ b/docs/data-plane/source-definition.md @@ -133,11 +133,11 @@ views: # --- Schema (mandatory) --- # The source owns its ClickHouse table schema: one source = one table = one schema. schema: - meta_schema: logs_beats_filebeat # Base field definitions (YAML) + meta_schema: meta/beats/filebeat # Base field definitions, a dfe-schemas path meta_schema_version: 1.0.0 - derived_schema: derived/beats/filebeat_auth # Column SELECTION from the meta schema (optional) - derived_schema_version: 1.0.0 # Unset follows the derived schema's own current - additional_fields: filebeat/add # Extra fields, indexes (optional) + # derived_schema: derived// # Column SELECTION from the meta schema (optional) + # derived_schema_version: 1.0.0 # Unset follows the derived schema's own current + # additional_fields: additional// # Extra fields, indexes (optional) ttl_days: 90 # Data retention engine: MergeTree # Base variant only (MergeTree, ReplacingMergeTree(...), ...) # The Replicated prefix is added at DDL time @@ -167,7 +167,7 @@ table, both on by default. Turning them off tells dfe-loader to stop populating A write carrying the 2.1 keys `sigma`, `mapping_standards` or `field_mappings` is rejected with a "removed in 2.2" error. Declare naming-standard views under `views`. -## The smallest source that works +The smallest source that works: ```yaml source: syslog @@ -188,16 +188,33 @@ header schema. Fields are added incrementally later. ## Schema ownership -Each source owns exactly one ClickHouse table schema, 1:1: +Each source owns exactly one ClickHouse table. Its columns come from the common header, then the meta or derived schema, then the additional fields. -```text -Source: filebeat - └── Schema: logs_beats_filebeat - ├── meta_schema.yaml (base columns, types, use cases) - ├── derived_schema.yaml (which of those columns this table keeps) - └── additional_fields.yaml (enrichment fields) +```mermaid +flowchart LR + accTitle: L4 Detail -- Source table columns + accDescr: The source table takes the common header profile's columns first. Then it takes the meta schema's columns, narrowed by the derived schema when there is one. Then it takes the additional fields. + + subgraph cols["Columns, in order"] + direction LR + header["Common header"] ~~~ meta["Meta schema, or its
derived selection"] ~~~ additional["Additional fields"] + end + + table[("Source table")] + + cols --> table + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class table store + class cols group ``` +*Figure 1. L4 Detail -- Source table columns. Read the columns left to right.* Up: [Data plane](README.md). + +With no derived schema the table takes every meta schema column. + The type registry (dfe-schemas `registries/types.yaml`) maps primitives to ClickHouse types and is global, not per-source. The schema section feeds `SchemaBuilderV2` (`src/dfe_engine/schema/schema_builder_v2.py`). Create and migrate @@ -210,7 +227,8 @@ columns: - name: user_name type: string use_case: dimension - comment: "@source: first(user_id/uid/id)" + expr: "@source: first(user_id/uid/id)" + comment: "User identifier" - name: source_ip type: ip use_case: range @@ -219,7 +237,8 @@ columns: use_case: word_search ``` -Loader directives in `comment` survive into the ClickHouse column comment. The +The loader directive in `expr` and the prose in `comment` land together in the +ClickHouse column COMMENT. The full type system, the primitive-to-ClickHouse mapping and the use cases are in [schema.md](schema.md). diff --git a/docs/data-plane/source-detections.md b/docs/data-plane/source-detections.md index da2a2f908..e0899fd44 100644 --- a/docs/data-plane/source-detections.md +++ b/docs/data-plane/source-detections.md @@ -26,8 +26,6 @@ where_clause: "event_id = 4625" original_sql: SELECT * FROM dfe.winaudit WHERE event_id = 4625 ``` -### Conditions - A `condition` is one test on a field, or `all` or `any` of several, nested to `x-dfe-max-nesting` groups. `negate` inverts a test or a group. The engine compiles it to `original_sql` and `where_clause` and stores it beside them, so `GET /api/v1/rules/{name}` returns the `condition` and the console reopens its builder on it. A rule saved as SQL has no condition, and a `PUT` with SQL drops one. ```json @@ -47,8 +45,6 @@ The `RuleCondition` schema in `openapi.json` publishes the operators, the kind o A field is a column of the source table when the table has one by that name, otherwise a path inside `_json` (`_json.` in front is accepted). A path is read as the type of its operand. A row without the field fails every test but `exists`, and `negate` does not change that, so `exists` negated finds the rows that lack it. Every value is quoted and every field name is split into identifier parts, so no operand reaches the SQL as text. The hunt window bounds time for every rule, so a condition cannot compare `_timestamp` or the other time columns with `lt`, `lte`, `gt` or `gte`, and a save or preview says so. A text operand cannot hold `{window}` or `{unseen}`, which the runner fills in wherever they sit. A condition is always `source_type` `raw`. -### Preview - `POST /api/v1/rules/preview` takes `{source, condition, cel_filter?, cost_window_minutes?}` and runs what a save runs under the same org fence, then stops. Send `name` when editing a rule so the count follows the hunts that run it. A condition the save would refuse still answers 200, with the reasons in `sql_errors` or `matches_everything`. ```json @@ -57,11 +53,11 @@ A field is a column of the source table when the table has one by that name, oth "matches_everything": null, "cost_estimate": {"estimated_rows": 2, "band": "ok"}} ``` -`GET /api/v1/hunts?rule=` lists the hunts whose `rules` include exactly that name. +`GET /api/v1/hunts?rule=` lists the hunts with exactly that name in `rules`. ## Hunts -A hunt groups rules, names the customers whose rows they read, and carries the +A hunt groups rules, names the customers the rules read for, and holds the schedule. The runner runs `rate` schedules: `anchored` hunts are recognised and skipped. @@ -75,13 +71,29 @@ customers: [acme] rules: [brute_force_login] ``` +The hunt runner runs each rule over the hunt's window and writes every matched row to the detection table. + ```mermaid -flowchart TD - R[rule: condition on a source table] --> H[hunt runner, on the hunt's interval] - H -->|runs each rule over the window| M[matches] - M --> A[(data database .detection)] +flowchart LR + accTitle: L4 Detail -- Rule to detection table + accDescr: A hunt groups rules and carries the schedule. The hunt runner runs each rule over the hunt's window and writes every matched row to the detection table. + + rules["Rules"] + hunt["Hunt"] + runner["Hunt runner"] + detection[("detection
table")] + + rules --> hunt + hunt --> runner + runner -->|matched rows| detection + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + class detection store ``` +*Figure 1. L4 Detail -- Rule to detection table.* Up: [Data plane](README.md). + Matches land in `.detection` unless the hunt (`global_target_table_name`) or a rule entry (`target_table_name`) names another table. How the runner scales is @@ -90,7 +102,7 @@ match is [hunt-match-delivery.md](hunt-match-delivery.md). ## Hunt customers -A hunt's `customers` are org names. The engine resolves each to the org's tenant ids when it saves the hunt and stores them as `customer_tenant_ids`, and the runner reads only rows whose `_org_id` is one of them. Every org create, update and delete stores that org's tenant ids afresh in each hunt naming it, as one deploy-repo change, before the org itself is written. A deleted org keeps its name in `customers` and loses its stored ids, so the hunt fails every fire as `unknown_customers` and holds its watermark until someone edits it. A hunt carrying a direct `query` instead of `rules` is trusted SQL from the operator-controlled deploy repo, and no customer holds it. +A hunt's `customers` are org names. The engine resolves each to the org's tenant ids when it saves the hunt and stores them as `customer_tenant_ids`, and the runner reads the rows with one of them in `_org_id` and no others. Every org create, update and delete stores that org's tenant ids afresh in each hunt naming it, as one deploy-repo change, before the org itself is written. A deleted org keeps its name in `customers` and loses its stored ids, so the hunt fails every fire as `unknown_customers` and holds its watermark until someone edits it. A hunt carrying a direct `query` instead of `rules` is trusted SQL from the operator-controlled deploy repo, and no customer holds it. In production+team the hunt change waits on review while the org is written at once. So an org create or update is refused with 409 `tenant_id_held` when it takes a tenant id a hunt reads for another org, on main or on an unmerged review branch. Merge or delete that branch, or edit the hunt, to release the id. A squash merge leaves the branch behind, so turn on the forge's auto-delete of merged head branches. The startup seed skips such an org, and one the forge cannot check, until the next boot. @@ -98,9 +110,9 @@ Accepted residual: the runner keeps the hunts it loaded until its next reload (` ## Lookup databases -A rule condition reads its own row and one more kind of table: a table of a database `hunts.lookup_databases` lists (`DFE_ENGINE_HUNTS_LOOKUP_DATABASES`, comma-separated, empty by default). The forms are `x IN (SELECT FROM .
[WHERE ])`, `x IN .
` and `x IN (.
)`, with `NOT` and `GLOBAL` as usual. Any other subquery, join, union or table after `IN` is refused when the rule is saved, in a governance action and when the runner compiles it. +A rule condition reads its row and one more kind of table: a table of a database `hunts.lookup_databases` lists (`DFE_ENGINE_HUNTS_LOOKUP_DATABASES`, comma-separated, empty by default). The forms are `x IN (SELECT FROM .
[WHERE ])`, `x IN .
` and `x IN (.
)`, with `NOT` and `GLOBAL` as usual. Any other subquery, join, union or table after `IN` is refused when the rule is saved, in a governance action and when the runner compiles it. -The runner's ClickHouse user reads every org, so a listed database must hold shared reference data, such as threat-intel indicators, and no org's rows. Each of its tables must be a plain store of its own rows, on a `MergeTree`-family, `Log`, `TinyLog`, `StripeLog`, `Memory` or `EmbeddedRocksDB` engine, with no `_org_id` column. So a view, `Merge`, `Distributed`, `Set` or `Join` table is refused. The runner refuses to start while one is there, and stops at the reload that first finds one. The engine refuses to start too when ClickHouse answers at its start. When it does not, the engine starts with the check unrun and logs a warning, and the runner's check still applies. `system`, `information_schema` and the data database cannot be listed. The runner's `hunt_runner` service role needs `SELECT` on each listed database, added to its grants as a governed-ops edit. +The runner's ClickHouse user reads every org, so a listed database must hold shared reference data, such as threat-intel indicators, and no org's rows. Each of its tables must hold its rows itself, on a `MergeTree`-family, `Log`, `TinyLog`, `StripeLog`, `Memory` or `EmbeddedRocksDB` engine, with no `_org_id` column. So a view, `Merge`, `Distributed`, `Set` or `Join` table is refused. The runner refuses to start while one is there, and stops at the reload that first finds one. The engine refuses to start too when ClickHouse answers at its start. When it does not, the engine starts with the check unrun and logs a warning, and the runner's check still applies. `system`, `information_schema` and the data database cannot be listed. The runner's `hunt_runner` service role needs `SELECT` on each listed database, added to its grants as a governed-ops edit. ## Sigma views diff --git a/docs/data-plane/source-flow.md b/docs/data-plane/source-flow.md index bd6f4909c..d979ca6cd 100644 --- a/docs/data-plane/source-flow.md +++ b/docs/data-plane/source-flow.md @@ -10,59 +10,82 @@ A source names where its records come in, whether they are transformed, and where they land. Everything else - topics, routing rules, endpoints, the instances that run the stages - is compiled from that definition by the engine and deployed through gitops. A record's path is predictable from the source alone, and a hand edit to a compiled artefact is drift the engine reports and re-syncs. -## The flow in one diagram +## The flow on each transport Two transports carry the same three stages. The bus (Kafka on the `single` and `scale` profiles) holds records between stages, so a stage can be down and nothing is lost. Direct gRPC (`slim`, `mesh`) stores nothing between stages, so the receiver's own buffer is the only slack. +On the bus, a source's records go through its landing topic, its transform and its transformed topic to the loader. + ```mermaid flowchart LR - subgraph INPUT - R[receiver pool] - F[fetcher
one per source] - end - subgraph BUS[bus form] - TL[(source_land)] - TO[(source_load)] - end - subgraph TRANSFORM - T[transform instance
one per source, optional] - end - subgraph OUTPUT - L[loader pool] - A[archiver] - end - R --> TL - F --> TL - TL --> T --> TO - TO --> L - TL -.->|no transform| L - TL --> A - L --> CH[(ClickHouse
source table, else main)] + accTitle: L4 Detail -- Source flow on the bus + accDescr: The receiver pool and the source's fetcher write the source's landing topic. The source's transform reads it and writes the transformed topic, which the loader pool reads. The archiver reads the landing topic. + + subgraph input["Input"] + receiver["Receiver pool"] + fetcher["Fetcher
one per source"] + end + + land[("source_land")] + + transform["Transform
one per source"] + archiver["Archiver"] + + load[("source_load")] + + loader["Loader pool"] + + input --> land + land --> transform + land -.-> archiver + transform --> load + load --> loader + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef async fill:#F6E6EF,stroke:#A8517F,stroke-width:1px,color:#3D1529 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class land,load async + class input group ``` +*Figure 1. L4 Detail -- Source flow on the bus. With no transform the loader reads `source_land` itself.* Up: [Data plane](README.md). + +The loader writes the source's own table, else `main`. The dotted arrow is the optional archive path. + +On direct gRPC the receiver pushes each matched record to the loader, and to the archiver when the source is archived. + ```mermaid flowchart LR - subgraph INPUT - R[receiver pool] - end - subgraph OUTPUT - L[loader pool
scalo Push listener] - A[archiver
scalo Push listener] - end - R -->|matched destination| L - R -.->|archive: true| A - L --> CH[(ClickHouse
source table, else main)] + accTitle: L4 Detail -- Source flow on direct gRPC + accDescr: The receiver pool pushes each matched record to the loader pool, and to the archiver when the source is archived. The loader writes the record to ClickHouse. + + receiver["Receiver pool"] + + loader["Loader pool"] + archiver["Archiver"] + + ch[("ClickHouse")] + + receiver -->|matched| loader + receiver -.->|archived| archiver + loader --> ch + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + class ch store ``` -Direct runs the receiver to loader path alone (the second diagram). A fetcher or a transform is an app of its own per source, and addressing a pool of those over gRPC needs the mesh, which is not GA. A source that needs one is refused at save on a direct deployment, and the compile leaves it out. `GET /api/v1/system/deployment` reports `transports.per_source_apps: false` there, and each `GET /api/v1/sources/catalogue` entry carries `runnable`, `runnable_reason` and `runnable_intakes`, judged by the same resolver as the save. +*Figure 2. L4 Detail -- Source flow on direct gRPC. The loader and the archiver each take the push on a scalo Push listener.* Up: [Data plane](README.md). -The archiver keeps the RAW record as it arrived, before any transform. On the bus it reads the landing topic. On direct the receiver fans the record out to it beside the loader. A profile that apps.yaml does not offer dfe-archiver in, or that does not run it unasked (`default_in`) while no archiver instance is deployed, refuses `archive: true` at save, naming the profile, while a source stored with it before still reads and compiles. +Direct runs the receiver to loader path alone (figure 2). A fetcher or a transform is one app per source, and addressing a pool of those over gRPC needs the mesh, which is not GA. On a direct deployment a source that needs one is rejected at save and left out of the compile. `GET /api/v1/system/deployment` reports `transports.per_source_apps: false` there, and each `GET /api/v1/sources/catalogue` entry has `runnable`, `runnable_reason` and `runnable_intakes` from the same resolver as the save. -The first hop parses only the transport. dfe-receiver and dfe-fetcher parse the transport layer (a syslog header, a beats envelope) into JSON with the payload in `message`. Content-dependent parsing, such as a product's message body through grok, happens in a transform reading `message`. Raw capture (`_raw`) is off by default because it doubles every record. A first-hop parse bug is fixed in the receiver or fetcher, never by turning raw capture on. +The archiver keeps the record as it arrived, before any transform. On the bus it reads the landing topic. On direct the receiver sends the record to it beside the loader. `archive: true` is rejected at save, naming the profile, where apps.yaml does not offer dfe-archiver in the profile, or does not run it by default (`default_in`) and no archiver instance is deployed. A source stored with it before still reads and compiles. + +The first hop parses the transport and stops. dfe-receiver and dfe-fetcher turn the transport layer (a syslog header, a beats envelope) into JSON with the payload in `message`. Parsing the content, such as a product's message body through grok, happens in a transform reading `message`. Raw capture (`_raw`) is off by default because it doubles every record. A first-hop parse bug is fixed in the receiver or fetcher, never by turning raw capture on. ## Two transports, one per deployment -A source declares `transport: bus` or `transport: direct`. Left unset, it takes the deployment's own, which follows the profile's `kafka.mode`: `disabled` means direct, anything else means bus. Every stage is bound to that one at deploy, so it is the only transport the deployment carries and a source naming the other is refused at save with the reason - `GET /api/v1/system/deployment` reports the same one transport, so a console never offers the other. Separately, a flow is never mixed: every stage of one source runs on the same transport. +A source declares `transport: bus` or `transport: direct`. Left unset, it takes the deployment's transport, which follows the profile's `kafka.mode`: `disabled` means direct, anything else means bus. Every stage is bound to that transport at deploy, so a source naming the other is rejected at save with the reason. `GET /api/v1/system/deployment` reports the same transport, so a console never offers the other. A flow is never mixed: every stage of one source runs on one transport. | Profile | Transport | Between stages | Balancing between pools | |---|---|---|---| @@ -73,7 +96,7 @@ A source declares `transport: bus` or `transport: direct`. Left unset, it takes | scale | bus | Kafka topics on a broker cluster | consumer groups | | mesh (in the charts, refused by dfe-infra's bootstrap) | direct | gRPC | a listener per stage pool, see [../deployment/transports.md](../deployment/transports.md) | -The source records bus-versus-direct only. Which bus (Kafka) and which direct protocol (gRPC) are deployment facts, so a second bus provider joins behind the same `transport: bus` value without a change to the source model. +The source records bus or direct and nothing more. Which bus (Kafka) and which direct protocol (gRPC) are deployment facts, so a second bus provider joins behind the same `transport: bus` value with no change to the source model. ## Naming is generated, never typed @@ -81,14 +104,16 @@ The source records bus-versus-direct only. Which bus (Kafka) and which direct pr |---|---|---| | landing topic | `
.sql`) reached the deploy repo with - the new column. + the new column. The engine writes no `ddl/` directory: it applies the DDL to + ClickHouse itself, so this test fails as written. ## test_data_evolution.py -- the acceptance path, step by step One test per step of `docs/data-evolution-acceptance.md`, named for its number, in the order the path runs: a record lands with nothing declared, a nested `_json` path answers, a shipped schema refuses a write, a field promotes, a -source gets its own routing and typed table, an index moves on the live table, +source gets its routing and typed table, an index moves on the live table, and a transform parses a syslog line into columns. A step that cannot pass yet is `xfail(strict=True)` naming its open issue, so -the run goes red when the cause is fixed rather than staying quietly green. 2.7 is -dfe-engine#1173. Nothing here is softened to make a run green. +the run goes red when the cause is fixed rather than passing quietly. 2.7 is +dfe-engine#1173. Nothing here is softened to make a run pass. The steps share one source and one table, so run this file in ONE worker -- pass `-p no:xdist`, because workers would split the sequence across processes that fight over it. Two waits are the product's, not -the harness's: a new source's first record waits on dfe-loader's 60 s topic +the harness's: a fresh source's first record waits on dfe-loader's 60 s topic refresh, and a promoted column cannot be judged inside the loader's 300 s directive cache, so 1.3 sends a fresh record each poll until the cache turns over. 2.7 additionally needs the per-source transform instance RUNNING and the -receiver restarted onto the new source's routing. A deploy does neither, and the +receiver restarted onto the fresh source's routing. A deploy does neither, and the API reports `restart_required: []` while the receiver still needs one. Each 1.3 run leaves a forked schema behind. `meta/syslog` is core, so promote -forks it to `_syslog` before adding the column; teardown removes the +forks it to `_syslog` before adding the column. Teardown removes the source and drops its table, and deliberately leaves the fork, because deleting a schema some other version may reference is not a test's call. On a long-lived stack they accumulate one per run and are cleared by hand. -## test_filebeat_pipeline.py -- the transform layer on real data +## test_filebeat_pipeline.py -- the transform layer on corpus data The whole source path with a transform in it: receiver -> `filebeat_land` -> a transform -> `filebeat_load` -> loader -> ClickHouse. Real cisco corpus data, read from an archive in memory and never unpacked (it is Elastic-licensed). -Four tests: an event with the routing discriminator reaches its own source +Four tests: an event with the routing discriminator reaches its source table, one without it stays on the default path, the transform populates ECS columns, and an unschema'd source still lands under the common header. It is NOT parameterised over dfe-transform-vrl and dfe-transform-vector. They run the same VRL, read the same land topic and emit to the same load topic, and a landed row names neither, so two cases would be one experiment run twice. -`DFE_E2E_TRANSFORM` names the deployed app in the failure message only. +`DFE_E2E_TRANSFORM` names the deployed app in the failure message and nowhere else. -`test_external_pipelines.py` puts the customer's own pipeline in both places it -plugs in: a real Vector as a source's origin, and a real Fluent Bit as its +`test_external_pipelines.py` puts the customer's pipeline in both places it +plugs in: a running Vector as a source's origin, and a running Fluent Bit as its transform, each configured from the source's `connection` block. A compliant record, a bare one and one claiming another tenant each land once, in the -source's own org under its own name. The containers run on the docker daemon the +source's org under its name. The containers run on the docker daemon the shell points at, so they need a broker they can reach: `DFE_E2E_KAFKA_BOOTSTRAP`, plus `DFE_E2E_DOCKER_NETWORK` for a dfe-docker stack's compose network. Config is copied in rather than mounted, so a remote daemon works. -## Assumed deployment +## Running it -A `single` or `standard` k8s tier (see docs/architecture.md "Deployment tiers"): +The suite assumes a `single` or `scale` k8s tier (see [docs/deployment/README.md](../../docs/deployment/README.md)): dfe-engine, dfe-ui, the receiver, ClickHouse, HyperDX, and a reachable deploy repo. The filebeat file additionally needs a transform app deployed for the `filebeat` source, and its table built from `meta/beats/filebeat.yaml`. -## Run - export DFE_E2E_RECEIVER_URL=https://dfe. # or the receiver LB export DFE_E2E_CH_HOST= # + _CH_USER/_CH_PASSWORD/_CH_DB export DFE_E2E_HYPERDX_URL=https://hyperdx. # + _HYPERDX_API_KEY @@ -129,26 +127,26 @@ repo. The filebeat file additionally needs a transform app deployed for the export DFE_E2E_TRANSFORM=dfe-transform-vrl # which transform is deployed uv run pytest tests/e2e -m live -o addopts="" -Each var that is missing skips only the tests that need it. `DFE_E2E_VERIFY=1` -enforces TLS verification; it is left OFF above because a deployment usually -fronts these with its own CA and the tests assert the data path, not the cert -chain. Set it when the endpoints carry a publicly-rooted certificate. +Each var that is missing skips the tests that need it and no others. +`DFE_E2E_VERIFY=1` enforces TLS verification. It is left OFF above because a +deployment usually fronts these with a private CA and the tests assert the data +path, not the cert chain. Set it when the endpoints have a publicly-rooted certificate. ## Status `test_filebeat_pipeline.py` has been run against a live deployment. The routing -test passed. The rest failed, on two real product defects rather than on the +test passed. The rest failed, on two product defects rather than on the harness: the loader rejects IPv4 literals for an IPv6 column so every event carrying `source.ip` is dropped (dfe-loader#127), and a batched receiver POST was forwarded whole so the loader was handed an array where one event belongs (dfe-loader#128, fixed in dfe-receiver). Worth knowing before reading a pass as proof: the ECS assertion rests on -`log_file_path`, which the bundled VRL sets only in its cisco_umbrella branch. -Of the events a run posts, roughly a third can satisfy it, so a green result +`log_file_path`, which the bundled VRL sets in its cisco_umbrella branch and nowhere else. +Of the events a run posts, roughly a third can satisfy it, so a passing result says the umbrella path transformed -- not that all three corpus modules did. -`test_live_pipeline.py` has NOT been run green end to end. +`test_live_pipeline.py` has NOT passed as a whole run. `test_data_evolution.py` has been run against the 2.2.2 compose stack. 1.3, 3.2 and 3.3 pass and 2.7 is the one xfail. 1.1 expects a 409 where the engine lets a From 4a81c36fa6d6f2c41285a2791f38fdcef9a83185 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 13:55:33 +1100 Subject: [PATCH 2/2] docs(engine): fix four documentation errors Restore "Buffering without a bus" in transports.md. The deployment README links to it. dfe-infra's mesh receiver profile still configures it. dfe-transform-vector scales on ScalingPressure on the single, scale and mesh profiles, as the archiver and the other transforms do. The charts poll the shim at /keda/pressure. That path is an alias for the pressure query on the generic /q/{name} route. The view execution sequence shows the API fetching the view through the executor. ClickHouse answers the executor rather than the API. --- docs/data-plane/query-api.md | 19 ++++++++++--------- docs/deployment/keda-scaling.md | 5 ++--- docs/deployment/transports.md | 10 ++++++++++ 3 files changed, 22 insertions(+), 12 deletions(-) diff --git a/docs/data-plane/query-api.md b/docs/data-plane/query-api.md index b560d84e1..7598b1114 100644 --- a/docs/data-plane/query-api.md +++ b/docs/data-plane/query-api.md @@ -25,25 +25,26 @@ A label-based query interface over ClickHouse parameterised views, with multi-te ## Security Model -A view execution checks scope, keeps the declared parameters, injects `org_id` and runs the view on ClickHouse. +The route checks `query:execute` and fetches the view through the executor. The executor then keeps the declared parameters, injects `org_id` and runs the view on ClickHouse. ```mermaid sequenceDiagram accTitle: L4 Detail -- View execution - accDescr: The client posts to a view's execute route. The API checks the caller's scope. The executor looks the view up in the catalog, keeps the parameters the view declares, injects org_id from the auth context and runs the view on ClickHouse, and the rows return to the client as JSON. + accDescr: The client posts to a view's execute route. The API asks the executor for the view, which reads it from the catalog. The API then calls execute, and the executor keeps the declared parameters, injects org_id and runs the view on ClickHouse. The rows return to the executor, the result to the API and JSON to the client. participant client as Client participant api as Engine API - participant catalog as ViewCatalog participant executor as ViewExecutor + participant catalog as ViewCatalog participant ch as ClickHouse client->>api: Execute view - api->>api: Check query:execute - api->>catalog: Look up view - catalog-->>executor: View and parameters - executor->>executor: Keep declared parameters - executor->>executor: Inject org_id + api->>executor: get_view + executor->>catalog: Look up view + catalog-->>executor: View definition + executor-->>api: View definition + api->>executor: execute executor->>ch: Run view - ch-->>api: Rows + ch-->>executor: Rows + executor-->>api: Result api-->>client: JSON ``` diff --git a/docs/deployment/keda-scaling.md b/docs/deployment/keda-scaling.md index 5d3996a04..647db81b2 100644 --- a/docs/deployment/keda-scaling.md +++ b/docs/deployment/keda-scaling.md @@ -79,7 +79,7 @@ scalo registers the composite as a bare `scaling_pressure` gauge. An app emits i With no matching rows the query answers 0. That is a successful answer, so it replaces the cached value rather than holding it, and the CPU trigger sets the replica count. A failed query (ClickHouse unreachable, a timeout, a bad query) alone returns the last-good value, or the `cold_hold` of 0 when there is none yet. -KEDA polls the shim at `GET /q/`. Its built-in queries are in `src/dfe_engine/keda_shim/queries.yaml`, and a YAML file at `DFE_ENGINE_KEDA_SHIM_QUERY_CONFIG` is deep-merged over them, so a changed query or a new signal is a config edit with no rebuild. A list in the override replaces the built-in list. Each query takes these keys: +The charts' pressure trigger polls `GET /keda/pressure?service=` (dfe-infra `argocd/values/apps/_fragments/pressure-trigger.yaml`), the alias for the `pressure` query of the generic `GET /q/{name}` route, and `/keda/hunt-backlog` is the alias for `backlog`. Its built-in queries are in `src/dfe_engine/keda_shim/queries.yaml`, and a YAML file at `DFE_ENGINE_KEDA_SHIM_QUERY_CONFIG` is deep-merged over them, so a changed query or a new signal is a config edit with no rebuild. A list in the override replaces the built-in list. Each query takes these keys: | Key | Meaning | |---|---| @@ -103,8 +103,7 @@ The docs site fills the table below from the dfe-infra chart renders at the stac | App | KEDA | Triggers by default | |-----|------|---------------------| | dfe-receiver / dfe-loader | on | CPU ~70% and scalo ScalingPressure (shim, target 70) | -| dfe-archiver, dfe-transform-elastic / vrl | on | CPU ~70%, plus ScalingPressure on the single, scale and mesh profiles | -| dfe-transform-vector | on | CPU ~70% | +| dfe-archiver, dfe-transform-elastic / vrl / vector | on | CPU ~70%, plus ScalingPressure on the single, scale and mesh profiles | | dfe-ui | on | CPU ~70% | diff --git a/docs/deployment/transports.md b/docs/deployment/transports.md index 55ec2ab87..4ce74d686 100644 --- a/docs/deployment/transports.md +++ b/docs/deployment/transports.md @@ -62,6 +62,16 @@ flowchart LR Values in the deploy repo select the broker. `kafka.provider` chooses which operator deploys it in-cluster: Strimzi by default, Redpanda behind a licence gate (`redpanda.acceptLicense`). `kafka.mode: external` with `kafka.external.bootstrap` points the stack at a broker you run, and `kafka.external.provider` names it: `msk`, `confluent-cloud` or `redpanda-cloud` (dfe-infra `helm/charts/kafka/values.yaml`). Deployed brokers and MSK use SCRAM-SHA-512, and Confluent Cloud takes its API key over SASL PLAIN. The apps reject a broker that takes IAM auth alone, such as MSK Serverless (`msk_iam`, scalo-rs `src/transport/kafka/config.rs`). Nothing in the engine changes for any of these. Sizing and the swap matrix: [backing-services.md](backing-services.md). +## Buffering without a bus + +On the direct transport no stage stores records, and no fetcher runs, so the receiver is the one stage that holds them. What it holds depends on the receiver's acknowledgement mode, and dfe-infra's `apps/dfe-receiver/profile-mesh.yaml` sets the figures for both. + +With acknowledgements on, the default, the receiver builds no buffer or spool. It holds each request until the loader confirms delivery. Once that hold expires the sender gets a 503 and retries from its copy. + +With acknowledgements off, the buffer and spool apply. mesh sets `buffer.memory_limit` to 3 GiB of the pod's 4Gi limit, so back-pressure starts at 2.4 GiB, and turns on `buffer.spillover` with 20 GiB of node-local spool per replica. A loader outage shorter than that is invisible to senders, and a longer one back-pressures them (503) until the loader returns. A record held there has had its 2xx, so it is lost if the pod dies, the trade for turning acknowledgements off. + +The knobs are the receiver's (`config.example.yaml`, `buffer:`), so a deployer tunes them in the overlay like any other dial. + ## The mesh tier balances between pools with a listener per pool Status: design built, never run against a live cluster. The charts, the `mesh` profile and the engine's pool addressing are on main, and dfe-infra's `bootstrap.sh` stops on `DFE_PROFILE=mesh` ([dfe-infra#875](https://github.com/hyperi-io/dfe-infra/issues/875)).