From bab0d9dc5bb72aeffded653bbee1972feb033e25 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 13:41:22 +1100 Subject: [PATCH 1/2] docs(console): correct docs and redraw all ten Mermaid diagrams Corrections cover every page under docs/, the README and CONTRIBUTING. They reflect the code, dfe-engine, dfe-hyperdx, dfe-infra and hyperi-ci. - Correct the documented LOG_LEVEL range from DEBUG to ERROR to TRACE to CRITICAL. - build-and-deploy.md explains the chart destination and `make build`. It says an empty NEXT_PUBLIC_API_URL sends the browser to the console's origin for the engine. - authentication.md says renewal sets the dfe_token cookie again. - The Flow card docs cover a source fed by a pipeline. - Redraw all ten Mermaid diagrams to the diagram standard. The L3 console diagram goes at the top of architecture.md. The rest are L4. Three become sequence diagrams and one a state diagram. - Cut config comments to pointers and short reasons. - Move the yarn age gate's CVE exception to CONTRIBUTING. - Remove the Dockerfile comment describing a cache order the file does not have. Signed-off-by: Derek --- .dockerignore | 1 - .github/workflows/ci.yml | 2 +- .github/workflows/macos-dev-smoke.yml | 5 +- .gitignore | 4 +- .yarnrc.yml | 19 +---- CONTRIBUTING.md | 62 ++++++-------- Dockerfile | 15 +--- README.md | 24 +++--- apps/dfe-core-ui/.gitignore | 4 +- docs/architecture.md | 112 +++++++++++++++++--------- docs/authentication.md | 61 +++++++------- docs/build-and-deploy.md | 42 +++++++--- docs/console-and-the-flow.md | 101 ++++++++++++----------- docs/create-schema-form.md | 44 ++++++---- docs/engine-api-client.md | 29 +++++-- docs/hyperdx-rule-create.md | 4 +- docs/local-development.md | 110 +++++++++++++------------ docs/observe-embed.md | 35 +++++--- osv-scanner.toml | 2 +- 19 files changed, 375 insertions(+), 301 deletions(-) diff --git a/.dockerignore b/.dockerignore index a733abfbd..707cbed76 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,4 +1,3 @@ -# Keep the build context lean + never ship host artefacts into the image. **/node_modules **/.next **/.turbo diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fc365b203..2479ba7dc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,5 +44,5 @@ jobs: tag: ${{ inputs.tag || '' }} from-head: ${{ inputs.from-head || '' }} bump: ${{ inputs.bump || 'auto' }} - # The called workflow is our own hyperi-io/hyperi-ci ts-ci.yml (not a third party) and needs the full secret set for publishing (registry tokens, R2 credentials, GH App token). Matches the documented hyperi-ci.yml template. + # hyperi-ci's reusable workflow publishes, so it needs every secret. secrets: inherit # nosemgrep: yaml.github-actions.security.secrets-inherit.secrets-inherit diff --git a/.github/workflows/macos-dev-smoke.yml b/.github/workflows/macos-dev-smoke.yml index de066a720..e89c0be62 100644 --- a/.github/workflows/macos-dev-smoke.yml +++ b/.github/workflows/macos-dev-smoke.yml @@ -5,10 +5,7 @@ # License: BUSL-1.1 # Copyright: (c) 2026 HYPERI PTY LIMITED # -# Runs scripts/macos-dev-smoke.py, the same file a developer runs on a Mac. The -# GitHub macOS runner has no docker, so the engine runs with no ClickHouse. A Linux -# leg runs the script's docker mode, the one a Mac with OrbStack or Docker Desktop picks. -# Dispatch it with an engine tag and a ui tag to gate a release candidate. +# docs/local-development.md "Running natively on a Mac, in VS Code" covers this workflow. name: macOS dev smoke diff --git a/.gitignore b/.gitignore index 3af0632fb..8ccc83e01 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,3 @@ -# See https://help.github.com/articles/ignoring-files/ for more about ignoring files. - # Dependencies node_modules .pnp @@ -70,7 +68,7 @@ TODO.md .cursor/ .cursorrules -# OS junk (never useful in repo) +# OS junk .DS_Store Thumbs.db *.swp diff --git a/.yarnrc.yml b/.yarnrc.yml index e49fd76ef..b01615943 100644 --- a/.yarnrc.yml +++ b/.yarnrc.yml @@ -1,22 +1,7 @@ nodeLinker: node-modules -# Supply-chain release-age gate, enforced at INSTALL time. Deliberate, and not -# a duplicate of the org Renovate cooldown: that one governs what Renovate -# proposes, this one governs what yarn will install, including a hand-run -# `yarn add` that Renovate never sees. -# -# The two do NOT share an exception. Renovate lets a CVE fix bypass its 7-day -# cooldown; this gate has no such notion, so a security PR raised early fails -# here with YN0016 "quarantined" until the version ages out (dfe-ui#129). -# -# Unblock a genuine CVE with the narrow escape, never by lowering the gate: -# - npmPreapprovedPackages, using an EXACT locator (brace-expansion@5.0.8), -# removed once the version ages past the window -# - or `yarn add --no-time-gate` for a single command -# -# Keep the value in the `d` form only while it demonstrably parses: yarn -# issue #6991 reports the suffix being dropped, which reads as protective and -# is not. Verified firing on yarn 4.13.0. +# Install-time release-age gate, separate from Renovate's cooldown. CONTRIBUTING.md "What tends to bite" covers a CVE fix it blocks. +# yarn#6991 reports the `d` suffix being dropped, so check the gate still fires after a yarn bump. npmMinimalAgeGate: '7d' # Peers these packages use through their own dependencies but do not declare; diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9f3f24fb8..5ba7f15c5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -36,8 +36,6 @@ versioning and changelog generation. All commits must follow this format: | `chore` | Other changes that don't modify src or test files | None | | `revert` | Reverts a previous commit | Varies | -### Breaking Changes - For breaking changes that require a major version bump, add `!` after the type or include `BREAKING CHANGE:` in the footer: @@ -47,7 +45,7 @@ feat!: remove deprecated API endpoints BREAKING CHANGE: The /v1/users endpoint has been removed. Use /v2/users instead. ``` -### Examples +Examples: ```text feat(auth): add OAuth2 support for Google login @@ -61,18 +59,14 @@ refactor(core)!: restructure module exports BREAKING CHANGE: Named exports are now used instead of default exports. ``` -### Scope - Scope is optional but recommended. Use it to indicate the area of the codebase affected (e.g., `api`, `auth`, `core`, `cli`, `docs`). -## Semantic Versioning - This project follows [Semantic Versioning 2.0.0](https://semver.org/): -- **MAJOR** (X.0.0): Breaking changes that require users to modify their code -- **MINOR** (0.X.0): New features that are backwards-compatible -- **PATCH** (0.0.X): Bug fixes and minor improvements +- MAJOR (X.0.0): Breaking changes that require users to modify their code +- MINOR (0.X.0): New features that are backwards-compatible +- PATCH (0.0.X): Bug fixes and minor improvements Versions are automatically determined by semantic-release based on commit messages. Do not manually update version numbers. @@ -85,11 +79,11 @@ contributors have the right to submit their contributions. By making a contribution to this project, you certify that: 1. The contribution was created in whole or in part by you and you have the - right to submit it under the license indicated in the file; or + right to submit it under the license indicated in the file, or 2. The contribution is based upon previous work that, to the best of your knowledge, is covered under an appropriate open source license and you - have the right under that license to submit that work with modifications; + have the right under that license to submit that work with modifications, or 3. The contribution was provided directly to me by some other person who @@ -136,13 +130,13 @@ anniversary of its release. ## How to Contribute -1. **Fork the repository** and create your branch from `main` -2. **Make your changes** following the commit message format above -3. **Sign off your commits** with the DCO -4. **Test your changes** to ensure they work as expected -5. **Submit a pull request** with a clear description of what you've done +1. Fork the repository and create your branch from `main`. +2. Make your changes, following the commit message format above. +3. Sign off your commits with the DCO. +4. Run the checks below. +5. Open a pull request that says what changed and why. -### Pull Request Checklist +Before you open it: - [ ] Commits follow the conventional commit format - [ ] All commits are signed off (DCO) @@ -163,8 +157,7 @@ hyperi-ci check it once with `uv tool install hyperi-ci` (or `pipx install hyperi-ci`), or run it ad hoc with `uvx hyperi-ci check`. It runs the project's full local validation -- the same suite CI runs -- and reports what to fix. Run it before every push and before opening a -pull request. It is the single best way to give your change the best chance of -surviving CI. +pull request. ## CI/CD Workflow @@ -176,11 +169,7 @@ When your pull request is merged to `main`: 4. Creates a new GitHub release with release notes 5. Publishes the package (if applicable) -This happens automatically - no manual intervention required. - -## Questions - -If you have questions about contributing, please open an issue or contact us. +This happens automatically - no manual intervention required. A question about contributing goes in an issue. ## Context @@ -188,9 +177,7 @@ If you have questions about contributing, please open an issue or contact us. The operator console: a Next.js app that reads and writes through one backend, the dfe-engine API, and frames the dfe-hyperdx fork for data exploration. It keeps no datastore. When the console needs data the engine grows an endpoint, rather than the UI reaching past it. -Not an internal tool. Other organisations run this console against their own -engine, so cluster names, our fleet and credential paths belong in the private -config repos. One artifact ships, the `ghcr.io/hyperi-io/dfe-ui` container. +Not an internal tool. Other organisations run this console against their engines, so cluster names, fleet details and credential paths belong in each deployment's private config repo. One artifact ships, the `ghcr.io/hyperi-io/dfe-ui` container. The console is one of the suite's user-facing components, with dfe-receiver and dfe-hyperdx, so an advisory here is graded on reachability before severity. The exposure model is declared once, in dfe-infra: @@ -228,15 +215,16 @@ The Playwright suite's setup is [apps/dfe-core-ui/e2e/README.md](apps/dfe-core-u ### What tends to bite -| Don't | Do | Why | -| ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| hand-patch a generated type to add a field the engine spec copy lacks | copy the engine spec again, then `yarn workspace @repo/dfe-engine-types generate` | #298: a hand patch kept a stale spec alive after the engine made the field required | -| re-derive what the engine already declares -- a picker's options, a name rule | read the engine's list, carry the engine's rule | #287, the engine picker was hardcoded and offered an engine that does not exist. #218, the form allowed `aws_cloudtrail` where the engine wants a DNS-1123 label | -| put a health or metrics route behind `withAuth` | keep `livez`, `readyz`, `metrics` in the `proxy.ts` matcher exclusions | probes carry no session, so `withAuth` returned 307 to `/login`. Kubernetes counts 3xx as success, so probes passed while the app was face down | -| read a failed engine call as a completed state | tell a broken config apart from an unreachable engine | #236, `proxy.ts` treated a failed setup-status call as setup complete | -| run the Playwright specs in parallel | leave `fullyParallel: false` and `workers: 1` | every spec shares one engine and calls `reset_all`. Workers race on seed-static, so a spec passes alone and hangs with the folder | -| add a per-file timeout when a `findBy` backstop expires | rely on the global `testTimeout: 20_000` | four suites died at vitest's 5s default on 4cpu runners, carrying 15s backstops the per-test budget beheads | -| merge a `feat:` with the GitHub squash button | default to `fix:` | #209, the validator wants `HYPERCI_ALLOW_FEAT=1` set at commit time and the squash button has no environment. Main was red from 2026-09-03 | +| Don't | Do | Why | +| ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| hand-patch a generated type to add a field the engine spec copy lacks | copy the engine spec again, then `yarn workspace @repo/dfe-engine-types generate` | #298: a hand patch kept a stale spec alive after the engine made the field required | +| re-derive what the engine already declares -- a picker's options, a name rule | read the engine's list, carry the engine's rule | #287, the engine picker was hardcoded and offered an engine that does not exist. #218, the form allowed `aws_cloudtrail` where the engine wants a DNS-1123 label | +| put a health or metrics route behind `withAuth` | keep `livez`, `readyz`, `metrics` in the `proxy.ts` matcher exclusions | probes carry no session, so `withAuth` returned 307 to `/login`. Kubernetes counts 3xx as success, so probes passed while the app was face down | +| read a failed engine call as a completed state | tell a broken config apart from an unreachable engine | #236, `proxy.ts` treated a failed setup-status call as setup complete | +| run the Playwright specs in parallel | leave `fullyParallel: false` and `workers: 1` | every spec shares one engine and calls `reset_all`. Workers race on seed-static, so a spec passes alone and hangs with the folder | +| add a per-file timeout when a `findBy` backstop expires | rely on the global `testTimeout: 20_000` | four suites died at vitest's 5s default on 4cpu runners, carrying 15s backstops the per-test budget beheads | +| lower `npmMinimalAgeGate` in `.yarnrc.yml` to land a CVE fix | pre-approve the exact locator in `npmPreapprovedPackages`, or `yarn add --no-time-gate` once | the gate applies at install and has no CVE exception, unlike Renovate's cooldown, so an early security PR fails with YN0016 "quarantined" (#129) | +| merge a `feat:` with the GitHub squash button | default to `fix:` | #209, the validator wants `HYPERCI_ALLOW_FEAT=1` set at commit time and the squash button has no environment. Main was red from 2026-09-03 | ### Where this sits diff --git a/Dockerfile b/Dockerfile index 02d055007..f703c8c8e 100644 --- a/Dockerfile +++ b/Dockerfile @@ -5,8 +5,7 @@ # License: BUSL-1.1 # Copyright: (c) 2026 HYPERI PTY LIMITED # -# Next.js standalone output (apps/dfe-core-ui/next.config.ts) gives a -# self-contained server plus the traced workspace deps. +# docs/build-and-deploy.md covers the three stages. # Trixie by name: the bare node:24-slim tag is still Debian 12. Renovate moves the digest. ARG NODE_IMAGE=node:24-trixie-slim@sha256:43b9ba1142b8dcf33fa8f922ca1be85714adc444e0586c7cf7726daaeabc1bea @@ -16,7 +15,7 @@ ARG NODE_IMAGE=node:24-trixie-slim@sha256:43b9ba1142b8dcf33fa8f922ca1be85714adc4 FROM --platform=$BUILDPLATFORM ${NODE_IMAGE} AS builder WORKDIR /app -# Toolchain for any native (node-gyp) transitive deps. Builder-only; discarded. +# node-gyp toolchain for native dependencies, in the builder stage alone. # Unpinned: Debian removes superseded versions from its archive, so a version pin breaks the build at the next point release. # hadolint ignore=DL3008 RUN apt-get update && apt-get install -y --no-install-recommends \ @@ -25,7 +24,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ RUN corepack enable -# Manifests first so the install layer caches independently of source. COPY package.json yarn.lock .yarnrc.yml turbo.json ./ COPY apps/ apps/ COPY packages/ packages/ @@ -36,11 +34,9 @@ COPY scripts/ scripts/ RUN yarn config set supportedArchitectures.cpu --json '["x64","arm64"]' \ && yarn install --immutable ENV NEXT_PUBLIC_API_URL="" -# hyperi-ci stamp-version writes VERSION before a publish image build. -# COPY of a missing file fails, so bind-mount the context and copy only when present. +# VERSION exists in a publish build alone, and COPY of a missing file fails. RUN --mount=type=bind,source=.,target=/src \ if [ -f /src/VERSION ]; then cp /src/VERSION /app/VERSION; fi -# turbo builds the workspace packages (@repo/*) then the Next app. RUN yarn build # --- App tree --- @@ -48,9 +44,7 @@ RUN yarn build FROM --platform=$BUILDPLATFORM ${NODE_IMAGE} AS app_tree WORKDIR /app -# Standalone bundle carries the monorepo layout: apps/dfe-core-ui/server.js plus a -# minimal node_modules + traced packages/. Neither the built static assets nor the -# public/ tree ride in standalone -- copy both alongside the app. +# The standalone output leaves out .next/static and public/, so both are copied beside it. COPY --from=builder /app/apps/dfe-core-ui/.next/standalone ./ COPY --from=builder /app/apps/dfe-core-ui/.next/static ./apps/dfe-core-ui/.next/static COPY --from=builder /app/apps/dfe-core-ui/public ./apps/dfe-core-ui/public @@ -67,7 +61,6 @@ ENV NODE_ENV=production ENV PORT=3000 ENV HOSTNAME=0.0.0.0 -# Non-root runtime. RUN groupadd -g 1001 nodejs && useradd -u 1001 -g nodejs -m nextjs COPY --from=app_tree --chown=nextjs:nodejs /app ./ diff --git a/README.md b/README.md index 3dc69ee99..7c3a2fbca 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ The DFE console, where an operator sets up and runs a deployment. [![Build Status](https://github.com/hyperi-io/dfe-ui/actions/workflows/ci.yml/badge.svg)](https://github.com/hyperi-io/dfe-ui/actions) [![License: BUSL-1.1](https://img.shields.io/badge/License-BUSL--1.1-blue)](LICENSE) -It manages the deployment through the dfe-engine API alone, never past it to the infrastructure. Its Observe pages are served from their own URL and framed inside the console, and the console sends its own telemetry to an OTLP endpoint when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. +It manages the deployment through the dfe-engine API alone, never past it to the infrastructure. Its Observe pages are served from a separate URL and framed inside the console, and the console sends its telemetry to an OTLP endpoint when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. ## What it does @@ -27,7 +27,7 @@ On a system-installed node (`/usr/bin/node`) `corepack enable` fails with `EACCES` on its symlink. Either `corepack enable --install-directory ~/.local/bin` (any writable directory on your PATH), or prefix each command with `corepack` -- `corepack yarn install` -- which needs no enable. A yarn 1 -already on your PATH is the wrong yarn for this lockfile. +on your PATH is the wrong yarn for this lockfile. Point it at an engine with `NEXT_PUBLIC_API_URL` and `INTERNAL_API_URL`. With neither set, every page says the engine URL is not configured. Full environment reference: [docs/local-development.md](docs/local-development.md). @@ -36,15 +36,15 @@ Full environment reference: [docs/local-development.md](docs/local-development.m A Turborepo/yarn-workspaces monorepo with one shipped app and six packages: -| Workspace | What it is | -| ----------------------------- | ------------------------------------------------------------------ | -| `apps/dfe-core-ui` | the Next.js app - the only artifact shipped (standalone container) | -| `packages/dfe-engine-types` | a committed copy of the engine's OpenAPI specs + generated types | -| `packages/dfe-icons` | SVG -> React icon codegen + storybook | -| `packages/dev-logger` | small shared logging helper | -| `packages/dfe-json-viewer-cn` | the JSON viewer component | -| `packages/dfe-shared-ui` | shared React components and utilities | -| `packages/typescript-config` | shared tsconfigs | +| Workspace | What it is | +| ----------------------------- | ----------------------------------------------------------------- | +| `apps/dfe-core-ui` | the Next.js app - the one artifact shipped (standalone container) | +| `packages/dfe-engine-types` | a committed copy of the engine's OpenAPI specs + generated types | +| `packages/dfe-icons` | SVG -> React icon codegen + storybook | +| `packages/dev-logger` | small shared logging helper | +| `packages/dfe-json-viewer-cn` | the JSON viewer component | +| `packages/dfe-shared-ui` | shared React components and utilities | +| `packages/typescript-config` | shared tsconfigs | ## Documentation @@ -63,7 +63,7 @@ The suite's published docs are at , and this reposit ## Help and contributing -How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in a new issue. +How to work in this repo, the commands that prove a change and what tends to bite are in [CONTRIBUTING.md](CONTRIBUTING.md). A question that is not a bug goes in an issue as well. Open bugs: [this repo](https://github.com/hyperi-io/dfe-ui/issues?q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor) | [whole DFE suite](https://github.com/search?type=issues&q=is%3Aopen+is%3Aissue+type%3ABug+-label%3Abacklog+-label%3Anext-minor+repo%3Ahyperi-io%2Fdfe-infra+repo%3Ahyperi-io%2Fdfe-docker+repo%3Ahyperi-io%2Fdfe-deploy+repo%3Ahyperi-io%2Fdfe-engine+repo%3Ahyperi-io%2Fdfe-loader+repo%3Ahyperi-io%2Fdfe-receiver+repo%3Ahyperi-io%2Fdfe-fetcher+repo%3Ahyperi-io%2Fdfe-archiver+repo%3Ahyperi-io%2Fdfe-transform-vrl+repo%3Ahyperi-io%2Fdfe-transform-vector+repo%3Ahyperi-io%2Fdfe-transform-elastic+repo%3Ahyperi-io%2Fdfe-ui+repo%3Ahyperi-io%2Fdfe-hyperdx+repo%3Ahyperi-io%2Fscalo-rs+repo%3Ahyperi-io%2Fscalo-py+repo%3Ahyperi-io%2Flogreducer+repo%3Ahyperi-io%2Fdfe-schemas+repo%3Ahyperi-io%2Fvector-vrl+repo%3Ahyperi-io%2Fculvert+repo%3Ahyperi-io%2Fdfe-clickhouse-rs+repo%3Ahyperi-io%2Ffactbook-rs) diff --git a/apps/dfe-core-ui/.gitignore b/apps/dfe-core-ui/.gitignore index d8a6cd08c..377d3a022 100644 --- a/apps/dfe-core-ui/.gitignore +++ b/apps/dfe-core-ui/.gitignore @@ -1,5 +1,3 @@ -# See https://help.github.com/articles/ignoring-files/ for more about ignoring files. - # dependencies /node_modules /.pnp @@ -33,7 +31,7 @@ yarn-debug.log* yarn-error.log* .pnpm-debug.log* -# env files (can opt-in for committing if needed) +# env files .env.* !.env.example diff --git a/docs/architecture.md b/docs/architecture.md index 3c1edec20..04a1c2238 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,52 +1,92 @@ # dfe-ui architecture -dfe-ui is the operator surface of the DFE suite: a Next.js app that talks to exactly one backend, the dfe-engine API, and frames dfe-hyperdx for data exploration. It holds no state of its own beyond the browser session. Everything it shows and changes goes through engine endpoints. +dfe-ui is the operator console of the DFE suite: a Next.js app that talks to one backend, the dfe-engine API, and frames dfe-hyperdx for data exploration. It has no datastore: what it shows and changes goes through engine endpoints, Observe content comes from dfe-hyperdx, and the browser keeps preferences such as the theme in local storage. The console exposes the well-trodden part of the engine API and never reaches past it. When the console needs data, the engine grows an endpoint. -## Where it sits in the DFE stack +## The console is five parts + +The browser reaches the console through the proxy, and the console reaches the engine through the NextAuth routes and the typed API client. ```mermaid -flowchart TB - USER[Operator browser] --> UI[dfe-core-ui
Next.js app] - UI -->|typed API client| ENG[dfe-engine API] - UI -->|iframe /observe| HDX[dfe-hyperdx fork] - HDX --> CH[(ClickHouse)] - ENG -->|governed commits| GIT[(gitops repos)] +flowchart LR + accTitle: L3 Components -- dfe-ui + accDescr: The operator's browser requests pages through the proxy. The console pages sign in and renew through the NextAuth routes, call the engine through the typed API client, and frame dfe-hyperdx from the Observe page. + + browser["Operator browser"] + + subgraph dfeui["dfe-ui"] + proxy["Proxy"] + pages[["Console pages"]] + nextauth[["NextAuth routes"]] + client[["API client"]] + observe[["Observe page"]] + end + + engine["dfe-engine API"] + hyperdx["dfe-hyperdx"] + + browser -->|requests| proxy + proxy --> pages + pages --> nextauth + pages --> client + pages --> observe + nextauth -->|sign-in| engine + client -->|API calls| engine + observe -->|frame| hyperdx + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class browser ext + class dfeui group ``` -The control pyramid rule: the UI exposes the well-trodden, high-value 80/20 -of the engine API and NEVER reaches past it. If the UI needs data, the -engine grows an endpoint. The whole-suite map lives in the dfe-engine repo -(`docs/architecture.md` there). +*Figure 1. L3 Components -- dfe-ui. The dashed box is external.* Up: top of dfe-ui. Down: [L4 Detail -- Console menu groups](console-and-the-flow.md), [L4 Detail -- Source flow card](console-and-the-flow.md#the-source-page-draws-the-engines-flow-resolver), [L4 Detail -- OIDC sign-in](authentication.md#sign-in-goes-through-the-engine), [L4 Detail -- Engine API client](engine-api-client.md#the-spec-becomes-typed-calls), [L4 Detail -- Observe embed](observe-embed.md), [L4 Detail -- Uploaded row lifecycle](create-schema-form.md#how-a-row-moves-from-import-to-submit). + +| Part | Code | Does | +| --------------- | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Proxy | `apps/dfe-core-ui/src/proxy.ts` | Guards the signed-in pages, sends an account on an issued password to the password change and an incomplete setup to the wizard, plants the `dfe_token` cookie and sets the Content-Security-Policy. | +| Console pages | `apps/dfe-core-ui/src/app`, one feature directory per domain | Route group `(auth)` for the signed-in app, and `(no-auth)` for login, first-run setup, the forced password change and the engine-unavailable page. | +| NextAuth routes | `apps/dfe-core-ui/src/app/api/auth/[...nextauth]/route.ts`, `apps/dfe-core-ui/src/core/config/auth.ts` | Sign-in, session renewal and sign-out against the engine ([authentication.md](authentication.md)). | +| API client | `apps/dfe-core-ui/src/core/config/api` | The one typed client feature code calls the engine through, with TanStack Query hooks per endpoint ([engine-api-client.md](engine-api-client.md)). | +| Observe page | `apps/dfe-core-ui/src/app/(auth)/observe/[[...feature]]/page.tsx` | Frames dfe-hyperdx in chromeless mode ([observe-embed.md](observe-embed.md)). | -## Monorepo map +On a signed-in page the proxy checks the session, then the forced password change, then asks the engine for the setup status, and rewrites the request to the engine-unavailable page when the engine does not answer. The NextAuth, health and metrics routes skip the proxy. Feature code makes no raw fetch. The calls that skip the client all run on the server: the NextAuth sign-in and session helpers, and the `getAuthPermissions` and `getSetupStatus` server actions. + +## One app ships, built from six packages + +Five workspace packages are bundled into the one app that ships. ```mermaid flowchart LR - subgraph apps - CORE[dfe-core-ui
the shipped app] + accTitle: L4 Detail -- Workspace packages + accDescr: Two generated packages and three hand-written packages are bundled into the dfe-core-ui app. The JSON viewer builds on the shared UI package. + + subgraph generated["Generated"] + direction LR + types["dfe-engine-types"] + icons["dfe-icons"] end - subgraph packages - TYPES[dfe-engine-types
engine spec copy + generated types] - ICONS[dfe-icons
SVG -> React codegen] - LOG[dev-logger] - TSC[typescript-config] - JSON[dfe-json-viewer-cn] - SHARED[dfe-shared-ui] + + subgraph written["Hand-written"] + direction LR + logger["dev-logger"] + shared["dfe-shared-ui"] + json["dfe-json-viewer-cn"] + shared --> json end - TYPES --> CORE - ICONS --> CORE - LOG --> CORE - JSON --> CORE - SHARED --> CORE -``` -Only `dfe-core-ui` ships (Next standalone output in a three-stage Dockerfile). The packages are bundled into it at build and never published on their own. + core["dfe-core-ui"] + + generated -->|bundled| core + written -->|bundled| core + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class core core + class generated,written group +``` -## App structure +*Figure 2. L4 Detail -- Workspace packages. The blue box is the app that ships.* Up: [dfe-ui documentation](README.md). -- Route groups: `(auth)` for the signed-in app, `(no-auth)` for login, first-run setup, the forced password change and the engine-unavailable page. -- Feature directories per domain (Hunts, Schemas, Settings, ...) with - TanStack Query hooks per endpoint. -- Feature code reaches the engine through one typed client ([engine-api-client.md](engine-api-client.md)), with no raw fetch in feature code. The raw fetch sites are all server-side: the NextAuth sign-in and session helpers, and the `getAuthPermissions` and `getSetupStatus` server actions. -- `/observe` iframes dfe-hyperdx ([observe-embed.md](observe-embed.md)). -- Auth is NextAuth over local accounts and the engine's OIDC hand-back ([authentication.md](authentication.md)). +`dfe-engine-types` is generated from the engine's OpenAPI spec and `dfe-icons` from SVG source. The sixth package, `typescript-config`, is the tsconfig the four React and logging packages extend, and nothing of it reaches the bundle. `dfe-core-ui` alone ships, as Next standalone output in a three-stage Dockerfile ([build-and-deploy.md](build-and-deploy.md)). No package is published by itself. diff --git a/docs/authentication.md b/docs/authentication.md index 28c648b06..4a6f8f84a 100644 --- a/docs/authentication.md +++ b/docs/authentication.md @@ -1,44 +1,43 @@ # Authentication -Two ways to sign in, both wired through NextAuth. The UI never runs its own identity provider handshake - identity always comes from the engine. +Two ways to sign in, both wired through NextAuth. The console never runs an identity provider handshake itself: identity always comes from the engine. -## Sign-in paths +## Sign-in goes through the engine -```mermaid -flowchart TB - subgraph creds["local account"] - L1[login form] --> E1[engine /api/v1/auth/login] --> S1[NextAuth session] - end - subgraph oidc["external IdP (OIDC)"] - O1[login page stores a nonce
in sessionStorage] --> O2[engine /api/v1/auth/oidc/.../login
return_to carries the nonce] --> O3[IdP] --> O4[engine callback
303 to /login/oidc#token] --> O5[nonce checked,
oidc-token provider] --> S2[NextAuth session] - end -``` - -- **Local account:** the login form posts to the engine's local auth. The console forwards the browser's `X-Forwarded-For` so the engine's audit trail can name the client. -- **OIDC:** the engine is the relying party and re-mints its own token. It returns the browser to `/login/oidc` with the token in the URL fragment. The console accepts that token only when the `login_nonce` on the return URL matches the one the same tab stored when it started the login, so a crafted link carrying someone else's token cannot sign a browser in. A login the engine refuses comes back the same way with the refusal's code instead (`#error=no_role_granted`), and the sign-in form says why. The code picks fixed words in `apps/dfe-core-ui/src/core/config/loginRefusal.ts`, so a link cannot put text on the page. - -## Session renewal - -The browser asks for a renewal by posting to `/api/auth/session` with no token data. The server refreshes the session's engine token against the engine, reads the roles from `/api/v1/auth/me`, and takes the expiry and the password-change flag from the engine's answer. Nothing the browser sends can set the token, roles, lifetime or flag. +A local account signs in on the login form, which posts to the engine's local login (`POST /api/v1/auth/login`) and then reads the roles from `GET /api/v1/auth/me`. The console forwards the browser's `X-Forwarded-For`, so the engine's audit trail can name the client. -A renewal the engine refuses marks the session expired, and the console signs out to the login page. That covers a session the engine has ended (`session_ended`) and one past its maximum age (`session_expired`). +For OIDC the engine is the relying party, and the console signs in with the token the engine re-mints, never the provider's. -## Changing your own password - -The form asks for the current password as well as the new one. On the forced first-login screen that is the password the deployment issued. A wrong one is marked on the field (`invalid_current_password`). The engine ends every session of the account on a change, so the console signs out and the login page says to sign in with the new password. - -## Signing out +```mermaid +sequenceDiagram + accTitle: L4 Detail -- OIDC sign-in + accDescr: The login page asks the engine for the provider's authorisation URL, with a return URL that has a nonce. The browser signs in at the identity provider, and the engine's callback returns it to the console with the engine token. The console checks the nonce and NextAuth checks the token with the engine. + autonumber + participant page as Login page + participant auth as NextAuth routes + participant engine as dfe-engine API + participant idp as Identity provider + page->>engine: OIDC login, return_to with nonce + engine-->>page: Authorisation URL + page->>idp: Sign in + idp->>engine: Redirect to callback + engine-->>page: 303 to /login/oidc, token in fragment + page->>auth: Sign in, oidc-token provider + auth->>engine: GET /api/v1/auth/me + engine-->>auth: User and roles + auth-->>page: Session +``` -`apps/dfe-core-ui/src/app/api/auth/[...nextauth]/route.ts` wraps NextAuth's sign-out. Once NextAuth has cleared its session, the route asks the engine to end every session of the account (`POST /api/v1/auth/logout`) with the session's token, before the response reaches the browser. An engine that is down, or too old to have the route, does not stop the local sign-out. +*Figure 1. L4 Detail -- OIDC sign-in. Solid arrows are requests, dashed arrows are answers.* Up: [L3 Components -- dfe-ui](architecture.md#the-console-is-five-parts). -## The dfe_token cookie +The login page stores a nonce in `sessionStorage` before it leaves for the provider, and the return URL has the same nonce as `login_nonce`. The console rejects the token unless the two match, so a crafted link with someone else's token cannot sign a browser in. A login the engine rejects comes back the same way with the refusal's code instead (`#error=no_role_granted`), and the sign-in form says why. The code picks fixed words in `apps/dfe-core-ui/src/core/config/loginRefusal.ts`, so a link cannot put text on the page. -`apps/dfe-core-ui/src/proxy.ts` mirrors the session's engine token into a `dfe_token` cookie scoped to `DFE_COOKIE_DOMAIN`, so the embedded dfe-hyperdx signs in as the same user. Signing out expires it with the same Domain and Path, along with the host-only copy the engine's OIDC callback sets. +## Sessions renew and end at the engine -## Redirects +The browser asks for a renewal by posting to `/api/auth/session` with no token data. The server refreshes the session's engine token against the engine, reads the roles from `/api/v1/auth/me`, and takes the expiry and the password-change flag from the engine's answer. Nothing the browser sends can set the token, roles, lifetime or flag. A renewal the engine rejects marks the session expired, and the console signs out to the login page. That covers a session the engine has ended (`session_ended`) and one past its maximum age (`session_expired`). -Every post-login redirect goes through `safeRedirectPath` (`apps/dfe-core-ui/src/core/config/loginCallback.ts`), including NextAuth's own `redirect` callback. A target that resolves off the console's origin - `//host`, `/\host`, another scheme - becomes `/`. +Changing a password asks for the current one as well. On the forced first-login screen that is the password the deployment issued. A wrong one is marked on the field (`invalid_current_password`). The engine ends every session of the account on a change, so the console signs out and the login page says to sign in with the changed password. -Key files: `apps/dfe-core-ui/src/proxy.ts`, `apps/dfe-core-ui/src/core/config/auth.ts`, `apps/dfe-core-ui/src/core/auth/renewEngineSession.ts`, `apps/dfe-core-ui/src/core/auth/oidcLoginNonce.ts` and the `(no-auth)` login page. +`apps/dfe-core-ui/src/app/api/auth/[...nextauth]/route.ts` wraps NextAuth's sign-out. Once NextAuth has cleared its session, the route asks the engine to end every session of the account (`POST /api/v1/auth/logout`) with the session's token, before the response reaches the browser. An engine that is down, or too old to have the route, does not stop the local sign-out. The same route expires the `dfe_token` cookie, which `apps/dfe-core-ui/src/proxy.ts` sets from the session's engine token, scoped to `DFE_COOKIE_DOMAIN`, so the embedded dfe-hyperdx signs in as the same user. A renewal replants that cookie, and sign-out expires it with the same Domain and Path, along with the host-only copy the engine's OIDC callback sets. -The suite-wide auth model, with dfe-engine as the one place access is decided, is in dfe-engine's [OIDC and RBAC architecture](https://github.com/hyperi-io/dfe-engine/blob/main/docs/control-plane/oidc-rbac-architecture.md). +Every post-login redirect goes through `safeRedirectPath` (`apps/dfe-core-ui/src/core/config/loginCallback.ts`), including NextAuth's `redirect` callback. A target that resolves off the console's origin, such as `//host`, `/\host` or another scheme, becomes `/`. Key files: `apps/dfe-core-ui/src/proxy.ts`, `apps/dfe-core-ui/src/core/config/auth.ts`, `apps/dfe-core-ui/src/core/auth/renewEngineSession.ts`, `apps/dfe-core-ui/src/core/auth/oidcLoginNonce.ts` and the `(no-auth)` login page. The suite-wide auth model, with dfe-engine as the one place access is decided, is in dfe-engine's [OIDC and RBAC architecture](https://github.com/hyperi-io/dfe-engine/blob/main/docs/control-plane/oidc-rbac-architecture.md). diff --git a/docs/build-and-deploy.md b/docs/build-and-deploy.md index 98c27d8f9..af720d2af 100644 --- a/docs/build-and-deploy.md +++ b/docs/build-and-deploy.md @@ -1,20 +1,40 @@ # Build and deploy -One shipped artifact: the `dfe-core-ui` Next.js app as a standalone-output container, published by hyperi-ci to `ghcr.io/hyperi-io/dfe-ui` with a thin Helm chart built from its deployment contract. The packages in the monorepo are bundled into the app at build, and none is published to npm (`release.destinations.npm: false` in `.hyperi-ci.yaml`). +One shipped artifact: the `dfe-core-ui` Next.js app as a standalone-output container, published by hyperi-ci to `ghcr.io/hyperi-io/dfe-ui` with a thin Helm chart built from its deployment contract. The workspace packages are bundled into the app at build, and none is published to npm (`release.destinations.npm: false` in `.hyperi-ci.yaml`). + +A release reaches a running pod as one image and one chart. ```mermaid flowchart LR - SRC[monorepo] -->|turbo build| STAND[Next standalone output
outputFileTracingRoot] - STAND -->|three-stage Dockerfile| IMG[container image] - IMG -->|hyperi-ci publish| REG[ghcr.io/hyperi-io/dfe-ui] - CON[deploy/contract.json] -->|hyperi-ci, on scalo-service| CHART[thin chart] - REG --> POD[dfe-ui pod] - CHART -->|dfe-infra values + deploy repo values| POD + accTitle: L4 Detail -- Build and deploy + accDescr: hyperi-ci builds the dfe-ui repo into a container image and a thin chart. dfe-infra deploys the chart with its values and the image tag and digest it pins, and the result is the dfe-ui pod. + + repo["dfe-ui repo"] + ci["hyperi-ci release"] + image[("Container image")] + chart[("Thin chart")] + infra["dfe-infra deploy"] + pod["dfe-ui pod"] + + repo --> ci + ci -->|push| image + ci -->|publish| chart + image -->|tag, digest| infra + chart -->|chart| infra + infra --> pod + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + class image core + class chart store ``` -- Build: `yarn build` (turbo). The app uses Next standalone output with `outputFileTracingRoot` at the monorepo root, so the container carries only what the app needs. -- Container: a three-stage `Dockerfile`. The builder installs and runs `yarn build`, `app_tree` assembles the standalone output with `.next/static` and `public/` beside it, and `runner` is the slim runtime that runs as uid 1001. `NEXT_PUBLIC_*` variables are inlined at BUILD time, and the image bakes `NEXT_PUBLIC_API_URL` as empty. Everything else the app reads at runtime comes from ordinary env vars ([observe-embed.md](observe-embed.md) has the embed case). -- Publish: hyperi-ci builds and pushes the image (`make build` wraps `hyperi-ci run build`). semantic-release derives the version from the commits, so the version is never bumped by hand. +*Figure 1. L4 Detail -- Build and deploy. The blue box is the one artifact this repo ships.* Up: [dfe-ui documentation](README.md). + +- Build: `yarn build` (turbo). The app uses Next standalone output with `outputFileTracingRoot` at the monorepo root, so the container has only what the app needs. +- Container: a three-stage `Dockerfile`. The builder installs and runs `yarn build`, `app_tree` assembles the standalone output with `.next/static` and `public/` beside it, and `runner` is the slim runtime that runs as uid 1001. `NEXT_PUBLIC_*` variables are inlined at BUILD time, and the image bakes `NEXT_PUBLIC_API_URL` as empty, so the browser calls the engine on the console's origin. Everything else the app reads at runtime comes from ordinary env vars ([observe-embed.md](observe-embed.md) has the embed case). +- Publish: in CI, hyperi-ci pushes the image to `ghcr.io/hyperi-io/dfe-ui` and the thin chart to its OCI chart registry. `make build` wraps `hyperi-ci run build`. semantic-release derives the version from the commits, so the version is never bumped by hand. - Deploy: dfe-infra deploys the thin chart with its integration values for the app (`argocd/values/apps/dfe-ui/values.yaml` in dfe-infra) and then the deploy repo's instance values file, which the engine writes. dfe-infra's `versions.yaml` pins the image tag and digest. - Deployment contract: `deploy/contract.json` (scalo-service contract schema v4) declares the port, probes, run-as identity, NextAuth secret and scaling the chart renders the pod from. Keep `security` in step with the Dockerfile `USER` and `health` with the route handlers. -- Headers: every page leaves `apps/dfe-core-ui/src/proxy.ts` with a Content-Security-Policy carrying a fresh script nonce, which is why no page is prerendered. `apps/dfe-core-ui/next.config.ts` adds `X-Content-Type-Options: nosniff`, `Referrer-Policy: strict-origin-when-cross-origin` and `X-Frame-Options: DENY` to every response and drops `X-Powered-By`. A proxy in front must pass these through rather than set its own. +- Headers: every page leaves `apps/dfe-core-ui/src/proxy.ts` with a Content-Security-Policy with a fresh script nonce, which is why no page is prerendered. `apps/dfe-core-ui/next.config.ts` adds `X-Content-Type-Options: nosniff`, `Referrer-Policy: strict-origin-when-cross-origin` and `X-Frame-Options: DENY` to every response and drops `X-Powered-By`. A proxy in front must pass these through rather than replace them. diff --git a/docs/console-and-the-flow.md b/docs/console-and-the-flow.md index 6219f9fe3..6959ed8e9 100644 --- a/docs/console-and-the-flow.md +++ b/docs/console-and-the-flow.md @@ -2,76 +2,85 @@ A record enters DFE, is optionally transformed, and lands in a table. The console's menu groups follow that flow: Data flow defines what comes in, Detect what to look for in it, Observe is where the landed data is read, and Stack is what is running. +Data flow feeds both Observe and Detect, and Stack runs the apps behind it. + ```mermaid flowchart LR - OB[Observe
look at the data] - DF[Data flow
define what comes in] - DT[Detect
define what to look for] - ST[Stack
see and tune what is running] - DF --> OB - DF --> DT - ST -.-> DF + accTitle: L4 Detail -- Console menu groups + accDescr: Stack runs the apps behind what Data flow defines. Data flow feeds what Observe reads and what Detect looks for. + + stack["Stack
what is running"] + dataflow["Data flow
what comes in"] + observe["Observe
read the data"] + detect["Detect
what to look for"] + + stack -.-> dataflow + dataflow --> observe + dataflow --> detect + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A ``` -## The four groups +*Figure 1. L4 Detail -- Console menu groups. The dotted arrow is Stack running and tuning the apps, not data moving.* Up: [L3 Components -- dfe-ui](architecture.md#the-console-is-five-parts). + +Each group is one RBAC family: a heading appears when at least one destination under it is the signed-in user's. The sidebar lists Observe first, then Data flow, Detect and Stack (`apps/dfe-core-ui/src/core/components/SidebarMenu/SidebarMenu/constants.tsx`). -Each group is one RBAC family: a heading appears exactly when at least one -destination under it is the signed-in user's. +| Group | Destinations | Route | What the user does there | RBAC family | +| --------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------ | ----------------------------------------------------------------------------------- | +| Observe | Search, Saved Searches, Chart Explorer, Dashboards, Hunt Results | `/observe/search`, `/observe/search/list`, `/observe/chart`, `/observe/dashboards`, `/observe/hunt-results` | look at the data | `dashboard_read` | +| Data flow | Sources, Meta Schemas, Library | `/sources`, `/schemas`, `/library` | define what comes in, its fields, where it lands | `source_*`, `schema_*`, `library_*` | +| Detect | Rules, Hunts | `/rules`, `/hunts` | define what to look for | `rule_*`, `hunt_*` | +| Stack | Components, Admin Tools | `/components`, `/admin-tools` | see and tune what is running | `deployment_read`, `helmvars_read`, `helmvars_write`, `deployment_admin_links_read` | -| Group | Destinations | Route | What the user does there | RBAC family | -| --------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ---------------------------------------------- | -| Observe | Search, Saved Searches, Chart Explorer, Dashboards, Hunt Results | `/observe/search`, `/observe/search/list`, `/observe/chart`, `/observe/dashboards`, `/observe/hunt-results` | look at the data | `dashboard_read` | -| Data flow | Sources, Meta Schemas, Library | `/sources`, `/schemas`, `/library` | define what comes in, its fields, where it lands | `source_*`, `schema_*`, `library_*` | -| Detect | Rules, Hunts | `/rules`, `/hunts` | define what to look for | `rule_*`, `hunt_*` | -| Stack | Components, Admin Tools | `/components`, `/admin-tools` | see and tune what is running | `deployment_read`, `helmvars_read`, `helmvars_write`, `deployment_admin_links_read` | +The Observe destinations are dfe-hyperdx embeds and appear once a dfe-hyperdx URL is configured. -The Observe destinations are dfe-hyperdx embeds and appear only once a dfe-hyperdx URL -is configured. +Settings and Platform are not in the menu. The settings button in the header opens them, at `/settings/admin` and `/settings/platform`. Some routes have no menu entry either: `/sources/ingest-rules`, `/schemas/meta-schemas`, `/schemas/other-schemas`, `/field-maps`, `/transforms`, `/account`, and the first-run `/setup/[step]`. -Settings and Platform are not in the menu. The settings button in the header -opens them, at `/settings/admin` and `/settings/platform`. Some routes have no -menu entry either: `/sources/ingest-rules`, `/schemas/meta-schemas`, -`/schemas/other-schemas`, `/field-maps`, `/transforms`, `/account`, and the -first-run `/setup/[step]`. +## The Source page draws the engine's flow resolver -## The Source page follows the flow +A source's page opens on its Flow tab: three stages and the two arrows between them. The `main` source, and a source with no meta schema or not yet deployed, shows no Transform stage. -A source's own page opens on its Flow tab: three stages and the two arrows between them. The `main` source, and a source with no meta schema or not yet deployed, shows no Transform stage. +The Flow tab draws a source's input, transform and output, with the carrier and target on each arrow. ```mermaid flowchart LR - IN[INPUT
receiver match, fetcher instance, or your own pipeline] - TR[TRANSFORM
app, variant, endpoint or topics] - OUT[OUTPUT
table, archive] - IN -->|carrier + topic or endpoint| TR - TR -->|carrier + topic or endpoint| OUT + accTitle: L4 Detail -- Source flow card + accDescr: The Flow tab draws an Input, a Transform and an Output stage. Each arrow between two stages names the carrier and the topic or endpoint the records move on. + + input["Input"] + transform["Transform"] + output["Output"] + + input -->|carrier, target| transform + transform -->|carrier, target| output + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A ``` -Every value on that card comes from `GET /api/v1/sources/{name}/flow`, which -returns the engine's own flow resolver. The compilers write each app's deployed -config from the same resolver, so the drawing and the running stack are one -answer rather than two implementations of a naming convention. The console -never builds a topic name or an endpoint of its own. +*Figure 2. L4 Detail -- Source flow card.* Up: [L3 Components -- dfe-ui](architecture.md#the-console-is-five-parts). -Three things follow from reading the resolver rather than the source's fields: +| Stage | Shows | +| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| Input | The receiver match, the fetcher instance, or the topic your pipeline writes, with the origin and the transport. | +| Transform | The app, its variant and its endpoint or topics. For your pipeline, the topics it reads and writes, its consumer group and its lag in records. | +| Output | The table, the topic or endpoint the loader reads it from, and whether the archive keeps the raw record. | -- The card shows the effective transport, never the declared one. A source that names no transport takes the deployment's default, and the card says which. -- A refusal is the engine's own sentence. When a source's stages cannot run as declared (archiving where the archiver does not carry the transport, a transform app that does not carry it, a transport the deployment does not offer) the resolver answers 422 and the card shows that message verbatim. -- Engine-owned stages are read-only. A fetcher instance and a transform instance are derived from the source, so the card marks them engine-owned and links to the app's page under `/components` rather than offering an edit that would be recompiled away. +The stages, transport and targets on that card come from `GET /api/v1/sources/{name}/flow`, which returns the engine's flow resolver, and the lag of your pipeline from `GET /api/v1/sources/{name}/lag`. The compilers write each app's deployed config from the same resolver, so the drawing and the running stack are one answer. The console never builds a topic name or an endpoint itself (`apps/dfe-core-ui/src/Sources/components/ViewSourceTabs/SourceFlowTabContent/flowStages.ts`). Three things follow: -With no source named in the URL, the Sources page opens the `main` source, where every record the receiver cannot place lands. So the card draws that default flow before anyone has defined a source. An engine with no `main` source shows an empty panel instead. +- The card shows the effective transport, never the declared one. A source that names no transport takes the deployment's default, and the card says which. +- A refusal is the engine's sentence. When a source's stages cannot run as declared (archiving where the archiver does not have the transport, a transform app that does not have it, a transport the deployment does not offer) the resolver answers 422 and the card shows that message verbatim. +- Engine-managed stages are read-only. A fetcher instance and a transform instance are derived from the source, so the card marks them engine-owned and links to the app's page under `/components` rather than offering an edit that would be recompiled away. -## Where a source's transport and archive are set +With no source in the URL, the Sources page opens the `main` source, where every record the receiver cannot place lands. So the card draws that default flow before anyone has defined a source. -Both live on the source form's first tab, beside the name: +Transport and archive are set on the source form's first tab, beside the name: -- Transport: leave it empty to inherit the deployment default, or pick one the deployment offers, `bus` or `direct`. The engine refuses at save a flow its resolver cannot run. +- Transport: leave it empty to inherit the deployment default, or pick one the deployment offers, `bus` or `direct`. The engine rejects at save a flow its resolver cannot run. - Archive: keep the raw record as it arrived. The archiver reads the landing topic, so this needs the bus. -## Reading an app's manifest rather than naming it +## The Stack pages read the app manifest -The Stack destinations render from `GET /api/v1/apps`, whose entries come from -the app manifest. Nothing in the console is keyed to an app's name: +The Stack destinations render from `GET /api/v1/apps`, and its entries come from the app manifest. These fields, not the app's name, decide what the console draws: - `multiplicity` decides where an app appears. A `single` app serves every source and belongs to the fleet. A `per_config` app belongs to its source and appears on that source's page. - `file_sets` decides whether an app instance gets a Files panel at all. diff --git a/docs/create-schema-form.md b/docs/create-schema-form.md index 09e97f410..9a9c058fa 100644 --- a/docs/create-schema-form.md +++ b/docs/create-schema-form.md @@ -1,6 +1,6 @@ # `CreateSchemaForm` (dfe-core-ui) -`apps/dfe-core-ui/src/core/components/CreateSchemaForm` builds a schema from its metadata fields plus columns, and pages the uploaded columns so a large import stays usable. Columns come from a manual table or from a CSV or Elastic index template upload. The full column lists live in React context for promotion and submit, and only the current page of uploaded rows is rendered. +`apps/dfe-core-ui/src/core/components/CreateSchemaForm` builds a schema from its metadata fields plus columns, and pages the uploaded columns so a large import stays usable. Columns come from a manual table or from a CSV or Elastic index template upload. The full column lists live in React context for promotion and submit, and the uploaded table renders the current page of rows alone. ## Component map @@ -12,36 +12,46 @@ Paths are relative to the component directory. | `CreateSchemaFormProvider` (`contexts/CreateSchemaForm.context.tsx`) | Holds `uploadedSchemaColumns`, `invalidUploadedSchemaColumns` and `schemaColumns`, with refs for synchronous reads, promotion and removal. | | `SchemaUploadCollapse` (`SchemaUploadCollapse/index.tsx`) | Upload type (DFE CSV or JSON), the file section, and the column tabs. | | `SchemaUploadFileSection` (`SchemaUploadCollapse/SchemaUploadFileSection.tsx`) | Calls `convertCsv` (CSV) or `useElasticConvert` (JSON) and hands the rows to `handleSetUploadedSchemaColumns`. | -| `UploadedSchemaTable` (`UploadedSchemaTable/index.tsx`) | The valid uploaded columns: an `AddSchemaTable` over `uploadedColumns`, searchable, paged at 50 rows by default. Renders `InvalidColumnsTable` above it when there are invalid rows. | -| `InvalidColumnsTable` (`InvalidColumnsTable/index.tsx`) | Rows that failed `rowSchema` on import, in their own `Form.List` (`invalidColumns`), unpaged, promoted back to valid as they are fixed. | +| `UploadedSchemaTable` (`UploadedSchemaTable/index.tsx`) | Every uploaded column, invalid ones included: an `AddSchemaTable` over `uploadedColumns`, searchable, paged at 50 rows by default. Renders `InvalidColumnsTable` above it when there are invalid rows. | +| `InvalidColumnsTable` (`InvalidColumnsTable/index.tsx`) | Rows that fail `rowSchema`, in a separate `Form.List` (`invalidColumns`), unpaged, each dropped from the list once it parses. | | `AddSchemaTable` (`AddSchemaTable/index.tsx`) | `Form.List` plus table. Syncs the list from `initialValues` when a `JSON.stringify` signature of those values changes. | ## How a row moves from import to submit +An uploaded row is valid or invalid from import, and submit is blocked while any row is invalid. + ```mermaid -flowchart LR - FILE[file] -->|convertCsv / useElasticConvert| SET[handleSetUploadedSchemaColumns] - SET -->|rowSchema.safeParse passes| UP[uploadedSchemaColumns] - SET -->|fails| INV[invalidUploadedSchemaColumns] - INV -->|fixed in InvalidColumnsTable| PROMOTE[handleUpdateInvalidUploadedSchemaColumn] - PROMOTE --> UP - UP --> SUBMIT[onFinish: rowSchema over every row] +stateDiagram-v2 + direction LR + accTitle: L4 Detail -- Uploaded row lifecycle + accDescr: A converted row is parsed on import. A row that parses is valid and one that fails is invalid. An invalid row is fixed or removed, and submit runs when no row is invalid, parsing every row again. + [*] --> Imported: file converted + Imported --> Valid: passes + Imported --> Invalid: fails + Invalid --> Valid: fixed + Invalid --> [*]: removed + Valid --> Submitted: none invalid + Submitted --> [*] + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class Invalid error ``` -1. Import. `SchemaUploadFileSection` converts the file (`transformDataToUploadedSchemaRow`) and passes the rows to `handleSetUploadedSchemaColumns`, which runs `rowSchema.safeParse` per row. Valid rows become `uploadedSchemaColumns`, failures `invalidUploadedSchemaColumns` with their Zod issues. +*Figure 1. L4 Detail -- Uploaded row lifecycle. Orange is a row that failed `rowSchema`.* Up: [L3 Components -- dfe-ui](architecture.md#the-console-is-five-parts). + +1. Import. `SchemaUploadFileSection` converts the file (`transformDataToUploadedSchemaRow`) and passes the rows to `handleSetUploadedSchemaColumns`, which runs `rowSchema.safeParse` per row. Every row goes into `uploadedSchemaColumns`, and the rows that fail are listed in `invalidUploadedSchemaColumns` with their Zod issues. 2. Form lists. `AddSchemaTable` uses `useLayoutEffect` to `setFieldsValue` its `Form.List` when the `initialValues` signature changes, so a parent passing a new array each render does not reset the list. -3. Promotion, two paths. `InvalidColumnsTable` watches `invalidColumns` (`Form.useWatch` plus `useEffect`) and promotes each row that now parses. `handleUpdateUploadedSchemaColumns`, reached from the form's `onValuesChange` through `handleFormValuesChange`, merges live form rows with the invalid payloads by row `id` and promotes those that parse. Both validate the moved paths in a `queueMicrotask`. +3. Promotion, two paths. `InvalidColumnsTable` watches `invalidColumns` (`Form.useWatch` plus `useEffect`) and promotes each row that parses after the edit. `handleUpdateUploadedSchemaColumns`, reached from the form's `onValuesChange` through `handleFormValuesChange`, merges live form rows with the invalid payloads by row `id` and promotes those that parse. Both validate the moved paths in a `queueMicrotask`. 4. Submit. `onFinish` runs `rowSchema.safeParse` over every `uploadedSchemaColumns` entry and every non-blank `schemaColumns` entry from context, and blocks the submit when any fails or `invalidUploadedSchemaColumns` is not empty. ## Paging limits rendering, not validation -The uploaded table passes `pagination` to the table, so only the current page of rows mounts its `Form.Item`s. The form's `preserve` keeps list values when rows unmount as the user pages. The authoritative full lists for submit are the context arrays and the `safeParse` in `onFinish`, never the visible cells. +The uploaded table passes `pagination` to the table, so rows off the current page do not mount their `Form.Item`s. The form's `preserve` keeps list values when rows unmount as the user pages. The authoritative full lists for submit are the context arrays and the `safeParse` in `onFinish`, never the visible cells. -`InvalidColumnsTable` and the manual `schemaColumns` table ("Schema Columns" with no upload, "Additional Columns" beside one) are not paged, so a large invalid set or a large manual schema renders every row at once. +`InvalidColumnsTable` and the manual `schemaColumns` table ("Schema Columns" with no upload, "Additional Columns" beside one) are not paged, so a large invalid set or a large manual schema renders every row at once. A file cleaned before upload leaves fewer rows in the unpaged invalid table. ## What costs time on a large import -- Whole-form parses. Every `Form.Item` rule from `useAntdZodResolver` (`apps/dfe-core-ui/src/core/utils/zod/useAntdZodResolver.ts`) asks for a parse of the whole form. Calls with deep-equal values share the last parse, but each check runs `isEqual` over the form and each new parse takes a `cloneDeep` snapshot, so every edit costs one full parse plus a deep compare and a deep copy of every column. +- Whole-form parses. Every `Form.Item` rule from `useAntdZodResolver` (`apps/dfe-core-ui/src/core/utils/zod/useAntdZodResolver.ts`) asks for a parse of the whole form. Calls with deep-equal values share the last parse, but each check runs `isEqual` over the form and each fresh parse takes a `cloneDeep` snapshot, so every edit costs one full parse plus a deep compare and a deep copy of every column. - Signatures. `AddSchemaTable` builds `JSON.stringify(initialValues)` on every render. - Repeat parses. Each row is parsed at import and again at submit. - Mounted tabs. Both `SchemaUploadCollapse` tabs set `forceRender: true`, so the uploaded and manual tables stay mounted together. @@ -50,9 +60,9 @@ CSV conversion (`convertCsv`, a server action) and Elastic conversion (`useElast ## What breaks if a change ignores the model -- Two sources of truth. Editing flows through Ant Design `Form.List`, and the arrays for promotion and submit live in context. The helpers keep them aligned on promotion and removal, so a change that writes only one side lets them diverge. +- Two sources of truth. Editing flows through Ant Design `Form.List`, and the arrays for promotion and submit live in context. The helpers keep them aligned on promotion and removal, so a change that writes one side and not the other lets them diverge. - Effect order. Promotion runs partly from `useWatch` plus `useEffect` and partly from `onValuesChange`, and `validateFields` runs after both in a microtask. Reordering them changes which errors show. -- The change filter. `onValuesChange` work for the column lists runs only when `changedValuesMayAffectTabLists` (`contexts/CreateSchemaForm.context.helpers.tsx`) says the change touches `schemaColumns`, `uploadedColumns` or `invalidColumns`. +- The change filter. `onValuesChange` work for the column lists is skipped unless `changedValuesMayAffectTabLists` (`contexts/CreateSchemaForm.context.helpers.tsx`) says the change touches `schemaColumns`, `uploadedColumns` or `invalidColumns`. - Row ids. Promotion and removal match rows by `id`, carried in a hidden `Form.Item`. A row without a stable `id` cannot be promoted or removed. Before adding another `Form.Item` with `formValidation` on a column row, weigh that each edit re-parses the whole form. A scoped validator on that path costs less. Paging or virtualising `InvalidColumnsTable` would match the uploaded table, provided promotion and removal still match by `id`. diff --git a/docs/engine-api-client.md b/docs/engine-api-client.md index 5b463a516..2ffe35ade 100644 --- a/docs/engine-api-client.md +++ b/docs/engine-api-client.md @@ -2,16 +2,35 @@ Feature code reaches the engine through one typed client generated from the engine's OpenAPI spec. The endpoint map is compile-time checked against the generated types, so a route that disappears from the spec fails the build, not the user. -## How the pieces connect +## The spec becomes typed calls + +The endpoint map sits between the generated types and the client, so a route the spec drops fails compilation there. ```mermaid flowchart LR - SPEC[packages/dfe-engine-types/specs/openapi.json
copied from dfe-engine] -->|openapi-typescript| TYPES[generated paths types] - TYPES --> MAP[API_CONFIG endpoint map
satisfies keyof paths] - MAP --> CLIENT[createApiClient] - CLIENT --> HOOKS[TanStack Query hooks
per feature dir] + accTitle: L4 Detail -- Engine API client + accDescr: openapi-typescript turns the copied engine spec into the paths types. The endpoint map is checked against those types, the API client sends requests from the map, and the TanStack Query hooks in each feature directory call the client. + + spec[("Engine spec copy")] + types["Paths types"] + map["Endpoint map"] + client["API client"] + hooks["Query hooks"] + + spec --> types + types --> map + map -->|routes| client + client -->|typed calls| hooks + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef core fill:#DCEBF7,stroke:#0072B2,stroke-width:2px,color:#0B2A3F + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + class spec store + class client core ``` +*Figure 1. L4 Detail -- Engine API client. The blue box is the client.* Up: [L3 Components -- dfe-ui](architecture.md#the-console-is-five-parts). + - Engine spec: `packages/dfe-engine-types/specs/openapi.json` and `openapi.e2e.json`, committed byte copies of dfe-engine's `openapi/openapi.json` and `openapi/openapi.e2e.json`. - Generated code: `yarn workspace @repo/dfe-engine-types generate` runs openapi-typescript over both specs (`types/index.d.ts`, `types/e2e.d.ts`) and writes the scope and limit constants (`scopes/index.ts`, `limits/index.ts`). - Endpoint map: `apps/dfe-core-ui/src/core/config/api/endpoints/index.ts`, typed `satisfies Record>`, so every entry must exist in the spec. diff --git a/docs/hyperdx-rule-create.md b/docs/hyperdx-rule-create.md index 6d7505643..3b65bb111 100644 --- a/docs/hyperdx-rule-create.md +++ b/docs/hyperdx-rule-create.md @@ -5,7 +5,7 @@ A search in the console's Observe pages can become a detection rule. dfe-hyperdx ## Flow 1. In dfe-hyperdx, the user clicks the Create Rule trigger, and dfe-hyperdx renders the current search to SQL (`POST /dfe/export-sql`). -2. dfe-hyperdx creates the rule through its own `POST /dfe/create-rule` route, which forwards to the engine's `POST /api/v1/rules/from-hyperdx` with the caller's engine token. -3. dfe-hyperdx opens `/rules/{id}`, the rule's own page in the console, in a new tab. +2. dfe-hyperdx creates the rule through its `POST /dfe/create-rule` route, which forwards to the engine's `POST /api/v1/rules/from-hyperdx` with the caller's engine token. +3. dfe-hyperdx opens `/rules/{id}`, the rule's page in the console, in another tab. The routes are in dfe-hyperdx's `packages/api/src/dfe/routers/query-export.ts`, and the trigger in `packages/app/src/dfe/components/CreateRuleFromSearch/index.tsx`. diff --git a/docs/local-development.md b/docs/local-development.md index 8000118bd..79c22c747 100644 --- a/docs/local-development.md +++ b/docs/local-development.md @@ -4,7 +4,7 @@ The app runs as a plain `next dev` process against a local dfe-engine -- no cont ## Boot the engine first, then the app -The engine is the only backend the console needs. Run it as a local process on port 8003 per [dfe-engine docs/local-dev.md](https://github.com/hyperi-io/dfe-engine/blob/main/docs/local-dev.md), then: +The engine is the one backend the console needs. Run it as a local process on port 8003 per [dfe-engine docs/local-dev.md](https://github.com/hyperi-io/dfe-engine/blob/main/docs/local-dev.md), then: ```bash corepack enable # once -- the repo pins yarn 4 via packageManager @@ -18,31 +18,66 @@ yarn workspace dfe-core-ui dev --port 3000 If `corepack enable` fails with `EACCES` (a system-installed node cannot symlink into `/usr/bin`), give it a writable directory on your PATH -- `corepack enable --install-directory ~/.local/bin` -- or run every yarn command through corepack (`corepack yarn install`), which needs no enable. Skipping corepack and using a yarn 1 from your PATH runs the wrong yarn against this lockfile. -The app answers on . Port 3000 is not a habit -- the CORS allowlist a dev engine falls back to breaks login on any other port (see the gotchas). `yarn dev` (turbo) runs the same app when you do not need to pass a port. +The app answers on . Port 3000 is not a habit -- the CORS allowlist a dev engine falls back to breaks the console on any other port (see the gotchas). `yarn dev` (turbo) runs the same app when you do not need to pass a port. -Sign in at `/login` with an engine account. On a fresh dev engine that is the bootstrap `admin` on the engine's shipped default password (`_DEFAULT_PASSWORD` in dfe-engine's `src/dfe_engine/auth/bootstrap.py`). Until initial setup completes, every signed-in page redirects to the `/setup` first-run wizard, after the forced password change when the account carries one. +Sign in at `/login` with an engine account. On a fresh dev engine that is the bootstrap `admin` on the engine's shipped default password (`_DEFAULT_PASSWORD` in dfe-engine's `src/dfe_engine/auth/bootstrap.py`). Until initial setup completes, every signed-in page redirects to the `/setup` first-run wizard, after the forced password change when the account has one. + +Two curls check both halves, no browser needed. `ADMIN_PASSWORD` is the engine admin's password: + +```bash +curl -s -X POST http://localhost:8003/api/v1/auth/login \ + -H 'Content-Type: application/json' \ + -d "{\"username\":\"admin\",\"password\":\"${ADMIN_PASSWORD}\"}" +# -> a JSON body with access_token, expires_in, user_id and roles + +curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3000/login +# -> 200 +``` + +Then check it as a user: open , sign in, and land in the app (the `/setup` wizard on a fresh engine). + +The rest of the daily loop: + +```bash +yarn test # the app's vitest suite, through turbo +yarn lint # eslint via turbo +yarn workspace dfe-core-ui storybook # component workbench, port 6006 +yarn workspace @repo/dfe-icons storybook # icon gallery, port 6007 +``` + +The engine's API spec arrives here by pull request from the engine. [engine-api-client.md](engine-api-client.md) has that and the manual sync. The Playwright suite is in [apps/dfe-core-ui/e2e/README.md](../apps/dfe-core-ui/e2e/README.md), and contribution mechanics (commit format, DCO, local CI) are in [CONTRIBUTING.md](../CONTRIBUTING.md). ## The browser and the server both talk straight to the engine -The browser fetches the engine cross-origin (CORS), the Next server calls it server-side, and NextAuth mints the session from the engine's login endpoint (`POST /api/v1/auth/login`, then `GET /api/v1/auth/me` for roles). The engine is also the token issuer (`https://dfe.local/api` by default) whose ES384 keys are published at `/.well-known/jwks.json`. The embedded dfe-hyperdx trusts those same keys, reading the `dfe_token` cookie `apps/dfe-core-ui/src/proxy.ts` plants. +The browser fetches the engine cross-origin (CORS), the Next server calls it server-side, and NextAuth mints the session from the engine's login endpoint (`POST /api/v1/auth/login`, then `GET /api/v1/auth/me` for roles). The engine issues the tokens (`https://dfe.local/api` by default) and publishes its ES384 keys at `/.well-known/jwks.json`. The embedded dfe-hyperdx trusts those keys, reading the `dfe_token` cookie `apps/dfe-core-ui/src/proxy.ts` plants. + +Locally the browser calls the engine directly, beside the calls `next dev` makes for it. ```mermaid -flowchart LR - browser["browser"] - subgraph local["local processes"] - ui["dfe-ui next dev :3000"] - engine["dfe-engine :8003"] +sequenceDiagram + accTitle: L4 Detail -- Local console and engine + accDescr: The browser loads the login page from next dev, which asks the engine for the setup status. Sign-in goes through NextAuth to the engine. After that the browser calls the engine cross-origin, and an embedded dfe-hyperdx checks the dfe_token cookie against the engine's keys. + autonumber + participant browser as Browser + participant ui as dfe-ui next dev 3000 + participant engine as dfe-engine 8003 + participant hdx as dfe-hyperdx + browser->>ui: GET /login + ui->>engine: GET setup-status + browser->>ui: Sign in + ui->>engine: POST /auth/login, GET /auth/me + browser->>engine: API calls, CORS + opt Observe configured + browser->>hdx: Frame, dfe_token cookie + hdx->>engine: GET /.well-known/jwks.json end - browser -->|app + session| ui - browser -->|API fetches, CORS| engine - ui -->|login, setup-status| engine - ui -.->|iframe embed, optional| fork["dfe-hyperdx fork"] - fork -.->|JWKS trust| engine ``` -The fork iframe is optional for console work. Wiring it up is covered in [observe-embed.md](observe-embed.md) and the fork's own [getting-started doc](https://github.com/hyperi-io/dfe-hyperdx/blob/main/docs/development/getting-started.md). +*Figure 1. L4 Detail -- Local console and engine.* Up: [dfe-ui documentation](README.md). + +The dfe-hyperdx frame is optional for console work. Wiring it up is covered in [observe-embed.md](observe-embed.md) and the fork's [getting-started doc](https://github.com/hyperi-io/dfe-hyperdx/blob/main/docs/development/getting-started.md). -## Every deployment fact is an environment variable +Every deployment fact is an environment variable. | Variable | When | Purpose | | --------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------ | @@ -51,11 +86,11 @@ The fork iframe is optional for console work. Wiring it up is covered in [observ | `NEXTAUTH_URL` / `NEXTAUTH_SECRET` | runtime | NextAuth session config -- any non-empty secret works for dev | | `DFE_COOKIE_DOMAIN` | runtime | parent domain for the `dfe_token` cookie -- leave unset locally | | `HYPERDX_URL` / `HYPERDX_PORT` | runtime | the embed pair, either variable -- see [observe-embed.md](observe-embed.md) | -| `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_SERVICE_NAME` | runtime | OTLP export of the console's own telemetry, off when the endpoint is unset or empty | -| `LOG_LEVEL` | runtime | server log threshold, `DEBUG` to `ERROR`, default `INFO` | +| `OTEL_EXPORTER_OTLP_ENDPOINT` / `OTEL_SERVICE_NAME` | runtime | OTLP export of the console's telemetry, off when the endpoint is unset or empty | +| `LOG_LEVEL` | runtime | server log threshold, scalo-py's names from `TRACE` to `CRITICAL`, default `INFO` | | `LOG_FORMAT` | runtime | `json` or `text`. Unset writes JSON in production or with an OTLP endpoint, text otherwise | -`NEXT_PUBLIC_*` values are baked in at build time -- `next dev` reads them at boot, so changing one needs a dev-server restart (a production image needs a rebuild). `next dev` also loads a gitignored `apps/dfe-core-ui/.env.local` if present, and explicit shell values win over it (Next.js env precedence). `apps/dfe-core-ui/.env.example` lists the rest, the session-refresh timings among them. +`NEXT_PUBLIC_*` values are baked in at build time -- `next dev` reads them at boot, so changing one needs a dev-server restart (a production image needs a rebuild). `next dev` loads a gitignored `apps/dfe-core-ui/.env.local` if present, and explicit shell values win over it (Next.js env precedence). `apps/dfe-core-ui/.env.example` lists the rest, the session-refresh timings among them. Two ports matter, and 3000 matters most: @@ -64,32 +99,16 @@ Two ports matter, and 3000 matters most: | 3000 | dfe-ui `next dev` | must be 3000, 5173 or 5174 unless the engine's CORS list is extended | | 8003 | dfe-engine API | Swagger at `/docs`, JWKS at `/.well-known/jwks.json` | -## Two curls verify the loop +## The gotchas that bite -Both halves, no browser needed. `ADMIN_PASSWORD` is the engine admin's password: - -```bash -curl -s -X POST http://localhost:8003/api/v1/auth/login \ - -H 'Content-Type: application/json' \ - -d "{\"username\":\"admin\",\"password\":\"${ADMIN_PASSWORD}\"}" -# -> a JSON body carrying access_token, expires_in, user_id and roles - -curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3000/login -# -> 200 -``` - -Then prove it as a user: open , sign in, and land in the app (the `/setup` wizard on a fresh engine). - -## The gotchas that actually bite - -- A non-default port breaks login silently. An engine on `DFE_ENGINE_ENV=dev` that names no CORS origins of its own allows only `http://localhost:3000`, `:5173` and `:5174`. On any other port the login page keeps spinning and the browser console fills with CORS errors on `/api/v1/auth/setup-status`. Set the engine's `DFE_ENGINE_API_CORS_ORIGINS` (comma-separated) or stay on port 3000. +- A non-default port breaks the console quietly. An engine on `DFE_ENGINE_ENV=dev` that names no CORS origins allows `http://localhost:3000`, `:5173` and `:5174` and nothing else. On any other port every browser call to the engine fails its CORS check, starting with `/api/v1/auth/setup-status` on the login page, so the OIDC option is missing and signed-in pages cannot load their data. Set the engine's `DFE_ENGINE_API_CORS_ORIGINS` (comma-separated) or stay on port 3000. - A fresh engine bounces `/` to `/setup`. `apps/dfe-core-ui/src/proxy.ts` asks the engine's setup-status on every signed-in request. Until initial setup completes, every signed-in page redirects to the wizard -- that is first-run behaviour, not a fault. -- No engine reads as an error page, served as 200. With no engine running, `/login`, `/setup` and every signed-in page read "Cannot reach the DFE engine", with a Try again button. A page cannot set its own status, so the proxy rewrites the request to `/engine-unavailable`. The rewrite asks for a 503, but Next serves a rewrite with the target page's status, so the HTTP answer is 200. Monitor the engine itself, not this page. A refused connection, an engine that says nothing within `ENGINE_CALL_TIMEOUT_MS` (`apps/dfe-core-ui/src/core/config/engineTimeout.ts`) and a 502, 503 or 504 count as no answer. A missing `INTERNAL_API_URL` is a different fault. `/login` and the signed-in shell both name it ("The engine URL is not configured") instead of failing. +- No engine reads as an error page, served as 200. With no engine running, `/login`, `/setup` and every signed-in page read "Cannot reach the DFE engine", with a Try again button. A page cannot set its status, so the proxy rewrites the request to `/engine-unavailable`. The rewrite asks for a 503, but Next serves a rewrite with the target page's status, so the HTTP answer is 200. Monitor the engine, not this page. A refused connection, an engine that says nothing within `ENGINE_CALL_TIMEOUT_MS` (`apps/dfe-core-ui/src/core/config/engineTimeout.ts`) and a 502, 503 or 504 count as no answer. A missing `INTERNAL_API_URL` is a different fault. `/login` and the signed-in shell both name it ("The engine URL is not configured") instead of failing. - One dev server per checkout. Next.js refuses a second `next dev` for the same app directory and exits with "Another next dev server is already running". ## Running natively on a Mac, in VS Code -The console and the engine run as plain processes. The Rust apps stay in docker or kind. To prove a machine, one command, one PASS or FAIL line and one log file to paste back: +The console and the engine run as plain processes. The Rust apps stay in docker or kind. To check a machine, one command, one PASS or FAIL line and one log file to paste back: ```bash python3 scripts/macos-dev-smoke.py # engine from ../dfe-engine, or DFE_ENGINE_DIR= @@ -100,15 +119,4 @@ It wants `uv`, node 24 and a dfe-engine checkout beside this repo, and prints wh In VS Code, Run and Debug has an `Engine + Console` compound (`Engine + Console (local ClickHouse)` when docker is up). It asks once for the engine checkout, `../dfe-engine` by default. Run the `Engine: e2e seed (setup complete)` task, then sign in at as `initial_user` with the engine's well-known e2e password (`_WELL_KNOWN_E2E_PASSWORD` in dfe-engine's `src/dfe_engine/api/e2e/seed/accounts.py`). -Two Mac traps. Point the console at `127.0.0.1:8003`, not `localhost`, because Node resolves `localhost` to `::1` first and the engine listens on IPv4 only. And leave ports 5000 and 7000 alone, macOS holds them for AirPlay Receiver. The `macOS dev smoke` workflow runs the same script on a GitHub macOS arm64 runner. Dispatch it with an engine tag and a ui tag (or `main`) to gate a release candidate. Any other ref is refused before a checkout. - -## The rest of the daily loop - -```bash -yarn test # the app's vitest suite, through turbo -yarn lint # eslint via turbo -yarn workspace dfe-core-ui storybook # component workbench, port 6006 -yarn workspace @repo/dfe-icons storybook # icon gallery, port 6007 -``` - -The engine's API spec arrives here by pull request from the engine. [engine-api-client.md](engine-api-client.md) has that and the manual sync. The Playwright suite is in [apps/dfe-core-ui/e2e/README.md](../apps/dfe-core-ui/e2e/README.md), and contribution mechanics (commit format, DCO, local CI) are in [CONTRIBUTING.md](../CONTRIBUTING.md). +Two Mac traps. Point the console at `127.0.0.1:8003`, not `localhost`, because Node resolves `localhost` to `::1` first and the engine listens on IPv4 alone. And leave ports 5000 and 7000 alone, macOS holds them for AirPlay Receiver. The `macOS dev smoke` workflow runs the same script on a GitHub macOS arm64 runner, with no ClickHouse because that runner has no docker, and a Linux leg runs its docker mode. Dispatch it with an engine tag and a ui tag (or `main`) to gate a release candidate. Any other ref is refused before a checkout. diff --git a/docs/observe-embed.md b/docs/observe-embed.md index e2f2ea7cb..3945d12e9 100644 --- a/docs/observe-embed.md +++ b/docs/observe-embed.md @@ -1,28 +1,39 @@ # The /observe embed - the Observe explore surface -`/observe/[[...feature]]` (`apps/dfe-core-ui/src/app/(auth)/observe/[[...feature]]/page.tsx`) frames dfe-hyperdx in chromeless mode (`?embed=1`), so the dfe-ui shell owns navigation while dfe-hyperdx provides search, charts and dashboards over the data in ClickHouse. The page maps its path to the dfe-hyperdx route (`/observe` to `/search`, `/observe/dashboards` to `/dashboards/list`, `/observe/hunt-results` to `/search?source=hunts`). The initial theme rides the URL, and the page posts it again as a `DFE_SET_THEME` message when the frame loads and on every theme toggle. +`/observe/[[...feature]]` (`apps/dfe-core-ui/src/app/(auth)/observe/[[...feature]]/page.tsx`) frames dfe-hyperdx in chromeless mode (`?embed=1`), so the dfe-ui shell owns navigation and dfe-hyperdx provides search, charts and dashboards over the data in ClickHouse. The page maps its path to the dfe-hyperdx route (`/observe` to `/search`, `/observe/dashboards` to `/dashboards/list`, `/observe/hunt-results` to `/search?source=hunts`) and passes its query parameters on. The dashboards page shows the usage headline above the frame. + +The page keeps the frame's theme in step with the console by posting it when the frame loads and on every toggle. ```mermaid -flowchart LR - SHELL[dfe-ui shell
owns nav + theme] -->|iframe ?embed=1| HDX[dfe-hyperdx
chromeless] - SHELL -->|postMessage DFE_SET_THEME| HDX +sequenceDiagram + accTitle: L4 Detail -- Observe embed + accDescr: The Observe page loads dfe-hyperdx in a frame with embed and theme in the URL. When the frame has loaded, and on every theme toggle, the page posts a DFE_SET_THEME message to it. + autonumber + participant page as Observe page + participant hdx as dfe-hyperdx + page->>hdx: Load frame, embed=1 and theme + hdx-->>page: Frame loaded + page->>hdx: postMessage DFE_SET_THEME + loop Each theme toggle + page->>hdx: postMessage DFE_SET_THEME + end ``` -## How it is configured +*Figure 1. L4 Detail -- Observe embed. Solid arrows are requests, the dashed arrow is the load event.* Up: [L3 Components -- dfe-ui](architecture.md#the-console-is-five-parts). + +The frame signs in from the `dfe_token` cookie the proxy plants ([authentication.md](authentication.md#sessions-renew-and-end-at-the-engine)). + +## HYPERDX_URL or HYPERDX_PORT points the frame at dfe-hyperdx One runtime mechanism, two ways to reach dfe-hyperdx. The `(auth)` layout reads both values per request and hands them to `HyperdxContext`. `useHyperdxUrl()` is the one source the frame URL and the sidebar gating read. | Variable | Points at | | -------------- | ---------------------------------------------------------------------------------------------------------------------------- | -| `HYPERDX_URL` | dfe-hyperdx on its own hostname (k8s gateway: `https://hyperdx.{domain}`). Wins when set. | +| `HYPERDX_URL` | dfe-hyperdx on a separate hostname (k8s gateway: `https://hyperdx.{domain}`). Wins when set. | | `HYPERDX_PORT` | Same host, different port (docker compose). Host comes from `window.location`, so it is correct for localhost and LAN alike. | -Neither is `NEXT_PUBLIC_`, and that is deliberate: a build-inlined read would -bake the build machine's value into the browser bundle. With neither set, -`/observe` renders "Search is not configured for this deployment." +Neither is `NEXT_PUBLIC_`, because a build-inlined read would bake the build machine's value into the browser bundle. With neither set, `/observe` renders "Search is not configured for this deployment." The console's Content-Security-Policy frames nothing but dfe-hyperdx: `frame-src` is `HYPERDX_URL`'s origin, or the host the browser used on `HYPERDX_PORT`. dfe-hyperdx reached any other way is blocked in the browser. -## Rule-create round trip - -dfe-hyperdx creates the rule through its `/dfe/create-rule` route, which forwards to the engine, and opens `/rules/{id}` in a new tab. dfe-ui holds no listener for it. Detail: [hyperdx-rule-create.md](hyperdx-rule-create.md). +dfe-hyperdx creates a rule from a search through its `/dfe/create-rule` route, which forwards to the engine, and opens `/rules/{id}` in another tab. dfe-ui holds no listener for it ([hyperdx-rule-create.md](hyperdx-rule-create.md)). diff --git a/osv-scanner.toml b/osv-scanner.toml index f09fb9134..b579212bb 100644 --- a/osv-scanner.toml +++ b/osv-scanner.toml @@ -5,7 +5,7 @@ # License: BUSL-1.1 # Copyright: (c) 2026 HYPERI PTY LIMITED -# Graded by reachability, per dfe-infra docs/THREAT-MODEL.md: dfe-ui takes +# Graded by reachability, per dfe-infra docs/threat-model.md: dfe-ui takes # untrusted input, so only an advisory that reaches no shipped code is accepted. From ff2493c436c926e0ceb77a217adeadb1919fb9fc Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 13:53:38 +1100 Subject: [PATCH 2/2] docs(console): name both engine URL variables in the gotcha The 'engine URL is not configured' message shows when INTERNAL_API_URL and NEXT_PUBLIC_API_URL are both unset. The server code reads either one. The image has NEXT_PUBLIC_API_URL baked empty. INTERNAL_API_URL decides whether the message shows in the image. Signed-off-by: Derek --- docs/local-development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/local-development.md b/docs/local-development.md index 79c22c747..afe8f9c4d 100644 --- a/docs/local-development.md +++ b/docs/local-development.md @@ -103,7 +103,7 @@ Two ports matter, and 3000 matters most: - A non-default port breaks the console quietly. An engine on `DFE_ENGINE_ENV=dev` that names no CORS origins allows `http://localhost:3000`, `:5173` and `:5174` and nothing else. On any other port every browser call to the engine fails its CORS check, starting with `/api/v1/auth/setup-status` on the login page, so the OIDC option is missing and signed-in pages cannot load their data. Set the engine's `DFE_ENGINE_API_CORS_ORIGINS` (comma-separated) or stay on port 3000. - A fresh engine bounces `/` to `/setup`. `apps/dfe-core-ui/src/proxy.ts` asks the engine's setup-status on every signed-in request. Until initial setup completes, every signed-in page redirects to the wizard -- that is first-run behaviour, not a fault. -- No engine reads as an error page, served as 200. With no engine running, `/login`, `/setup` and every signed-in page read "Cannot reach the DFE engine", with a Try again button. A page cannot set its status, so the proxy rewrites the request to `/engine-unavailable`. The rewrite asks for a 503, but Next serves a rewrite with the target page's status, so the HTTP answer is 200. Monitor the engine, not this page. A refused connection, an engine that says nothing within `ENGINE_CALL_TIMEOUT_MS` (`apps/dfe-core-ui/src/core/config/engineTimeout.ts`) and a 502, 503 or 504 count as no answer. A missing `INTERNAL_API_URL` is a different fault. `/login` and the signed-in shell both name it ("The engine URL is not configured") instead of failing. +- No engine reads as an error page, served as 200. With no engine running, `/login`, `/setup` and every signed-in page read "Cannot reach the DFE engine", with a Try again button. A page cannot set its status, so the proxy rewrites the request to `/engine-unavailable`. The rewrite asks for a 503, but Next serves a rewrite with the target page's status, so the HTTP answer is 200. Monitor the engine, not this page. A refused connection, an engine that says nothing within `ENGINE_CALL_TIMEOUT_MS` (`apps/dfe-core-ui/src/core/config/engineTimeout.ts`) and a 502, 503 or 504 count as no answer. An unset engine URL is a different fault. With both `INTERNAL_API_URL` and `NEXT_PUBLIC_API_URL` unset, `/login` and the signed-in shell name it ("The engine URL is not configured") instead of failing. The image bakes `NEXT_PUBLIC_API_URL` as empty, so there an unset `INTERNAL_API_URL` is enough. - One dev server per checkout. Next.js refuses a second `next dev` for the same app directory and exits with "Another next dev server is already running". ## Running natively on a Mac, in VS Code