From f7ed65e99be1ad0c13367c79d4378ffa511146a2 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 16:07:31 +1100 Subject: [PATCH 1/2] docs(fork): match the docs to the code and redraw every diagram The architecture pages said Envoy runs OIDC in front of HyperDX and passes identity headers. The gateway has no OIDC policy on the HyperDX route. The console sets the engine token in the dfe_token cookie. The API verifies that token against the engine's JWKS. The auth, deployment and catalogue pages describe this auth flow. The dated security records correct their claims about gateway auth. The alert checker still starts in the image. The engine-only /alerts and /webhooks routes and the embed's route block keep alerting off. ADR 0002 and the pages that cite it describe these blocks. ADR 0003 is marked superseded. docker-compose.caps.yml still caps two services removed by the DocumentDB move. compose rejects every command that layers it. docker-local.md opens with this warning. The dev flow points at yarn dev:dfe, since the DFE dev override scales ClickHouse to zero. All 27 mermaid diagrams pass the diagram checker on Mermaid 12.3.0. The text arrow chains became prose or a table. The sync track record moved to its own page. --- docs/README.md | 60 ++-- docs/architecture/README.md | 178 +++++----- docs/architecture/authorization.md | 96 ++--- docs/architecture/backend.md | 184 +++++----- docs/architecture/clickhouse-data-model.md | 159 ++++----- docs/architecture/data-flow.md | 182 +++++----- docs/architecture/deployment.md | 134 +++---- docs/architecture/frontend.md | 95 +++-- docs/architecture/oidc-authentication.md | 262 +++++++------- docs/architecture/query-to-rule-pipeline.md | 97 +++--- docs/architecture/team-identity.md | 111 ++++-- .../0002-alerting-disabled-for-dfe-rules.md | 9 + .../0003-additive-only-fork-strategy.md | 10 +- docs/decisions/0004-casbin-removed.md | 12 +- docs/decisions/0005-console-typography.md | 13 +- docs/decisions/README.md | 4 +- docs/development/docker-local.md | 152 ++++---- docs/development/getting-started.md | 101 +++--- docs/fork/README.md | 107 +++--- docs/fork/design.md | 171 ++++----- docs/fork/leaving-upstream.md | 78 +++-- docs/fork/security-sync.md | 147 ++++---- docs/fork/sync-cycle.md | 327 ++++++++---------- docs/fork/sync-track-record.md | 96 +++++ docs/fork/what-we-changed.md | 285 +++++++-------- 25 files changed, 1602 insertions(+), 1468 deletions(-) create mode 100644 docs/fork/sync-track-record.md diff --git a/docs/README.md b/docs/README.md index 492189b92..ef356fe67 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3,7 +3,7 @@ DFE's fork of [HyperDX](https://github.com/hyperdxio/hyperdx), embedded in the DFE platform as its visualisation and search layer. -Two questions bring people here, and they are answered in different places: +Four questions bring people here, and they are answered in different places: | Question | Start at | | --- | --- | @@ -12,44 +12,48 @@ Two questions bring people here, and they are answered in different places: | How do I run it locally? | [development/](development/README.md) | | Why was it built this way? | [decisions/](decisions/README.md) | ---- - -## What is ours and what is upstream's - -This matters more here than in a normal repo, because the answer decides whether -a change is free or expensive. +Which files are ours and which are upstream's matters more here than in a normal repo, because the answer decides whether +a change is free or expensive. Our files sit beside upstream's, and HyperI +standards apply to ours alone. ```mermaid -flowchart TB - subgraph ours["OURS - HyperI standards apply"] - d["docs/"] +flowchart LR + accTitle: L4 Detail -- Ours and upstream's files + accDescr: Ours are our pages under docs, every dfe directory under packages, the security and githooks directories, our scripts and the workflows we added. Upstream's are agent_docs, the root agent and contributor docs, and everything under packages outside a dfe directory. Ownership is per file, and .fork-surface lists the upstream files we edit. + + subgraph ours["Ours, HyperI standards apply"] + direction TB + d["Our pages
under docs/"] dfe["packages/**/dfe/**"] - tools["scripts/, security/, .githooks/"] - wf[".github/workflows/
ci, fork-*, upstream-*"] + tools["security/, .githooks/,
our scripts/"] + wf["Our workflows"] end - subgraph theirs["UPSTREAM - left exactly as shipped"] + subgraph theirs["Upstream's"] + direction TB ad["agent_docs/"] - rootmd["AGENTS.md, MCP.md,
LOCAL.md, DEPLOY.md,
CONTRIBUTING.md"] - code["everything under packages/
outside dfe/"] + rootmd["AGENTS.md, MCP.md, LOCAL.md,
DEPLOY.md, CONTRIBUTING.md"] + code["packages/ outside dfe/"] end - classDef mine fill:#0072B2,stroke:#00456b,color:#ffffff - classDef up fill:#009E73,stroke:#005f45,color:#ffffff - class d,dfe,tools,wf mine - class ad,rootmd,code up -``` + ours ~~~ theirs -**Our files follow HyperI documentation standards. Upstream's files are left -untouched, including their docs.** Applying our conventions to `agent_docs/` or -`AGENTS.md` would buy nothing and create conflict surface on every sync. + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class ad,rootmd,code ext + class ours,theirs group +``` -`README.md` and `CLAUDE.md` are the two shared files: both exist upstream, both -carry a small catalogued delta of ours, and both are listed in `.fork-surface`. +*Figure 1. L4 Detail -- Ours and upstream's files. Dashed boxes are upstream's.* Up: [dfe-hyperdx](../README.md). ---- +Applying our conventions to `agent_docs/` or `AGENTS.md` would buy nothing and +create conflict surface on every sync. Where an upstream file does have a delta +of ours, `.fork-surface` lists it and [fork/what-we-changed.md](fork/what-we-changed.md) +says why. `README.md`, `CLAUDE.md` and `docs/alert-webhook-template-variables.md` +are the shared docs: upstream ships each, and each has a small delta of ours. -## The documentation set +The documentation set: ```text docs/ @@ -60,4 +64,4 @@ docs/ ``` Everything here is plain CommonMark plus fenced mermaid, so it renders on GitHub -and drops into a docs generator without a rewrite. +without a build step. diff --git a/docs/architecture/README.md b/docs/architecture/README.md index aeea6c79c..b13660821 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -1,21 +1,23 @@ # Architecture HyperDX is DFE's **visualisation and search layer**: engineers use it to search -logs, view traces, build dashboards and replay sessions. Detection, alerting and -response are the DFE platform's job, not HyperDX's. +logs, view traces and build dashboards, inside the DFE console. Detection, +alerting and response are the DFE platform's job, not HyperDX's. Five things make our deployment different from stock HyperDX, and everything else follows from them: -- Authentication is external: Envoy terminates OIDC and passes identity - headers, and HyperDX never sees a password. -- Authorization is the DFE engine's: it is the policy decision point, and - ClickHouse GRANTs enforce data access. -- MongoDB is replaced by DocumentDB, PostgreSQL with the DocumentDB - extensions and its Mongo wire-protocol gateway in one container +- Authentication is the engine's. The DFE console signs the user in through the + DFE engine and hands the embedded app the engine's token in the `dfe_token` + cookie. The API verifies that token itself, and HyperDX's own password login + is switched off. +- Authorization is the engine's as well: it decides which ClickHouse user a + session reads as, and ClickHouse GRANTs enforce data access. +- MongoDB is replaced by DocumentDB, PostgreSQL with the DocumentDB extensions + and its Mongo wire-protocol gateway in one container ([ADR 0006](../decisions/0006-documentdb-document-store.md)). -- The DFE 2.1 discovery tool's enhancements are ported in. -- Rules are generated from the query you are iterating in front of you. +- The DFE 2.1 discovery tool's enhancements are ported in, including turning the + search in front of you into a DFE rule. - JSON is native throughout, the ClickHouse JSON type included. What we changed to achieve that, and how we keep it working across upstream @@ -25,106 +27,100 @@ releases, is [the fork documentation](../fork/README.md). ## The system +The console frames the HyperDX UI, and the API behind it answers to the engine +for every session and reads ClickHouse as the team's own user. + ```mermaid -flowchart TB - browser(["User browser"]) - oidc(["OIDC provider
Google / Entra ID"]) +flowchart LR + accTitle: L2 Services -- dfe-hyperdx + accDescr: The DFE console frames the HyperDX UI, which calls the HyperDX API. The API asks the DFE engine about each session, queries ClickHouse and keeps its documents in DocumentDB. The dashboard provisioner writes shipped dashboards to DocumentDB. - subgraph apps["Instrumented applications"] - sdks["OTel SDKs
browser, Node, Python"] - end + console["DFE console
dfe-ui"] - subgraph dfe["DFE platform"] - envoy["Envoy
OIDC proxy"] - ui["DFE UI"] - rules["DFE rules engine
detection + hunts"] + subgraph hdx["dfe-hyperdx"] + app[["HyperDX UI
Next.js"]] + api[["HyperDX API
Express"]] + provisioner["Dashboard
provisioner"] + end - subgraph hdx["Embedded HyperDX"] - otel["OTel collector"] - api["HyperDX API
Express"] - app["HyperDX UI
Next.js"] - end + engine["DFE engine"] + ch[("ClickHouse")] + docdb[("DocumentDB")] - ch[(ClickHouse
telemetry)] - docdb[("DocumentDB
on CloudNativePG")] - end + console -->|frames| app + app --> api + api -->|sessions| engine + api -->|queries| ch + api -->|documents| docdb + provisioner -->|dashboards| docdb - sdks -->|OTLP| otel - otel -->|writes| ch - api <-->|OpAMP| otel - api <-->|MongoDB protocol| docdb - api -->|proxies queries| ch - rules -->|queries| ch - browser -->|HTTPS| envoy - envoy -->|OIDC| oidc - envoy -->|identity headers| api - envoy -->|identity headers| ui - browser --> ui - browser --> app - - classDef external fill:#F0E442,stroke:#8a8200,color:#000000 - classDef ours fill:#0072B2,stroke:#00456b,color:#ffffff - classDef upstream fill:#009E73,stroke:#005f45,color:#ffffff - classDef store fill:#E69F00,stroke:#8a6100,color:#000000 - class browser,oidc,sdks external - class envoy,ui,rules ours - class otel,api,app upstream + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 class ch,docdb store + class hdx group ``` -Colour encodes ownership: blue is DFE's, green is upstream HyperDX, orange is a -datastore, yellow is outside our boundary. +*Figure 1. L2 Services -- dfe-hyperdx.* Up: top of dfe-hyperdx. Down: [L3 Components -- HyperDX UI](frontend.md#structure), [L3 Components -- HyperDX API](backend.md#structure). + +One container runs the UI, the API and upstream's alert checker, and the dashboard provisioner when `DASHBOARD_PROVISIONER_DIR` is set. The engine calls the API as well, with +a service token of its own, to seed teams, sources and connections. No OTel +collector of HyperDX's runs in DFE: the DFE pipeline writes the telemetry +tables. --- ## Where things live +`app`, `api` and `cli` all build on `common-utils`, which holds the query +engine. + ```mermaid flowchart LR - app["packages/app
Next.js frontend"] - api["packages/api
Express backend"] - cu["packages/common-utils
query engine"] - cli["packages/cli
terminal UI"] - oc["packages/otel-collector
Go, schema migrations"] - eval["packages/hdx-eval
MCP eval harness"] + accTitle: L4 Detail -- Package dependencies + accDescr: The app, api and cli packages each depend on common-utils. + + app["packages/app"] + api["packages/api"] + cli["packages/cli"] + cu["packages/
common-utils"] app --> cu api --> cu cli --> cu -``` -| Package | Role | -| ------------------------- | ---------------------------------------------------------------- | -| `@hyperdx/app` | Frontend: search, dashboards, session replay | -| `@hyperdx/api` | REST API, OpAMP server, MCP server, ClickHouse proxy | -| `@hyperdx/common-utils` | Isomorphic query engine, Lucene-to-SQL parser, Zod types | -| `@hyperdx/cli` | Terminal CLI and TUI (`hdx`) | -| `@hyperdx/otel-collector` | Custom OTel Collector build, plus ClickHouse schema migrations | -| `@hyperdx/hdx-eval` | Benchmarks the MCP server against observability scenarios | - -Our additions live under `packages/api/src/dfe/**` and `packages/app/src/dfe/**` -so they cannot collide with upstream. Compose files and collector config are in -`docker/`. - ---- - -## Read next - -| Topic | Doc | -| --- | --- | -| How telemetry gets from an SDK to a chart | [data-flow.md](data-flow.md) | -| How a search becomes SQL | [clickhouse-data-model.md](clickhouse-data-model.md) | -| Frontend structure and state | [frontend.md](frontend.md) | -| API structure and the OpAMP/MCP servers | [backend.md](backend.md) | -| Envoy, the `x-oidc-*` contract, JWT verification | [oidc-authentication.md](oidc-authentication.md) | -| Who is allowed to see what | [authorization.md](authorization.md) | -| Which team a user lands on, its connection, the startup repair | [team-identity.md](team-identity.md) | -| Turning a saved search into a DFE rule | [query-to-rule-pipeline.md](query-to-rule-pipeline.md) | -| DocumentDB, service topology | [deployment.md](deployment.md) | -| Why we chose what we chose | [../decisions/](../decisions/) | - ---- - -## Provenance + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A +``` -These pages describe what the code does now. Where a design has changed, the superseded reasoning lives in [../decisions/](../decisions/). +*Figure 2. L4 Detail -- Package dependencies.* Up: [L2 Services -- dfe-hyperdx](#the-system). + +| Package | Role | +| ------------------------- | -------------------------------------------------------------- | +| `@hyperdx/app` | Frontend: search, dashboards, session replay | +| `@hyperdx/api` | REST API, OpAMP server, MCP server, ClickHouse proxy | +| `@hyperdx/common-utils` | Isomorphic query engine, Lucene-to-SQL parser, Zod types | +| `@hyperdx/cli` | Terminal CLI and TUI (`hdx`) | +| `@hyperdx/otel-collector` | Custom OTel Collector build, plus ClickHouse schema migrations | +| `@hyperdx/hdx-eval` | Benchmarks the MCP server against observability scenarios | + +Our additions live under `packages/api/src/dfe/**`, `packages/app/src/dfe/**` +and `packages/common-utils/src/dfe/**`, so they cannot collide with upstream. +Compose files and collector config are in `docker/`. + +The rest of this section, page by page: + +| Topic | Doc | +| -------------------------------------------------------------- | ------------------------------------------------------ | +| How telemetry gets from an SDK to a chart | [data-flow.md](data-flow.md) | +| How a search becomes SQL | [clickhouse-data-model.md](clickhouse-data-model.md) | +| Frontend structure and state | [frontend.md](frontend.md) | +| API structure and the OpAMP and MCP servers | [backend.md](backend.md) | +| The engine token and how the API verifies it | [oidc-authentication.md](oidc-authentication.md) | +| Who is allowed to see what | [authorization.md](authorization.md) | +| Which team a user lands on, its connection, the startup repair | [team-identity.md](team-identity.md) | +| Turning a saved search into a DFE rule | [query-to-rule-pipeline.md](query-to-rule-pipeline.md) | +| DocumentDB, service topology | [deployment.md](deployment.md) | +| Why we chose what we chose | [../decisions/](../decisions/README.md) | + +These pages describe what the code does. Where a design has changed, the +superseded reasoning lives in [../decisions/](../decisions/README.md). diff --git a/docs/architecture/authorization.md b/docs/architecture/authorization.md index 6e2a1bebe..88957a25a 100644 --- a/docs/architecture/authorization.md +++ b/docs/architecture/authorization.md @@ -1,65 +1,69 @@ # Authorization -**The DFE engine decides, ClickHouse enforces, HyperDX trusts the edge.** +**The DFE engine decides, ClickHouse enforces, HyperDX scopes by team.** -There is no authorization middleware inside HyperDX in this fork. Casbin RBAC -was removed as redundant - see [ADR 0004](../decisions/0004-casbin-removed.md) -for why. +HyperDX holds no policy store of its own in this fork. Casbin RBAC was removed +as redundant, see [ADR 0004](../decisions/0004-casbin-removed.md) for why. --- ## Who decides what +The engine names the session's team, the API scopes every document by that team, +and ClickHouse holds the team's user to its GRANTs. + ```mermaid flowchart LR - user(["user"]) --> envoy["Envoy
authenticates"] - envoy -->|identity headers| api["HyperDX API
trusts the edge"] - api -->|scopes every query by team| mongo["DocumentDB"] - api -->|team connection credentials| ch[(ClickHouse
GRANTs enforce)] - engine["DFE engine
policy decision point"] -.->|owns policy| ch - - classDef edge fill:#F0E442,stroke:#8a8200,color:#000000 - classDef app fill:#009E73,stroke:#005f45,color:#ffffff - classDef ours fill:#0072B2,stroke:#00456b,color:#ffffff - classDef store fill:#E69F00,stroke:#8a6100,color:#000000 - class envoy,user edge - class api,mongo app - class engine ours - class ch store + accTitle: L4 Detail -- Authorization layers + accDescr: A signed-in request reaches the HyperDX API. The API asks the DFE engine which team the session is on, scopes every DocumentDB query by that team, and queries ClickHouse as the team's own ClickHouse user, whose GRANTs decide what it reads. + + user["Signed-in user"] + api["HyperDX API"] + engine["DFE engine"] + docdb[("DocumentDB")] + ch[("ClickHouse
GRANTs")] + + user -->|engine token| api + api -->|which team| engine + api -->|team scoped| docdb + api -->|team's user| ch + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + class user ext + class docdb,ch store ``` -Three layers, each doing one thing: - -1. **Envoy** proves who the user is and forwards `x-oidc-*` / `X-Forwarded-*` - headers. See [oidc-authentication.md](oidc-authentication.md). -2. **HyperDX** maps that identity to a User and a Team, then scopes every query - by `team`. This is upstream behaviour, not something we added - every route - handler already does it, which is precisely why Casbin was redundant. -3. **ClickHouse** enforces data access with GRANTs. Each Team's `Connection` - record carries a team-specific ClickHouse username and password, so a query - that somehow escaped the application-level scoping still cannot read another - team's data. + +*Figure 1. L4 Detail -- Authorization layers. The dashed box is outside dfe-hyperdx.* Up: [L3 Components -- HyperDX API](backend.md#structure). -The DFE engine is the policy decision point for the wider platform. HyperDX does -not consult it per request; the decision is expressed as which ClickHouse -credentials a Team's connection holds. - ---- +Three layers, each doing one thing: -## Why the last layer matters +1. The engine says who the user is and which ClickHouse user the session reads + as. dfe-hyperdx verifies the engine's token and asks the engine about every + session. See [oidc-authentication.md](oidc-authentication.md). +2. HyperDX maps that identity to a User and a Team, then scopes every query by + `team`. Every route handler does this, which is why Casbin was redundant, and + the fork corrects `getTeam(id)` to filter by that id. Our admin lockdown on + top refuses human writes to teams, sources and connections, and every human + request to alerts and webhooks. +3. ClickHouse enforces data access with GRANTs. Each Team's `Connection` record + holds that team's ClickHouse username and password, so a query that escaped + the application-level scoping still cannot read another team's data. + +The engine is the policy decision point for the wider platform. HyperDX consults +it per session, cached for 30 seconds, and the decision is expressed as which +ClickHouse user a Team's connection holds +([team-identity.md](team-identity.md)). Application-level scoping is a correctness property of upstream's code, and upstream can change it. Data-layer GRANTs are ours, and they hold regardless. -That is the useful division: we do not try to re-implement upstream's tenancy -inside their code (which would be conflict surface we then have to defend on -every sync), we put our enforcement somewhere upstream cannot reach. - ---- - -## What this means when you add a route +That is the useful division. We do not re-implement upstream's tenancy inside +their code, which would be conflict surface to defend on every sync. We put our +enforcement where upstream cannot reach. -If you add a route under `packages/api/src/dfe/`, it inherits nothing -automatically. Scope your query by `req.user.team` the way upstream's handlers -do, and assume the ClickHouse credentials in play are already the right ones for -that team. +A route under `packages/api/src/dfe/` inherits nothing automatically. Scope your +query by `req.user.team` the way upstream's handlers do, and assume the +ClickHouse credentials in play are the right ones for that team. diff --git a/docs/architecture/backend.md b/docs/architecture/backend.md index 92d07bc35..6c1dff968 100644 --- a/docs/architecture/backend.md +++ b/docs/architecture/backend.md @@ -1,118 +1,116 @@ # Backend -> **Scope.** This describes UPSTREAM HyperDX's architecture, which we do not -> modify. It is here so the fork's own docs are readable on their own; -> upstream's [`agent_docs/`](../../agent_docs/) is the authority, and anything -> specific to DFE is called out as such. +> **Scope.** This describes UPSTREAM HyperDX's API, which we do not modify, and +> where our `dfe/` layer sits in it. It is here so the fork's own docs are +> readable on their own. Upstream's [`agent_docs/`](../../agent_docs/) is the +> authority for upstream's parts. + +## Structure + +With DFE auth on, application routes run after upstream's session middleware and +our identity middleware. Health probes and `/metrics` mount before both. ```mermaid -flowchart TB - subgraph Express_API_packages_api["Express API (packages/api)"] - MW["Middleware Stack
compression -> json -> session -> passport -> CORS"] - - subgraph Internal_Routes_Session_Auth["Internal Routes (Session Auth)"] - R1["/health, /login, /register"] - R2["/dashboards"] - R3["/alerts"] - R4["/saved-search"] - R5["/connections"] - R6["/sources"] - R7["/clickhouse-proxy"] - R8["/ai"] - R9["/team, /me, /webhooks"] - end - - subgraph External_API_Bearer_Token["External API (Bearer Token)"] - E1["/api/v2/alerts"] - E2["/api/v2/charts"] - E3["/api/v2/dashboards"] - E4["/api/v2/sources"] - end - - subgraph OpAMP_Server["OpAMP Server"] - OPAMP["OpAMP Endpoint
(separate port)"] - end +flowchart LR + accTitle: L3 Components -- HyperDX API + accDescr: The HyperDX UI calls the API. Inside, upstream's session middleware runs first, then the DFE identity middleware, then the internal routes or the ClickHouse proxy. The API asks the DFE engine about each session, keeps documents in DocumentDB and queries ClickHouse. + + app[["HyperDX UI
Next.js"]] + + subgraph api["HyperDX API"] + direction LR + session["Session
Passport"] --> identity[["DFE identity"]] + identity --> routes["Internal routes"] + identity --> proxy[["ClickHouse proxy"]] end - MW --> R1 & R2 & R3 & R4 & R5 & R6 & R7 & R8 & R9 - MW --> E1 & E2 & E3 & E4 + engine["DFE engine"] + ch[("ClickHouse")] + docdb[("DocumentDB")] + + app --> api + api -->|sessions| engine + api -->|queries| ch + api -->|documents| docdb - R7 -->|Reverse proxy| CH[(ClickHouse)] - R2 & R3 & R4 & R5 & R6 & R9 --> DOCDB["DocumentDB"] - OPAMP <-->|Protobuf| COLLECTOR["OTel Collector"] + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class docdb,ch store + class api group ``` -Authentication: + +*Figure 1. L3 Components -- HyperDX API.* Up: [L2 Services -- dfe-hyperdx](README.md#the-system). Down: [L4 Detail -- Engine token check](oidc-authentication.md#the-flow), [L4 Detail -- Query read path](data-flow.md#read-path). -- **Internal routes** use Passport.js session auth (`passport-local-mongoose`) - with sessions stored via `connect-mongo` (through DocumentDB in production). - In local mode (single user), authentication is bypassed entirely. -- **External API** (`/api/v2/*`) uses Bearer token auth matching the user's - `accessKey` field. Rate limited to 100 req/min. +| Part | What it is | +| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Session | Upstream's Passport.js session auth (`passport-local-mongoose`), sessions stored through `connect-mongo`. Bypassed in local mode (`IS_LOCAL_APP_MODE`) | +| DFE identity | Ours, `dfe/middleware/jwt-verify.ts`. Verifies the engine token and signs the user in, see [oidc-authentication.md](oidc-authentication.md) | +| Internal routes | `/dashboards`, `/saved-search`, `/sources`, `/connections`, `/team`, `/me`, `/ai` and the rest, plus our `/dfe` routes | +| ClickHouse proxy | `/clickhouse-proxy`, which sends the browser's SQL to ClickHouse with the team connection's credentials | +| External API | `/api/v2/*`, Bearer auth on the user's `accessKey`, rate limited per key (`EXTERNAL_API_RATE_LIMIT_MAX` a minute). Engine-only in DFE mode | +| OpAMP server | A separate port that serves collector configuration, below | -## OTel Collector & OpAMP +In DFE mode `/alerts`, `/webhooks`, `/api/v2`, `/mcp` and personal access keys +refuse every human session, and `/connections`, `/sources` and `/team` let a +member read but not write. The engine's service token reaches team, source and +connection routes alone. The list is in +[what-we-changed.md](../fork/what-we-changed.md#authorization). + +## OTel collector and OpAMP + +Upstream's collector takes its receivers, connectors, exporters and pipeline +wiring from the API over OpAMP. DFE runs no such collector: the DFE pipeline +writes the telemetry tables, so this path is idle in a DFE deployment. ```mermaid sequenceDiagram - participant API as HyperDX API
OpAMP Server - participant SUP as OpAMP Supervisor - participant COL as OTel Collector
child process + accTitle: L4 Detail -- OpAMP collector configuration + accDescr: The OpAMP supervisor reports to the API, the API answers with a generated collector configuration, and the supervisor starts the collector with it. Every later heartbeat gets the generated configuration and its hash. + participant API as HyperDX API + participant SUP as OpAMP supervisor + participant COL as OTel collector participant CH as ClickHouse - Note over API,COL: Startup - SUP->>API: AgentToServer (heartbeat) - API->>API: buildOtelCollectorConfig()
Generate full pipeline YAML - API-->>SUP: ServerToAgent
(RemoteConfig) - SUP->>COL: Start with generated config - - Note over API,COL: Steady State - loop Every heartbeat interval - SUP->>API: AgentToServer (status) - API-->>SUP: ServerToAgent (config if changed) + SUP->>API: AgentToServer, first heartbeat + API->>API: buildOtelCollectorConfig + API-->>SUP: ServerToAgent with RemoteConfig + SUP->>COL: start with the generated config + loop every heartbeat + SUP->>API: AgentToServer, status + API-->>SUP: ServerToAgent, config and hash end - - COL->>CH: Write telemetry data - COL->>CH: Scrape Prometheus metrics + COL->>CH: write telemetry ``` -The OTel collector runs under an **OpAMP supervisor** that manages its -lifecycle. The HyperDX API dynamically generates the collector's full pipeline -configuration (receivers, processors, connectors, exporters, and service -pipelines) based on team settings. This enables: + +*Figure 2. L4 Detail -- OpAMP collector configuration. Dashed arrows are answers.* Up: [L3 Components -- HyperDX API](#structure). -- **Remote configuration** - pipeline changes without collector restarts -- **Auth enforcement** - collector can require API keys when - `collectorAuthenticationEnforced` is enabled on the team -- **Session replay routing** - the routing connector separates rrweb events into - a dedicated pipeline and table +The remote configuration covers receivers, connectors, exporters and pipeline +wiring, built from team settings. Processors and each pipeline's processor list +come from the bootstrap `docker/otel-collector/config.yaml` and an optional +custom file. That gives pipeline changes without a collector restart, API-key +enforcement when a team turns on `collectorAuthenticationEnforced`, and the +routing connector that sends rrweb events to their own pipeline and table. With +no `OPAMP_SERVER_URL` the collector runs standalone on +`docker/otel-collector/config.yaml` plus a standalone config file. -In **standalone mode** (no OpAMP), the collector uses static config files from -`docker/otel-collector/config.yaml`. - -## Key Integrations - -## Session Replay +## Features Browser SDKs capture DOM mutations as **rrweb** events, shipped as OTLP logs with a `rr-web.event` attribute. The collector routes these to `hyperdx_sessions`. The frontend reconstructs playback using rrweb's `Replayer` -class, correlated to logs and traces via `rum.sessionId`. - -## Dashboards - -Stored as documents (in DocumentDB) as a set of **tiles**, each containing a -`SavedChartConfig` (source, display type, select/where/groupBy). Dashboard-level -filters apply across all tiles. Import/export is supported via versioned JSON -templates. - -## Saved Searches - -Persisted queries referencing a Source, with Lucene or SQL `where` clauses. -Stored as documents in DocumentDB. Displayed in the sidebar navigation grouped -by tags. Can have associated alerts. - -## AI Assistant - -Uses the Vercel AI SDK with Anthropic as the provider. The backend introspects -ClickHouse table metadata to provide schema context, then returns structured -chart/search/table configurations that the frontend renders directly. +class, correlated to logs and traces via `rum.sessionId`. The DFE console does +not offer session replay: the embed blocks `/sessions`. + +A dashboard is a document in DocumentDB holding a set of **tiles**, each a +`SavedChartConfig` (source, display type, select, where, groupBy). +Dashboard-level filters apply across all tiles, and import and export use +versioned JSON templates. A saved search is a query against one Source, with a +Lucene or SQL `where`, kept in DocumentDB and listed in the sidebar by tag. + +The AI assistant uses the Vercel AI SDK with Anthropic or OpenAI as the provider +(`AI_PROVIDER`). The backend introspects ClickHouse table metadata for schema +context, then returns chart, search and table configurations the frontend +renders directly. diff --git a/docs/architecture/clickhouse-data-model.md b/docs/architecture/clickhouse-data-model.md index 472a07c8b..3212ec404 100644 --- a/docs/architecture/clickhouse-data-model.md +++ b/docs/architecture/clickhouse-data-model.md @@ -1,128 +1,105 @@ # ClickHouse data model > **Scope.** This describes UPSTREAM HyperDX's architecture, which we do not -> modify. It is here so the fork's own docs are readable on their own; -> upstream's [`agent_docs/`](../../agent_docs/) is the authority, and anything +> modify. It is here so the fork's own docs are readable on their own. +> Upstream's [`agent_docs/`](../../agent_docs/) is the authority, and anything > specific to DFE is called out as such. -The central architectural innovation: HyperDX is not tied to any specific table -schema. The **Source** model maps ClickHouse table columns to semantic roles via -SQL expressions. +HyperDX is not tied to any table schema. The **Source** model maps ClickHouse +table columns to semantic roles through SQL expressions. A connection holds the +ClickHouse login, and each source points one kind of telemetry at one table +through that connection. ```mermaid flowchart LR - subgraph Configuration_DocumentDB["Configuration (DocumentDB)"] - CONN["Connection
host, user, password"] - SRC_LOG["Source (Log)
from: default.otel_logs"] - SRC_TRACE["Source (Trace)
from: default.otel_traces"] - SRC_SESSION["Source (Session)
from: default.hyperdx_sessions"] - SRC_METRIC["Source (Metric)
from: default.otel_metrics_*"] - end + accTitle: L4 Detail -- Sources and tables + accDescr: A connection holds the ClickHouse login. Log, trace, session and metric sources each read their own table through it. - subgraph Expression_Mapping["Expression Mapping"] - direction TB - TS["timestampValueExpression
-> TimestampTime"] - BODY["bodyExpression
-> Body"] - SEV["severityTextExpression
-> SeverityText"] - TID["traceIdExpression
-> TraceId"] - SID["spanIdExpression
-> SpanId"] - SVC["serviceNameExpression
-> ServiceName"] - end + conn["Connection
host, user, password"] - subgraph ClickHouse_Tables["ClickHouse Tables"] - T1["otel_logs"] - T2["otel_traces"] - T3["hyperdx_sessions"] - T4["otel_metrics_*"] + subgraph sources["Sources"] + srclog["Log source"] + srctrace["Trace source"] + srcsession["Session source"] + srcmetric["Metric source"] end - CONN --- SRC_LOG & SRC_TRACE & SRC_SESSION & SRC_METRIC - SRC_LOG --> TS & BODY & SEV & TID - SRC_LOG --> T1 - SRC_TRACE --> T2 - SRC_SESSION --> T3 - SRC_METRIC --> T4 + subgraph tables["ClickHouse tables"] + logs[("otel_logs")] + traces[("otel_traces")] + sessions[("hyperdx_sessions")] + metrics[("otel_metrics_*")] + end - SRC_LOG -.->|cross-references| SRC_TRACE - SRC_LOG -.->|cross-references| SRC_SESSION + conn -->|login| sources + srclog -->|reads| logs + srctrace -->|reads| traces + srcsession -->|reads| sessions + srcmetric -->|reads| metrics + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class logs,traces,sessions,metrics store + class sources,tables group ``` -Each Source has 20+ expression fields that map semantic concepts (timestamp, -body, severity, trace ID, span ID, service name, etc.) to arbitrary SQL -expressions over the underlying table. Sources cross-reference each other -(`logSourceId`, `traceSourceId`, `sessionSourceId`, `metricSourceId`), enabling -navigation between telemetry types. + +*Figure 1. L4 Detail -- Sources and tables.* Up: [Architecture](README.md). + +A source's expression fields map semantic concepts to SQL over its table: +`timestampValueExpression`, `bodyExpression`, `severityTextExpression`, +`traceIdExpression`, `spanIdExpression`, `serviceNameExpression` and many more. +Sources cross-reference each other (`logSourceId`, `traceSourceId`, +`sessionSourceId`, `metricSourceId`), which is how the UI moves between +telemetry types. So HyperDX can work on top of any ClickHouse table: point it at +your schema and map the columns. -This means HyperDX can work on top of **any** ClickHouse table - you point it at -your existing schema and map the columns. +In DFE the engine owns sources. A team is seeded with `main` and `hunts`, and +the platform team with the OTel tables and ClickHouse's `system` database as +well +([what-we-changed.md](../fork/what-we-changed.md#dfe-integration-features)). -## ClickHouse Data Model +## Tables + +Logs and traces join on `TraceId`, and traces join a browser session through the +`rum.sessionId` attribute. ```mermaid erDiagram + direction LR + accTitle: L4 Detail -- Upstream telemetry tables + accDescr: otel_logs and otel_traces share TraceId, and otel_traces links to hyperdx_sessions through the rum.sessionId attribute. otel_logs { - DateTime TimestampTime PK DateTime64 Timestamp String TraceId - String SpanId - UInt8 TraceFlags - String SeverityText - Int32 SeverityNumber String ServiceName - String Body - Map ResourceAttributes Map LogAttributes - String ScopeName } - otel_traces { - DateTime Timestamp PK + DateTime64 Timestamp String TraceId String SpanId - String ParentSpanId - String ServiceName - String SpanName - String SpanKind - String StatusCode - Map ResourceAttributes - Map SpanAttributes UInt64 Duration - Map Events_Timestamp - Map Events_Name - Map Events_Attributes - Map Links_TraceId - Map Links_SpanId } - hyperdx_sessions { - DateTime TimestampTime PK + DateTime TimestampTime String TraceId - String Body - String ServiceName - Map ResourceAttributes Map LogAttributes } - - otel_metrics_gauge { - DateTime TimeUnix PK - String MetricName - String ServiceName - Float64 Value - Map ResourceAttributes - Map MetricAttributes - } - - otel_logs ||--o{ otel_traces : "TraceId" - otel_logs ||--o{ hyperdx_sessions : "rum.sessionId" - otel_traces ||--o{ hyperdx_sessions : "rum.sessionId" + otel_logs }o--o{ otel_traces : "TraceId" + otel_traces }o--o{ hyperdx_sessions : "rum.sessionId" ``` -All tables use: - -- **MergeTree** engine with ZSTD compression -- **Partitioning** by `toDate(Timestamp)` -- **TTL** based on timestamp for automatic data expiry -- **Bloom filter indexes** on attribute map keys/values for fast filtering -- **tokenbf_v1** full-text index on Body/SpanName for text search -- **Materialized columns** for frequently-accessed nested attributes (e.g., - Kubernetes metadata, `rum.sessionId`) + +*Figure 2. L4 Detail -- Upstream telemetry tables. Key columns only.* Up: [Architecture](README.md). + +The schema is in `docker/otel-collector/schema/seed/`. The telemetry tables are +MergeTree with ZSTD codecs, partitioned by day and expired by a TTL on their +timestamp. Rollups are SummingMergeTree, and `00008_otel_metrics_timeseries.sql` +adds a TimeSeries table. The primary `otel_logs` and `otel_traces` schemas use +`text` indexes on attribute maps, trace ids and the body or span name. The +`_compat` schemas and `hyperdx_sessions` use bloom filter and `tokenbf_v1` +indexes. Kubernetes resource attributes, and `rum.sessionId` on traces, are +materialised columns. The metrics land in five tables, `otel_metrics_gauge`, +`_sum`, `_histogram`, `_exponential_histogram` and `_summary`. diff --git a/docs/architecture/data-flow.md b/docs/architecture/data-flow.md index 85d285af2..b0125042a 100644 --- a/docs/architecture/data-flow.md +++ b/docs/architecture/data-flow.md @@ -1,112 +1,134 @@ # Data flow > **Scope.** This describes UPSTREAM HyperDX's architecture, which we do not -> modify. It is here so the fork's own docs are readable on their own; -> upstream's [`agent_docs/`](../../agent_docs/) is the authority, and anything +> modify. It is here so the fork's own docs are readable on their own. +> Upstream's [`agent_docs/`](../../agent_docs/) is the authority, and anything > specific to DFE is called out as such. -## Write Path (Telemetry Ingestion) +## Write path + +Upstream's collector takes OTLP and Fluentd input, and its routing connector +sends rrweb events to their own table and everything else to the OTel tables. ```mermaid flowchart LR - subgraph Sources["Sources"] - A1["App (OTLP gRPC)"] - A2["App (OTLP HTTP)"] - A3["App (Fluentd)"] - A4["Browser (rrweb events)"] - end + accTitle: L4 Detail -- Telemetry write path + accDescr: Applications send OTLP and Fluentd input and browsers send rrweb events to the collector receivers. Processors transform and batch them, and the routing connector writes rrweb events to hyperdx_sessions and everything else to the OTel tables. - subgraph OTel_Collector["OTel Collector"] - direction TB - RX["Receivers
otlp, fluentforward"] - TX["Processors
transform, batch,
memory_limiter"] - RT["Routing Connector
(logs)"] - EX1["Exporter
clickhouse"] - EX2["Exporter
clickhouse/rrweb"] + subgraph sources["Sources"] + otlp["Apps, OTLP"] + fluent["Apps, Fluentd"] + rrweb["Browsers, rrweb"] end - subgraph ClickHouse["ClickHouse"] - T1["otel_logs"] - T2["otel_traces"] - T3["otel_metrics_*"] - T4["hyperdx_sessions"] - end - - A1 -->|:4317| RX - A2 -->|:4318| RX - A3 -->|:24225| RX - A4 -->|:4318| RX - RX --> TX --> RT - RT -->|rr-web.event present| EX2 - RT -->|default| EX1 - EX1 --> T1 & T2 & T3 - EX2 --> T4 + receivers["Receivers"] + processors["Processors"] + routing{"rr-web.event
present?"} + sessions[("hyperdx_sessions")] + otel[("otel_logs, traces,
metrics")] + + sources --> receivers + receivers --> processors + processors --> routing + routing -->|yes| sessions + routing -->|no| otel + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class otlp,fluent,rrweb ext + class routing decision + class sessions,otel store + class sources group ``` -Key details: + +*Figure 1. L4 Detail -- Telemetry write path. Dashed boxes are external.* Up: [Architecture](README.md). + +- Receivers accept OTLP (gRPC on 4317, HTTP on 4318) and Fluentd (24225). +- The transform processor parses JSON log bodies, infers severity and normalises + case. Batch and memory limiter processors follow. +- The routing connector checks the `rr-web.event` log attribute. +- The ClickHouse exporter writes `otel_logs`, `otel_traces` and five metric + tables over the native protocol. -- **Receivers** accept OTLP (gRPC on 4317, HTTP on 4318) and Fluentd (24225) -- **Transform processor** parses JSON log bodies, infers severity, normalizes - case -- **Routing connector** inspects log attributes - events with `rr-web.event` are - routed to the session replay pipeline (`hyperdx_sessions` table) -- **ClickHouse exporter** writes to `otel_logs`, `otel_traces`, and five metric - tables via the native TCP protocol (port 9000) +DFE runs none of this. The DFE pipeline writes the telemetry tables, and no +OpAMP-managed collector of HyperDX's is deployed. -## Read Path (Query Execution) +## Read path + +The browser renders the SQL, and the API adds the team connection's credentials +on the way to ClickHouse. ```mermaid sequenceDiagram - participant Browser + accTitle: L4 Detail -- Query read path + accDescr: The browser renders a chart config to SQL and posts it to the ClickHouse proxy. The API checks the session, loads the connection's credentials and forwards the query to ClickHouse, then returns the results. + participant Browser as Browser participant API as HyperDX API participant CH as ClickHouse - Browser->>Browser: renderChartConfig(chartConfig)
to parameterized SQL - Browser->>API: POST /api/clickhouse-proxy
(SQL + connection ID) - API->>API: Validate session
Load Connection credentials - API->>CH: Proxy HTTP request
(injected auth headers) - CH-->>API: Query results (JSON) - API-->>Browser: Query results + Browser->>Browser: renderChartConfig to parameterised SQL + Browser->>API: POST /api/clickhouse-proxy, SQL and connection id + API->>API: check session, load connection credentials + API->>CH: forward with the connection's ClickHouse user + CH-->>API: query results + API-->>Browser: query results ``` + +*Figure 2. L4 Detail -- Query read path. Dashed arrows are answers.* Up: [L3 Components -- HyperDX API](backend.md#structure). + The query engine (`renderChartConfig` in `common-utils`) runs **in the -browser**, generating parameterized ClickHouse SQL. The API acts as an -authenticated proxy - it never interprets the SQL, only validates the session -and injects ClickHouse credentials. +browser**, generating parameterised ClickHouse SQL. The API is an authenticated +proxy: it never interprets the SQL, it checks the session and sets +`X-ClickHouse-User` and `X-ClickHouse-Key` from the connection. In DFE mode it +first drops every credential the caller sent (`dropClientCredentials`). -In **local mode** (single-user deployment), the browser queries ClickHouse -directly, bypassing the proxy entirely. +In **local mode** (`IS_LOCAL_MODE`, a single-user deployment), the browser +queries ClickHouse directly, bypassing the proxy. -## Alert Evaluation (Disabled in DFE - See DFE Rules and Hunts) +## Alert evaluation -> **DFE note:** The HyperDX alert checker is **not started** in DFE deployments. -> Detection and alerting are handled by the DFE platform's rules engine and hunt -> workflows, which query ClickHouse directly. The alert API routes are blocked -> via Casbin RBAC. The code below documents upstream HyperDX's built-in alerting -> for reference only. +The alert checker task starts with the image in DFE as well, but nobody can +create an alert for it to evaluate. In DFE mode `/alerts` and `/webhooks` refuse +every human session, the engine's service token reaches team, source and +connection routes alone, and the console sends a visit to `/alerts` back to +`/search`. Detection and alerting are the DFE rules engine's and its hunts, +which query ClickHouse directly +([ADR 0002](../decisions/0002-alerting-disabled-for-dfe-rules.md)). The flow +below is upstream's, for reference. ```mermaid flowchart LR - subgraph check_alerts_Background_Task_DISABLED_IN_DFE["check-alerts (Background Task) - DISABLED IN DFE"] - LOAD["Load active alerts"] - BUILD["Build ChartConfig
from alert source"] - QUERY["Execute query
on ClickHouse"] - EVAL["Compare result
vs threshold"] - FIRE["Send webhook
notification"] - HIST["Write AlertHistory"] - end - - DB["DocumentDB"] --> LOAD - LOAD --> BUILD --> QUERY - CH[(ClickHouse)] <--> QUERY - QUERY --> EVAL - EVAL -->|Triggered| FIRE --> HIST - EVAL -->|OK| HIST - FIRE --> WH["Slack / Generic /
Incident.io Webhook"] - HIST --> DB + accTitle: L4 Detail -- Upstream alert evaluation + accDescr: The alert checker loads active alerts from DocumentDB, builds each one's query, runs it on ClickHouse and compares the result with the threshold. A triggered alert sends a webhook and records history, and an alert within its threshold records history alone. + + alerts[("Alerts
DocumentDB")] + build["Load and
build query"] + query["Query
ClickHouse"] + check{"Threshold
crossed?"} + fire(["Send webhook,
record history"]) + record(["Record history"]) + + alerts --> build + build --> query + query --> check + check -->|yes| fire + check -->|no| record + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + class alerts store + class check decision ``` -The alert checker runs as a separate Node.js process on a per-minute schedule. -Each alert references either a Saved Search or a Dashboard tile, from which a -`ChartConfig` is derived and evaluated against ClickHouse. Webhook notifications -use Mustache templates with full alert context. + +*Figure 3. L4 Detail -- Upstream alert evaluation.* Up: [Architecture](README.md). + +The checker runs once a minute. Each alert references a saved search or a +dashboard tile, from which a `ChartConfig` is derived and evaluated against +ClickHouse. Webhook bodies are templates with the alert's context. diff --git a/docs/architecture/deployment.md b/docs/architecture/deployment.md index 67c03f903..02201aa0e 100644 --- a/docs/architecture/deployment.md +++ b/docs/architecture/deployment.md @@ -2,52 +2,56 @@ **The document store is DocumentDB, and HyperDX does not know.** Mongoose, `connect-mongo`, `passport-local-mongoose`, `migrate-mongo` and every model and -route work unchanged - `MONGO_URI` points at the DocumentDB gateway. +route work unchanged: `MONGO_URI` points at the DocumentDB gateway. --- ## The data layer +The API's models and its session store both reach DocumentDB over an ordinary +`mongodb://` connection. + ```mermaid flowchart LR + accTitle: L4 Detail -- Document store + accDescr: The unmodified HyperDX API models and the connect-mongo session store connect to DocumentDB through its Mongo wire-protocol gateway. + subgraph app["HyperDX, unmodified"] - API["API
Mongoose ODM"] - SESS["session store
connect-mongo"] + models["API models
Mongoose"] + sessions["Session store
connect-mongo"] end - DOCDB[("DocumentDB
PostgreSQL 18 + extensions
Mongo wire gateway")] + docdb[("DocumentDB
PostgreSQL 18")] - API -->|mongodb://| DOCDB - SESS -->|mongodb://| DOCDB + app -->|mongodb://| docdb - classDef upstream fill:#009E73,stroke:#005f45,color:#ffffff - classDef ours fill:#0072B2,stroke:#00456b,color:#ffffff - classDef store fill:#E69F00,stroke:#8a6100,color:#000000 - class API,SESS upstream - class DOCDB store + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class docdb store + class app group ``` + +*Figure 1. L4 Detail -- Document store.* Up: [L3 Components -- HyperDX API](backend.md#structure). + The gateway speaks the MongoDB wire protocol, so standard drivers and tools (`mongosh`, Compass, `mongodump`) connect with an ordinary `mongodb://` string. -Documents land in PostgreSQL as BSON via the DocumentDB extensions, which brings -PostgreSQL's backup, replication and monitoring ecosystem to the data. On +Documents land in PostgreSQL as BSON through the DocumentDB extensions, which +brings PostgreSQL's backup, replication and monitoring to the data. On Kubernetes the DocumentDB operator runs the cluster on CloudNativePG (dfe-infra `helm/charts/documentdb`). ---- - -## Running it - -The override is `docker-compose.dfe.yml`, which replaces the `db` service and -repoints the app. **Read that file for the current values** - reproducing them -here just creates a second copy to go stale. +In compose, the override is `docker-compose.dfe.yml`, which replaces the `db` +service and repoints the app. **Read that file for the values**: reproducing +them here makes a second copy to go stale. -Two things about it are worth knowing, because they are easy to get wrong: +Two things about it are easy to get wrong: -- **TLS and SCRAM-SHA-256.** The gateway refuses plain connections. `MONGO_URI` - sets `tls=true`, a `tlsCAFile` for the CA that the `documentdb-tls` service - mints, and `authMechanism=SCRAM-SHA-256`. -- **The gateway listens on 10260 in the container.** Compose reaches it at +- TLS and SCRAM-SHA-256. The gateway rejects plain connections. `MONGO_URI` sets + `tls=true`, a `tlsCAFile` for the CA that the `documentdb-tls` service mints, + and `authMechanism=SCRAM-SHA-256`. +- The gateway listens on 10260 in the container. Compose reaches it at `documentdb:10260`, and the dev override publishes it as 27017. Credentials come from the environment (`POSTGRES_USER`, `POSTGRES_PASSWORD`), @@ -57,48 +61,48 @@ not from literals in the compose file. ## Service topology -```mermaid -flowchart TB - ext(["external traffic"]) - - subgraph net["compose network"] - APP["app
UI + API + OpAMP server"] - OTEL["otel-collector
OpAMP supervisor mode"] - CH[(ClickHouse
telemetry)] - DOCDB[("DocumentDB")] - end +In the compose stack, applications send telemetry to the collector and users +reach the app, and both write to or read ClickHouse. - ext -->|OTLP :4317 / :4318| OTEL - ext -->|UI + API| APP - OTEL -->|writes| CH - OTEL -->|scrapes| CH - APP -->|queries| CH - APP -->|metadata| DOCDB - APP <-->|OpAMP| OTEL - - classDef edge fill:#F0E442,stroke:#8a8200,color:#000000 - classDef svc fill:#009E73,stroke:#005f45,color:#ffffff - classDef ours fill:#0072B2,stroke:#00456b,color:#ffffff - classDef store fill:#E69F00,stroke:#8a6100,color:#000000 - class ext edge - class APP,OTEL svc - class CH,DOCDB store +```mermaid +flowchart LR + accTitle: L4 Detail -- Compose services + accDescr: Instrumented applications send OTLP to the otel-collector, which writes ClickHouse. Users reach the app container, which queries ClickHouse and keeps its documents in DocumentDB. + + senders["Instrumented
applications"] + users["Users"] + otel["otel-collector"] + app["app
UI, API, OpAMP"] + ch[("ClickHouse")] + docdb[("DocumentDB")] + + senders -->|OTLP| otel + users -->|UI and API| app + otel -->|writes| ch + app -->|queries| ch + app -->|documents| docdb + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + class senders,users ext + class ch,docdb store ``` -The `app` container bundles the Next.js frontend and the Express API. The -collector runs in **OpAMP supervisor mode** - it takes its pipeline -configuration from the API at runtime rather than from a static file, which is -why the API is both a query server and a control plane. - -Image tags and ports live in the compose files; see -[../development/docker-local.md](../development/docker-local.md) for running it -locally. + +*Figure 2. L4 Detail -- Compose services. Dashed boxes are external.* Up: [Architecture](README.md). ---- - -## What DFE adds on top - -In the DFE platform this sits behind Envoy, which terminates OIDC - see -[oidc-authentication.md](oidc-authentication.md). Alerting is not started; the -DFE rules engine owns detection +The `app` container bundles the Next.js frontend and the Express API. The +collector runs in **OpAMP supervisor mode**: it takes its pipeline from the +API's OpAMP server at runtime rather than from a static file, which is why the +API is both a query server and a control plane. It scrapes ClickHouse's +Prometheus metrics as well as writing telemetry. Image tags and ports live in +the compose files, and +[../development/docker-local.md](../development/docker-local.md) covers running +it locally. + +In the DFE platform the console frames the app and the API verifies the engine's +token, see [oidc-authentication.md](oidc-authentication.md). The DFE pipeline +writes the telemetry, so no collector of HyperDX's runs there. Alerting stays +upstream's code with nobody able to create an alert ([ADR 0002](../decisions/0002-alerting-disabled-for-dfe-rules.md)). diff --git a/docs/architecture/frontend.md b/docs/architecture/frontend.md index 5e2459089..ad9d46af5 100644 --- a/docs/architecture/frontend.md +++ b/docs/architecture/frontend.md @@ -1,63 +1,56 @@ # Frontend -> **Scope.** This describes UPSTREAM HyperDX's architecture, which we do not -> modify. It is here so the fork's own docs are readable on their own; -> upstream's [`agent_docs/`](../../agent_docs/) is the authority, and anything -> specific to DFE is called out as such. +> **Scope.** This describes UPSTREAM HyperDX's frontend, which we do not modify, +> as the DFE console frames it. It is here so the fork's own docs are readable +> on their own. Upstream's [`agent_docs/`](../../agent_docs/) is the authority, +> and anything specific to DFE is called out as such. + +## Structure + +Charts build their SQL in the browser and send it through the API's ClickHouse +proxy, and everything else goes through the API's REST routes. ```mermaid -flowchart TB - subgraph Next_js_Pages_Router["Next.js (Pages Router)"] - SEARCH["Search Page
Log/Trace search"] - DASH["Dashboard Page
Chart tiles"] - ALERTS["Alerts Page
Alert management"] - SESSIONS["Sessions Page
Session replay"] - SERVICES["Services Page
APM overview"] - SVCMAP["Service Map
Trace topology"] - CHART["Chart Explorer
Ad-hoc charting"] - CHSQL["SQL Page
Direct ClickHouse SQL"] - end +flowchart LR + accTitle: L3 Components -- HyperDX UI + accDescr: The DFE console frames the UI pages. Pages render charts through the browser ClickHouse client, which builds SQL with renderChartConfig, and load everything else through the API hooks. Both reach the HyperDX API. - subgraph Core_Components["Core Components"] - NAV["AppNav
Sidebar navigation"] - TIMECHART["DBTimeChart
Time-series visualization"] - TABLE["DBTableChart
Tabular data"] - SIDEPANEL["DBRowSidePanel
Log/trace detail"] - WATERFALL["DBTraceWaterfallChart
Span waterfall"] - PLAYER["DOMPlayer
rrweb session replay"] - SQLEDITOR["SQLEditor
CodeMirror SQL input"] - SEARCHINPUT["SearchInputV2
Lucene/SQL autocomplete"] - end + console["DFE console
dfe-ui"] - subgraph State_Data["State & Data"] - URL["URL Params
(nuqs)"] - TQ["TanStack Query
(server state)"] - CHC["ClickhouseClient
(browser)"] - API_HOOKS["API Hooks
(ky + /api/*)"] + subgraph app["HyperDX UI"] + pages["Pages
search, dashboards"] + chc["ClickHouse client
renderChartConfig"] + hooks["API hooks
ky, TanStack Query"] end - SEARCH & DASH & ALERTS & SESSIONS --> NAV - SEARCH --> SEARCHINPUT & TABLE & SIDEPANEL - DASH --> TIMECHART & TABLE - SESSIONS --> PLAYER - SERVICES --> TIMECHART - SVCMAP --> WATERFALL - - TIMECHART & TABLE --> CHC - CHC -->|common-utils
renderChartConfig| CHC - CHC -->|/api/clickhouse-proxy| API_HOOKS - ALERTS --> API_HOOKS - API_HOOKS --> TQ - SEARCH & DASH --> URL + api[["HyperDX API
Express"]] + + console -->|frames| pages + pages --> chc + pages --> hooks + chc --> api + hooks --> api + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class app group ``` + +*Figure 1. L3 Components -- HyperDX UI.* Up: [L2 Services -- dfe-hyperdx](README.md#the-system). + +The DFE embed turns on four features, search, saved searches, the chart explorer +and dashboards (`packages/app/src/dfe/embedFeatures.ts`). A direct visit to +`/alerts`, `/sessions`, `/service-map` or `/team` is sent back to `/search`. + Key patterns: -- **URL-driven state**: All search filters, time ranges, and dashboard contexts - are encoded in URL parameters (via `nuqs`), making every view deep-linkable -- **Server state**: TanStack Query manages all API data with custom hooks in - `api.ts`, `dashboard.ts`, `savedSearch.ts`, `source.ts`, `sessions.ts` -- **Query engine in browser**: `renderChartConfig()` from `common-utils` runs - client-side, generating parameterized SQL sent through the ClickHouse proxy -- **UI library**: Mantine components throughout, with Recharts and uPlot for - charts, CodeMirror for SQL editing, and rrweb for session replay +- URL-driven state. Search filters, time ranges and dashboard contexts are + encoded in URL parameters (`nuqs`), so every view is deep-linkable. +- Server state. TanStack Query manages API data, with hooks in `api.ts`, + `dashboard.ts`, `savedSearch.ts`, `source.ts` and `sessions.ts`. +- Query engine in the browser. `renderChartConfig()` from `common-utils` runs + client-side and generates parameterised SQL, sent through + `/api/clickhouse-proxy`. +- UI library. Mantine components throughout, with Recharts and uPlot for charts, + CodeMirror for SQL editing, and rrweb for session replay. diff --git a/docs/architecture/oidc-authentication.md b/docs/architecture/oidc-authentication.md index 252861e67..691cbef02 100644 --- a/docs/architecture/oidc-authentication.md +++ b/docs/architecture/oidc-authentication.md @@ -3,171 +3,185 @@ **The DFE engine is the single JWT issuer. dfe-hyperdx verifies its token rather than trusting a header.** -Envoy terminates OIDC at the edge and verifies the engine's ES384 token. -dfe-hyperdx verifies it _again_ against the engine's JWKS, because dfe-hyperdx -is a policy enforcement point and an unauthenticated header is not an identity. - -None of this touches upstream's auth. It is one middleware registered ahead of -Passport, and with `DFE_AUTH_MODE` unset it is never registered at all - the -fork behaves exactly as upstream ships. +The user signs in on the DFE console, through the engine, which is the OIDC +relying party to the customer's identity provider. The console plants the +engine's ES384 token in the `dfe_token` cookie, scoped by `DFE_COOKIE_DOMAIN` to +the parent domain so the HyperDX subdomain receives it. The gateway puts no OIDC +policy on the HyperDX route, because a provider redirect cannot complete inside +the console's iframe. So dfe-hyperdx verifies the token against the engine's +JWKS: it is the policy enforcement point, and an unverified header is not an +identity. + +In DFE mode HyperDX's password login, registration and invite acceptance answer +404 (`dfe/middleware/legacy-auth-lockdown.ts`). None of this touches upstream's +auth code. It is one middleware, registered after Passport's session middleware +and ahead of every router, and with `DFE_AUTH_MODE` unset it is never registered +at all: the fork behaves exactly as upstream ships. --- ## The flow +The API verifies the token's signature itself, then asks the engine what the +session may do, and only an engine answer signs the user in. + ```mermaid sequenceDiagram - participant User as Browser - participant Envoy as Envoy at the edge - participant Engine as DFE engine + accTitle: L4 Detail -- Engine token check + accDescr: The browser sends the engine token to the dfe-hyperdx API. The API fetches the engine JWKS, verifies the token, asks the engine for the session's groups, ClickHouse identity and role, then finds or creates the user and team and signs the user in for this request. + participant Browser as Browser participant HDX as dfe-hyperdx API + participant Engine as DFE engine - User->>Envoy: request - Envoy->>Envoy: OIDC flow, verify engine token - Envoy->>HDX: forward with Bearer token or dfe_token cookie - HDX->>Engine: fetch JWKS (cached) - HDX->>HDX: jwtVerify ES384, check iss - HDX->>Engine: GET /api/v1/auth/me with the token (cached) - Engine-->>HDX: groups, ClickHouse identity and role - HDX->>HDX: sub claim to email, ClickHouse identity to team - HDX->>HDX: find-or-provision user and team, ensure its connection - HDX->>HDX: req.login(session false) - HDX-->>User: upstream isUserAuthenticated now passes + Browser->>HDX: request with dfe_token cookie or Bearer token + HDX->>Engine: fetch JWKS, cached + HDX->>HDX: verify ES384 signature and issuer + HDX->>Engine: GET /api/v1/auth/me, cached + Engine-->>HDX: groups, ClickHouse identity, role + HDX->>HDX: find or create user and team, req.login + HDX-->>Browser: the route answers ``` + +*Figure 1. L4 Detail -- Engine token check. Dashed arrows are answers.* Up: [L3 Components -- HyperDX API](backend.md#structure). + Three things about the flow are worth knowing: -- **`req.login(..., { session: false })`** populates `req.user` exactly as - Passport would, so every upstream route guard works unmodified. That is why - the fork needs no changes to upstream's auth code. -- **A missing or invalid token falls THROUGH, it does not 401.** The route-level - `isUserAuthenticated` guard is what rejects the request. Failing open here - would be a hole; failing closed here would break upstream's own session login, - which still has to work. -- **A valid token the engine refuses is answered here.** The engine's own 401 or - 403 passes straight through, a session the engine grants no group or no - ClickHouse identity gets 403, and an engine that is unreachable, times out or - answers with anything unreadable gets 401. None of them falls back to the - token's claims. +- `req.login(..., { session: false })` populates `req.user` exactly as Passport + would, so every upstream route guard works unmodified. That is why the fork + needs no changes to upstream's auth code. +- A missing or invalid token falls THROUGH, it does not 401. The route-level + `isUserAuthenticated` guard is what rejects the request. Routes upstream + leaves public stay public, and the guard stays the one place that rejects. +- A valid token the engine rejects is answered here. The engine's own 401 or 403 + passes straight through, a session the engine grants no group or no ClickHouse + identity gets 403, and an engine that is unreachable, times out or answers + with anything unreadable gets 401. None of them falls back to the token's + claims. --- ## Where the identity comes from -```mermaid -flowchart TB - entry["dfeIdentityMiddleware
jwt-verify.ts"] --> mode{"DFE_AUTH_MODE"} - mode -->|oidc-proxy, the default| jwt["engineJwtMiddleware
verify ES384 against JWKS"] - mode -->|header-dev| hdr["oidcIdentityMiddleware
trust headers UNVERIFIED"] - mode -->|unset| off["never registered
upstream behaviour"] +`DFE_AUTH_MODE` picks the path when the API starts, and both DFE paths end in +the same user and team lookup. - jwt --> resolve["find-or-provision
user and team"] +```mermaid +flowchart LR + accTitle: L4 Detail -- Identity mode + accDescr: At start the API reads DFE_AUTH_MODE. oidc-proxy verifies the engine token, header-dev trusts identity headers unverified, and unset leaves upstream auth alone. Both DFE paths find or create the user and team. + + start(["API starts"]) + mode{"DFE_AUTH_MODE?"} + jwt["Verify engine
ES384 token"] + hdr["Trust identity
headers"] + off(["Upstream
auth only"]) + resolve(["Find or create
user and team"]) + + start --> mode + mode -->|oidc-proxy| jwt + mode -->|header-dev| hdr + mode -->|unset| off + jwt --> resolve hdr --> resolve - classDef safe fill:#009E73,stroke:#005f45,color:#ffffff - classDef danger fill:#D55E00,stroke:#7a3500,color:#ffffff - classDef neutral fill:#0072B2,stroke:#00456b,color:#ffffff - class jwt,off safe - class hdr danger - class entry,mode,resolve neutral + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + class mode decision ``` + +*Figure 2. L4 Detail -- Identity mode.* Up: [L3 Components -- HyperDX API](backend.md#structure). + **`header-dev` is dev-only and trusts headers without any verification.** It exists so local development works without a running engine. Anything that can -reach the API can assert any identity under it. Never set it in production. - -### Token source - -`Authorization: Bearer ` if present, otherwise a `dfe_token` cookie. The -cookie header is parsed directly rather than adding `cookie-parser`, to avoid a -dependency for one lookup. - -In `oidc-proxy` mode a request that does not authenticate through an engine -token is anonymous, even when it carries an old dfe-hyperdx session cookie. The -middleware clears the user passport restored before it runs. - -### Claims - -| Claim | Used for | -| ------ | -------------------------------------------------------------- | -| `sub` | the user's email; without it the request falls through | -| `iss` | enforced when `DFE_ENGINE_ISSUER` is set | -| `exp` | caps how long the engine's answer for the token is kept | -| `role` | the dashboard role, only when the engine's answer carries none | - -### Team - -The `groups` claim is never read. Since dfe-engine#591 a token's own groups -grant nothing: the engine resolves a session's groups from the account the token +reach the API can assert any identity under it, so it will not start with +`NODE_ENV=production`. + +The token is `Authorization: Bearer ` if present, otherwise the +`dfe_token` cookie. The cookie header is parsed directly rather than adding +`cookie-parser`, to avoid a dependency for one lookup. In `oidc-proxy` mode a +request that does not authenticate through an engine token is anonymous, even +when it has an old dfe-hyperdx session cookie: the middleware clears the user +Passport restored before it runs. + +The middleware reads these claims: + +| Claim | Used for | +| -------------------------- | --------------------------------------------------------------- | +| `sub` | the user's email. Without it the request falls through | +| `iss` | enforced when `DFE_ENGINE_ISSUER` is set | +| `exp` | caps how long the engine's answer for the token is kept | +| `role` | the dashboard role, only when the engine's answer has none | +| `password_change_required` | when true the request falls through and the engine is not asked | + +The `groups` claim is never read. Since dfe-engine#591 a token's groups grant +nothing: the engine resolves a session's groups from the account the token binds. So the middleware asks the engine, `GET /api/v1/auth/me` with the caller's token. The team is named after the ClickHouse user the engine hands the caller (`hyperdx_identity`): one team per org, plus one platform team. An empty answer -is a refusal, never `DFE_AUTH_DEFAULT_TEAM`. A user already on another team is -moved onto this one, because upstream never moves a user off the team it was -created on. The rule, the connection seeding and the startup repair are in -[team-identity.md](team-identity.md). - -The dashboard role is the engine's `hyperdx_role` from the same answer, and the -token's `role` claim only when the engine sends none. +is a refusal, never `DFE_AUTH_DEFAULT_TEAM`. A user on another team is moved +onto this one, because upstream never moves a user off the team it was created +on. The rule, the connection seeding and the startup repair are in +[team-identity.md](team-identity.md). The dashboard role is the engine's +`hyperdx_role` from the same answer, and the token's `role` claim when the +engine sends none. The answer is cached per token for 30 seconds, never past the token's `exp`, at most 1000 entries, and concurrent requests for one token share one lookup. That 30 seconds is how long a removed group, a changed role or a deleted account takes to reach dfe-hyperdx. A lookup the engine could not answer is never -cached. - -Both the user and the team are find-or-create, so there is no registration or -invite step in DFE mode. - ---- - -## Configuration - -| Variable | Default | Purpose | -| ------------------------ | -------------------- | --------------------------------------------------------------------------------------------- | -| `DFE_AUTH_MODE` | unset | `oidc-proxy`, `header-dev`, or unset for stock upstream. Any other value stops the API | -| `DFE_ENGINE_JWKS_URL` | - | engine JWKS URL, and `oidc-proxy` mode will not start without it | -| `DFE_ENGINE_ISSUER` | unset | enforced as the `iss` claim when set | -| `DFE_ENGINE_TIMEOUT_MS` | `5000` | how long any request to the engine may take before it is abandoned | -| `DFE_AUTH_HEADER_EMAIL` | `x-forwarded-email` | `header-dev` only, which will not start with `NODE_ENV=production` | -| `DFE_AUTH_HEADER_GROUPS` | `x-forwarded-groups` | `header-dev` only | -| `DFE_AUTH_DEFAULT_TEAM` | unset | the engine service identity's team, and the `header-dev` team when no group header is present | +cached. Both the user and the team are find-or-create, so there is no +registration or invite step in DFE mode. -Read in `packages/api/src/dfe/config.ts`; `isDfeEnabled` is what `api-app.ts` -gates the middleware registration on. +A token whose `sub` is `svc:dfe-engine` and whose audience includes +`dfe-hyperdx` is the engine's service identity. It signs in as an internal +principal on the default team, and reaches `/team`, `/sources`, `/connections` +and `/dfe/sources` only. Anywhere else it gets 403. --- -## The files - -| File | Role | -| -------------------------------------- | ------------------------------------------------------------ | -| `dfe/middleware/jwt-verify.ts` | entry point, mode switch, ES384 verification | -| `dfe/middleware/oidc-identity.ts` | the `header-dev` path | -| `dfe/controllers/engine-session.ts` | the engine's answer for a token, cached; the team and role | -| `dfe/controllers/user-provisioning.ts` | find-or-create the user, and move it to its team | -| `dfe/controllers/team-provisioning.ts` | find-or-create the team | -| `dfe/controllers/org-connection.ts` | the team's one connection, as the team's own ClickHouse user | -| `dfe/tasks/team-connection-repair.ts` | the startup delete of connections that are not their team's | -| `dfe/config.ts` | the variables above | -| `api-app.ts` | the one upstream file touched - a guarded `app.use` | +## Configuration and files + +| Variable | Default | Purpose | +| ------------------------ | -------------------- | --------------------------------------------------------------------------------------------------------- | +| `DFE_AUTH_MODE` | unset | `oidc-proxy`, `header-dev`, or unset for stock upstream. Any other value stops the API | +| `DFE_ENGINE_JWKS_URL` | - | engine JWKS URL, and `oidc-proxy` mode will not start without it | +| `DFE_ENGINE_ISSUER` | unset | enforced as the `iss` claim when set | +| `DFE_ENGINE_TIMEOUT_MS` | `5000` | how long a session, connection or rule request to the engine may take. The JWKS fetch uses jose's timeout | +| `DFE_AUTH_HEADER_EMAIL` | `x-forwarded-email` | `header-dev` only, which will not start with `NODE_ENV=production` | +| `DFE_AUTH_HEADER_GROUPS` | `x-forwarded-groups` | `header-dev` only | +| `DFE_AUTH_DEFAULT_TEAM` | unset | the engine service identity's team, and the `header-dev` team when no group header is present | + +They are read in `packages/api/src/dfe/config.ts`, and `isDfeEnabled` is what +`api-app.ts` gates the middleware registration on. + +| File | Role | +| -------------------------------------- | --------------------------------------------------------------- | +| `dfe/middleware/jwt-verify.ts` | entry point, mode switch, ES384 verification | +| `dfe/middleware/oidc-identity.ts` | the `header-dev` path | +| `dfe/controllers/engine-session.ts` | the engine's answer for a token, cached, with team and role | +| `dfe/controllers/user-provisioning.ts` | find-or-create the user, and move it to its team | +| `dfe/controllers/team-provisioning.ts` | find-or-create the team | +| `dfe/controllers/org-connection.ts` | the team's one connection, as the team's ClickHouse user | +| `dfe/tasks/team-connection-repair.ts` | the startup delete of connections that are not their team's | +| `dfe/config.ts` | the variables above | +| `api-app.ts` | upstream's file, where a guarded block registers the middleware | The JWKS resolver is built once and reused. `createRemoteJWKSet` does its own fetch caching, coalescing and cooldown, so there is no key cache of ours to get wrong. ---- - -## AI steering - -| Don't | Do | Why | -| -------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------ | -| Trust `x-forwarded-email` in new code | Read `req.user`, set by the middleware | The header is only authoritative in `header-dev`, which is not production | -| Return 401 for a missing or invalid token | Fall through and let `isUserAuthenticated` decide | Upstream session login must keep working alongside DFE mode | -| Read the token's `groups` claim for a team | Take `resolveEngineSession`'s team | The claim grants nothing; the account the token binds decides | -| Name a team after a group, or reuse its connection | Name it after the ClickHouse identity; seed only that user | A team's connection is what every member queries as | -| Fall back to a claim when the engine is down | Refuse | A team is ClickHouse access, so an unanswered lookup must not grant one | -| Add auth logic to upstream's `middleware/auth.ts` | Add it under `dfe/` and register it in `api-app.ts` | Editing upstream's auth is permanent conflict surface, and the guard blocks it | -| Widen `algorithms` beyond `['ES384']` | Leave it pinned | Algorithm confusion is the classic JWT verification bug | +The mistakes a change here tends to make, and what to do instead: + +| Don't | Do | Why | +| -------------------------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------ | +| Trust `x-forwarded-email` in new code | Read `req.user`, set by the middleware | The header is authoritative in `header-dev` alone, which is not production | +| Return 401 for a missing or invalid token | Fall through and let `isUserAuthenticated` decide | The route guard is the one place that rejects, so public routes stay public | +| Read the token's `groups` claim for a team | Take `resolveEngineSession`'s team | The claim grants nothing. The account the token binds decides | +| Name a team after a group, or reuse its connection | Name it after the ClickHouse identity, and seed that user | A team's connection is what every member queries as | +| Fall back to a claim when the engine is down | Refuse | A team is ClickHouse access, so an unanswered lookup must not grant one | +| Add auth logic to upstream's `middleware/auth.ts` | Add it under `dfe/` and register it in `api-app.ts` | Editing upstream's auth is permanent conflict surface, and the guard blocks it | +| Widen `algorithms` beyond `['ES384']` | Leave it pinned | Algorithm confusion is the classic JWT verification bug | diff --git a/docs/architecture/query-to-rule-pipeline.md b/docs/architecture/query-to-rule-pipeline.md index 7ed6e5303..bda15c8ce 100644 --- a/docs/architecture/query-to-rule-pipeline.md +++ b/docs/architecture/query-to-rule-pipeline.md @@ -1,40 +1,48 @@ # Query-to-rule pipeline **Turn a HyperDX search into a DFE detection rule without retyping it.** An -analyst who has already found the thing in HyperDX presses a button; the engine -gets the SQL that search runs and saves it as a rule a hunt can use. +analyst who has found the thing in HyperDX presses a button, and the engine gets +the SQL that search runs and saves it as a rule a hunt can use. Two routes (`POST /dfe/export-sql`, `POST /dfe/create-rule`) and one button, all -under `dfe/` - no upstream file is modified. +under `dfe/`. No upstream file is modified. --- ## The path +The API renders the search with the same renderer the search page uses, and the +engine decides whether the result is a rule. + ```mermaid sequenceDiagram + accTitle: L4 Detail -- Search to rule + accDescr: The analyst refines a search. The UI asks the API to export it, and the API reads column metadata from ClickHouse and returns the rendered SQL. The UI posts the raw SQL to create-rule, the API forwards it to the engine as the caller, and the engine's answer comes back to the UI. participant User as Analyst participant App as HyperDX UI - participant API as HyperDX API /dfe + participant API as HyperDX API participant CH as ClickHouse participant Engine as DFE engine - User->>App: refine a search until it is right - App->>API: POST /dfe/export-sql with the search's chart config - API->>API: resolve source and connection for the team + User->>App: refine a search + App->>API: POST /dfe/export-sql API->>CH: fetch column metadata - API->>API: renderChartConfig then parameterizedQueryToSql then format API-->>App: sql, rawSql, config, source App->>API: POST /dfe/create-rule with rawSql - API->>Engine: POST /api/v1/rules/from-hyperdx as the caller - Engine-->>API: the rule id, or 422 when the SQL cannot be compiled - API-->>App: the engine's status and body - App-->>User: open the rule in the DFE UI, or show the engine's reason + API->>Engine: POST /api/v1/rules/from-hyperdx + Engine-->>API: rule id, or 422 + API-->>App: engine status and body ``` + +*Figure 1. L4 Detail -- Search to rule. Dashed arrows are answers.* Up: [L3 Components -- HyperDX API](backend.md#structure). + +The UI then opens the created rule in the DFE console, or shows why it was +refused. + --- -## What the export renders +## The routes and the button `packages/api/src/dfe/routers/query-export.ts`, mounted at `/dfe` behind upstream's `isUserAuthenticated`. @@ -51,62 +59,53 @@ plus optional `startTime` / `endTime` in milliseconds. | `config` | the original chart config, for structured consumption | | `source` | `{ name, kind, from, connection }` | -The query is the one the search runs. `from` is the source's own table, and the +The query is the one the search runs. `from` is the source's table, and the search bar, its language and every side-panel filter reach `renderChartConfig` -as the view holds them, so the renderer that runs the search also writes the -rule. The engine reads the `FROM` as the table the hunt scans and the `WHERE` as -the detection logic. +as the view holds them, so the renderer that runs the search writes the rule. +The engine reads the `FROM` as the table the hunt scans and the `WHERE` as the +detection logic. -There is no time bound. The saved chart config carries no -`timestampValueExpression`, so the renderer emits none, and the hunt runner -supplies its own window. The engine strips the display-only rest (`LIMIT`, -`SETTINGS`, time buckets). +The button sends no `startTime` or `endTime`, so the rule has no time bound. The +saved chart config has no `timestampValueExpression`, so the renderer emits +none, and the hunt runner supplies its window. The engine strips the +display-only rest (`LIMIT`, `SETTINGS`, time buckets). The engine expands no placeholder. A templated table such as `{{org_id}}.{{source_table_name}}` or a `{timestamp_condition}` does not parse, -and the engine refuses a rule it cannot compile with 422 and saves nothing. - -### Rendering - -`renderChartConfig` (needs ClickHouse column metadata, hence the client), then -`parameterizedQueryToSql`, then `format`. **A formatting failure is not -fatal** - it falls back to the raw SQL, because a valid query that reads badly -beats no query at all. - ---- +and the engine rejects with 422, saving nothing, a rule it cannot compile or one +that matches every event. -## The button +The render is `renderChartConfig` (it needs ClickHouse column metadata, hence +the client), then `parameterizedQueryToSql`, then `format`. A formatting failure +is not fatal: it falls back to the raw SQL, because a valid query that reads +badly beats no query at all. `packages/app/src/dfe/components/CreateRuleFromSearch/` exports the search, posts the `rawSql` to `/dfe/create-rule`, and opens the created rule at -`DFE_UI_BASE_URL/rules/`. A refusal shows the engine's own reason. +`DFE_UI_BASE_URL/rules/`. A 422 shows the engine's reason, a 403 says the +user may not create rules, and an engine that rejects the session's token (which +the route answers as 502, so the app does not redirect to a login) says the +engine did not accept the session. It deliberately uses plain `useState` rather than react-query's `useMutation`. -The component is injected into upstream's `DBSearchPage`, and upstream's own -tests render that page behind a _partial_ `@tanstack/react-query` mock - pulling -a second hook out of that module makes their pristine tests explode. Our delta -has to stay invisible to upstream's test suite. - ---- - -## Version sensitivity +The component is injected into upstream's `DBSearchPage`, and upstream's tests +render that page behind a PARTIAL `@tanstack/react-query` mock. Pulling a second +hook out of that module makes their pristine tests explode. Our delta has to +stay invisible to upstream's test suite. Two things moved in 2.29 and the code guards for both: - `SavedChartConfig` became a **union**, so `chartConfig.source` is optional and - `where` only exists on the builder variant. Both are checked before use. + `where` exists on the builder variant alone. Both are checked before use. - `source.connection` is a **connection-id string**, not a populated object. -Neither guard is decorative; removing either produces a runtime failure on some -chart types only. - ---- +Removing either guard produces a runtime failure on some chart types. -## AI steering +The mistakes a change here tends to make, and what to do instead: | Don't | Do | Why | | ------------------------------------------------ | ----------------------------------------- | ---------------------------------------------------------- | -| Assume `chartConfig.where` exists | Check `'where' in chartConfig` first | It is a union; raw-SQL and PromQL variants have no `where` | -| Swap the table or the time filter for a template | Render against `source.from` as it stands | The engine expands no placeholder and refuses the rule | +| Assume `chartConfig.where` exists | Check `'where' in chartConfig` first | It is a union. Raw-SQL and PromQL variants have no `where` | +| Swap the table or the time filter for a template | Render against `source.from` as it stands | The engine expands no placeholder and rejects the rule | | Rewrite the search into SQL by hand | Let `renderChartConfig` render it | The search page runs the same renderer | | Add a react-query hook to this component | Use plain state | Upstream's partial mock of that module breaks their tests | diff --git a/docs/architecture/team-identity.md b/docs/architecture/team-identity.md index 77ff82c05..717336bf9 100644 --- a/docs/architecture/team-identity.md +++ b/docs/architecture/team-identity.md @@ -7,11 +7,7 @@ caller holding `query:execute` at system scope through a role other than `org_viewer`, else `dfe_org_` for a caller that resolves to exactly one org. The team is named after that username, so there is one team per org plus one platform team, and two users share a team only when the engine gives them -the same identity. - ---- - -## The rule +the same identity. How the API learns it: - `GET /api/v1/auth/me` answers `hyperdx_identity`, the username `GET /api/v1/hyperdx/connection` hands the same caller, without its password. @@ -29,53 +25,92 @@ the same identity. ## Seeding the connection `dfe/controllers/org-connection.ts` runs on every request, after the user is -placed on the team. +placed on the team. A team holding its user goes on, a team holding any other +user rejects the session, and a team holding none is seeded. + +```mermaid +flowchart LR + accTitle: L4 Detail -- Team connection check + accDescr: On every request the API checks the team's connection. A connection as the team's own ClickHouse user lets the request carry on, one as another user is refused with 403, and no connection starts a seed. + + request(["Request on
a team"]) + held{"Team holds
a connection?"} + carry(["Carry on"]) + refuse(["Reject, 403"]) + seed["Seed the
connection"] + + request --> held + held -->|team's user| carry + held -->|other user| refuse + held -->|none| seed + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class held decision + class refuse error +``` + + +*Figure 1. L4 Detail -- Team connection check.* Up: [L3 Components -- HyperDX API](backend.md#structure). + +A seed takes a claim first. One request per team holds the seed claim and asks +the engine, and the rest wait up to 30 seconds for it to settle. ```mermaid -flowchart TB - held{"team holds a connection?"} - held -->|as the team name| present["present: carry on once any seed in flight settles"] - held -->|as anyone else| mismatch["mismatch: 403"] - held -->|none| claim{"seed claim taken?"} - claim -->|this team failed under 30s ago| unavailable - claim -->|another request holds it| wait{"that seed settles within 30s?"} - wait -->|yes| present - wait -->|no| unavailable - claim -->|taken| ask["ask the engine for the caller's connection"] - ask -->|401 or 403| refused["refused: 403"] - ask -->|unreachable, 5xx, unreadable| unavailable["unavailable: log in, retry once the claim lapses"] - ask -->|username is not the team name| mismatch - ask -->|username is the team name| create["create the connection and seed sources"] - create -->|another replica won| present - - classDef bad fill:#D55E00,stroke:#7a3500,color:#ffffff - classDef ok fill:#009E73,stroke:#005f45,color:#ffffff - class mismatch,refused bad - class present,create,unavailable ok +flowchart LR + accTitle: L4 Detail -- Seeding a team connection + accDescr: A request takes the team's seed claim if it is free and asks the engine for the caller's connection. The team's own user creates the connection and its sources. An engine 401 or 403, or another ClickHouse user, is refused with 403. Any other answer lets the request in and retries later. A request that finds the claim held waits for that seed, then carries on if the connection exists or retries later. + + claim{"Seed claim
free?"} + ask{"Engine
answer?"} + wait["Wait up to
30 seconds"] + create(["Create connection
and sources"]) + refuse(["Reject, 403"]) + later(["Let in,
retry later"]) + carry(["Carry on"]) + + claim -->|free| ask + claim -->|held| wait + ask -->|team's user| create + ask -->|refused| refuse + ask -.->|no answer| later + wait -.->|not seeded| later + wait -->|seeded| carry + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class claim,ask decision + class refuse error ``` -- A connection is created only as the team's own name, so a platform admin + +*Figure 2. L4 Detail -- Seeding a team connection. Dotted arrows let the request in without a connection.* Up: [L3 Components -- HyperDX API](backend.md#structure). + +- A connection is created as the team's name and no other, so a platform admin arriving first on an org team cannot seed it with the platform reader. - The `dfe_one_connection_per_team` unique index on `connections.team` makes a seed racing on another replica fail with a duplicate key, which reads as - `present`. It is ensured from the dfe layer in `oidc-proxy` mode only; - `header-dev` and upstream seed several `DEFAULT_CONNECTIONS` onto one team. + `present`. The dfe layer creates it in `oidc-proxy` mode alone, and a failure + logs a warning without stopping a seed, because `header-dev` and upstream seed + several `DEFAULT_CONNECTIONS` onto one team. - A team holding no connection is seeded whatever happened before, so a failed first seed or a connection since deleted is restored. - `dfe/models/team-seed.ts` holds one claim per team, atomic across replicas. A login's parallel requests make one seeding attempt, and a failed attempt is - retried once the 30-second claim lapses. The attempt marks the claim settled + retried once the 30-second claim lapses. The attempt sets the claim settled when it ends, and every other request on the team waits for that mark, at most until the claim lapses, so none answers for a team holding its connection but not yet its sources. - The `dfe_one_source_name_per_team` index is started once Mongo connects. On DocumentDB a unique index on a non-empty collection builds in the background, and a first seed that started the build would wait on it. -- A query ClickHouse refuses with code 516 marks the team's connection for a +- A query ClickHouse rejects with code 516 flags the team's connection for a re-check. The team's next request asks the engine for its material and updates the stored host and password in place when they differ. A replica asks at most - once every 30 seconds per team, and ClickHouse's own error reaches the caller - in the meantime. + once every 30 seconds per team, and ClickHouse's error reaches the caller in + the meantime. ## The startup repair @@ -83,7 +118,7 @@ Before this rule a team was named after the first of its members' IdP groups and seeded from whichever member arrived first, so a group team could hold a ClickHouse user wider than some of its members were handed. `dfe/tasks/team-connection-repair.ts` runs once per process when Mongo connects, -in `oidc-proxy` mode only, and deletes every connection whose username is not +in `oidc-proxy` mode alone, and deletes every connection whose username is not its team's name, orphans included. - It never touches users or content. Members move onto their identity team on @@ -98,7 +133,7 @@ its team's name, orphans included. Each deletion logs the team, the username and the member count, then each emptied team's content: -```text +```console WARN DFE: deleting a team connection that is not the team identity teamId: "6abb3e94a0702ad8a4b5265c" team: "acme-team" username: "dfe_query_reader" connectionId: "6abb3e94a0702ad8a4b52670" members: 2 @@ -114,9 +149,9 @@ INFO DFE: team connection repair complete ## Evidence before the repair Run these in `mongosh` against the dfe-hyperdx database before deploying the -release that carries the repair, and keep the output. They are read-only, and -every projection is an inclusion list, so `password` and the team `apiKey` are -never returned. +release that has the repair, and keep the output. They are read-only, and every +projection is an inclusion list, so `password` and the team `apiKey` are never +returned. Connections whose username is not their team's name, the set the repair deletes: diff --git a/docs/decisions/0002-alerting-disabled-for-dfe-rules.md b/docs/decisions/0002-alerting-disabled-for-dfe-rules.md index 2ba1a491c..ace1d42a1 100644 --- a/docs/decisions/0002-alerting-disabled-for-dfe-rules.md +++ b/docs/decisions/0002-alerting-disabled-for-dfe-rules.md @@ -1,5 +1,14 @@ # HyperDX alerting disabled in favour of DFE rules and hunts +**Status:** accepted, mechanism superseded. Casbin was removed +([ADR 0004](0004-casbin-removed.md)), and the alert checker still starts with +the image. What holds the decision: in DFE mode `/alerts` and `/webhooks` refuse +every human session (`dfe/middleware/admin-lockdown.ts`), the engine's service +token reaches team, source and connection routes alone, and the console sends a +visit to `/alerts` back to `/search` (`packages/app/src/dfe/embedFeatures.ts`), +so nobody can create an alert for the checker to run. The record below is kept +as written. + HyperDX includes a built-in alerting system (the `checkAlerts` background task, Alert/AlertHistory models, webhook notifications). **We disable this entirely** and use the DFE platform's own rules engine and hunt workflows instead. diff --git a/docs/decisions/0003-additive-only-fork-strategy.md b/docs/decisions/0003-additive-only-fork-strategy.md index cd40b6a55..45f97d6ab 100644 --- a/docs/decisions/0003-additive-only-fork-strategy.md +++ b/docs/decisions/0003-additive-only-fork-strategy.md @@ -1,5 +1,11 @@ # Additive-only fork strategy +**Status:** superseded. The fork edits upstream files in place where a change +cannot go under `dfe/`, and catalogues each one in `.fork-surface` and +[../fork/what-we-changed.md](../fork/what-we-changed.md). How that is held +together is [../fork/design.md](../fork/design.md). Casbin was removed +([ADR 0004](0004-casbin-removed.md)). The assessment below is kept as written. + A realistic assessment of where the additive-only principle holds, where it gets difficult, and what the actual upstream merge cost looks like. @@ -12,7 +18,7 @@ Zero upstream application files modified. The only changes are to Docker Compose **Upstream merge cost: zero.** -### OIDC Identity Middleware: Effectively Additive +### OIDC Identity Middleware: Mostly Additive All DFE logic lives in new files under `packages/api/src/dfe/`. The only upstream file touched is `api-app.ts` with a single conditional block appended @@ -97,7 +103,7 @@ upstream conflicts are unlikely but possible. | Work Stream | Files modified in upstream | Truly additive? | Merge cost per release | | ----------------- | -------------------------- | ------------------------ | ------------------------------- | | DocumentDB | 0 | Yes | Zero | -| OIDC middleware | 1 (`api-app.ts`) | Effectively yes | Trivial (one delimited block) | +| OIDC middleware | 1 (`api-app.ts`) | Mostly | Trivial (one delimited block) | | Casbin RBAC | 0 | Yes | Zero (review new routes) | | Multi-tenancy | 0 | Yes (with review caveat) | Zero (review `getTeam()` calls) | | Alerting disabled | 0 | Yes | Zero | diff --git a/docs/decisions/0004-casbin-removed.md b/docs/decisions/0004-casbin-removed.md index 86a80e074..4bdf5057b 100644 --- a/docs/decisions/0004-casbin-removed.md +++ b/docs/decisions/0004-casbin-removed.md @@ -1,6 +1,8 @@ # 0004 - Casbin RBAC removed as redundant -**Status:** accepted +**Status:** accepted. One line below has since changed: HyperDX does not take an +identity injected at the edge. It verifies the engine's token itself, see +[../architecture/oidc-authentication.md](../architecture/oidc-authentication.md). ## Context @@ -46,11 +48,11 @@ services in different languages. **Keep Casbin as defence in depth.** Rejected: it was not defending anything the query scoping did not already cover, and a second authorization layer that is -never the one that fires is a liability rather than a safety net - it invites -the assumption that it is load-bearing. +never the one that fires is a liability rather than a safety net. It invites the +assumption that something depends on it. -**Move Casbin into the DFE engine only.** This is effectively what happened. The -engine owns policy; what was removed is the HyperDX-side enforcement copy. +**Move Casbin into the DFE engine only.** This is what happened. The engine owns +policy, and what was removed is the HyperDX-side enforcement copy. ## Notes diff --git a/docs/decisions/0005-console-typography.md b/docs/decisions/0005-console-typography.md index ce61179f8..b17a79220 100644 --- a/docs/decisions/0005-console-typography.md +++ b/docs/decisions/0005-console-typography.md @@ -27,8 +27,9 @@ so it stops being re-litigated. ## How it is wired (do not bypass) - `src/config/fonts.ts` - `MANTINE_FONT_MAP['Inter']` resolves to - `var(--font-inter), sans-serif`; the variables come from next/font in - `pages/_document.tsx` / `pages/_app.tsx`. + `var(--font-inter), sans-serif`. The variables come from `src/fonts.ts`, which + loads the copied-in font files through `next/font/local` (`src/dfe/fonts.ts`), + and `pages/_document.tsx` and `pages/_app.tsx` apply them. - `pages/_app.tsx` pins the DFE theme to Inter regardless of user font preference (upstream's font picker only applies to the hyperdx theme). - Mono surfaces use `var(--font-ibm-plex-mono)` (LogTable, HyperJson, SQL/Lucene @@ -38,9 +39,9 @@ so it stops being re-litigated. ## Consequences -- New surfaces take Inter by default (Mantine theme cascade); reach for +- New surfaces take Inter by default (Mantine theme cascade). Reach for `var(--font-ibm-plex-mono)` only when rendering query text, code, or raw data values. -- Never hardcode a family name; always the next/font variables, or the font - falls back to whatever the viewer's OS ships and the "old hyperdx fonts" - report comes back. +- Never hardcode a family name. Use the next/font variables, or the font falls + back to whatever the viewer's OS ships and the "old hyperdx fonts" report + comes back. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 89f2be6be..8df72f5cc 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -9,8 +9,8 @@ point - a rewritten ADR is worth nothing. | ADR | Decision | Status | | --- | --- | --- | -| [0002](0002-alerting-disabled-for-dfe-rules.md) | HyperDX alerting disabled in favour of DFE rules and hunts | accepted | -| [0003](0003-additive-only-fork-strategy.md) | Additive-only fork strategy | accepted | +| [0002](0002-alerting-disabled-for-dfe-rules.md) | HyperDX alerting disabled in favour of DFE rules and hunts | accepted, mechanism superseded | +| [0003](0003-additive-only-fork-strategy.md) | Additive-only fork strategy | superseded | | [0004](0004-casbin-removed.md) | Casbin RBAC removed as redundant | accepted | | [0005](0005-console-typography.md) | Console typography: Inter + IBM Plex Mono | accepted | | [0006](0006-documentdb-document-store.md) | DocumentDB as the document store | accepted | diff --git a/docs/development/docker-local.md b/docs/development/docker-local.md index 84bec5e53..f883ad0fa 100644 --- a/docs/development/docker-local.md +++ b/docs/development/docker-local.md @@ -1,107 +1,95 @@ -# DFE HyperDX - local docker (capped) + reuse for dfe-docker / k8s - -How to run the patched (upstream-2.29) HyperDX fork locally, with **hard memory -caps** so it never OOMs a dev laptop. The compose caps layer is the resource -SSoT that dfe-docker reuses and k8s mirrors. - -## TL;DR - two modes - -Both source the base compose + the DFE override + the caps layer. The only -difference is whether the app runs locally (dev, hot reload) or in a container. -Dev also layers `docker-compose.caps.dev.yml`, for the two services only the dev -compose defines. - -### Dev (fast iteration - app local, infra in docker) +# DFE HyperDX - local docker, capped + +**Status: the caps layer does not load.** `docker-compose.caps.yml` still caps +two services the move to DocumentDB removed, `postgres` among them, so compose +rejects every command below that names it +(`service "postgres" has neither an image nor a build context specified`). Until +those blocks go, run the dev command without the caps files, since DocumentDB +has no cap either way. The container command stays blocked: without the caps +layer it starts ClickHouse and the collector uncapped. + +How to run the fork locally with hard memory caps on ClickHouse and the +collectors. DocumentDB and the app stay uncapped. Both modes source the base +compose and the DFE override, and the caps layer goes last. The difference is +whether the app runs locally (dev, hot reload) or in a container. + +For dev, `yarn dev:dfe` is the flow ([getting-started.md](getting-started.md)): +it starts DocumentDB in Docker and runs the API and app locally. The DFE dev +override scales ClickHouse and both collectors to zero, so ingest and ClickHouse +come from dfe-docker or a ClickHouse you point `DEFAULT_CONNECTIONS` at. To cap +what it starts, bring the infra up yourself with the caps layers, then run the +app: ```sh -# 1. infra for the UI (ClickHouse + DocumentDB), capped. Name the 2 -# services so compose does NOT build otel-collector/prometheus (heavy build, -# not needed to iterate on look/feel; the app's self-telemetry is non-fatal): docker compose --env-file .env.dfe \ -f docker-compose.dev.yml -f docker-compose.dfe.dev.yml \ -f docker-compose.caps.yml -f docker-compose.caps.dev.yml \ - up -d ch-server documentdb + up -d documentdb -# 2. app + api locally (hot reload): -yarn app:dev:dfe +dotenvx run -f .env.dfe -f .env -- yarn app:dev:dfe ``` -(Add `otel-collector otel-collector-json prometheus` to step 1 only if you need -telemetry ingestion; they require `OTEL_COLLECTOR_VERSION` from the base -`.env`.) App: - API: . Needs -`.env.dfe` (dev creds; gitignored - copy from `.env.dfe.example`). +App: , API: . It needs `.env.dfe` +(dev credentials, gitignored, copied from `.env.dfe.example`). -### Prod image (containerised app - closest to k8s) +To run the app in a container, the closest to k8s: ```sh docker compose --env-file .env --env-file .env.dfe \ - -f docker-compose.yml -f docker-compose.dfe.yml -f docker-compose.caps.yml up -d --build + -f docker-compose.yml -f docker-compose.dfe.yml -f docker-compose.caps.yml up -d ``` -Builds the fork app image (`docker/hyperdx/Dockerfile`) and runs everything in -containers. No compose layer caps the `app` container: the dev compose does not -define it, and Compose rejects a capped service that has no image. Its -Kubernetes limit is `resources` in `dfe/deploy/contract.json` (2Gi limit, 512Mi -request). - -## Memory caps (docker-compose.caps.yml) - the resource SSoT +The `app` service runs the image `docker-compose.yml` names from `.env` +(`HDX_IMAGE_REPO`, `IMAGE_NAME_DOCKERHUB`, `IMAGE_VERSION`). No compose layer +caps it. Its Kubernetes limit is `resources` in `dfe/deploy/contract.json`. -Layer this file LAST. `docker compose up` (v2) honours -`deploy.resources.limits.memory` outside swarm. Current ceilings: +## Memory caps -| Service | limit | reservation | k8s note | -| ---------------------- | ----- | ----------- | -------------------------------- | -| ch-server (ClickHouse) | 2g | 512m | dominant; drop to 1536m if tight | -| otel-collector / -json | 300m | - | one each | -| prometheus | 300m | - | dev metrics | +Layer `docker-compose.caps.yml` last. `docker compose up` (v2) honours +`deploy.resources.limits.memory` outside swarm. -The `documentdb` service has no cap in this file. +| Service | limit | reservation | Set in | +| ---------------------- | ----- | ----------- | ----------------------------- | +| ch-server (ClickHouse) | 2g | 512m | `docker-compose.caps.yml` | +| otel-collector | 300m | - | `docker-compose.caps.yml` | +| otel-collector-json | 300m | - | `docker-compose.caps.dev.yml` | +| prometheus | 300m | - | `docker-compose.caps.dev.yml` | -`otel-collector-json` and `prometheus` are capped in -`docker-compose.caps.dev.yml`, because the prod image stack defines neither. - -Dev ceiling ~= 3.4 GB. **Never run this stack uncapped on a laptop.** - -These map 1:1 to Kubernetes: - -```text -deploy.resources.limits.memory -> containers[].resources.limits.memory -deploy.resources.reservations.memory -> containers[].resources.requests.memory -``` +The `documentdb` service has no cap. `otel-collector-json` and `prometheus` are +capped in the dev file because the prod stack defines neither. **Never run this +stack uncapped on a laptop.** -## What is DFE-specific (what dfe-docker + k8s must replicate) +The compose keys map to Kubernetes one to one: -The fork keeps upstream HyperDX code pristine; DFE behaviour is additive -(`packages/*/src/dfe/**`) and driven by env + compose overrides: +| Compose | Kubernetes | +| -------------------------------------- | ---------------------------------------- | +| `deploy.resources.limits.memory` | `containers[].resources.limits.memory` | +| `deploy.resources.reservations.memory` | `containers[].resources.requests.memory` | -- **DocumentDB as the document store.** `docker-compose.dfe*.yml` replaces `db` - (mongo) with `documentdb`; the app speaks Mongo-wire to its gateway unchanged. - k8s: a DocumentDB cluster on CloudNativePG. -- **OIDC trusted-header auth** (`DFE_AUTH_MODE=oidc-proxy`): an OIDC proxy - (Envoy in k8s / oauth2-proxy in docker) terminates auth and forwards - `x-forwarded-email` / `x-forwarded-groups`. Empty `DFE_AUTH_MODE` = upstream - local login (fine for look/feel iteration). -- **ClickHouse connection + sources** via `DEFAULT_CONNECTIONS` / - `DEFAULT_SOURCES` env (see `.env.dfe.example`). +## What is DFE-specific -## Reuse for Kaz (dfe-docker) +The fork keeps upstream's code pristine wherever it can, and DFE behaviour is +under `packages/*/src/dfe/**`, driven by env and compose overrides: -dfe-docker layers the SAME `docker-compose.caps.yml` (or copies the table above) -onto the HyperDX service. Pin the fork image by `tag@sha256` (per the DFE -SHA-pin rule). +- DocumentDB as the document store. `docker-compose.dfe*.yml` replaces `db` + (mongo) with `documentdb`, and the app speaks Mongo-wire to its gateway + unchanged. On k8s it is a DocumentDB cluster on CloudNativePG. +- Engine token auth (`DFE_AUTH_MODE=oidc-proxy`). The API verifies the engine's + token from the `dfe_token` cookie against the engine's JWKS, see + [oidc-authentication.md](../architecture/oidc-authentication.md). An empty + `DFE_AUTH_MODE` is upstream's local login, fine for look-and-feel work. +- The ClickHouse connection and sources, through `DEFAULT_CONNECTIONS` and + `DEFAULT_SOURCES` (see `.env.dfe.example`). `ghcr.io/hyperi-io/dfe-hyperdx` is the one registry path the fork publishes to. -hyperi-ci builds it from the root `Dockerfile` (`.hyperi-ci.yaml`, -`publish.container`), and the `dfe-hyperdx` chart hyperi-ci publishes from -`dfe/deploy/contract.json` deploys that same name. The -`docker.hyperdx.io/hyperdx/hyperdx` targets in the Makefile are upstream's own -release targets: we never run them, and rewriting them would buy a merge -conflict on every sync for nothing. - -## Notes - -- CH dev port is `HDX_DEV_CH_HTTP_PORT` (default 8123; no clash with the - always-on dev CH on :18123). -- Empty ClickHouse still renders the themed UI - point `DEFAULT_CONNECTIONS` at - a CH with real otel tables to see data. -- Teardown: `yarn dev:down` (or `docker compose ... down`). +hyperi-ci builds it from the root `Dockerfile` (`release.container` in +`.hyperi-ci.yaml`), and the `dfe-hyperdx` chart hyperi-ci publishes from +`dfe/deploy/contract.json` deploys that same name. dfe-docker pins it by tag and +sets its own memory limit (`compose/limits.yml` there). The image targets in the +Makefile are upstream's release targets, named from `.env`: we never run them, +and rewriting them would buy a merge conflict on every sync for nothing. + +The ClickHouse dev port is `HDX_DEV_CH_HTTP_PORT` (default 8123). An empty +ClickHouse still renders the themed UI, so point `DEFAULT_CONNECTIONS` at a +ClickHouse with OTel tables to see data. Tear down with +`docker compose --env-file .env.dfe -f docker-compose.dev.yml -f docker-compose.dfe.dev.yml down`. diff --git a/docs/development/getting-started.md b/docs/development/getting-started.md index 8d60a32d5..c0efac177 100644 --- a/docs/development/getting-started.md +++ b/docs/development/getting-started.md @@ -1,60 +1,84 @@ -# DFE Development Guide +# DFE development guide -Quick reference for developing against the DFE fork of HyperDX. - -## Prerequisites match upstream +Developing against the DFE fork of HyperDX. The fork's default branch is the DFE +branch and releases are cut from it. Upstream syncs are in +[sync-cycle.md](../fork/sync-cycle.md), and the design in +[the architecture docs](../architecture/README.md). -See [CLAUDE.md](../../CLAUDE.md) for the full development setup. The DFE +Upstream's [AGENTS.md](../../AGENTS.md) covers the development setup. The DFE differences are the compose invocation and one env file: ```bash cp .env.dfe.example .env.dfe ``` -`.env.dfe` is gitignored and required -- `yarn dev:dfe` fails without it. The -example carries working dev values; the DocumentDB credentials are the only -required keys. +`.env.dfe` is gitignored and required: `yarn dev:dfe` fails without it. The +example has working dev values, and the DocumentDB credentials are the keys it +needs. -## Local dev: `yarn dev:dfe` runs the app locally on DocumentDB +## Local dev: `yarn dev:dfe` ```bash yarn dev:dfe ``` -This starts DocumentDB in Docker (replacing upstream MongoDB) and runs the API -and Next.js app as local processes. It does NOT start ClickHouse or the otel -collectors -- the DFE override scales them to zero because ingest belongs to the -DFE stack (dfe-docker or a local ClickHouse you point `DEFAULT_CONNECTIONS` at, -in `.env.dfe`). The collectors are never built either: the override pins the -prebuilt image, so there is no from-source collector compile. +This starts DocumentDB in Docker (replacing upstream MongoDB) and runs the API, +the Next.js app and the common-utils watcher as local processes. It does NOT +start ClickHouse or the otel collectors: the DFE override scales them to zero, +because ingest belongs to the DFE stack (dfe-docker, or a local ClickHouse you +point `DEFAULT_CONNECTIONS` at in `.env.dfe`). The collectors are never built +either, because the override drops their build and names the prebuilt image. + +The local API keeps its documents in DocumentDB in Docker and reads whatever +ClickHouse `DEFAULT_CONNECTIONS` names. With `DFE_AUTH_MODE=oidc-proxy` it +trusts the engine's signing key as well. ```mermaid flowchart LR - subgraph local["local processes"] - app["Next.js app"] -->|api proxy| api["API"] + accTitle: L4 Detail -- Local dev with dev:dfe + accDescr: The Next.js app proxies API calls to the local API. The API keeps documents in DocumentDB running in Docker, fetches the engine's JWKS in oidc-proxy mode, and reads ClickHouse when DEFAULT_CONNECTIONS names one. + + subgraph local["Local processes"] + app["Next.js app"] + api["API"] end - subgraph docker["docker"] - docdb["DocumentDB"] + + subgraph docker["Docker"] + docdb[("DocumentDB")] end - api --> docdb - api -->|JWKS trust| engine["dfe-engine :8003"] - api -.->|DEFAULT_CONNECTIONS| ch[(ClickHouse)] + + engine["dfe-engine :8003"] + ch[("ClickHouse")] + + app -->|api proxy| api + api -->|documents| docdb + api -.->|JWKS| engine + api -.->|queries| ch + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class docdb,ch store + class engine ext + class local,docker group ``` + +*Figure 1. L4 Detail -- Local dev with dev:dfe. Dotted arrows depend on settings in .env.dfe.* Up: [Development](README.md). + Ports are FIXED under `dev:dfe`: API 8000, app 8080, DocumentDB 27017 (published -on all interfaces), from `.env` defaults -- the per-worktree port slots and the +on all interfaces), from `.env` defaults. The per-worktree port slots and the :9900 dev portal belong to upstream `yarn dev`, not this flow, so two `dev:dfe` checkouts collide. To retarget by hand: the API and the app BOTH read `PORT`, so run them as separate processes with distinct `PORT` values and set `HYPERDX_API_PORT` so the app's `/api` proxy finds the API. -## Wiring against dfe-engine - `DFE_AUTH_MODE` in `.env.dfe` is the master switch: empty disables the DFE identity layer entirely, `oidc-proxy` verifies engine-signed JWTs, and `header-dev` trusts dev headers. With it enabled, the fork trusts the engine's ES384 signing key over JWKS and the `/dfe/*` routes forward the caller's -`dfe_token` to the engine -- both settings also live in `.env.dfe`: +`dfe_token` to the engine. Both settings live in `.env.dfe`: ```sh DFE_ENGINE_JWKS_URL=http://localhost:8003/.well-known/jwks.json @@ -69,26 +93,21 @@ curl -s http://localhost:8003/.well-known/jwks.json # -> {"keys":[{"kty":"EC","crv":"P-384",...,"alg":"ES384",...}]} ``` -The engine's own local boot is documented in -[dfe-engine docs/LOCAL-DEV.md](https://github.com/hyperi-io/dfe-engine/blob/main/docs/LOCAL-DEV.md). +The engine's local boot is documented in +[dfe-engine docs/local-dev.md](https://github.com/hyperi-io/dfe-engine/blob/main/docs/local-dev.md). -## Production (all services in Docker) +## Production and config ```bash -docker compose -f docker-compose.yml -f docker-compose.dfe.yml up -d +docker compose --env-file .env --env-file .env.dfe \ + -f docker-compose.yml -f docker-compose.dfe.yml up -d ``` -The `app` service `MONGO_URI` is automatically overridden in -`docker-compose.dfe.yml`. - -## All DFE work lands on main - -The fork's default branch is the DFE branch and releases are cut from it. -Upstream syncs are covered in [docs/fork/sync-cycle.md](../fork/sync-cycle.md); -the full design (DocumentDB, OIDC, additive-only strategy) is in -[the architecture docs](../architecture/README.md). +`docker-compose.dfe.yml` overrides the `app` service's `MONGO_URI`. Name both +env files: `.env.dfe` holds the DocumentDB login, and without it compose starts +DocumentDB with a blank user and password. -## DocumentDB serves the MongoDB wire protocol +DocumentDB serves the MongoDB wire protocol: - The MongoDB DATA LAYER is unchanged: Mongoose, connect-mongo and passport-local-mongoose all work through DocumentDB (the fork's additions live @@ -101,7 +120,7 @@ the full design (DocumentDB, OIDC, additive-only strategy) is in `docker-compose.dfe.dev.yml` - Images are pinned in `docker-compose.dfe.dev.yml` -## Where config lives +Where config lives: | File | Purpose | | ------------------------------- | -------------------------------------------- | diff --git a/docs/fork/README.md b/docs/fork/README.md index 885e3b4bf..1f14b5118 100644 --- a/docs/fork/README.md +++ b/docs/fork/README.md @@ -6,19 +6,16 @@ repeatedly, forever, and everything in these docs exists to keep that cheap. **The governing rule: minimise what we change in anything that comes from upstream.** Every edit to a pristine upstream file becomes permanent -merge-conflict surface. Getting it wrong does not fail loudly - it silently +merge-conflict surface. Getting it wrong does not fail loudly. It makes every future sync more expensive, and the bill arrives months later for whoever runs one. ---- - -## Start here - | You want to | Read | | --- | --- | | Know what we changed and why | [what-we-changed.md](what-we-changed.md) | | Understand how the fork is held together | [design.md](design.md) | | Run an upstream sync | [sync-cycle.md](sync-cycle.md) | +| See what each sync cost | [sync-track-record.md](sync-track-record.md) | | Know how this ends | [leaving-upstream.md](leaving-upstream.md) | | Make a change without adding conflict surface | [working-guide.md](working-guide.md) | @@ -26,64 +23,72 @@ whoever runs one. ## What we own, in one diagram +Every upstream merge takes four kinds of content, and the catalogued in-place +edits are the only kind that can conflict. + ```mermaid -flowchart TB +flowchart LR + accTitle: L4 Detail -- What an upstream merge takes + accDescr: Our dfe directories never conflict, catalogued in-place edits conflict and rerere replays them, the security layer is stripped before the merge and rebuilt after, and the rest of the tree is taken whole from upstream. The build and tests after the merge are the proof. + subgraph ours["What we own"] - dfedir["packages/**/dfe/**
additive, cannot conflict"] - inplace["catalogued in-place edits
.fork-surface"] + dfedir["packages/**/dfe/**"] + inplace["In-place edits
.fork-surface"] temp["security/overrides.yaml
security/patches/"] end - subgraph theirs["What upstream owns"] - rest["everything else
~99% of the tree"] - end - - dfedir -->|no conflict, ever| merge{{"upstream merge"}} - inplace -->|conflicts, rerere replays| merge - temp -->|stripped first, rebuilt after| merge - rest -->|taken wholesale| merge - merge --> gate["build + test
the only real proof"] - - classDef safe fill:#009E73,stroke:#005f45,color:#ffffff - classDef risk fill:#E69F00,stroke:#8a6100,color:#000000 - classDef up fill:#0072B2,stroke:#00456b,color:#ffffff - classDef gatecls fill:#CC79A7,stroke:#7d4567,color:#ffffff - class dfedir,temp safe - class inplace risk - class rest,merge up - class gate gatecls + rest["Upstream's tree"] + merge["Upstream
merge"] + gate(["Build and test"]) + + dfedir -->|never conflicts| merge + inplace -->|rerere replays| merge + temp -->|stripped first| merge + rest -->|taken whole| merge + merge --> gate + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class inplace error + class rest ext + class ours group ``` -Only the orange box costs us anything on a sync. Keeping it small is the whole -game, and [design.md](design.md) is how. - ---- - -## The five rules - -1. **New code goes under a `dfe/` directory.** Those paths do not exist - upstream, so they carry zero conflict risk. This is the default. -2. **Editing an upstream file in place is an EXCEPTION** - it must be - identified, documented in [what-we-changed.md](what-we-changed.md), and - listed in `.fork-surface` so the guard passes. -3. **Never bulk-reformat an upstream file.** A repo-wide formatter rewrites - lines we do not own, which destroys rerere's ability to replay and makes - every later sync a fresh hand-resolve. -4. **Never merge upstream onto `main` by hand.** The sync workflow does it on a - bot branch and opens a PR. -5. **All our work happens in the fork.** Features, fixes and security fixes land here, and raising them upstream is not part of the process. When upstream fixes something we carry, `--check`, `--apply` and `--drift` say so and we delete our copy. The long-term plan is in [leaving-upstream.md](leaving-upstream.md). - -Our files follow HyperI documentation standards. Upstream's files are left -exactly as they are, including their docs - `agent_docs/`, `AGENTS.md` and the -rest are theirs, and adding our conventions to them would be conflict surface -bought for nothing. +*Figure 1. L4 Detail -- What an upstream merge takes. The orange box is the one that costs us on a sync.* Up: [dfe-hyperdx documentation](../README.md). + +Keeping that box small is the whole game, and [design.md](design.md) is how. + +The five rules that follow from it: + +1. New code goes under a `dfe/` directory. Those paths do not exist upstream, + so they have zero conflict risk. This is the default. +2. Editing an upstream file in place is an EXCEPTION. It must be identified, + documented in [what-we-changed.md](what-we-changed.md), and listed in + `.fork-surface` so the guard passes. +3. Never bulk-reformat an upstream file. A repo-wide formatter rewrites lines + we do not own, which destroys rerere's ability to replay and makes every + later sync a fresh hand-resolve. +4. Never merge upstream onto `main` by hand. The sync workflow does it on a bot + branch and opens a PR. +5. All our work happens in the fork. Features, fixes and security fixes land + here, and raising them upstream is not part of the process. When + upstream fixes something we hold, `--check`, `--apply` and `--drift` say so + and we delete our copy. The long-term plan is in + [leaving-upstream.md](leaving-upstream.md). + +Our files follow HyperI documentation standards. Upstream's files, docs +included, are left as upstream ships them unless `.fork-surface` lists a delta: +`agent_docs/`, `AGENTS.md` and the rest are theirs, and adding our conventions +to them would be conflict surface bought for nothing. --- ## The tooling -Run once per clone - rerere, the conflict-surface hook and the lockfile merge -driver are all per-clone git config, and a fresh clone starts unprotected: +Run `fork-setup.sh` once per clone. Rerere, the conflict-surface hook and the +lockfile merge driver are all per-clone git config, and a fresh clone starts unprotected: ```bash ./scripts/fork-setup.sh diff --git a/docs/fork/design.md b/docs/fork/design.md index 07e5efaba..7e94a1d94 100644 --- a/docs/fork/design.md +++ b/docs/fork/design.md @@ -1,69 +1,75 @@ # How the fork is held together -Four mechanisms, each answering a different failure. Read the ladder first - it +Four mechanisms, each answering a different failure. Read the ladder first: it is the one that changes what you type today. ---- - -## Three questions, before every change - -Before the ladder, before anything: +Three questions come before every change, before the ladder, before anything: -1. **Do I really need it?** -2. **Can it be smaller and simpler?** -3. **Can it be done with less upstream drift risk?** +1. Do we need it? +2. Can it be smaller and simpler? +3. Can it be done with less upstream drift risk? The governing goal of this fork is to MINIMISE the file-level variation from -upstream we have to maintain, while still carrying our changes, our -integrations, and the occasional security fix. Every question above serves that -one sentence. +upstream we have to maintain, while still keeping our changes, our integrations, +and the occasional security fix. Every question above serves that one sentence. It is pointed at models in particular, because they reliably fail it: given a small brief they produce a large diff, since a large diff reads as effort. Here -it is not effort. A change touching no upstream file is free; a change touching +it is not effort. A change touching no upstream file is free. A change touching an upstream file is a bill paid at every future sync, by whoever runs it. -Applies to removal too. When a sync leaves one of our deltas identical to +It applies to removal as well. When a sync leaves one of our deltas identical to upstream again, take it OFF `.fork-surface` - the file is pristine and the entry -now costs a check for nothing. +costs a check for nothing. --- ## The cheapest-edit ladder -**Before changing an upstream file, stop at the first option that fits.** - This is the single highest-leverage habit in the repo, and the one every model gets wrong by default: the instinct is to open the upstream file and rewrite the function body. That works, passes tests, and costs a hand-resolve on every sync forever. +Each rung costs more on every sync than the one before it, so take the first +that fits. + ```mermaid -flowchart TB - start(["a change is needed"]) --> q1{"can it live
under dfe/ ?"} - q1 -->|yes| a1["put it there
COSTS NOTHING"] - q1 -->|no| q2{"can the upstream file
change by ONE TOKEN?"} - q2 -->|yes| a2["logic in dfe/, same signature,
swap the identifier at the call site"] - q2 -->|no| q3{"can it be an ADDITION
next to a brace?"} - q3 -->|yes| a3["append an else-if
merges cleanly far more often"] - q3 -->|no| a4["edit in place,
and CATALOGUE it"] - - classDef free fill:#009E73,stroke:#005f45,color:#ffffff - classDef cheap fill:#F0E442,stroke:#8a8200,color:#000000 - classDef costly fill:#D55E00,stroke:#7a3500,color:#ffffff - classDef ask fill:#0072B2,stroke:#00456b,color:#ffffff - class a1 free - class a2,a3 cheap - class a4 costly - class q1,q2,q3 ask +flowchart LR + accTitle: L4 Detail -- The cheapest-edit ladder + accDescr: If the change can live under dfe it goes there and costs nothing. If not and the upstream file can change by one token, the logic goes under dfe and the call site swaps one identifier. If not and it can be an addition next to a brace, append an else-if. Otherwise edit in place and catalogue it. + + q1{"Can it live
under dfe/?"} + q2{"Can the upstream file
change by ONE TOKEN?"} + a1(["Put it there,
costs nothing"]) + q3{"Can it be an
ADDITION?"} + a2(["Logic in dfe/,
swap the identifier"]) + a3(["Append
an else-if"]) + a4(["Edit in place
and CATALOGUE"]) + + q1 -->|no| q2 + q1 -->|yes| a1 + q2 -->|no| q3 + q2 -->|yes| a2 + q3 -->|yes| a3 + q3 -->|no| a4 + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class q1,q2,q3 decision + class a4 error ``` + +*Figure 1. L4 Detail -- The cheapest-edit ladder. Orange is the costliest rung.* Up: [The fork](README.md). + Rule 2 is the one worth internalising. Give your function the SAME SIGNATURE as upstream's and swap the identifier at the call site: `foo(a, b, c)` becomes `dfeFoo(a, b, c)`. The argument list stays byte-identical, so if upstream later changes those arguments the conflict is trivial instead of structural. -### It is worth real money +### It pays on every sync A native-ClickHouse-JSON fix was first written the obvious way: rewrite `buildJSONExtractQuery`'s body in `DBRowJsonViewer.tsx` and edit upstream's test @@ -71,9 +77,9 @@ expectations. That is 53 insertions across the two files upstream churns hardest in that area. Redone as rule 2 (four call sites differing by one identifier) plus rule 3 -(three `else if (isJsonColumn)` additions reusing upstream's own predicate): -**15 insertions**, `buildJSONExtractQuery` left byte-pristine, and the test file -back to pristine and off the catalogue entirely. +(three `else if (isJsonColumn)` additions reusing upstream's predicate): **15 +insertions**, `buildJSONExtractQuery` left byte-pristine, and the test file back +to pristine and off the catalogue entirely. Same feature. A quarter of the standing cost, forever. @@ -99,7 +105,7 @@ CANNOT do, and asking it to is the trap: preimage never matches twice and rerere accumulates one-shot resolutions that never replay. - `yarn.lock` is a megabyte of generated text. "Replay these hunks" is not a - meaningful answer for a lockfile; "take upstream's and regenerate" is the only + meaningful answer for a lockfile. "Take upstream's and regenerate" is the correct one, every time. So the temporary layer is GENERATED, never merged - stripped before an upstream @@ -109,47 +115,48 @@ again" a command rather than an afternoon. The procedure is --- -## The security posture is INVERTED here +## Security scanning is INVERTED here The house standard is scanners-on, dependencies-current. In this repo that is backwards, deliberately: **off by default, on only for what WE added.** -- `renovate.json` disables every manager and re-enables exactly one thing: the - SHA-pinned actions in the workflows we own. Upstream's workflows are excluded - BY NAME so a sync that adds one does not silently opt it in. +- `renovate.json` extracts dependencies from the files it names in + `includePaths` and nothing else: the SHA-pinned actions in our six workflows, + our three compose files and the root `Dockerfile`. A workflow or Dockerfile a + sync adds is outside that list, so it is never opted in. - CodeQL default setup is ON. Dependabot's automatic fix PRs are OFF. - Dependabot vulnerability ALERTS stay ON - they are the input to `scripts/security-triage.py`. - gitleaks stays ON and blocking. It scans OUR commits for OUR secrets and has nothing to do with upstream's dependency tree. -**Why.** We do not own upstream's dependency tree. Patching a vendored dep -diverges us for a fix we did not write, converts a pristine file into permanent -rerere conflict surface, and gets re-conflicted on the next sync regardless. -Upstream patches upstream; we get it when we sync. And an alert queue full of -items nobody can action trains everyone to ignore the queue, including the one -that matters. +**Why.** We do not own upstream's dependency tree. Patching a dependency of +theirs diverges us for a fix we did not write, converts a pristine file into +permanent rerere conflict surface, and gets re-conflicted on the next sync +regardless. Upstream patches upstream, and we get it when we sync. And an alert +queue full of items nobody can action trains everyone to ignore the queue, +including the one that matters. **What this does NOT mean.** Muting per-PR noise does not make an inherited CVE unreal - the fork ships as a container image and those CVEs ship with it. The image scan at publish still applies, and a critical finding there is an argument for **syncing upstream now**, not for hand-patching a dependency we do not own. -### The exception: a reachable HIGH or CRITICAL +## The exception: a reachable HIGH or CRITICAL -**Ask what the pending sync already clears, before triaging anything.** Alerts -are raised against the default branch, so an unmerged sync branch is invisible -to them. In August 2026, 51 of 136 open alerts were already answered by a sync -sitting in a PR -- including every `next` advisory, because 2.33.0 moved it from -16.1.7 to 16.2.11. Triaging that list first would have been most of a day spent -on items the merge closes, and any pin written for one is surface held forever -for nothing. +**Ask what the pending sync clears, before triaging anything.** Alerts are +raised against the default branch, so an unmerged sync branch is invisible to +them. In August 2026, 51 of 136 open alerts were answered by a sync sitting in a +PR -- including every `next` advisory, because 2.33.0 moved it from 16.1.7 to +16.2.11. Triaging that list first would have been most of a day spent on items +the merge closes, and any pin written for one is surface held forever for +nothing. -We do patch, but only when BOTH hold: +We do patch, but when BOTH hold and not otherwise: 1. severity is **high or critical**, AND -2. there is a **real vector** - the vulnerable path is actually reachable in how - DFE runs this fork. +2. there is a **real vector** - the vulnerable path is reachable in how DFE runs + this fork. Point 2 is the one that gets skipped, and it is the one that matters. Most advisories against a transitive dependency are unreachable here: the package is @@ -157,7 +164,7 @@ present but the vulnerable function is never imported, or it is only reachable from a path we do not ship. **"npm audit says high" is not a vector.** If you cannot write down how an attacker gets there, there is nothing to patch. -### Standing verdicts +## Standing verdicts Triaged 2026-08-05 against 136 open alerts. Kept by PACKAGE rather than by alert number, because the numbers churn and the reasoning does not. Re-check a row @@ -168,9 +175,9 @@ only when what pulls the package changes. | `tar` | `node-gyp`, `cacache` | Install-time toolchain. Not in the shipped image, no request path. | | `protobufjs` 6.x | `@hyperdx/otel-web-session-recorder` (`~6.11.2`) | Browser recorder, upstream's own package. A force to >=7.5.5 is a major bump across a pinned range. Upstream's to fix. | | `systeminformation` | `@opentelemetry/host-metrics` | Command injection needs the attacker to already control host network config. Root range `^5.24.0` admits the fix, so any refresh clears it. | -| `@opentelemetry/*` | upstream's telemetry stack | 0.57 -> 0.217 is the otel-js renumbering. Forcing it breaks the stack; upstream carries it. | +| `@opentelemetry/*` | upstream's telemetry stack | 0.57 -> 0.217 is the otel-js renumbering. Forcing it breaks the stack, and upstream maintains it. | | `ws` | `ink`, `jsdom`, `storybook` | Dev, test and CLI. Not the server's socket path. | -| `lodash` 4.17.x | `@stoplight/*`, `html-webpack-plugin` | Lint and build only. The workspaces themselves are already on 4.18.x. | +| `lodash` 4.17.x | `@stoplight/*`, `html-webpack-plugin` | Lint and build only. The workspaces themselves are on 4.18.x. | | `sharp` | `next` (`^0.34.5`) | Next's image optimiser, pinned by Next. Moves when Next moves. | | `rollup`, `postcss`, `serialize-javascript`, `semver`, `minimatch`, `brace-expansion` | build toolchain | Build-time, processing our own sources. Nothing attacker-supplied reaches them. | | `path-to-regexp` | `express` (`~0.1.12`) | The ReDoS needs attacker-controlled ROUTE DEFINITIONS. Ours are static. | @@ -180,16 +187,15 @@ only when what pulls the package changes. **`ip-address` is the one worth reading.** It looks like the strongest candidate on paper: `packages/api/src/utils/validators.ts` uses it to build `isPrivateIp()`, the SSRF guard on user-configured webhook URLs, and the -advisory is _precisely_ an SSRF bypass -- leading-zero octets decoded as decimal -while the resolver decodes them as octal. Advisory plus call site reads as an +advisory is exactly an SSRF bypass -- leading-zero octets decoded as decimal and +the resolver decodes them as octal. Advisory plus call site reads as an open-and-shut hit. It is not reachable, and only a probe shows why. Both call sites pass `new URL(host).hostname`, and the WHATWG URL parser normalises IPv4 hosts before the guard sees them: `0177.0.0.1`, `0x7f.0.0.1` and `2130706433` all arrive as -`127.0.0.1` and are blocked, which is also what the socket layer does on -connect. The vulnerable decode never runs on a string an attacker chose the -encoding of. +`127.0.0.1` and are blocked, which is what the socket layer does on connect. The +vulnerable decode never runs on a string an attacker chose the encoding of. Two things follow. Reading the advisory against the call site would have produced a pin that bought nothing and cost a check every sync -- **run the path @@ -198,24 +204,23 @@ instead of a parsed hostname, this row flips: `isPrivateIp` returns `false` for anything `ip-address` cannot parse, so the fall-through is the thing to watch, not the octet decoding. -### Code scanning: keep it all, sort it by who wrote the line +## Code scanning: keep it all, sort it by who wrote the line The muting above is the DEPENDENCY axis, where authorship is not a useful question - the whole tree is upstream's regardless of who pulled a package in. -Code findings are different: they carry a file and a line, so they can be +Code findings are different: they have a file and a line, so they can be attributed. `fork-security.yml` scans the whole tree and sorts the results: -- **OURS** - a line we added or changed versus the merge base. **Gates the - build.** -- **ACCEPTED** - ours, reviewed, listed in `security/accepted.yaml` with a - reason. Suppressed from the gate but PRINTED every run, because a suppression - that has become wrong should be visible. -- **INHERITED** - upstream's lines. Reported, never gates. +| Bucket | Which lines | Effect | +| --------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | +| OURS | a line we added or changed versus the merge base | gates the build | +| ACCEPTED | ours, reviewed, listed in `security/accepted.yaml` with a reason | kept off the gate but PRINTED every run, so a suppression that has become wrong is visible | +| INHERITED | upstream's lines | reported, never gates | -Measured on a real run: 108 findings, 0 ours, 5 accepted, 103 inherited. Gating -on 108 is unworkable; muting 108 loses the ones that matter. Sorting is the only -answer that keeps both. +On one run: 108 findings, 0 ours, 5 accepted, 103 inherited. Gating on 108 is +unworkable, and muting 108 loses the ones that matter. Sorting is the answer +that keeps both. Attribution is `scripts/attribute-findings.py`, SARIF in, so it serves semgrep and CodeQL alike. It compares **line content against the merge base**: a line we @@ -238,10 +243,10 @@ reconnects them: git replace --graft e58f01d3f4ede7b691ee4cf2873ad8548d93f210 fbeaf152... ``` -**Non-destructive.** No object is rewritten and no SHA changes; `git replace -d` -undoes it. The merge base is unchanged, so the sync workflow, the drift report -and the attribution script all behave identically - blame and `git log` simply -stop lying. +**Non-destructive.** No object is rewritten and no SHA changes, and +`git replace -d` undoes it. The merge base is unchanged, so the sync workflow, +the drift report and the attribution script all behave identically. Blame and +`git log` stop lying. It is per-clone config, applied by `fork-setup.sh` from the two recorded SHAs, rather than a `refs/replace/*` ref that no clone fetches by default. diff --git a/docs/fork/leaving-upstream.md b/docs/fork/leaving-upstream.md index c3c96f2af..4bfd2f6aa 100644 --- a/docs/fork/leaving-upstream.md +++ b/docs/fork/leaving-upstream.md @@ -2,33 +2,43 @@ **Forking is a position, not a destination.** The intent is to stop tracking hyperdxio/hyperdx once HyperDX's rate of change slows and stabilises, most -likely by merging the code into DFE and vendoring it there (C below). Until then -all our work stays in the fork. This doc exists so that decision gets made +likely by merging the code into DFE and keeping a copy there (C below). Until +then all our work stays in the fork. This doc exists so that decision gets made deliberately rather than drifted into. --- ## Three ways out +From where we stand there are three ways out, and only a de-fork ends with less +to carry. + ```mermaid flowchart LR - now(["forked, syncing
periodically"]) - now --> a["A. STAY FORKED
keep syncing"] - now --> b["B. DE-FORK
upstream absorbs our delta,
we become a consumer"] - now --> c["C. HARD FORK
stop merging, vendor into DFE"] - - a -->|delta stays small,
upstream stays fast| a - b -->|best outcome| done(["plain dependency"]) - c -->|only when the merge
stops being worth it| own(["we own an
observability platform"]) - - classDef good fill:#009E73,stroke:#005f45,color:#ffffff - classDef ok fill:#F0E442,stroke:#8a8200,color:#000000 - classDef costly fill:#D55E00,stroke:#7a3500,color:#ffffff - class b,done good - class a,now ok - class c,own costly + accTitle: L4 Detail -- Three ways out of the fork + accDescr: From forked and syncing, A stays forked and keeps syncing, B lets upstream absorb our delta and ends as a plain dependency, and C stops merging and copies the code into DFE, ending with us owning an observability platform. + + now(["Forked,
syncing"]) + a["A. Stay forked,
keep syncing"] + b["B. De-fork,
upstream absorbs
our delta"] + c["C. Hard fork,
stop merging"] + done(["Plain
dependency"]) + own(["We own an
observability
platform"]) + + now --> a + now --> b + now --> c + b -->|best outcome| done + c -->|expected exit| own + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + class c,own error ``` + +*Figure 1. L4 Detail -- Three ways out of the fork. Orange is the expensive way out.* Up: [The fork](README.md). + **A. Stay forked (today's position).** Cheapest while upstream moves fast and our delta is small. The cost is a recurring sync, and the whole apparatus in [design.md](design.md) exists to keep that cost roughly flat as upstream @@ -37,15 +47,15 @@ accelerates. **B. Upstream absorbs our delta, and we de-fork to a plain dependency.** The best outcome, and not hypothetical. In 2.33.0 upstream added a "kiosk mode" that hides the nav on dashboards - our embed feature arriving under a different name. -`packages/app/src/layout.tsx` now carries both conditions. Every time upstream -lands something we already hold, our delta shrinks and the case for B -strengthens. We do not push our delta upstream to get there, so B only happens -when upstream lands what we hold on its own. +`packages/app/src/layout.tsx` has both conditions. Every time upstream lands +something we hold, our delta shrinks and the case for B strengthens. We do not +push our delta upstream to get there, so B only happens when upstream lands what +we hold on its own. -**C. Hard fork - stop merging, vendor the code into DFE.** The expected way out, +**C. Hard fork - stop merging, copy the code into DFE.** The expected way out, and the expensive one. Right once upstream's rate of change has slowed enough -that its releases stop being worth the merge, and we have the people to carry a -full observability platform. Taken as a decision, never slid into by neglect. +that its releases stop being worth the merge, and we have the people to maintain +a full observability platform. Taken as a decision, never slid into by neglect. --- @@ -53,8 +63,8 @@ full observability platform. Taken as a decision, never slid into by neglect. | Signal | Where to get it | Argues for | | --------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------ | -| Delta size and files touched | `.githooks/fork-surface-check.py --audit`, [what-we-changed.md](what-we-changed.md) | shrinking -> B; growing into upstream's hot files -> C | -| Sync cost over time | the measurement table in [sync-cycle.md](sync-cycle.md) | rising despite the ladder -> C | +| Delta size and files touched | `.githooks/fork-surface-check.py --audit`, [what-we-changed.md](what-we-changed.md) | shrinking -> B, growing into upstream's hot files -> C | +| Sync cost over time | the table in [sync-track-record.md](sync-track-record.md) | rising despite the ladder -> C | | Upstream velocity | `--drift` and the release cadence | slowing -> C cheaper, B less urgent | | What we would lose | the package table in [../architecture/README.md](../architecture/README.md) | more we do not maintain -> A or B | | Whether the delta is ours to keep | the catalogue | branding and config evaporate under B | @@ -65,17 +75,17 @@ full observability platform. Taken as a decision, never slid into by neglect. Not a rename. It means: -- **Taking ownership of the dependency tree.** Upstream currently patches it for - us - fourteen pins in their `resolutions` block that we inherit for free. -- **Maintaining what we do not touch today** - the OTel collector build, the MCP +- Taking ownership of the dependency tree. Upstream patches it for us, through + the pins in its `resolutions` block that we inherit for free. +- Maintaining what we do not touch today: the OTel collector build, the MCP server, the CLI, the eval harness. -- **Losing the delta gate's meaning.** Its whole job is proving an upstream - change did not break us. With no upstream, it is just a test suite. +- Losing the delta gate's meaning. Its whole job is proving an upstream change + did not break us. With no upstream, it is a test suite. The security layer is the honest measure of that first cost. Every pin and patch -it carries is work upstream is currently doing that we would inherit -permanently. An empty `security/overrides.yaml` is the goal state precisely -because it means upstream is still carrying that load. +it holds is work upstream does that we would inherit for good. An empty +`security/overrides.yaml` is the goal state precisely because it means upstream +is still carrying that load. > **This is a human decision, taken once, with the numbers in front of us.** It > is not an agent's call, and not something to conclude in the middle of a sync. diff --git a/docs/fork/security-sync.md b/docs/fork/security-sync.md index 01c375f05..9f646c908 100644 --- a/docs/fork/security-sync.md +++ b/docs/fork/security-sync.md @@ -160,7 +160,7 @@ returns an `apiKey`. That was not needed to clear it. **#199 and #150, both high, command injection in `networkInterfaces()`.** -This one is genuinely reached. `packages/api/src/index.ts` instantiates +This one is reached. `packages/api/src/index.ts` instantiates `@opentelemetry/host-metrics`, which requires `systeminformation/lib/network` (`si.js:22`) and calls `networkStats()` on the metrics timer. `networkStats()` calls `networkInterfaces()` at `network.js:1302` and `:1518`. @@ -182,7 +182,7 @@ PIN just produced, so it cannot tell an upstream fix from ours. `--verify` is the one CI gates on and it is correct. Do not act on `--check` calling a fresh pin redundant. -### ip-address - the only genuinely remote one, already closed +### ip-address - the one remote advisory, closed **#248, high, SSRF via leading-zero octets read as decimal.** @@ -206,7 +206,7 @@ came back with 15 packages. That was wrong: a scoped package sits at and never matches. Depth 2 is the real number - **21**, not 15. Six more turned out to be in the image: `@hono/node-server` and five -`@opentelemetry/*`. Only `@babel/runtime` was genuinely absent. +`@opentelemetry/*`. Only `@babel/runtime` was absent. ### The OpenTelemetry six @@ -297,7 +297,7 @@ Measured 2026-08-26: 99 Dependabot alerts, but only **26** audit findings at moderate-and-above. Fewer because yarn dedupes by package, and different because it walks the whole workspace including devDependencies - so it finds nested copies the hoisted `node_modules` view hides. `lodash` reads 4.18.1 at the top -level and the audit finds a 4.17.21 copy; `semver` reads 6.3.1 and the audit +level and the audit finds a 4.17.21 copy. `semver` reads 6.3.1 and the audit finds 5.7.1. **How to silence one, properly.** Yarn Berry's own settings, confirmed present @@ -336,12 +336,12 @@ written reason per entry. Follow that format. ### Tooling warts found on the way -- **`hyperi-ci deps drift` never read `yarn.lock`.** It looked for +- `hyperi-ci deps drift` never read `yarn.lock`. It looked for `package-lock.json` only, so it compared nothing and reported clean - on this repo and on dfe-ui. Fixed in hyperi-ci 2.9.23. First honest run here: 12 floors a whole major behind the lock, including `zod` declared `3.25` against `4.4.3` locked, in four packages. -- **husky is not installed on a fresh clone**, so `lint-staged` never runs and +- husky is not installed on a fresh clone, so `lint-staged` never runs and formatting errors reach the commit. Two of ours got through that way this pass. @@ -363,17 +363,17 @@ written reason per entry. Follow that format. upstream's. Until then the hook stays off and lint-staged has to be run by hand. -- **There are TWO ratchets, and the second one catches you fixing the first - wrongly.** `--max-warnings` per package is one. `scripts/ci/ratchet.mjs` is - the other, and it counts `as any` and `eslint-disable` occurrences against a +- There are TWO ratchets, and the second one catches you fixing the first + wrongly. `--max-warnings` per package is one. `scripts/ci/ratchet.mjs` is the + other, and it counts `as any` and `eslint-disable` occurrences against a committed baseline. So silencing an eslint warning with a disable comment passes the first and fails the second, by design. Good design - it is the guard against exactly that reflex. The right move is a real fix. `sql[i]` in a loop trips - `security/detect-object-injection` on every read; `sql.charAt(i)` does not, - and is the same operation. `as jest.Mock` trips `no-unsafe-type-assertion`; - `jest.mocked()` is the typed helper and needs no cast. Eleven + `security/detect-object-injection` on every read, and `sql.charAt(i)` does not + and is the same operation. `as jest.Mock` trips `no-unsafe-type-assertion`, + and `jest.mocked()` is the typed helper and needs no cast. Eleven `(dfeConfig as any).X =` assignments collapse to one `dfeConfig as Record` alias. Twenty-one casts removed that way this pass, none of them muted. @@ -419,7 +419,7 @@ alerts arrive regardless. ### What is done as of this pass -- 99 alerts -> 4. The 95 dismissed each carry their reason on the alert; +- 99 alerts -> 4. The 95 dismissed each carry their reason on the alert, and `scripts/dismiss-triaged-alerts.py` holds the verdicts as data and re-runs after a sync. - The 4 left open are systeminformation, the one real finding. They close when @@ -453,7 +453,7 @@ type-checks the mock properly, which is the point, and that is exactly why it fails: `createConnection` resolves a full mongoose `Document` - 50-plus methods - so `mockResolvedValue({ _id: someObjectId })` is rejected. Minting real - ObjectIds does not help; the Document type is the blocker, not the id type. + ObjectIds does not help. The Document type is the blocker, not the id type. 17 tsc errors, and no honest way through short of building real Documents in every fixture. The loose `x as jest.Mock` cast exists for that reason and the @@ -554,7 +554,7 @@ in the reusable workflow purely to dodge anonymous pull limits. Upstream's Five open PRs went red on `ci / Quality` overnight with the same two lines, while `ci / Test` stayed green: -```text +```console audit: failed YN0001: RequestError: Timeout awaiting 'socket' for 60000ms ``` @@ -625,10 +625,9 @@ done the `security/overrides.yaml` bar, so nothing is pinned and the register is unchanged. -The interesting part is not the tally. Two entries in -`scripts/dismiss-triaged-alerts.py` were written against advisories that have -since been superseded, and both would have re-dismissed a new alert with a -reason that is no longer true. +Two entries in `scripts/dismiss-triaged-alerts.py` were written against +advisories that have since been superseded, and both would have re-dismissed a +new alert with a reason that is no longer true. ### fast-uri - the range moved up past us, not the other way round @@ -644,11 +643,9 @@ wrote the mirror note, which is that a new advisory can raise the ceiling over a version that never moved. A version-comparison verdict has a shelf life and the advisory, not the lockfile, decides when it expires. -Still not reachable, for a different reason than before: - -```text -@hyperdx/api -> @modelcontextprotocol/sdk@1.29.0 -> ajv@8.20.0 -> fast-uri@3.1.4 -``` +Still not reachable, for a different reason than before. The one path is +`@hyperdx/api`, which pulls `@modelcontextprotocol/sdk@1.29.0`, which pulls +`ajv@8.20.0`, which pulls `fast-uri@3.1.4`. ajv is the only parent. It uses fast-uri to resolve `$id` and `$ref` and to validate `format: uri`, against schemas that are our own source, and it issues @@ -657,15 +654,12 @@ need a requester downstream of the parse. There is none. The fork also puts `requireServicePrincipal` in front of `/mcp` (`api-app.ts:137`), so the transport that drags ajv in is not reachable by a human user at all. -### qs - the old reason was simply wrong +### qs - the old reason was wrong The old entry read "No call site passes request-derived input to the sink." -Express's own query parser is a call site, and `req.query` is request-derived: - -```text -@hyperdx/api -> express@4.22.1 -> qs@6.14.2 -@hyperdx/api -> express@4.22.1 -> body-parser@1.20.6 -> qs@6.15.3 -``` +Express's query parser is a call site, and `req.query` is request-derived. +`@hyperdx/api` pulls `express@4.22.1`, which pulls `qs@6.14.2` directly and +`qs@6.15.3` through `body-parser@1.20.6`. Every request that carries a query string or an urlencoded body reaches qs before any of our code does. GHSA-4mjr-xmp4-gh2g (DoS via attacker-controlled @@ -677,7 +671,12 @@ entry in that table to use it. Both advisories are moderate, so they do not clear THE BAR, and the caller is an OIDC-authenticated tenant user rather than the open internet - HyperDX sits behind Envoy or oauth2-proxy in every DFE deployment. What it buys an attacker is a self-inflicted DoS on the tenant they -already have an account on. +already have an account on. Corrected 2026-11-16: the k8s gateway puts no OIDC +policy on the HyperDX route (dfe-infra `helm/edge/gateway/values.yaml`, +`routes.hyperdx`), and express parses the query string before any auth runs, so +any caller that reaches the route reaches qs. The route is off the public +listener by default (`ui.public.hyperdx: false`). Both advisories are moderate, +so the verdict below THE BAR stands. **Watch this one.** The parents pin `~6.14.0` and `~6.15.1`, so a `resolutions` bump to 6.16.0 is not free - express and body-parser would each be off their @@ -719,7 +718,7 @@ write it has already won. `@hyperdx/otel-web-session-recorder`, both imported through `@hyperdx/browser` (`_app.tsx:9`, `DBSearchPage.tsx:28`, `AppNav.tsx:6`, `AppNavFeedback.tsx:3`). The advisory is an infinite loop in `unzipSync` on a malformed ZIP64 archive. A -session recorder compresses; nothing in the bundle parses a ZIP. Grepping +session recorder compresses, and nothing in the bundle parses a ZIP. Grepping `packages/` for `unzipSync`, `unzlibSync`, `decompressSync` and `fflate` returns nothing, so no code of ours reaches it either. NOT VERIFIED against the library source: this clone has an empty `node_modules` and no `.yarn/cache`, so the @@ -793,23 +792,20 @@ a sync. ### The lockfile copy is dev only -`yarn why image-size --recursive` gives one path: - -```text -@hyperdx/app -> @storybook/nextjs@10.1.4 -> image-size@2.0.2 -``` +`yarn why image-size --recursive` gives one path: `@hyperdx/app` pulls +`@storybook/nextjs@10.1.4`, which pulls `image-size@2.0.2`. `@storybook/nextjs` is an app devDependency. Dependabot calls it `runtime` because it reads the lockfile, not the image - the same mistake as every earlier pass. After `yarn workspaces focus @hyperdx/api --production` there is no `image-size` directory anywhere in `node_modules`. -### Next vendors the same code, and that copy is in the image +### Next ships the same code, and that copy is in the image `next@16.3.4` ships its own image-size at `next/dist/compiled/image-size` and the type detector at `next/dist/compiled/image-detector/detector.js`. Neither is in `yarn.lock`, so no alert will ever fire on them. Both carry the ICNS, JXL and -HEIF parsers, and the vendored JXL parser still throws 2.0.2's +HEIF parsers, and Next's copied-in JXL parser still throws 2.0.2's `No codestream found in JXL container`, so it predates the fix. The standalone server that `Dockerfile:106` copies into the image loads them from `next/dist/server/image-optimizer.js`. @@ -826,8 +822,11 @@ size or by 8 on a zero, so it always terminates. Next splits the two: placeholder sharp has just produced. So production never parses request-derived bytes with the vulnerable code, and -`/_next/image` sits behind the OIDC gateway anyway. The vendored copy moves when -a Next release vendors 2.0.3 or later. Check it on the next sync with: +`/_next/image` sits behind the OIDC gateway anyway. Next's copy moves when a +Next release ships 2.0.3 or later. Corrected 2026-11-16: the k8s gateway puts no +OIDC policy on the HyperDX route (dfe-infra `helm/edge/gateway/values.yaml`, +`routes.hyperdx`), so "behind the OIDC gateway" does not hold. The verdict rests +on the first sentence alone. Check it on the next sync with: ```sh rg -o 'No codestream found in JXL container' node_modules/next/dist/compiled/image-size/index.js @@ -893,17 +892,17 @@ do not bite, but because nobody opts in, not because nothing starts the SDK. ### #184 - reachable, bounded, tolerable risk -With the SDK running, every inbound request on the api port reaches the sink: - -```text -http server 'request' - -> instrumentation-http 0.57.2 http.js:385 propagation.extract(ROOT_CONTEXT, headers) - -> CompositePropagator -> W3CBaggagePropagator.extract (@opentelemetry/core 1.30.1) -``` +With the SDK running, every inbound request on the api port reaches the sink. +The http server's `request` event runs instrumentation-http 0.57.2, which calls +`propagation.extract(ROOT_CONTEXT, headers)` (`http.js:385`). That reaches the +`CompositePropagator` and then `W3CBaggagePropagator.extract` in +`@opentelemetry/core` 1.30.1. That runs before express, so before any auth middleware. In a DFE deployment the OIDC gateway stops an unauthenticated request first, but anything that reaches -the pod port directly gets to the propagator. +the pod port directly gets to the propagator. Corrected 2026-11-16: the k8s +gateway puts no OIDC policy on the HyperDX route, so any request that reaches +the route reaches the propagator too. What bounds it is the advisory's own Impact section. Node caps the combined headers at 16 KB by default, the header is already in memory, and the extra cost @@ -928,16 +927,12 @@ Re-dismissing needs repo admin. Alert 213, GHSA-23hp-3jrh-7fpw, npm 1123940. Decompression and parse DoS, `<= 7.5.18`. The only parent was Yarn's implicit `node-gyp@npm:latest`, locked -at 10.2.0: - -```text -@hyperdx/app -> @storybook/nextjs -> node-polyfill-webpack-plugin -> crypto-browserify - -> browserify-cipher -> browserify-aes -> evp_bytestokey -> node-gyp@10.2.0 -> tar@6.2.1 -node-gyp@10.2.0 -> make-fetch-happen@13.0.1 -> cacache@18.0.4 -> tar@6.2.1 -``` - -plus `fsevents` under rollup, jest, nodemon and playwright, which never installs -on Linux. +at 10.2.0. `@hyperdx/app` reached it through `@storybook/nextjs`, +`node-polyfill-webpack-plugin`, `crypto-browserify`, `browserify-cipher`, +`browserify-aes`, `evp_bytestokey` and `node-gyp@10.2.0`, which pulls +`tar@6.2.1` directly and again through `make-fetch-happen@13.0.1` and +`cacache@18.0.4`. The same `node-gyp` sits under `fsevents` under rollup, jest, +nodemon and playwright, which never installs on Linux. It never ran. A full Linux install builds nx, esbuild, msw, protobufjs, core-js-pure, @scarf/scarf and unrs-resolver, and nothing through node-gyp. The @@ -1007,12 +1002,9 @@ copy, and the published `ghcr.io/hyperi-io/dfe-hyperdx:latest` (0.2.8, revision ### axios - fixed in the image, still pinned by nx -Two copies, and all 12 ranges end at `< 1.20.0`: - -```text -@hyperdx/api -> @slack/webhook@7.0.7 -> axios@1.20.0 -hyperdx (root) -> nx@23.1.1 -> axios@1.18.1 nx pins "1.18.1" exactly -``` +Two copies, and all 12 ranges end at `< 1.20.0`. `@hyperdx/api` pulls +`@slack/webhook@7.0.7`, which pulls `axios@1.20.0`. The root workspace pulls +`nx@23.1.1`, which pins `axios` at exactly `1.18.1`. The production focus holds one axios, 1.20.0, which #111 put there. @@ -1041,9 +1033,9 @@ describe main, and the image catches up at the next release. `publicPath` has no trailing slash. The range is `< 7.4.5`, and the advisory's own text puts the fix at 8.3.0 with no backport to 6.x or 7.x. -```text -@hyperdx/app -> @storybook/nextjs@10.1.4 -> @storybook/builder-webpack5@10.1.4 -> webpack-dev-middleware@6.1.3 -``` +`@hyperdx/app` pulls `@storybook/nextjs@10.1.4`, which pulls +`@storybook/builder-webpack5@10.1.4`, which pulls +`webpack-dev-middleware@6.1.3`. `@storybook/nextjs` is an app devDependency (`packages/app/package.json:115`), and only `storybook dev` serves the middleware. It is in neither the production @@ -1136,7 +1128,7 @@ range. 5.30.7, under the floor. The synced tree resolves 5.33.14. No patches. - ip-address: `packages/api` now declares upstream's `^10.7.1` and resolves 10.7.2, so the SSRF guard stays above the 2026-08-26 range. -- Next 16.3.6 still vendors the pre-fix image-size: `rg` finds +- Next 16.3.6 still ships the pre-fix image-size: `rg` finds `No codestream found in JXL container` in `next/dist/compiled/image-size/index.js`. The 2026-09-27 verdict stands, since production only runs `validate()`. @@ -1147,11 +1139,8 @@ range. exhaustion on deeply nested brace patterns, `<= 3.0.3`. 3.0.3, from 2024-05-21, is the latest release, so there is no fix to take. -It is in the image: - -```text -@hyperdx/api -> http-proxy-middleware@4.1.1 -> micromatch@4.0.8 -> braces@3.0.3 -``` +It is in the image: `@hyperdx/api` pulls `http-proxy-middleware@4.1.1`, which +pulls `micromatch@4.0.8`, which pulls `braces@3.0.3`. It does not bite. braces expands the PATTERN, never the string being matched, and the one proxy the api mounts (`routers/api/clickhouseProxy.ts:209`) passes a @@ -1395,7 +1384,7 @@ Alert 350 is the one left open by reading. Apart from `login` and `register` static HTML. Whether that counts as the advisory's SSG for cache replacement was not traced. 16.3.8 carries the fix either way. -The vendored image-size in 16.3.8 still carries the pre-fix JXL parser: +The image-size copied into 16.3.8 still carries the pre-fix JXL parser: `rg -c 'No codestream found in JXL container'` finds it once in `next/dist/compiled/image-size/index.js`. The 2026-09-27 verdict stands. @@ -1554,7 +1543,7 @@ the fix, since the 2026-10-09 re-resolve, and 16.4.1 raises next's own floor to Pinned rather than re-resolved because the merge driver takes upstream's lockfile, which is how 2.40.0 took next 16.3.6 back to 16.3.4. Upstream has the bump open as hyperdxio/hyperdx#3460. `npmMinimalAgeGate` holds 16.4.3 -(2026-10-23), so the range takes 16.4.2 (2026-10-15). 16.4.2 still vendors the +(2026-10-23), so the range takes 16.4.2 (2026-10-15). 16.4.2 still ships the pre-fix image-size, so the 2026-09-27 verdict stands. ### storybook 10.2.13 -> 10.6.2, and @vitest/mocker gone @@ -1577,7 +1566,7 @@ yarn up -R storybook @storybook/nextjs @storybook/addon-docs @storybook/addon-li The family moves together because `@storybook/nextjs` pins its builder exactly, and builder-webpack5 10.6.2 drops @vitest/mocker. A newer babel preset and -probe-image-size make the lockfile diff about 2,900 lines, every added version +probe-image-size made the lockfile diff about 2,900 lines, every added version reached only through the new storybook packages. `storybook build` completes on 10.6.2. `osv-scanner.toml` loses GHSA-82fw-gwwq-j7x9, which the scanner no longer reports. @@ -1726,7 +1715,7 @@ osv-scanner reports all four copies. The code it names is v4 only. production tree. It is not one: the SDK has no nested copy and resolves the hoisted root, measured with `node -p "require.resolve('zod/package.json')"` from the SDK's own directory, which answers `node_modules/zod` 3.25.76. -- Next's vendored `next/dist/compiled/zod` carries `ZodArray` sixteen times and +- Next's copy at `next/dist/compiled/zod` carries `ZodArray` sixteen times and `handleArrayResult` never, so the copy in the standalone trace is v3 too. **The residual, and it is not this advisory.** v3's `ZodArray._parse` diff --git a/docs/fork/sync-cycle.md b/docs/fork/sync-cycle.md index 7fddbdd82..e5853f5ec 100644 --- a/docs/fork/sync-cycle.md +++ b/docs/fork/sync-cycle.md @@ -13,30 +13,43 @@ procedure. The fork holds two kinds of content, and they need opposite treatment. +Our permanent changes ride merged history for rerere to replay, and the +temporary security layer is declared and generated, so rerere never sees it. + ```mermaid -flowchart TB - subgraph permanent["PERMANENT -- must survive every update"] - feat["DFE features
embed, OIDC, theme, authz"] - dfedir["packages/**/dfe/**
cannot conflict"] - inplace["catalogued in-place edits
.fork-surface"] +flowchart LR + accTitle: L4 Detail -- Permanent and temporary content + accDescr: Permanent content, the DFE features, the dfe directories and the catalogued in-place edits, is carried as merged history and git rerere replays its resolutions. Temporary content, the dependency pins and code patches, is declared, stripped before a merge and rebuilt after. + + subgraph permanent["PERMANENT, survives every update"] + direction LR + feat["DFE features"] + dfedir["packages/**/dfe/**"] + inplace["In-place edits
.fork-surface"] + feat ~~~ dfedir ~~~ inplace end - subgraph temporary["TEMPORARY -- upstream will fix these"] - pins["dependency pins
security/overrides.yaml"] - patches["code fixes
security/patches/*.patch"] + subgraph temporary["TEMPORARY, upstream will fix"] + direction LR + pins["Dependency pins
security/overrides.yaml"] + patches["Code fixes
security/patches/"] + pins ~~~ patches end - permanent -->|carried as MERGED HISTORY| rerere["git rerere
replays our resolutions"] - temporary -->|declared, then generated| gen["strip before merge,
rebuild after"] + rerere["git rerere replays
our resolutions"] + gen["Strip before merge,
rebuild after"] - classDef keep fill:#009E73,stroke:#005f45,color:#ffffff - classDef temp fill:#E69F00,stroke:#8a6100,color:#000000 - classDef mech fill:#0072B2,stroke:#00456b,color:#ffffff - class feat,dfedir,inplace keep - class pins,patches temp - class rerere,gen mech + permanent -->|merged history| rerere + temporary -->|declared| gen + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class permanent,temporary group ``` + +*Figure 1. L4 Detail -- Permanent and temporary content.* Up: [The fork](README.md). + Our features are permanent, so they belong in merged history where rerere can replay their conflict resolutions. Security fixes are temporary by design: upstream ships their own within weeks. Committing one in place puts a delta we @@ -50,7 +63,7 @@ never matches twice. A lockfile has no meaningful "replay these hunks" answer at all. So the temporary layer is GENERATED. `security/overrides.yaml` and -`security/patches/` are the source; the tree is derived from them. +`security/patches/` are the source, and the tree is derived from them. --- @@ -59,24 +72,32 @@ So the temporary layer is GENERATED. `security/overrides.yaml` and `.github/workflows/upstream-sync.yml` runs this weekly and on demand, and opens a PR. Run it by hand when you want to drive a specific upstream tag. +The cycle strips and rebuilds the security layer around the merge, then proves +the result. + ```mermaid -flowchart LR - strip["1. --unapply
strip the layer"] --> merge["2. git merge
rerere replays"] - merge --> apply["3. --apply
rebuild the layer"] - apply --> pin["4. --set
move the pin"] - pin --> install["5. yarn install
re-derive the lock"] - install --> verify["6. --verify
the invariants"] - verify --> gate["7. build + test
THE GUARANTEE"] - gate --> triage["8. triage
drafted for a human"] - - classDef layer fill:#E69F00,stroke:#8a6100,color:#000000 - classDef git fill:#0072B2,stroke:#00456b,color:#ffffff - classDef check fill:#009E73,stroke:#005f45,color:#ffffff - class strip,apply,pin layer - class merge,install git - class verify,gate,triage check +flowchart TB + accTitle: L4 Detail -- The sync cycle + accDescr: Merge runs four steps, unapply the security layer, git merge with rerere replaying, apply the layer and set the pin. Prove runs four more, yarn install to re-derive the lock, verify the invariants, build and test, and triage for a human. + + subgraph merge["1 Merge"] + direction LR + strip["1. --unapply"] --> gitmerge["2. git merge"] --> apply["3. --apply"] --> pin["4. --set pin"] + end + subgraph prove["2 Prove"] + direction LR + install["5. yarn install"] --> verify["6. --verify"] --> gate["7. build + test"] --> triage["8. triage"] + end + merge --> prove + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class merge,prove group ``` + +*Figure 2. L4 Detail -- The sync cycle. Read each stage left to right, then down.* Up: [The fork](README.md). + ```bash git switch -c sync/upstream-2.34.0 main scripts/security-override.py --unapply # 1 @@ -96,27 +117,28 @@ otherwise. --- -## What each step is actually doing +## What steps 1 to 3 do ### 1. Strip, so there is nothing to conflict on `--unapply` rewrites `resolutions` back to upstream's block at our current base -and reverse-applies every patch. The tree now matches upstream where the -temporary layer used to be, so git takes upstream's version of those hunks whole -and the merge never sees a delta there. +and reverse-applies every patch. The tree matches upstream where the temporary +layer used to be, so git takes upstream's version of those hunks whole and the +merge never sees a delta there. ### 2. Merge rerere replays previously-recorded resolutions for our PERMANENT deltas. `yarn.lock` is routed to `scripts/merge-lockfile.sh`, which takes upstream's -copy outright. Anything left conflicting is a genuine three-way conflict on a -catalogued file. +copy outright. Anything left conflicting is a three-way conflict on a catalogued +file. **A replayed resolution is not a correct one.** rerere reproduces the edit that -resolved a conflict whose preimage matched; it has no idea whether the result -still means anything. The failure it produces is specific and worth recognising: -our side of a hunk lands over upstream's side of the surrounding code, so the -file references names nothing imports, or declares parameters nobody passes. +resolved a conflict whose preimage matched, and it has no idea whether the +result still means anything. The failure it produces is specific and worth +recognising: our side of a hunk lands over upstream's side of the surrounding +code, so the file references names nothing imports, or declares parameters +nobody passes. 2.33.0 did exactly this to `OnboardingModal.tsx` - our import block, upstream's body, seven type errors. It survived six green workflow runs, because the gate @@ -126,33 +148,37 @@ which is the check that was missing. The general form is worth keeping in mind, because it bit twice in one week: a gate that does not exercise the thing it claims to cover reports success from an -empty run. The other instance was `--unapply` against an empty register - real -workflow runs, genuinely green, proving only that nothing happened. +empty run. The other instance was `--unapply` against an empty register - +workflow runs, green, proving only that nothing happened. ### 3. Rebuild, and find out what upstream took off our hands `--apply` layers the register back over the NEW upstream block, and applies each entry ONLY where it raises the floor. -That clause is load-bearing. Upstream pins the same packages we would. An entry -left in the register after upstream passes it does not go quietly stale: a -`^2.0.2` pin against an upstream now shipping 2.1.2 drags the whole tree -BACKWARDS onto the vulnerable line. Taking the higher of the two makes a -forgotten entry inert rather than harmful, and `--check` still nags until -someone deletes it. +Drop that clause and a forgotten entry does harm. Upstream pins the same +packages we would. An entry left in the register after upstream passes it does +not go quietly stale: a `^2.0.2` pin against an upstream shipping 2.1.2 drags +the whole tree BACKWARDS onto the vulnerable line. Taking the higher of the two +makes a forgotten entry inert rather than harmful, and `--check` still nags +until someone deletes it. A patch that no longer applies is REPORTED, never forced. That is the signal the cycle exists to produce. +--- + +## What steps 5 and 6 do + ### 5. The lockfile The merge took upstream's lockfile whole, so our fork-only entries are missing until `yarn install` re-derives them. Use `--mode=update-lockfile` first if you -want the resolution step on its own. Do NOT use `--immutable` here; it is the +want the resolution step on its own. Do NOT use `--immutable` here. It is the right flag everywhere else and the wrong one at exactly this point. `yarn install` re-derives only what our `package.json` asks for. A fix main -carries in the lockfile alone - a `yarn up -R` inside a declared range - is gone +holds in the lockfile alone - a `yarn up -R` inside a declared range - is gone after the merge, and nothing fails. Diff the resolved versions against `origin/main` and run `yarn up -R` again on every package main had moved past upstream. The 2.40.0 sync lost seven PRs of them, Next's RCE fix among them. @@ -174,12 +200,12 @@ Generated state drifts silently the moment somebody hand-edits the block, and a sync is the worst possible moment to discover it. "In one direction only" is not pedantry. `git apply` searches outward from the -hunk header, so a patch whose post-image already sits elsewhere in the file -- a -second call site upstream ships already safe -- reverse-applies THERE while the -real target sits untouched. Such a patch reads as applied while the fix is -absent, and stripping it before a merge rewrites upstream's correct code into -the vulnerable code. The tools call that `ambiguous` and refuse it; the fix is -to re-derive the patch against the current tree so it names one site. +hunk header, so a patch whose post-image sits elsewhere in the file -- a second +call site upstream ships safe -- reverse-applies THERE while the intended target +sits untouched. Such a patch reads as applied while the fix is absent, and +stripping it before a merge rewrites upstream's correct code into the vulnerable +code. The tools call that `ambiguous` and reject it. The fix is to re-derive the +patch against the current tree so it names one site. The same workflow runs the tooling's own test suites before any of the above: @@ -216,30 +242,39 @@ git checkout -- scripts/security-override.py --apply ``` -The numeric prefix is the apply order and reverses for unapply, so it is -load-bearing. See [the patch series contract](../../security/patches/README.md). +The numeric prefix is the apply order and reverses for unapply, so renumbering a +patch changes what applies first. See +[the patch series contract](../../security/patches/README.md). --- ## When a patch stops applying -That is the cycle working, not failing. Two possibilities, and telling them -apart needs judgement rather than a comparison: +That is the cycle working, not failing. Telling the two possible causes apart +needs judgement rather than a comparison, and each has its own fix. ```mermaid -flowchart TB - stale["patch no longer applies"] --> q{"did upstream fix it,
or did the code move?"} - q -->|upstream fixed it| drop["delete the patch
and its register entry"] - q -->|the code moved| rederive["re-derive the hunk
against the new file"] - - classDef signal fill:#E69F00,stroke:#8a6100,color:#000000 - classDef decision fill:#F0E442,stroke:#8a8200,color:#000000 - classDef action fill:#009E73,stroke:#005f45,color:#ffffff - class stale signal +flowchart LR + accTitle: L4 Detail -- A patch that stops applying + accDescr: A patch that no longer applies means either upstream fixed it, so the patch and its register entry are deleted, or the code moved, so the hunk is re-derived against the new file. + + stale(["Patch no
longer applies"]) + q{"Fixed upstream,
or code moved?"} + drop(["Delete the patch
and its entry"]) + rederive(["Re-derive
the hunk"]) + + stale --> q + q -->|fixed| drop + q -->|moved| rederive + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef decision fill:#FCF1D6,stroke:#9A6700,stroke-width:1px,color:#3D2A00 class q decision - class drop,rederive action ``` + +*Figure 3. L4 Detail -- A patch that stops applying.* Up: [The fork](README.md). + `scripts/security-triage.py --patches` drafts which, with evidence. It PROPOSES only -- see below. @@ -248,7 +283,7 @@ only -- see below. ## The triage step, and its limits `scripts/security-triage.py` answers the questions the mechanical tooling -cannot. It reads the signals this repo already produces rather than rescanning: +cannot. It reads the signals this repo produces rather than rescanning: | Flag | Question | Source it reads | | ----------- | --------------------------------------------------- | ------------------------------------ | @@ -261,44 +296,49 @@ cannot. It reads the signals this repo already produces rather than rescanning: no key, no network and no human, so the existing signals do as much as they can before anything reaches the model: -- Dependabot carries `dependency.scope`. A development-scoped advisory is build - tooling that never reaches the shipped image, so it is answered outright. - Measured on this repo: 13 of 72 settled for free. -- `attribute-findings.py` has already sorted scanner findings into OURS / - ACCEPTED / INHERITED, so only OURS is drafted. That bucket is usually empty. +- Dependabot has `dependency.scope`. A development-scoped advisory is build + tooling that never reaches the shipped image, so it is answered outright. On + this repo: 13 of 72 settled for free. +- `attribute-findings.py` has sorted scanner findings into OURS / ACCEPTED / + INHERITED, so only OURS is drafted. That bucket is usually empty. The model is asked the residue, not the pile. It writes a markdown report and nothing else. It does not edit the register, apply a patch, or clear a finding. Reachability is a human verdict: a model that -could clear its own findings would rebuild exactly the alert queue the fork's -inverted posture exists to avoid. +could clear its own findings would rebuild the alert queue the fork's inverted +scanner settings exist to avoid. With no `ANTHROPIC_API_KEY` or no `anthropic` SDK it prints the mechanical facts and marks each unjudged section `NOT JUDGED`. A sync is never blocked by an expired secret. +--- + +## Where the triage reads its signals + **CI reads Dependabot through a GitHub App, not a PAT.** The workflow `GITHUB_TOKEN` cannot reach the alerts endpoint whatever `permissions:` says - it wants the app-level Dependabot-alerts permission, and the 403 reads exactly like a missing scope on your own account. -The sync workflow mints a token from `hyperi-container-mgt`, whose app id and -private key are already org-wide secrets, so no new secret is needed here. What -IS needed, once, from an org owner: add **Dependabot alerts: Read** to that -app's permissions and accept it on the installation. +The sync workflow mints a token from `hyperi-container-mgt`, whose client id +(`vars.CONTAINER_MGT_CLIENT_ID`) and private key +(`secrets.CONTAINER_MGT_APP_PRIVATE_KEY`) are org-wide, so no new secret is +needed here. What IS needed, once, from an org owner: add **Dependabot alerts: +Read** to that app's permissions and accept it on the installation. Until then the triage falls back to `yarn npm audit`, loses the `scope` field, and says so in the report rather than looking like a repo with 18 advisories instead of 72. A `DEPENDABOT_TOKEN` secret is still honoured as a fallback if someone would rather use a PAT, but the App is the house standard. -**CodeQL is off** here (the -[inverted security posture](design.md#the-security-posture-is-inverted-here)), -so there is nothing to read from it. `attribute-findings.py` takes SARIF and -serves CodeQL as readily as semgrep, so turning it on later needs no change to -the triage. **Renovate** is scoped to the action pins in our own workflows and -produces no dependency signal by design. +**CodeQL default setup is on**, but the triage reads code findings as SARIF +through `attribute-findings.py`, and `fork-security.yml` feeds it semgrep's. +`attribute-findings.py` serves CodeQL's SARIF as readily, so adding it needs no +change to the triage. **Renovate** is scoped to the files its `includePaths` +names ([design.md](design.md#security-scanning-is-inverted-here)) and produces +no signal on upstream's dependency tree by design. **Never use `gh api repos/{owner}/{repo}/...` in this repo.** A fork has two remotes, and with no default set gh resolves the placeholders to UPSTREAM: the @@ -336,8 +376,8 @@ the cheap moment to shrink a delta down the turns it into a conflict. `upstream-drift.yml` runs it daily. **A stale `upstream/main` lies quietly.** One measurement taken against a ref -that had not been fetched in weeks reported 34 commits of drift and 2 conflicts; -a fresh fetch showed 122 commits and 11 conflicts. Nothing announces the +that had not been fetched in weeks reported 34 commits of drift and 2 conflicts, +and a fresh fetch showed 122 commits and 11 conflicts. Nothing announces the staleness - it just tells you the sync is smaller than it is. The daily CI number fetches every run, so trust it over whatever your laptop says. @@ -373,103 +413,7 @@ why it has to stay accurate. --- -## Track record - -Sync cost over time is the evidence the [de-fork decision](leaving-upstream.md) -should be taken against, so it gets recorded. - -| | 2026-07-23 (dry run) | 2026-08-04 (2.29.0 -> 2.33.0, full run) | 2026-09-21 (2.36.0 -> 2.39.1, aborted) | 2026-10-06 (2.36.0 -> 2.40.0, full run) | 2026-11-01 (2.41.0 -> 2.43.0, full run) | -| ---------------------- | -------------------- | --------------------------------------- | -------------------------------------- | --------------------------------------- | --------------------------------------- | -| upstream commits ahead | 122 | 164 | 116 | 164 | 105 | -| conflicting files | 11 | 12 | 14 | 18 | 9 | -| replayed by rerere | n/a | n/a | **0 of 9** | **0 of 13** | **0 of 7** | -| `yarn.lock` conflicted | yes | **no** | no | no | no | -| delta gate | 81 `src/dfe` tests | 211 suites / 4795 tests | not reached | 427 suites / 9159 tests, 5 packages | 529 suites / 10780 tests, 5 packages | - -What changed between the two rows is the merge driver and the generated security -layer. The lockfile stopped conflicting outright: the driver took upstream's -copy and `yarn install` re-derived exactly ONE entry, `jose`, our OIDC -dependency. - -Of the 12 conflicts, 7 carried content; the other 5 were modify/delete on the -workflows hyperi-ci replaced, whose resolution is to stay deleted. Two are worth -remembering: - -- `packages/api/tsconfig.build.json` - our only delta turned out to be - FORMATTING; the `include` content was upstream's all along. Taking theirs - wholesale removed the file from the conflict surface entirely. -- `packages/app/src/layout.tsx` - upstream added a kiosk mode that also hides - the nav. Both conditions now sit side by side, and that convergence is the - first real evidence for the de-fork path. - -### 2026-09-21 - rerere replayed nothing, and the alarm had been off for a month - -The 2.39.1 attempt produced 14 conflicts and rerere replayed **none** of them. -It was enabled and configured, and it had 11 recorded resolutions sitting in -`rr-cache`. Every one missed, because a preimage has to match EXACTLY and 116 -commits of upstream had moved the context around every hunk. - -That is the mechanism worth taking from this row: **rerere's value decays with -backlog.** A resolution recorded against a 20-commit gap is worthless at 116. -The weekly cadence is not a nicety, it is what keeps the preimages matching. - -Which is why the alarm mattered. `gh issue create --label 'upstream-sync'` needs -the label to exist and this repo never had one, so the call failed on every -conflicted run and `|| echo "Issue may already exist."` turned the failure into -a success. Five consecutive Mondays reported green, no issue was filed, and the -drift number nobody was reading is the number that decides when re-forking beats -merging. - -The step now creates the label, tolerates only an already-open issue for the -same ref, and fails the run otherwise. - -Of the 14: 5 are modify/delete on the upstream workflows that must stay deleted -(see the 2026-08-27 entry in security-sync.md - `release.yml` would publish to -hyperdxio's npm scopes and Docker Hub namespaces), and 9 carry content - -`.gitignore`, `.prettierignore`, -`docker/otel-collector/config.standalone.auth.yaml`, `DBChartPage.tsx` (2 -hunks), `DBDashboardPage.tsx` (2), `AppNav.tsx`, `SQLInlineEditor.tsx`, -`useRowWhere.tsx` and `scripts/ci/ratchet-baseline.json` (2). Thirteen hunks in -all. - -### 2026-10-06 - the merge was the easy part - -2.40.0 hit the 2.39.1 attempt's nine content conflicts plus four more: -`package.json`, `packages/api/package.json`, `DBRowJsonViewer.tsx` and -`DashboardsListPage.tsx`, which upstream rewrote with tabs and a header action -group. The other five were the deleted upstream workflows again. `rr-cache` held -25 entries and replayed none of the 13 content conflicts. -`config.standalone.auth.yaml` left the surface, since our only delta was quote -style, and upstream's new `jscpd.yml` joined `.fork-deleted`. - -Three breakages got past a merge git called resolved: - -- **The lockfile.** The driver took upstream's, so every security fix main held - in `yarn.lock` alone went back to upstream's version, Next 16.3.6 to 16.3.4 - among them. Nothing failed. See step 5. -- **`provisionDashboards/index.ts`** merged clean and failed `tsc`: upstream's - new `STATIC_LIST` filter has no `source` for our reference resolver to read. -- **Upstream's new `DashboardsListPage.test.tsx`** mounts the page with no - QueryClient, and both components we inject there query. All 21 of its tests - failed until they went behind `dfe/QueryClientOnly.tsx`. - -### 2026-11-01 - two pins unwound, 120 packages put back - -2.43.0 conflicted on seven content files and the deleted `release.yml` and -`vouch.yml`. `rr-cache` held 26 entries and replayed none. Upstream now ships -next 16.4.2 and undici 7.30.1, so `--check` called both register entries -redundant and they were removed. - -The lockfile regressed again, and wider: 120 packages main had moved with -`yarn up -R` came back older, Storybook 10.6.2 to 10.1.4 and tar 7.5.22 to 6.2.1 -among them. osv-scanner read 10 unfiltered packages until `yarn up -R` on the -whole list put them back. - -Upstream moved the filter-key limit read into `useFilterKeysFetchLimit`, which -our `dfe/__tests__/jsonFilterSidebar.test.tsx` mocks partially, so its suite -failed until the mock gained the new hook. - -### Expect `Build All-in-One Image` to fail on a sync PR +## Expect `Build All-in-One Image` to fail on a sync PR It is upstream's workflow, and it dies with `no space left on device` partway through the collector's Go build. Not a regression, and not something to fix by @@ -478,7 +422,7 @@ editing upstream's workflow. The job builds the whole Node app AND the Go collector into one image on a hosted runner, with no disk-reclaim step. The collector alone pulls the full prometheus and cloud-SDK dependency set. On an ordinary upstream PR the path -filters keep it dormant; a sync touches those paths, so it runs, and it does not +filters keep it dormant. A sync touches those paths, so it runs, and it does not fit. It is worth knowing this is capacity and nothing else, because the error looks @@ -486,7 +430,7 @@ alarming. Our only deltas anywhere near it are config and entry-script - `docker/otel-collector/*.yaml`, `docker/hyperdx/entry.prod.sh` - so the Go compile that runs out of room is byte-identical to upstream's. Built on adequate hardware (12 cores, 540G free), the same commit produces a 2.62GB -`all-in-one-auth` image with zero disk errors. Verified for 2.33.0; do that +`all-in-one-auth` image with zero disk errors. Verified for 2.33.0. Do that again rather than trusting this paragraph if the failure ever looks different. The standing options are a self-hosted runner, a disk-reclaim step (upstream @@ -498,6 +442,7 @@ surface, so no), or accepting the red check on sync PRs. We take the last. - [design.md](design.md) - why the fork is built this way - [what-we-changed.md](what-we-changed.md) - the change catalogue +- [sync-track-record.md](sync-track-record.md) - what each sync cost - [leaving-upstream.md](leaving-upstream.md) - how this ends - [../../security/patches/README.md](../../security/patches/README.md) - the patch series contract diff --git a/docs/fork/sync-track-record.md b/docs/fork/sync-track-record.md new file mode 100644 index 000000000..c392d1684 --- /dev/null +++ b/docs/fork/sync-track-record.md @@ -0,0 +1,96 @@ +# Sync track record + +Sync cost over time is the evidence the [de-fork decision](leaving-upstream.md) +should be taken against, so each run is recorded here. The procedure is +[sync-cycle.md](sync-cycle.md). + +| | 2026-07-23 (dry run) | 2026-08-04 (2.29.0 -> 2.33.0, full run) | 2026-09-21 (2.36.0 -> 2.39.1, aborted) | 2026-10-06 (2.36.0 -> 2.40.0, full run) | 2026-11-01 (2.41.0 -> 2.43.0, full run) | +| ---------------------- | -------------------- | --------------------------------------- | -------------------------------------- | --------------------------------------- | --------------------------------------- | +| upstream commits ahead | 122 | 164 | 116 | 164 | 105 | +| conflicting files | 11 | 12 | 14 | 18 | 9 | +| replayed by rerere | n/a | n/a | **0 of 9** | **0 of 13** | **0 of 7** | +| `yarn.lock` conflicted | yes | **no** | no | no | no | +| delta gate | 81 `src/dfe` tests | 211 suites / 4795 tests | not reached | 427 suites / 9159 tests, 5 packages | 529 suites / 10780 tests, 5 packages | + +What changed between the first two runs is the merge driver and the generated +security layer. The lockfile stopped conflicting outright: the driver took +upstream's copy and `yarn install` re-derived exactly ONE entry, `jose`, our +OIDC dependency. + +Of the 12 conflicts, 7 had content, and the other 5 were modify/delete on the +workflows hyperi-ci replaced, whose resolution is to stay deleted. Two are worth +remembering: + +- `packages/api/tsconfig.build.json` - our only delta turned out to be + FORMATTING. The `include` content was upstream's all along. Taking theirs + wholesale removed the file from the conflict surface entirely. +- `packages/app/src/layout.tsx` - upstream added a kiosk mode that hides the + nav. Both conditions sit side by side, and that convergence is the first + evidence for the de-fork path. + +## 2026-09-21 - rerere replayed nothing, and the alarm had been off for a month + +The 2.39.1 attempt produced 14 conflicts and rerere replayed **none** of them. +It was enabled and configured, and it had 11 recorded resolutions sitting in +`rr-cache`. Every one missed, because a preimage has to match EXACTLY and 116 +commits of upstream had moved the context around every hunk. + +That is the mechanism worth taking from this row: **rerere's value decays with +backlog.** A resolution recorded against a 20-commit gap is worthless at 116. +The weekly cadence is not a nicety, it is what keeps the preimages matching. + +Which is why the alarm mattered. `gh issue create --label 'upstream-sync'` needs +the label to exist and this repo never had one, so the call failed on every +conflicted run and `|| echo "Issue may already exist."` turned the failure into +a success. Five consecutive Mondays reported green, no issue was filed, and the +drift number nobody was reading is the number that decides when re-forking beats +merging. + +The step creates the label, tolerates only an already-open issue for the same +ref, and fails the run otherwise. + +Of the 14: 5 are modify/delete on the upstream workflows that must stay deleted +(see the 2026-08-27 entry in [security-sync.md](security-sync.md): `release.yml` +would publish to hyperdxio's npm scopes and Docker Hub namespaces), and 9 have +content: `.gitignore`, `.prettierignore`, +`docker/otel-collector/config.standalone.auth.yaml`, `DBChartPage.tsx` (2 +hunks), `DBDashboardPage.tsx` (2), `AppNav.tsx`, `SQLInlineEditor.tsx`, +`useRowWhere.tsx` and `scripts/ci/ratchet-baseline.json` (2). Thirteen hunks in +all. + +## 2026-10-06 - the merge was the easy part + +2.40.0 hit the 2.39.1 attempt's nine content conflicts plus four more: +`package.json`, `packages/api/package.json`, `DBRowJsonViewer.tsx` and +`DashboardsListPage.tsx`, which upstream rewrote with tabs and a header action +group. The other five were the deleted upstream workflows again. `rr-cache` held +25 entries and replayed none of the 13 content conflicts. +`config.standalone.auth.yaml` left the surface, since our only delta was quote +style, and upstream's new `jscpd.yml` joined `.fork-deleted`. + +Three breakages got past a merge git called resolved: + +- The lockfile. The driver took upstream's, so every security fix main held in + `yarn.lock` alone went back to upstream's version, Next 16.3.6 to 16.3.4 among + them. Nothing failed. See [step 5](sync-cycle.md#5-the-lockfile). +- `provisionDashboards/index.ts` merged clean and failed `tsc`: upstream's new + `STATIC_LIST` filter has no `source` for our reference resolver to read. +- Upstream's new `DashboardsListPage.test.tsx` mounts the page with no + QueryClient, and both components we inject there query. All 21 of its tests + failed until they went behind `dfe/QueryClientOnly.tsx`. + +## 2026-11-01 - two pins unwound, 120 packages put back + +2.43.0 conflicted on seven content files and the deleted `release.yml` and +`vouch.yml`. `rr-cache` held 26 entries and replayed none. Upstream ships next +16.4.2 and undici 7.30.1, so `--check` called both register entries redundant +and they were removed. + +The lockfile regressed again, and wider: 120 packages main had moved with +`yarn up -R` came back older, Storybook 10.6.2 to 10.1.4 and tar 7.5.22 to 6.2.1 +among them. osv-scanner read 10 unfiltered packages until `yarn up -R` on the +whole list put them back. + +Upstream moved the filter-key limit read into `useFilterKeysFetchLimit`, which +our `dfe/__tests__/jsonFilterSidebar.test.tsx` mocks partially, so its suite +failed until the mock gained the new hook. diff --git a/docs/fork/what-we-changed.md b/docs/fork/what-we-changed.md index a6348cfdd..c8eae1894 100644 --- a/docs/fork/what-we-changed.md +++ b/docs/fork/what-we-changed.md @@ -16,19 +16,19 @@ newer upstream is [sync-cycle.md](sync-cycle.md). ## Upstream base -- **Upstream:** (HyperDX v2) -- **Synced to:** whatever `.upstream-version` records. It is verified against +- Upstream: (HyperDX v2) +- Synced to whatever `.upstream-version` records. It is verified against `git merge-base` on every PR, so it cannot quietly drift from the tree. -- **Imported at:** commit `e58f01d` "Initial HyperDX commit", a FLATTENED copy - of upstream rather than a continuation of their history. - `scripts/fork-setup.sh` grafts it back on so blame and ancestry work - see +- Imported at commit `e58f01d` "Initial HyperDX commit", a FLATTENED copy of + upstream rather than a continuation of their history. `scripts/fork-setup.sh` + grafts it back on so blame and ancestry work - see [design.md](design.md#the-squashed-import-and-the-graft-that-repairs-it). -- **The `upstream` remote is required.** Without it the conflict-surface guard +- The `upstream` remote is required. Without it the conflict-surface guard cannot tell our files from upstream's, and the verify checks decline to run rather than pass on nothing. -- **Our changes live on `main`**, merged in as feature PRs, not as a curated - patch series rebased on a tracked upstream. That is why an upstream bump is a - merge with real conflict surface rather than a clean rebase. +- Our changes live on `main`, merged in as feature PRs, not as a curated patch + series rebased on a tracked upstream. That is why an upstream bump is a merge + with real conflict surface rather than a clean rebase. **RETIRED - the `.dfe[CHG]` shadow-copy convention.** Modified copies of upstream files were once kept beside the originals with a `.dfe[CHG]` suffix. It @@ -41,26 +41,38 @@ must be catalogued in `.fork-surface`. ## Where our code lives +Additive paths cannot conflict, and every in-place edit is listed in +`.fork-surface`. + ```mermaid flowchart LR - subgraph additive["Additive - cannot conflict"] - api["packages/api/src/dfe/**"] - app["packages/app/src/dfe/**"] - theme["packages/app/src/theme/themes/dfe/**"] - tools["scripts/, security/, .githooks/"] - docs["docs/"] + accTitle: L4 Detail -- Where our code lives + accDescr: Additive paths, the api, app and common-utils dfe directories, the dfe theme, the security and githooks directories, and our docs and scripts, cannot conflict. In-place edits to upstream files conflict on a sync, and .fork-surface lists every one. + + subgraph additive["Additive, cannot conflict"] + direction TB + dfedirs["packages/*/src/dfe/**"] + theme["packages/app/src/
theme/themes/dfe/**"] + tools["security/, .githooks/,
our docs and scripts"] end - subgraph inplace["In place - catalogued, conflicts on sync"] + subgraph inplace["In place, conflicts on sync"] + direction TB surface[".fork-surface
lists every one"] end - classDef safe fill:#009E73,stroke:#005f45,color:#ffffff - classDef risk fill:#E69F00,stroke:#8a6100,color:#000000 - class api,app,theme,tools,docs safe - class surface risk + additive ~~~ inplace + + classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A + classDef error fill:#FBE3D6,stroke:#C2410C,stroke-width:1px,color:#4A1A05 + classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4 + class surface error + class additive,inplace group ``` + +*Figure 1. L4 Detail -- Where our code lives. Orange is the conflict surface.* Up: [The fork](README.md). + --- ## Data layer: DocumentDB @@ -77,17 +89,18 @@ to the gateway unmodified. Rationale: [../decisions/0006-documentdb-document-store.md](../decisions/0006-documentdb-document-store.md). -## External OIDC authentication +## Engine token authentication -The auth boundary moves out of HyperDX. Envoy (Kubernetes) or oauth2-proxy -(docker) runs the OIDC flow and forwards identity headers. HyperDX trusts -`x-oidc-*` / `X-Forwarded-*`, find-or-creates the user, and names its team after -the ClickHouse identity the engine hands it. It falls back to upstream session -auth when the headers are absent. +The engine signs users in, and the API verifies the engine's ES384 token from +the `dfe_token` cookie or a Bearer header against the engine's JWKS. It +find-or-creates the user, and names its team after the ClickHouse identity the +engine hands it. A request with no valid token falls through to upstream's +session auth, which the route guards then refuse. -- `packages/api/src/dfe/middleware/oidc-identity.ts` - trusted-header identity - `packages/api/src/dfe/middleware/jwt-verify.ts` - ES384 verification against - the provider JWKS + the engine's JWKS, and the mode switch +- `packages/api/src/dfe/middleware/oidc-identity.ts` - the dev-only `header-dev` + path, which trusts identity headers unverified - `packages/api/src/dfe/controllers/user-provisioning.ts` - JIT user creation - `packages/api/src/dfe/controllers/team-provisioning.ts` - find-or-create the team by name @@ -95,7 +108,8 @@ auth when the headers are absent. a session: the team (its ClickHouse identity) and the dashboard role - `packages/api/src/dfe/middleware/legacy-auth-lockdown.ts` - 404s HyperDX's own password login, registration and invite-acceptance routes in OIDC mode -- `packages/api/src/api-app.ts` - wires both middlewares behind `AUTH_MODE` +- `packages/api/src/api-app.ts` - registers the identity middleware and the + legacy-auth lockdown in one block guarded by `isDfeEnabled` (`DFE_AUTH_MODE`) Detail: [../architecture/oidc-authentication.md](../architecture/oidc-authentication.md). @@ -114,8 +128,8 @@ expression, `findOne({})` to `findOne({ _id: id })`. `dfe/__tests__/team-scope.int.test.ts` seeds two teams and fails without it. Authorization is owned by the DFE engine as the policy decision point, and -enforced at the data layer by ClickHouse GRANTs. HyperDX trusts the identity -injected at the edge. +enforced at the data layer by ClickHouse GRANTs. HyperDX verifies the engine's +token itself rather than trusting a header. Rationale: [../decisions/0004-casbin-removed.md](../decisions/0004-casbin-removed.md). @@ -174,8 +188,8 @@ or team. **`requireDfeMode`** takes a DFE-only router out of the app when `DFE_AUTH_MODE` is unset - `next('router')`, so the request 404s as it would upstream. -`dfe/routers/shipped-dashboards.ts` mounts it, inside our own router rather than -at the `api-app.ts` mount, which keeps the gate off the upstream file. +`dfe/routers/shipped-dashboards.ts` mounts it, inside our router rather than at +the `api-app.ts` mount, which keeps the gate off the upstream file. **Only `admin` and `owner` change what the whole team sees (`dfe/middleware/role-claim.ts`).** Deleting or restoring a shipped dashboard @@ -183,14 +197,14 @@ changes it for every member. `jwt-verify.ts` takes the role from the engine's `hyperdx_role` on `GET /api/v1/auth/me`, cached for 30 seconds, and falls back to the token's `role` claim for an engine that sends none. The check is gated on the role's PRESENCE, so a session with none keeps team membership as the whole -authorisation; header-dev mode sets none, so local dev is unchanged. +authorisation. header-dev mode sets none, so local dev is unchanged. **The engine's `viewer` role is read-only (`refuseReadOnlyWrites`).** A team is one ClickHouse identity, so a viewer shares its team with accounts that write, and HyperDX routes check team membership alone. `dfeIdentityMiddleware` runs the gate once the identity is known, so it covers every route, ones upstream adds included. A `viewer` gets 403 on any non-read request outside the query proxy, -PromQL, the chart assistant, its own favourites and profile, and the +PromQL, the chart assistant, its favourites and profile, and the engine-authorised SQL export and rule creation. **A read-only session is not offered Save, Update or Create Rule @@ -217,12 +231,12 @@ allowed to set, so a hostile page could aim one at The console's button sends the header. It is the second layer, not the only one. dfe-ui plants the engine token in the -`dfe_token` cookie from its own proxy (`apps/dfe-core-ui/src/proxy.ts`): -`HttpOnly`, `Path=/`, `SameSite=Lax`, `Secure` whenever the UI is served over -https, `Domain` set to `DFE_COOKIE_DOMAIN` so the cookie reaches the embedded -HyperDX subdomain, and `Max-Age` from the token's own expiry. `SameSite=Lax` -already keeps that cookie off a cross-site POST, so the exposure was thin; the -header check is what holds when a deployment authenticates some other way. +`dfe_token` cookie from its proxy (`apps/dfe-core-ui/src/proxy.ts`): `HttpOnly`, +`Path=/`, `SameSite=Lax`, `Secure` whenever the UI is served over https, +`Domain` set to `DFE_COOKIE_DOMAIN` so the cookie reaches the embedded HyperDX +subdomain, and `Max-Age` from the token's expiry. `SameSite=Lax` already keeps +that cookie off a cross-site POST, so the exposure was thin. The header check is +what holds when a deployment authenticates some other way. ## Request validation @@ -245,12 +259,12 @@ expects undeclared static-filter keys to persist is skipped by name in - `packages/api/src/dfe/controllers/org-connection.ts` - a team holds ONLY one connection, as the ClickHouse user it is named after, fetched from the engine and seeded on any request that finds none, plus its seed sources. `main` and - `hunts` for every team (org-fenced by their row policies); `otel_logs`, + `hunts` for every team (org-fenced by their row policies), and `otel_logs`, `otel_traces`, `otel_metrics` and `clickhouse_system` for the platform team - alone, since operator telemetry and ClickHouse's own `system` database must - never reach an org_viewer. That source set is also the RBAC fence for the - pre-canned dashboards - see the provisioner note below. Seeding a NEW team - also runs `syncDashboards` for it when `DASHBOARD_PROVISIONER_DIR` is set: the + alone, since operator telemetry and ClickHouse's `system` database must never + reach an org_viewer. That source set is also the RBAC fence for the pre-canned + dashboards - see the provisioner note below. Seeding a NEW team also runs + `syncDashboards` for it when `DASHBOARD_PROVISIONER_DIR` is set: the provisioner cron fires once a minute with no run at start, so the page's first `GET /dashboards`, a second after the team is created, saw `[]` and the SPA cached it until a reload. Every request reads the team's connections, so a @@ -283,7 +297,7 @@ expects undeclared static-filter keys to persist is skipped by name in Keep our delta invisible to them. - `packages/app/src/dfe/clickhouseJsonPath.ts` - native ClickHouse JSON columns. `col['k']` on a JSON column is `arrayElement`, which ClickHouse rejects, so we - emit `JSONExtractString`; a JSON sub-path yields `Dynamic`, which + emit `JSONExtractString`. A JSON sub-path yields `Dynamic`, which `JSONExtract*` also rejects, so that ONE case is wrapped in `toString()`. Narrow by design - wrapping unconditionally changed the SQL for String and Map columns too. @@ -296,11 +310,12 @@ expects undeclared static-filter keys to persist is skipped by name in - `packages/common-utils/src/dfe/jsonPath.ts` - native ClickHouse JSON for charts and filters. A JSON sub-path is `Dynamic`, which ClickHouse refuses in `IN`, aggregates, `GROUP BY` and `ORDER BY`, so it needs an explicit coercion. - All the logic is here; **`core/renderChartConfig.ts` (catalogued)** carries - four added calls and nothing else - one in `renderWhereExpressionStr`, the - seam every SQL filter already passes through, one in `renderSelectList`'s - raw-string early return, which `renderSelect` and `renderGroupBy` BOTH feed, - so the SELECT list and the GROUP BY key cannot disagree, one wrapping the + All the logic is here. **`core/renderChartConfig.ts` (catalogued)** has five + added calls and nothing else - one in `renderWhereExpressionStr`, the function + every SQL filter passes through, one in `renderSelectList`'s raw-string early + return, which `renderSelect` and `renderGroupBy` BOTH feed, so the SELECT list + and the GROUP BY key cannot disagree, one on the group-by columns in + `renderSeriesLimitCte`, so its tuple matches that GROUP BY, one wrapping the `renderOrderBy` argument, and `dfeJsonAggregates` after `renderSelectList` builds each aggregate. Coercion is `toString()`, NOT the `.:String` sub-column upstream #2549 proposes - see the module header for the measurement that rules @@ -314,7 +329,7 @@ expects undeclared static-filter keys to persist is skipped by name in before the materialized-column pass re-prints the SQL. Editing the cast in `aggFnExpr` itself was rejected: it has no column metadata, so the change would hit every column and move about 170 upstream assertions and snapshots. - Pinned end to end by `dfe/__tests__/jsonAggregates.test.ts`. + Pinned from config to rendered SQL by `dfe/__tests__/jsonAggregates.test.ts`. ORDER BY was left alone until 2026-08-26 and now coerces via `dfeCoerceOrderBy`, which walks both the string and the `valueExpression[]` @@ -324,13 +339,12 @@ expects undeclared static-filter keys to persist is skipped by name in It now sorts, but `toString` sorts lexically - "99.5" above "1000.25". Sorting a numeric JSON path correctly needs the path's concrete TYPE, and `getJSONKeys` keeps only `typeArr[0]`, which for a mixed-type path is whatever - ClickHouse happened to list first. That decision is still open and is tracked - with max/min and facet values in the plan, since all three are the same - question. + ClickHouse happened to list first. That decision is open, along with max/min + and facet values, since all three are the same question. - Two cases the seam must handle, both found by e2e and pinned by tests. A - trailing `.:String` is a TYPE, not a path segment: quoting it as one reads a - JSON key literally named `:String`, empty for every row. And + Two cases the WHERE coercion must handle, both found by e2e and pinned by + tests. A trailing `.:String` is a TYPE, not a path segment: quoting it as one + reads a JSON key literally named `:String`, empty for every row. And `findJsonExpressions` returns the ENCLOSING call's closing paren attached when a path ends in a type specifier, so consuming it unbalances the aggregate around it - a syntax error that fails the whole facet batch and empties every @@ -353,17 +367,17 @@ expects undeclared static-filter keys to persist is skipped by name in Facet VALUES still render through upstream's `.:String`, so a path storing a non-String type in some rows lists an incomplete value set. Selecting a value - is unaffected - the WHERE seam coerces with `toString()`, which matches a + is unaffected - the WHERE coercion uses `toString()`, which matches a superset - so this costs completeness, never correctness. Changing it means editing upstream's own `metadata.test.ts` assertions, which is the most expensive kind of delta we have. Lucene `>`, `>=`, `<`, `<=` and ranges on a JSON path are numeric. Upstream's - serializer guards them as `dynamicType(P) in (...) and P`, and the seam gives - that operand `toFloat64OrNull(toString(P))` rather than `toString(P)`: a text - operand compares lexically, so `> '100'` matched 50 and 99, and a text BETWEEN - against numbers fails with NO_COMMON_TYPE. Upstream's `range()` also read - `column`, which is empty for a JSON path, so `_json.bytes:[100 TO 500]` + serializer guards them as `dynamicType(P) in (...) and P`, and our coercion + gives that operand `toFloat64OrNull(toString(P))` rather than `toString(P)`: a + text operand compares lexically, so `> '100'` matched 50 and 99, and a text + BETWEEN against numbers fails with NO_COMMON_TYPE. Upstream's `range()` also + read `column`, which is empty for a JSON path, so `_json.bytes:[100 TO 500]` rendered `( BETWEEN 100 AND 500)`. **`queryParser.ts` (catalogued)** wraps its `getColumnForField` call in `dfeRangeField`, which hands over the same numeric operand its comparison operators already use: one import and one wrapped call. @@ -385,19 +399,19 @@ expects undeclared static-filter keys to persist is skipped by name in test files to accommodate the rebrand. Mirrors `packages/api/jest.dfe.config.js`. The setup file is loaded by the jest config rather than compiled, so no tsconfig project covers it and typed linting - cannot parse it; **`packages/app/eslint.config.mjs`** carries one added - `ignores` entry for it, beside upstream's own `global-setup.js` line. The env - pin reaches the CHART palette and `getTheme`'s fallback too, because both - resolve an unmatched brand to the configured `DEFAULT_THEME`: - `detectActiveTheme` in **`src/utils.ts`** for a document carrying no theme - class, and `getTheme` in **`src/theme/index.ts`** for an unknown name. A DFE - deployment, where `NEXT_PUBLIC_THEME` is unset or `dfe`, resolves both to - `dfe` as before, and upstream's three theme assertions pass unedited. + cannot parse it. **`packages/app/eslint.config.mjs`** has one added `ignores` + entry for it, beside upstream's own `global-setup.js` line. The env pin + reaches the CHART palette and `getTheme`'s fallback too, because both resolve + an unmatched brand to the configured `DEFAULT_THEME`: `detectActiveTheme` in + **`src/utils.ts`** for a document carrying no theme class, and `getTheme` in + **`src/theme/index.ts`** for an unknown name. A DFE deployment, where + `NEXT_PUBLIC_THEME` is unset or `dfe`, resolves both to `dfe` as before, and + upstream's three theme assertions pass unedited. `src/dfe/__tests__/themeFallback.test.ts` pins the DFE side. - `packages/app/src/dfe/defaultSource.ts` - which source `/search` opens on cold. The embed opens on `main`, the landing every record falls into, because - a first Observe load that shows an empty window reads as a broken deployment; - a standalone page opens on `hunts`, since an analyst working the console + a first Observe load that shows an empty window reads as a broken deployment. + A standalone page opens on `hunts`, since an analyst working the console directly is working detections. Both are matched by NAME, which is the contract - the console never sees a HyperDX id - and both are overridable per deployment (`NEXT_PUBLIC_DFE_EMBED_DEFAULT_SOURCE`, @@ -422,7 +436,7 @@ expects undeclared static-filter keys to persist is skipped by name in Playwright 1.57.0 ships no bundled Chromium for Ubuntu 26.04 and `playwright install chromium` refuses for that platform, so a DFE dev host can never fetch the pinned revision. The config sets `channel: 'chrome'` per - project and raises the webServer budget (`E2E_APP_SERVER_TIMEOUT_MS`); the + project and raises the webServer budget (`E2E_APP_SERVER_TIMEOUT_MS`). The setup file exists because global setup calls `chromium.launch()` directly and so never sees the project config, and Playwright 1.57 honours no environment override for that call. @@ -451,7 +465,7 @@ expects undeclared static-filter keys to persist is skipped by name in `pages/_document.tsx` carries one added inline head script (`EMBED_INIT_SCRIPT`, alongside upstream's own `THEME_INIT_SCRIPT`) that sets `html.dfe-embed` from the `embed=1` URL param / persisted flag before - hydration; `styles/globals.css` hides `.dfe-appnav-slot` (the layout.tsx + hydration, and `styles/globals.css` hides `.dfe-appnav-slot` (the layout.tsx wrapper) under that class, so the full chrome sidebar never flashes before React removes it. - `packages/app/src/dfe/embedCsp.ts` + **`packages/app/proxy.ts`** - who may put @@ -481,9 +495,11 @@ expects undeclared static-filter keys to persist is skipped by name in each would buy two more conflict points for images nobody here builds. An image built from either sends no framing header. -- **Alerting is disabled, not removed.** HyperDX's alert checker is not started - and its routes are hidden by the embed nav gating. Detection is the DFE rules - engine's job. Rationale: +- **Alerting is disabled, not removed.** HyperDX's alert checker still starts + with the image, but nobody can create an alert for it. `/alerts` and + `/webhooks` refuse every human session, the engine's service token reaches + team, source and connection routes alone, and the embed sends `/alerts` back + to `/search`. Detection is the DFE rules engine's job. Rationale: [../decisions/0002-alerting-disabled-for-dfe-rules.md](../decisions/0002-alerting-disabled-for-dfe-rules.md). - `packages/api/src/dfe/middleware/provisioned-lockdown.ts` - 403 on PATCH and PUT against a dashboard the provisioner owns. Not polish: the provisioner runs @@ -500,9 +516,9 @@ expects undeclared static-filter keys to persist is skipped by name in item that POSTs a team-owned copy through the existing create call. - **Delete is allowed, and tombstoned.** `blockProvisionedWrites` lets DELETE through and records `{name, team, deletedAt}` in - `dfe/models/dashboard-tombstone.ts` once the route has actually deleted the - row; `syncDashboards` skips a tombstoned name, so the delete survives the next - cron tick AND an upgrade that ships a newer version of that dashboard. + `dfe/models/dashboard-tombstone.ts` once the route has deleted the row. + `syncDashboards` skips a tombstoned name, so the delete survives the next cron + tick AND an upgrade that ships a newer version of that dashboard. `dfe/routers/shipped-dashboards.ts` (`POST /dfe/dashboards/restore-shipped`) is the way back - it clears the team's tombstones and reprovisions inside the request, driven from `app/src/dfe/components/RestoreShippedDashboards/`. Team @@ -511,11 +527,11 @@ expects undeclared static-filter keys to persist is skipped by name in - **`packages/api/src/tasks/provisionDashboards/index.ts` - an edit to a pristine upstream file** (catalogued in `.fork-surface`). `syncDashboards` wrote tiles verbatim, so a file naming a source by name stored a name where an - ObjectId belongs and the tile rendered dead; upstream's own - `AUTO_PROVISION.md` example has the same bug. `resolveDashboardRefs` matches - source and connection names case-insensitively, as `DBDashboardImportPage` - does interactively. Unresolvable references pass through unchanged by default, - so upstream's behaviour and its dangling-source fixture still hold; + ObjectId belongs and the tile rendered dead. Upstream's `AUTO_PROVISION.md` + example has the same bug. `resolveDashboardRefs` matches source and connection + names case-insensitively, as `DBDashboardImportPage` does interactively. + Unresolvable references pass through unchanged by default, so upstream's + behaviour and its dangling-source fixture still hold, and `DASHBOARD_PROVISIONER_REQUIRE_REFS=true` skips the dashboard instead. That flag is the RBAC mechanism for the DFE set: a tenant team holds no otel source, so the platform dashboards never resolve for it. Content is @@ -557,9 +573,9 @@ expects undeclared static-filter keys to persist is skipped by name in failed request "Request failed with status code 400 Bad Request", and the reason sits in the response body, so every toast built from `error.message` hid it. A 4xx error now carries the body's `message`, a schema rejection's - issues, or the fork's own `{ error }`. **`api.ts` (catalogued)** gains one - import and one entry in the `ky.create` hooks. **`DBSearchPage.tsx` - (catalogued)** passes its two saved-search error toasts through upstream's own + issues, or the fork's `{ error }`. **`api.ts` (catalogued)** gains one import + and one entry in the `ky.create` hooks. **`DBSearchPage.tsx` (catalogued)** + passes its two saved-search error toasts through upstream's own `getApiErrorMessage`, which keeps the existing copy as the fallback. `dfe/__tests__/apiErrorDetail.test.ts` covers the response bodies and `forkDeltas.test.ts` pins the wiring. @@ -574,20 +590,20 @@ expects undeclared static-filter keys to persist is skipped by name in ## Config and bootstrap - `packages/api/src/dfe/config.ts` - auth mode, header names, default team -- `.env.dfe-example` - DFE env template +- `.env.dfe.example` - DFE env template ## Branding and frontend - `packages/app/src/theme/themes/dfe/**` - tokens, Mantine theme, logomark and wordmark assets. Typography is Inter + IBM Plex Mono - see ADR [0005](../decisions/0005-console-typography.md). -- `packages/app/src/dfe/components/**` - AppNav, SidebarMenu, ThemeToggle, - UserActionsButton, LandingPage +- `packages/app/src/dfe/components/**` - LandingPage, CreateRuleFromSearch and + RestoreShippedDashboards - Search language defaults to SQL, not upstream's Lucene (an explicit user - selection still wins). One seam: `getStoredLanguage()` in + selection still wins). One function: `getStoredLanguage()` in `components/SearchInput/SearchWhereInput.tsx` returns `'sql'` instead of null - when nothing is stored, which flips every `?? 'lucene'` fallback at once; the - three sites that bypassed the seam (`DBSearchPage.tsx` saved-search default, + when nothing is stored, which flips every `?? 'lucene'` fallback at once. The + three sites that bypassed it (`DBSearchPage.tsx` saved-search default, `DBDashboardPage.tsx` URL-parser default, `ContextSidePanel.tsx` destructure default) now consult it. Callers' dead `?? 'lucene'` tails are deliberately left to minimise the upstream diff. Pinned by @@ -597,14 +613,15 @@ expects undeclared static-filter keys to persist is skipped by name in - `.hyperi-ci.yaml`, `.github/workflows/ci.yml` - migrated to hyperi-ci - `.github/workflows/{upstream-sync,upstream-drift,fork-surface,fork-security}.yml` - - the fork machinery + - the fork machinery, and `dfe-documentdb.yml`, which runs the api integration + suite on DocumentDB - `.gitleaks.toml`, `scripts/audit.sh` - `packages/api/jest.dfe.config.js` - upstream's unit config plus one transform: `jose` is ESM-only and Jest's CJS loader cannot load it, so the tests exercise real ES384 verification rather than a crypto double. It deliberately does NOT - narrow `testMatch` - it used to, back when upstream's api package had no unit - target, and 2.33.0 added one. A scoped config would run our handful of tests - while silently skipping upstream's several hundred. + narrow `testMatch`: upstream's api package has a unit target since 2.33.0, and + a scoped config would run our handful of tests while skipping upstream's + several hundred. - `packages/api/jest.dfe.int.config.js` - upstream's integration config with `jose` added to its transform list, derived from upstream's list so packages they add keep loading. `api-app.ts` reaches `jose` through the DFE routers, so @@ -631,16 +648,16 @@ expects undeclared static-filter keys to persist is skipped by name in is the guard patch 0004 mounts, and goes when that patch does. - `.gitattributes` - ONE added line routing `yarn.lock` to `scripts/merge-lockfile.sh` -- `packages/app/src/dfe/fonts.ts` + `dfe/fonts/` - the four UI fonts vendored as - woff2 and loaded with `next/font/local`, so `next build` fetches nothing from - Google Fonts. Every subset Google serves is vendored with its unicode-range, - and `scripts/vendor-fonts.py` regenerates the files and `dfe/fonts.ts`. The - delta in **`packages/app/src/fonts.ts`** is its import line: the loaders come - from `@/dfe/fonts` instead of `next/font/google`, and upstream's four calls - stay byte-identical. Those calls are checked at build time, so a weight or CSS - variable upstream adds that we do not vendor fails the build instead of - rendering a synthesised weight. Sources and licences: `dfe/fonts/README.md`. - Pinned by `dfe/__tests__/forkDeltas.test.ts`. +- `packages/app/src/dfe/fonts.ts` + `dfe/fonts/` - the four UI fonts copied in + as woff2 and loaded with `next/font/local`, so `next build` fetches nothing + from Google Fonts. Every subset Google serves is copied in with its + unicode-range, and `scripts/vendor-fonts.py` regenerates the files and + `dfe/fonts.ts`. The delta in **`packages/app/src/fonts.ts`** is its import + line: the loaders come from `@/dfe/fonts` instead of `next/font/google`, and + upstream's four calls stay byte-identical. Those calls are checked at build + time, so a weight or CSS variable upstream adds that we do not copy in fails + the build instead of rendering a synthesised weight. Sources and licences: + `dfe/fonts/README.md`. Pinned by `dfe/__tests__/forkDeltas.test.ts`. - `knip.json` - added ignores so the pre-commit hook can run. Two classes, and neither is ours to fix. Upstream page components orphaned because this fork removed their routes (`pages/{benchmark,clickhouse,join-team,kubernetes,` @@ -656,9 +673,7 @@ expects undeclared static-filter keys to persist is skipped by name in by advisory id, never by package name - `npmAuditExcludePackages` would mute the next advisory against the same package too. Each id carries its reason and the traces are in [security-sync.md](security-sync.md). Upstream churns this - file rarely, so the conflict is small. `npmPreapprovedPackages` also carries a - temporary `body-parser` line that admits 1.20.9 past `npmMinimalAgeGate`, and - it goes on 2026-10-29. + file rarely, so the conflict is small. - `.github/workflows/docker-build.yml` - three deltas. The first two are for the same failure (#51). A `Free disk space` step calling @@ -679,10 +694,10 @@ expects undeclared static-filter keys to persist is skipped by name in describes ARC NODES for the ClickHouse build and does not apply to this pod. So the script keeps a lever for each and the assert covers both. A hosted - runner has ~8 GB of toolchains for languages we never build; a shared daemon - has a buildkit cache instead. It measures the filesystem holding docker's - data-root rather than `/`, since those are not always the same mount. Gated - exactly like the steps around it, so a skipped run pays nothing. + runner has ~8 GB of toolchains for languages we never build, and a shared + daemon has a buildkit cache instead. It measures the filesystem holding + docker's data-root rather than `/`, since those are not always the same mount. + Gated exactly like the steps around it, so a skipped run pays nothing. The third delta pins the seven `uses:` lines by commit SHA: three of `docker/setup-buildx-action` at v4.4.1 (2026-09-16) and four of @@ -697,7 +712,7 @@ expects undeclared static-filter keys to persist is skipped by name in the 16 upstream workflows we do not carry. `.fork-surface` cannot cover a deletion: it reads `--diff-filter=ACMR` against the merge base, where a file we removed is unchanged and therefore invisible, so a sync reinstates it in - silence. The check simply fails when a listed path exists. + silence. The check fails when a listed path exists. - `scripts/ci/__tests__/**` in `knip.json` - upstream's ratchet test. Only upstream's `main.yml` ever ran it, and we do not carry that workflow, so knip is correct that nothing uses it. Ignored rather than deleted, on the same @@ -724,27 +739,27 @@ expects undeclared static-filter keys to persist is skipped by name in ## Docs Our documentation lives under [docs/](../README.md) and follows HyperI -standards. Upstream's docs - `AGENTS.md`, `agent_docs/`, `MCP.md`, `LOCAL.md`, -`DEPLOY.md`, `CONTRIBUTING.md` - are left exactly as upstream ships them. - -`CLAUDE.md` and `README.md` are the two shared files - both exist upstream, both -carry a catalogued delta, both are listed in `.fork-surface`. +standards. Upstream's docs, `AGENTS.md`, `agent_docs/`, `MCP.md`, `LOCAL.md`, +`DEPLOY.md` and `CONTRIBUTING.md`, are upstream's, and the files below are the +ones that carry a delta, each listed in `.fork-surface`. - **`CLAUDE.md`** - upstream ships a one-line file, and we prepend the fork's governing rule above it so it is unmissable at the top of every agent session. Upstream rarely touches that line, so the conflict is small and rerere replays it. -- **`README.md`** - rewritten to describe this fork rather than the upstream - product, plus a `## Context` section carrying the working guide the HyperI doc - standard specifies. Its five subheadings are fixed and checked by - `hyperi-ai docs context --check`, so the section cannot drift. The section is - APPENDED at end of file, which is the cheapest placement we have: upstream's - own README edits land above it, so an upstream change and ours rarely occupy - the same hunk. +- **`README.md`** - rewritten whole to describe this fork rather than the + upstream product. The working guide that once sat at its end is + [working-guide.md](working-guide.md). Any upstream edit to the README + conflicts with it. - **`docs/alert-webhook-template-variables.md`** - an upstream page the DFE docs - site publishes to customers. Its three mentions of HyperDX say "the console" - instead, and nothing else on the page changes. It is listed in - `.fork-surface`. + site publishes to customers. Three of its mentions of HyperDX say "the + console" instead, and one "today" is gone. The whole file is also + prettier-formatted, so every line upstream touches conflicts. +- `LOCAL.md`, `agent_docs/tech_stack.md`, `docker/nginx/README.md`, + `docker/otel-collector/schema/README.md`, `packages/app/tests/e2e/README.md` + and `packages/common-utils/README.md` differ from the merge base by prettier + formatting alone. Formatting is no reason to hold a delta, so each is a + candidate to put back to upstream's text and take off `.fork-surface`. --- @@ -777,5 +792,3 @@ built on" for any release we have cut. - [security-sync.md](security-sync.md) - what we have learnt about the inherited dependency tree, one dated section per sync - [leaving-upstream.md](leaving-upstream.md) - how this ends -- The `x-oidc-*` contract is the universal seam shared with the rest of DFE - - see the dfe-engine OIDC dual-mode design From 021d4fbc95691a8ffbab280d6e7388c0e581a6b2 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 16 Nov 2026 16:25:34 +1100 Subject: [PATCH 2/2] docs(fork): clarify edge OIDC defaults and drop caps from dev command The gateway chart puts no OIDC policy on the HyperDX route by default. The OIDC page names the three settings that change this default: routes.hyperdx.edgePolicy, oidc.targetRoutes and jwtAuthn. The docker-local dev command drops the caps files. compose rejects those files. The container command is marked as needing a caps file that loads. --- docs/architecture/oidc-authentication.md | 12 +++++++----- docs/development/docker-local.md | 8 +++----- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/architecture/oidc-authentication.md b/docs/architecture/oidc-authentication.md index 691cbef02..cb0137414 100644 --- a/docs/architecture/oidc-authentication.md +++ b/docs/architecture/oidc-authentication.md @@ -6,11 +6,13 @@ than trusting a header.** The user signs in on the DFE console, through the engine, which is the OIDC relying party to the customer's identity provider. The console plants the engine's ES384 token in the `dfe_token` cookie, scoped by `DFE_COOKIE_DOMAIN` to -the parent domain so the HyperDX subdomain receives it. The gateway puts no OIDC -policy on the HyperDX route, because a provider redirect cannot complete inside -the console's iframe. So dfe-hyperdx verifies the token against the engine's -JWKS: it is the policy enforcement point, and an unverified header is not an -identity. +the parent domain so the HyperDX subdomain receives it. By default the gateway +chart puts no OIDC policy on the HyperDX route, because a provider redirect +cannot complete inside the console's iframe. An operator changes that by setting +`routes.hyperdx.edgePolicy`, by adding hyperdx to `oidc.targetRoutes`, or by +enabling `jwtAuthn`, a Gateway-wide filter on `/`. Either way dfe-hyperdx +verifies the token against the engine's JWKS: it is the policy enforcement +point, and an unverified header is not an identity. In DFE mode HyperDX's password login, registration and invite acceptance answer 404 (`dfe/middleware/legacy-auth-lockdown.ts`). None of this touches upstream's diff --git a/docs/development/docker-local.md b/docs/development/docker-local.md index f883ad0fa..f59a34285 100644 --- a/docs/development/docker-local.md +++ b/docs/development/docker-local.md @@ -16,14 +16,12 @@ whether the app runs locally (dev, hot reload) or in a container. For dev, `yarn dev:dfe` is the flow ([getting-started.md](getting-started.md)): it starts DocumentDB in Docker and runs the API and app locally. The DFE dev override scales ClickHouse and both collectors to zero, so ingest and ClickHouse -come from dfe-docker or a ClickHouse you point `DEFAULT_CONNECTIONS` at. To cap -what it starts, bring the infra up yourself with the caps layers, then run the -app: +come from dfe-docker or a ClickHouse you point `DEFAULT_CONNECTIONS` at. To +bring the infra up yourself and run the app separately: ```sh docker compose --env-file .env.dfe \ -f docker-compose.dev.yml -f docker-compose.dfe.dev.yml \ - -f docker-compose.caps.yml -f docker-compose.caps.dev.yml \ up -d documentdb dotenvx run -f .env.dfe -f .env -- yarn app:dev:dfe @@ -32,7 +30,7 @@ dotenvx run -f .env.dfe -f .env -- yarn app:dev:dfe App: , API: . It needs `.env.dfe` (dev credentials, gitignored, copied from `.env.dfe.example`). -To run the app in a container, the closest to k8s: +To run the app in a container, the closest to k8s, once the caps file loads: ```sh docker compose --env-file .env --env-file .env.dfe \