Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 32 additions & 28 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
DFE's fork of [HyperDX](https://github.com/hyperdxio/hyperdx), embedded in the
DFE platform as its visualisation and search layer.

Two questions bring people here, and they are answered in different places:
Four questions bring people here, and they are answered in different places:

| Question | Start at |
| --- | --- |
Expand All @@ -12,44 +12,48 @@ Two questions bring people here, and they are answered in different places:
| How do I run it locally? | [development/](development/README.md) |
| Why was it built this way? | [decisions/](decisions/README.md) |

---

## What is ours and what is upstream's

This matters more here than in a normal repo, because the answer decides whether
a change is free or expensive.
Which files are ours and which are upstream's matters more here than in a normal repo, because the answer decides whether
a change is free or expensive. Our files sit beside upstream's, and HyperI
standards apply to ours alone.

```mermaid
flowchart TB
subgraph ours["OURS - HyperI standards apply"]
d["docs/"]
flowchart LR
accTitle: L4 Detail -- Ours and upstream's files
accDescr: Ours are our pages under docs, every dfe directory under packages, the security and githooks directories, our scripts and the workflows we added. Upstream's are agent_docs, the root agent and contributor docs, and everything under packages outside a dfe directory. Ownership is per file, and .fork-surface lists the upstream files we edit.

subgraph ours["Ours, HyperI standards apply"]
direction TB
d["Our pages<br/>under docs/"]
dfe["packages/**/dfe/**"]
tools["scripts/, security/, .githooks/"]
wf[".github/workflows/<br/>ci, fork-*, upstream-*"]
tools["security/, .githooks/,<br/>our scripts/"]
wf["Our workflows"]
end

subgraph theirs["UPSTREAM - left exactly as shipped"]
subgraph theirs["Upstream's"]
direction TB
ad["agent_docs/"]
rootmd["AGENTS.md, MCP.md,<br/>LOCAL.md, DEPLOY.md,<br/>CONTRIBUTING.md"]
code["everything under packages/<br/>outside dfe/"]
rootmd["AGENTS.md, MCP.md, LOCAL.md,<br/>DEPLOY.md, CONTRIBUTING.md"]
code["packages/ outside dfe/"]
end

classDef mine fill:#0072B2,stroke:#00456b,color:#ffffff
classDef up fill:#009E73,stroke:#005f45,color:#ffffff
class d,dfe,tools,wf mine
class ad,rootmd,code up
```
ours ~~~ theirs

**Our files follow HyperI documentation standards. Upstream's files are left
untouched, including their docs.** Applying our conventions to `agent_docs/` or
`AGENTS.md` would buy nothing and create conflict surface on every sync.
classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef ext fill:#EEF0F2,stroke:#6B7280,stroke-width:1px,stroke-dasharray:4 3,color:#1F2937
classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4
class ad,rootmd,code ext
class ours,theirs group
```

`README.md` and `CLAUDE.md` are the two shared files: both exist upstream, both
carry a small catalogued delta of ours, and both are listed in `.fork-surface`.
*Figure 1. L4 Detail -- Ours and upstream's files. Dashed boxes are upstream's.* Up: [dfe-hyperdx](../README.md).

---
Applying our conventions to `agent_docs/` or `AGENTS.md` would buy nothing and
create conflict surface on every sync. Where an upstream file does have a delta
of ours, `.fork-surface` lists it and [fork/what-we-changed.md](fork/what-we-changed.md)
says why. `README.md`, `CLAUDE.md` and `docs/alert-webhook-template-variables.md`
are the shared docs: upstream ships each, and each has a small delta of ours.

## The documentation set
The documentation set:

```text
docs/
Expand All @@ -60,4 +64,4 @@ docs/
```

Everything here is plain CommonMark plus fenced mermaid, so it renders on GitHub
and drops into a docs generator without a rewrite.
without a build step.
178 changes: 87 additions & 91 deletions docs/architecture/README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,23 @@
# Architecture

HyperDX is DFE's **visualisation and search layer**: engineers use it to search
logs, view traces, build dashboards and replay sessions. Detection, alerting and
response are the DFE platform's job, not HyperDX's.
logs, view traces and build dashboards, inside the DFE console. Detection,
alerting and response are the DFE platform's job, not HyperDX's.

Five things make our deployment different from stock HyperDX, and everything
else follows from them:

- Authentication is external: Envoy terminates OIDC and passes identity
headers, and HyperDX never sees a password.
- Authorization is the DFE engine's: it is the policy decision point, and
ClickHouse GRANTs enforce data access.
- MongoDB is replaced by DocumentDB, PostgreSQL with the DocumentDB
extensions and its Mongo wire-protocol gateway in one container
- Authentication is the engine's. The DFE console signs the user in through the
DFE engine and hands the embedded app the engine's token in the `dfe_token`
cookie. The API verifies that token itself, and HyperDX's own password login
is switched off.
- Authorization is the engine's as well: it decides which ClickHouse user a
session reads as, and ClickHouse GRANTs enforce data access.
- MongoDB is replaced by DocumentDB, PostgreSQL with the DocumentDB extensions
and its Mongo wire-protocol gateway in one container
([ADR 0006](../decisions/0006-documentdb-document-store.md)).
- The DFE 2.1 discovery tool's enhancements are ported in.
- Rules are generated from the query you are iterating in front of you.
- The DFE 2.1 discovery tool's enhancements are ported in, including turning the
search in front of you into a DFE rule.
- JSON is native throughout, the ClickHouse JSON type included.

What we changed to achieve that, and how we keep it working across upstream
Expand All @@ -25,106 +27,100 @@ releases, is [the fork documentation](../fork/README.md).

## The system

The console frames the HyperDX UI, and the API behind it answers to the engine
for every session and reads ClickHouse as the team's own user.

```mermaid
flowchart TB
browser(["User browser"])
oidc(["OIDC provider<br/>Google / Entra ID"])
flowchart LR
accTitle: L2 Services -- dfe-hyperdx
accDescr: The DFE console frames the HyperDX UI, which calls the HyperDX API. The API asks the DFE engine about each session, queries ClickHouse and keeps its documents in DocumentDB. The dashboard provisioner writes shipped dashboards to DocumentDB.

subgraph apps["Instrumented applications"]
sdks["OTel SDKs<br/>browser, Node, Python"]
end
console["DFE console<br/>dfe-ui"]

subgraph dfe["DFE platform"]
envoy["Envoy<br/>OIDC proxy"]
ui["DFE UI"]
rules["DFE rules engine<br/>detection + hunts"]
subgraph hdx["dfe-hyperdx"]
app[["HyperDX UI<br/>Next.js"]]
api[["HyperDX API<br/>Express"]]
provisioner["Dashboard<br/>provisioner"]
end

subgraph hdx["Embedded HyperDX"]
otel["OTel collector"]
api["HyperDX API<br/>Express"]
app["HyperDX UI<br/>Next.js"]
end
engine["DFE engine"]
ch[("ClickHouse")]
docdb[("DocumentDB")]

ch[(ClickHouse<br/>telemetry)]
docdb[("DocumentDB<br/>on CloudNativePG")]
end
console -->|frames| app
app --> api
api -->|sessions| engine
api -->|queries| ch
api -->|documents| docdb
provisioner -->|dashboards| docdb

sdks -->|OTLP| otel
otel -->|writes| ch
api <-->|OpAMP| otel
api <-->|MongoDB protocol| docdb
api -->|proxies queries| ch
rules -->|queries| ch
browser -->|HTTPS| envoy
envoy -->|OIDC| oidc
envoy -->|identity headers| api
envoy -->|identity headers| ui
browser --> ui
browser --> app

classDef external fill:#F0E442,stroke:#8a8200,color:#000000
classDef ours fill:#0072B2,stroke:#00456b,color:#ffffff
classDef upstream fill:#009E73,stroke:#005f45,color:#ffffff
classDef store fill:#E69F00,stroke:#8a6100,color:#000000
class browser,oidc,sdks external
class envoy,ui,rules ours
class otel,api,app upstream
classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
classDef store fill:#DDF3EC,stroke:#00785A,stroke-width:1px,color:#0B3327
classDef group fill:none,stroke:#64748B,stroke-width:1px,stroke-dasharray:2 4
class ch,docdb store
class hdx group
```

Colour encodes ownership: blue is DFE's, green is upstream HyperDX, orange is a
datastore, yellow is outside our boundary.
*Figure 1. L2 Services -- dfe-hyperdx.* Up: top of dfe-hyperdx. Down: [L3 Components -- HyperDX UI](frontend.md#structure), [L3 Components -- HyperDX API](backend.md#structure).

One container runs the UI, the API and upstream's alert checker, and the dashboard provisioner when `DASHBOARD_PROVISIONER_DIR` is set. The engine calls the API as well, with
a service token of its own, to seed teams, sources and connections. No OTel
collector of HyperDX's runs in DFE: the DFE pipeline writes the telemetry
tables.

---

## Where things live

`app`, `api` and `cli` all build on `common-utils`, which holds the query
engine.

```mermaid
flowchart LR
app["packages/app<br/>Next.js frontend"]
api["packages/api<br/>Express backend"]
cu["packages/common-utils<br/>query engine"]
cli["packages/cli<br/>terminal UI"]
oc["packages/otel-collector<br/>Go, schema migrations"]
eval["packages/hdx-eval<br/>MCP eval harness"]
accTitle: L4 Detail -- Package dependencies
accDescr: The app, api and cli packages each depend on common-utils.

app["packages/app"]
api["packages/api"]
cli["packages/cli"]
cu["packages/<br/>common-utils"]

app --> cu
api --> cu
cli --> cu
```

| Package | Role |
| ------------------------- | ---------------------------------------------------------------- |
| `@hyperdx/app` | Frontend: search, dashboards, session replay |
| `@hyperdx/api` | REST API, OpAMP server, MCP server, ClickHouse proxy |
| `@hyperdx/common-utils` | Isomorphic query engine, Lucene-to-SQL parser, Zod types |
| `@hyperdx/cli` | Terminal CLI and TUI (`hdx`) |
| `@hyperdx/otel-collector` | Custom OTel Collector build, plus ClickHouse schema migrations |
| `@hyperdx/hdx-eval` | Benchmarks the MCP server against observability scenarios |

Our additions live under `packages/api/src/dfe/**` and `packages/app/src/dfe/**`
so they cannot collide with upstream. Compose files and collector config are in
`docker/`.

---

## Read next

| Topic | Doc |
| --- | --- |
| How telemetry gets from an SDK to a chart | [data-flow.md](data-flow.md) |
| How a search becomes SQL | [clickhouse-data-model.md](clickhouse-data-model.md) |
| Frontend structure and state | [frontend.md](frontend.md) |
| API structure and the OpAMP/MCP servers | [backend.md](backend.md) |
| Envoy, the `x-oidc-*` contract, JWT verification | [oidc-authentication.md](oidc-authentication.md) |
| Who is allowed to see what | [authorization.md](authorization.md) |
| Which team a user lands on, its connection, the startup repair | [team-identity.md](team-identity.md) |
| Turning a saved search into a DFE rule | [query-to-rule-pipeline.md](query-to-rule-pipeline.md) |
| DocumentDB, service topology | [deployment.md](deployment.md) |
| Why we chose what we chose | [../decisions/](../decisions/) |

---

## Provenance
classDef default fill:#F8FAFC,stroke:#64748B,stroke-width:1px,color:#0F172A
```

These pages describe what the code does now. Where a design has changed, the superseded reasoning lives in [../decisions/](../decisions/).
*Figure 2. L4 Detail -- Package dependencies.* Up: [L2 Services -- dfe-hyperdx](#the-system).

| Package | Role |
| ------------------------- | -------------------------------------------------------------- |
| `@hyperdx/app` | Frontend: search, dashboards, session replay |
| `@hyperdx/api` | REST API, OpAMP server, MCP server, ClickHouse proxy |
| `@hyperdx/common-utils` | Isomorphic query engine, Lucene-to-SQL parser, Zod types |
| `@hyperdx/cli` | Terminal CLI and TUI (`hdx`) |
| `@hyperdx/otel-collector` | Custom OTel Collector build, plus ClickHouse schema migrations |
| `@hyperdx/hdx-eval` | Benchmarks the MCP server against observability scenarios |

Our additions live under `packages/api/src/dfe/**`, `packages/app/src/dfe/**`
and `packages/common-utils/src/dfe/**`, so they cannot collide with upstream.
Compose files and collector config are in `docker/`.

The rest of this section, page by page:

| Topic | Doc |
| -------------------------------------------------------------- | ------------------------------------------------------ |
| How telemetry gets from an SDK to a chart | [data-flow.md](data-flow.md) |
| How a search becomes SQL | [clickhouse-data-model.md](clickhouse-data-model.md) |
| Frontend structure and state | [frontend.md](frontend.md) |
| API structure and the OpAMP and MCP servers | [backend.md](backend.md) |
| The engine token and how the API verifies it | [oidc-authentication.md](oidc-authentication.md) |
| Who is allowed to see what | [authorization.md](authorization.md) |
| Which team a user lands on, its connection, the startup repair | [team-identity.md](team-identity.md) |
| Turning a saved search into a DFE rule | [query-to-rule-pipeline.md](query-to-rule-pipeline.md) |
| DocumentDB, service topology | [deployment.md](deployment.md) |
| Why we chose what we chose | [../decisions/](../decisions/README.md) |

These pages describe what the code does. Where a design has changed, the
superseded reasoning lives in [../decisions/](../decisions/README.md).
Loading
Loading