Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ repo, not here.
## Events arrive two ways and leave one

Pushed to `dfe-receiver` or pulled by `dfe-fetcher`, and out through `dfe-loader`
into ClickHouse. Everything between is optional and profile-selected. This diagram shows the running parts of the data path on a Kafka profile.
into ClickHouse. Everything between is optional and profile-selected. On a Kafka profile, the receiver and fetcher produce to Kafka, and the loader reads the topics into ClickHouse.

```mermaid
flowchart LR
Expand Down Expand Up @@ -97,7 +97,7 @@ A source with a transform stops sharing `main_land`: the receiver routes it to `

## Two independent selectors decide what runs

Compose profiles select INFRASTRUCTURE. `service_profiles.yaml` selects DFE SERVICES. `scripts/resolve_profile.py` reads the second and derives the first, writing `.tmp/profile.mk` for the Makefile. This diagram shows where the resolver reads from and what reads its output.
Compose profiles select INFRASTRUCTURE. `service_profiles.yaml` selects DFE SERVICES. `scripts/resolve_profile.py` reads the second and derives the first, writing `.tmp/profile.mk` for the Makefile. The resolver reads the profile and the environment, and the Makefile includes the `profile.mk` it writes.

```mermaid
flowchart LR
Expand Down Expand Up @@ -137,7 +137,7 @@ flowchart LR

A profile declares its whole footprint through the optional `clickhouse`, `core`, `kafbat`, `hyperdx`, `otel` and `auth` keys, each overridable by the matching `.env` flag. Each service entry names the config file it mounts, which is how one service gets a Kafka config and another a gRPC one without a second compose file.

The `core` footprint is the management plane, independent of transport. `dfe-proxy` gives the console and the engine API one origin, because the console's client calls the engine with a relative `/api/v1/...` base URL. This diagram shows how dfe-proxy splits that origin.
The `core` footprint is the management plane, independent of transport. `dfe-proxy` gives the console and the engine API one origin, because the console's client calls the engine with a relative `/api/v1/...` base URL. dfe-proxy sends `/api/v1` to dfe-engine and every other path to dfe-ui.

```mermaid
flowchart LR
Expand Down Expand Up @@ -197,7 +197,7 @@ Dev builds fetch source into a managed git cache, or from your own checkouts whe

dfe-transform-vector mounts `resources/enrichment/` at `/etc/dfe-transform-vector/data`, where its bundled filebeat pipeline reads `timezones.csv`. dfe-transform-elastic compiles its transforms in, and the engine uploads the enrichment tables a dfe-transform-vrl program reads, so neither mounts them. `make test-source` drives the filebeat, Vector and Elastic cases through every hop. dfe-transform-vector's one port serves everything: the wrapper scrapes Vector's `internal_metrics` off its `prometheus_exporter` on loopback and registers the samples on scalo's own recorder (dfe-transform-vector#70), so `vector_*` reads on its metrics port and rides the OTLP push. The exporter's 9598 is not published.

Two gRPC listeners share port 6000, and they are not the same thing. This diagram shows both on a `*-grpc` profile.
Two gRPC listeners share port 6000, and they are not the same thing. A `*-grpc` profile runs both.

```mermaid
flowchart LR
Expand Down
2 changes: 1 addition & 1 deletion docs/developing.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ make ci # pull the pinned GHCR images, then start
`make dev` builds the `:local` images, then starts with the overlay the build writes, `.tmp/docker-compose.local.yml`, which repoints each component it built at `<service>:local` and empties the engine's `DFE_ENGINE_STACK_IMAGES`, so the console shows `DFE_STACK_VERSION`. `make ci` names no overlay. Both run `make env-files` first, which creates any `env/<service>.env` a release added and warns, never edits, when a template adds a key.

`scripts/build_dev_images.py` builds the `:local` images in two stages for every
Rust component. This diagram shows the two stages.
Rust component. The repo's Rust builder compiles the binary, and the component's Dockerfile packages it.

```mermaid
flowchart LR
Expand Down
2 changes: 1 addition & 1 deletion docs/known-issues.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ KAFKA_BACKEND=apache make -e dev # -e flips the precedence

## A `_load` topic changes what the loader reads

The receiver produces to `main_land` and a transform to `main_load`. This diagram shows which of the two the loader reads.
The receiver produces to `main_land` and a transform to `main_load`. The loader reads `main_load` when it exists, and `main_land` when it does not.

```mermaid
flowchart LR
Expand Down
2 changes: 1 addition & 1 deletion docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Kubernetes has separate probes and ignores `HEALTHCHECK`, so it never chooses.
## Self-telemetry is pushed, bar four pulls

The stack's own telemetry leaves by a different door from the data it ingests, so
a broken ingest pipeline cannot take the reporting on it down. This diagram shows what feeds the collector and who reads what it writes.
a broken ingest pipeline cannot take the reporting on it down. Services push to the collector, the collector pulls ClickHouse and the host, and dfe-hyperdx queries what it writes.

```mermaid
flowchart LR
Expand Down
2 changes: 1 addition & 1 deletion docs/operating.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The engine issues `admin` with a forced change at first login, which `make post`

dfe-hyperdx, the console's Search, authenticates by verifying the engine's token and holds no logins of its own. ClickHouse takes its generated password. Nothing else authenticates. The ingest edges, every metrics port and Kafka answer whoever can route to them, and so does Kafbat unless the `auth` profile is armed, where the gate is at the proxy, not inside the UI.

This diagram shows which host ports a client off the box can reach at the defaults.
By default, off-box clients reach the receiver's ingest ports, and every other port answers on the box alone.

```mermaid
flowchart TB
Expand Down
4 changes: 2 additions & 2 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ dfe-hyperdx logs there. ClickHouse, the broker and the proxies keep `json-file`.
The ingest edge returning 2xx only means the event was accepted. Work down this
order.

This diagram shows the loader checked before the schema.
Check the loader is ready before checking the schema.

```mermaid
flowchart LR
Expand Down Expand Up @@ -91,7 +91,7 @@ If the engine has not provisioned, the loader holds every message pending-schema
and dead-letters after a timeout. Check `dfe.main` exists and that the engine
is healthy.

On Kafka, this diagram shows the subscription checked before the dead letters.
On Kafka, check the loader's subscription before the dead letters.


```mermaid
Expand Down
Loading