diff --git a/docs/architecture.md b/docs/architecture.md index 628c20c..9511eff 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -22,7 +22,7 @@ repo, not here. ## Events arrive two ways and leave one Pushed to `dfe-receiver` or pulled by `dfe-fetcher`, and out through `dfe-loader` -into ClickHouse. Everything between is optional and profile-selected. This diagram shows the running parts of the data path on a Kafka profile. +into ClickHouse. Everything between is optional and profile-selected. On a Kafka profile, the receiver and fetcher produce to Kafka, and the loader reads the topics into ClickHouse. ```mermaid flowchart LR @@ -97,7 +97,7 @@ A source with a transform stops sharing `main_land`: the receiver routes it to ` ## Two independent selectors decide what runs -Compose profiles select INFRASTRUCTURE. `service_profiles.yaml` selects DFE SERVICES. `scripts/resolve_profile.py` reads the second and derives the first, writing `.tmp/profile.mk` for the Makefile. This diagram shows where the resolver reads from and what reads its output. +Compose profiles select INFRASTRUCTURE. `service_profiles.yaml` selects DFE SERVICES. `scripts/resolve_profile.py` reads the second and derives the first, writing `.tmp/profile.mk` for the Makefile. The resolver reads the profile and the environment, and the Makefile includes the `profile.mk` it writes. ```mermaid flowchart LR @@ -137,7 +137,7 @@ flowchart LR A profile declares its whole footprint through the optional `clickhouse`, `core`, `kafbat`, `hyperdx`, `otel` and `auth` keys, each overridable by the matching `.env` flag. Each service entry names the config file it mounts, which is how one service gets a Kafka config and another a gRPC one without a second compose file. -The `core` footprint is the management plane, independent of transport. `dfe-proxy` gives the console and the engine API one origin, because the console's client calls the engine with a relative `/api/v1/...` base URL. This diagram shows how dfe-proxy splits that origin. +The `core` footprint is the management plane, independent of transport. `dfe-proxy` gives the console and the engine API one origin, because the console's client calls the engine with a relative `/api/v1/...` base URL. dfe-proxy sends `/api/v1` to dfe-engine and every other path to dfe-ui. ```mermaid flowchart LR @@ -197,7 +197,7 @@ Dev builds fetch source into a managed git cache, or from your own checkouts whe dfe-transform-vector mounts `resources/enrichment/` at `/etc/dfe-transform-vector/data`, where its bundled filebeat pipeline reads `timezones.csv`. dfe-transform-elastic compiles its transforms in, and the engine uploads the enrichment tables a dfe-transform-vrl program reads, so neither mounts them. `make test-source` drives the filebeat, Vector and Elastic cases through every hop. dfe-transform-vector's one port serves everything: the wrapper scrapes Vector's `internal_metrics` off its `prometheus_exporter` on loopback and registers the samples on scalo's own recorder (dfe-transform-vector#70), so `vector_*` reads on its metrics port and rides the OTLP push. The exporter's 9598 is not published. -Two gRPC listeners share port 6000, and they are not the same thing. This diagram shows both on a `*-grpc` profile. +Two gRPC listeners share port 6000, and they are not the same thing. A `*-grpc` profile runs both. ```mermaid flowchart LR diff --git a/docs/developing.md b/docs/developing.md index 4f82953..da3f43c 100644 --- a/docs/developing.md +++ b/docs/developing.md @@ -89,7 +89,7 @@ make ci # pull the pinned GHCR images, then start `make dev` builds the `:local` images, then starts with the overlay the build writes, `.tmp/docker-compose.local.yml`, which repoints each component it built at `:local` and empties the engine's `DFE_ENGINE_STACK_IMAGES`, so the console shows `DFE_STACK_VERSION`. `make ci` names no overlay. Both run `make env-files` first, which creates any `env/.env` a release added and warns, never edits, when a template adds a key. `scripts/build_dev_images.py` builds the `:local` images in two stages for every -Rust component. This diagram shows the two stages. +Rust component. The repo's Rust builder compiles the binary, and the component's Dockerfile packages it. ```mermaid flowchart LR diff --git a/docs/known-issues.md b/docs/known-issues.md index 3289c06..5c7749a 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -32,7 +32,7 @@ KAFKA_BACKEND=apache make -e dev # -e flips the precedence ## A `_load` topic changes what the loader reads -The receiver produces to `main_land` and a transform to `main_load`. This diagram shows which of the two the loader reads. +The receiver produces to `main_land` and a transform to `main_load`. The loader reads `main_load` when it exists, and `main_land` when it does not. ```mermaid flowchart LR diff --git a/docs/observability.md b/docs/observability.md index daa91b9..f27fbca 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -67,7 +67,7 @@ Kubernetes has separate probes and ignores `HEALTHCHECK`, so it never chooses. ## Self-telemetry is pushed, bar four pulls The stack's own telemetry leaves by a different door from the data it ingests, so -a broken ingest pipeline cannot take the reporting on it down. This diagram shows what feeds the collector and who reads what it writes. +a broken ingest pipeline cannot take the reporting on it down. Services push to the collector, the collector pulls ClickHouse and the host, and dfe-hyperdx queries what it writes. ```mermaid flowchart LR diff --git a/docs/operating.md b/docs/operating.md index fd5e016..5656bd8 100644 --- a/docs/operating.md +++ b/docs/operating.md @@ -24,7 +24,7 @@ The engine issues `admin` with a forced change at first login, which `make post` dfe-hyperdx, the console's Search, authenticates by verifying the engine's token and holds no logins of its own. ClickHouse takes its generated password. Nothing else authenticates. The ingest edges, every metrics port and Kafka answer whoever can route to them, and so does Kafbat unless the `auth` profile is armed, where the gate is at the proxy, not inside the UI. -This diagram shows which host ports a client off the box can reach at the defaults. +By default, off-box clients reach the receiver's ingest ports, and every other port answers on the box alone. ```mermaid flowchart TB diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 51fb643..17236d4 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -55,7 +55,7 @@ dfe-hyperdx logs there. ClickHouse, the broker and the proxies keep `json-file`. The ingest edge returning 2xx only means the event was accepted. Work down this order. -This diagram shows the loader checked before the schema. +Check the loader is ready before checking the schema. ```mermaid flowchart LR @@ -91,7 +91,7 @@ If the engine has not provisioned, the loader holds every message pending-schema and dead-letters after a timeout. Check `dfe.main` exists and that the engine is healthy. -On Kafka, this diagram shows the subscription checked before the dead letters. +On Kafka, check the loader's subscription before the dead letters. ```mermaid