The fastest way to work on the engine is to run it as a plain local process -- a
uv virtualenv and one command, no containers and no Kubernetes.
make dev # -> Swagger UI at http://localhost:8003/docsmake dev exports DFE_CONFIG_DIR=./config and runs uv run dfe-engine run.
The console script dfe-engine maps to dfe_engine.api:run_dev_server, which
serves the API on DFE_ENGINE_API_PORT: 8000 in code, 8003 from .env.example, and
8003 is what make dev and the other repos' local docs assume.
Two things trip people up on a fresh clone.
- The schemas come from the
dfe-schemaspackage, whichuv syncinstalls, so there is nothing to check out.make devleavesDFE_ENGINE_SCHEMAS_DIRunset. Set it to point the engine at a tree of yours, and for nothing else.config/is a plain directory.make devpointsDFE_CONFIG_DIRat./config, the engine seeds its auth store and registries into whatever that directory is, and tolerates it being empty or a throwaway. It does NOT need to pre-exist with content. The minted signing key and per-group ClickHouse passwords go to.secrets/in the CURRENT directory (DFE_ENGINE_SECRETS_PATH, default./.secrets), not underDFE_CONFIG_DIR, so a second engine started from the same directory reuses that key. make devneeds a.env(cp .env.example .env). For auth or attributes work with no ClickHouse, setDFE_ENGINE_CLICKHOUSE_BOOTSTRAP_TABLES=falseand the engine boots with no ClickHouse at all.
To exercise the auth, accounts, groups or attributes endpoints with nothing else running -- no ClickHouse, no deploy repo:
env DFE_ENGINE_ENV=dev \
DFE_ENGINE_AUTH_ENABLED=false \
DFE_ENGINE_GITOPS_ENABLED=false \
DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_TABLES=false \
DFE_ENGINE_API_PORT=8003 \
DFE_CONFIG_DIR=/tmp/dfe-config \
uv run dfe-engine run- No token is needed.
DFE_ENGINE_AUTH_ENABLED=falsewithDFE_ENGINE_ENV=devhands every unauthenticated request theadminrole, so you can call any endpoint straight fromcurl/Swagger. The engine still loads or mints its ES384 signing key at boot. You just don't need it on this path. (The offlineauthsubcommands live on the separatedfeCLI, not thedfe-enginedaemon.) - Accounts and groups are YAML files under
DFE_CONFIG_DIR/auth. - Attribute request bodies are wrapped.
PUT /api/v1/auth/accounts/{username}/attributestakes{"attributes": {...}}, not the bare object, and theGETreturns the same wrapped object. A bare object is a 422.
With a local ClickHouse, the client is clickhouse-connect over HTTP, so the port is the HTTP one and a plain docker ClickHouse is not TLS:
env DFE_ENGINE_ENV=dev \
DFE_ENGINE_CLICKHOUSE_HOST=127.0.0.1 \
DFE_ENGINE_CLICKHOUSE_PORT=8123 \
DFE_ENGINE_CLICKHOUSE_SECURE=false \
DFE_ENGINE_CLICKHOUSE_PASSWORD=... \
DFE_CONFIG_DIR=/tmp/dfe-config \
uv run dfe-engine runGet it wrong and the engine logs SSL: WRONG_VERSION_NUMBER (TLS against a
plain port) or connection refused (the native 9000), retries for up to 180
seconds (DFE_ENGINE_CLICKHOUSE_BOOTSTRAP_WAIT_SECONDS), then serves degraded with
/readyz reporting clickhouse false while /livez stays alive. Tenant isolation
reads the ClickHouse custom setting SQL_current_tenant_id.
dfe-engine run binds the observability listener (/metrics, /livez,
/readyz) on 0.0.0.0:9090 before the API. With the dfe-docker stack up that
port is taken and the process dies with fatal: [Errno 98] Address already in use and no port named. The knob is scalo's, not DFE_ENGINE_-prefixed:
env METRICS_ADDR=127.0.0.1:9197 DFE_ENGINE_ENV=dev uv run dfe-engine runWith no collector running, the OTLP exporters aim at localhost:4317 and log
Transient error ... retrying every few seconds. scalo reads a whitespace-only
endpoint as "no endpoint", which is its off switch:
env OTEL_EXPORTER_OTLP_ENDPOINT=" " DFE_ENGINE_ENV=dev uv run dfe-engine runFront-end developers run the engine and the console as plain processes on a Mac
and keep the Rust apps in docker. The engine needs uv (it fetches the Python .python-version pins)
and make. One command boots it with the e2e helpers the console journeys call:
cp .env.e2e.example .env.e2e # once
make e2e-server E2E_NO_CLICKHOUSE=1 # no docker: ClickHouse pages answer 503
docker compose -f docker-compose.dev.yaml up -d --wait clickhouse && make e2e-server # with dockermake e2e-server starts clean every time. Config, schema copies, secrets and
artefacts live under tmp/e2e, and it drops every ClickHouse database under the
dfe_e2e prefix before it starts, so a crashed run leaves nothing for the next.
E2E_KEEP=1 reuses the last run, make e2e-clean wipes on demand, and
E2E_ENV_FILE and E2E_WORKSPACE move the env file and workspace out of the
checkout. ClickHouse users and roles are server-wide and are not swept. It stops
when something else listens on the API port, because a loopback bind there takes
the traffic. Playwright polls DFE_ENGINE_READY_URL=http://127.0.0.1:8003/readyz.
.vscode/launch.json and tasks.json hold both, plus the seed. dfe-ui has the
Engine + Console compound. Three things to know on macOS:
- Set
DFE_ENGINE_API_HOST=127.0.0.1in.env.e2eand point clients at127.0.0.1, notlocalhost. Node resolveslocalhostto::1first and the engine listens on IPv4 only. A0.0.0.0bind makes macOS ask to accept incoming connections. - Ports 5000 and 7000 belong to AirPlay Receiver. The engine uses 8003 and 9090, so nothing here collides.
METRICS_ADDR=127.0.0.1:19090moves the health listener if 9090 is busy.
dfe-ui's scripts/macos-dev-smoke.py runs the whole loop and ends in one PASS
or FAIL line. Its macOS dev smoke workflow runs it on a GitHub macOS arm64
runner.
The engine is the OIDC issuer and control plane the rest of the local stack trusts. Running dfe-engine, dfe-hyperdx, dfe-ui and a local dfe-deploy clone together, all as git clones rather than containers, the wiring is:
- dfe-hyperdx (the fork) trusts the engine's signing key over JWKS. Point the
fork at
DFE_ENGINE_JWKS_URL=http://localhost:8003/.well-known/jwks.json(the standard OIDC discovery path. ItsengineOrigin()derives the engine base URL from that). SetDFE_ENGINE_ISSUER=https://dfe.local/apito match the engine's advertised issuer (/.well-known/openid-configuration), enforced as the tokeniss. The fork's/dfe/*routes forward the caller'sdfe_tokento the engine. Under a devDFE_ENGINE_ENVthat cookie is set withoutSecure, so a plain-http stack keeps it. Production setsSecure, like the session cookie. - dfe-ui talks to the engine API at
NEXT_PUBLIC_API_URL(http://localhost:8003in its.env.example) and embeds the fork. The fork's create-rule button opens<NEXT_PUBLIC_DFE_UI_BASE_URL>/rules/{id}. - The deploy repo (dfe-deploy) is the gitops target. Either disable gitops
(
DFE_ENGINE_GITOPS_ENABLED=false) or point it at your local clone withDFE_ENGINE_GITOPS_REPO_URL/DFE_ENGINE_GITOPS_LOCAL_PATH. With it off the live auth store IS the durable store, so there is nothing to commit.
To work on dfe-ui with no engine at all, docker compose -f docker-compose.dev.yaml up prism serves a mock of openapi/openapi.json on port 4010. Point dfe-ui's NEXT_PUBLIC_API_URL at http://localhost:4010.
Bring the stack up one repo at a time, engine first, and confirm each speaks to the
engine before adding the next. Each repo's CONTRIBUTING.md has its local setup.