Repository navigation
test(e2e): kill hops under load and count the loss - #143
Merged
Merged
Conversation
This was referenced Sep 26, 2026
make ci on the single profile, from empty volumes, failed its own self test every time: "dfe-loader is fetching no topic within 150s". The loader was fine. The broker had the dfe-loader group Stable with all 12 main_land partitions assigned. The subscription claim read rdkafka_topic_partition_consumer_lag before injecting anything. scalo emits that series only when librdkafka reports a lag of 0 or more, and librdkafka reports -1 until the group has committed an offset on the partition. A topic nothing has been written to never has one, so the claim could not pass on a fresh stack. It only ever passed on a broker with history. The claim now runs after the injected events land, where it confirms the subscription, and as the diagnosis when they do not. Measured on the rc.14 pins: from zero it failed twice; after one committed record the same stack passed 8/8. Two tests pin the order, and both fail on the old code. Signed-off-by: Derek <derek@hyperi.io>
make test-source crashed before the runner started, on a .env straight out of make init: "invalid literal for int() with base 10: ''". The Makefile exports CLICKHOUSE_HTTP_PORT and CLICKHOUSE_USERNAME for the helpers, and exporting a key .env never set hands the script an EMPTY variable. test_source.py and test_flows.py read them with os.environ.get(key, default), which returns '' and never reaches the default. Both now use _pipeline.env_or, which post.py already did. _pipeline.clickhouse_url() dialled localhost whatever DFE_POST_HOST said, so on a second stack published beside another every POST ClickHouse query went to the first stack. It now follows DFE_POST_HOST like every other POST claim. CLICKHOUSE_URL still wins. test_source.py also refused --case vector, which dfe-infra's runner has had for a while. Found standing up the rc.14 pins from a fresh clone on a box that already had another stack's ports on 127.0.0.1. Signed-off-by: Derek <derek@hyperi.io>
* fix: add the make apply target the engine hints name dfe-engine tells a Compose operator to run `make apply SERVICES=<service>` after every config write the running app cannot take in place (appmgmt/appconfig.py APPLY_COMMAND, in the engine rc.14 pins). This repo had no apply target on any branch, so the hint failed with "No rule to make target 'apply'", and a per-source transform the engine had just declared never got a container. dfe-ui's corpus specs call the same target. apply resolves the profile, which regenerates the instances fragment, then runs `up -d --force-recreate --no-deps` on the named services only: a new instance is created, an existing one is recreated so it rereads its mounted config and its env_file. It refuses an empty SERVICES rather than recreating the whole stack. DEV=1 uses the dev chain for a stack `make dev` started. It joins the goals that need DFE_EXTERNAL_ORIGIN under DFE_BIND_SCOPE=all. docs/architecture.md still quoted the old `docker compose restart` hint. Signed-off-by: Derek <derek@hyperi.io> * fix: check apply's services against the re-resolved stack make parses its include of .profile.mk before the FORCE rule rewrites it, so the SERVICES check ran against the previous run's service list. A per-source instance the engine declared since then could never pass it: make apply SERVICES=dfe-transform-vrl-cisco-meraki stopped with 'not in the resolved stack' while env/dfe-transform-vrl.instances named it. apply now hands off to a second make, which reads the .profile.mk the first one just wrote. Signed-off-by: Derek <derek@hyperi.io> --------- Signed-off-by: Derek <derek@hyperi.io>
A container name is daemon-wide, so `COMPOSE_PROJECT_NAME` and a fresh port block are not enough for a second stack: the start fails on `Conflict. The container name "/dfe-contract-dfe-loader" is already in use`, and an EXITED container from a project nobody is running still holds the name. All 34 `container_name` values in docker-compose.yml now read `${DFE_CONTAINER_PREFIX:-}<name>`, so an empty default is today's names and one `.env` line re-prefixes a whole stack.
The prefix is safe because compose DNS is the SERVICE key, not `container_name`, and the service keys are untouched. Every in-network address stays put: `INTERNAL_API_URL=http://dfe-engine:8000` (env.example/ui.env), `http://dfe-loader:50051` (config/receiver/grpc.yaml, config/fetcher/aws-grpc.yaml), the envoy clusters in config/proxy/envoy.yaml, and post.py's two network bases. All 21 healthchecks target localhost, 127.0.0.1 or an in-container binary, so none of them resolves a container name either.
Four places do resolve a container name, through the raw docker CLI, and they now apply the same prefix through `_common._container_name()`: post.py's `docker exec` into the engine, test_source.py's `docker inspect`/`docker exec` on the archiver, and test_e2e.py's `report_unready` (`docker inspect`, `docker logs`). `docker compose exec/ps/logs` call sites are addressed by service key and need nothing.
scripts/instances.py is a second container_name generator -- it templates `container_name: {service}-{instance}` for per-source app instances straight from the service key, so the docker-compose.yml edit alone would have left every per-source container colliding across two stacks. Its fragment now carries the same variable.
Proved by rendering the real model: `docker compose config` over the base file with the 19 image pins placeholdered gives 30 services whose names are byte-identical to today's when the variable is unset, and each exactly `accept-` plus today's name when it is set. `make check-compose` resolves all 8 path/backend combinations, `make check-hardfail`, `check-docs`, `check-python` and 228 unit tests pass.
Closes #127
* fix(compose): dial ClickHouse on its container ports CLICKHOUSE_HTTP_PORT and CLICKHOUSE_NATIVE_PORT were the bundled container's host publish AND the port dfe-engine, dfe-hunt-runner and the collector dialled inside the network. Move the publish for a second stack and all three dial a port nothing listens on. Same conflation HyperDX already had fixed. - The three services now dial 8123/9000, the container ports, whatever the host publish says. CLICKHOUSE_HTTP_PORT and CLICKHOUSE_NATIVE_PORT go back to being the publish and nothing else, so they are safe to move. - An external CLICKHOUSE_HOST gets its own pair, CLICKHOUSE_EXTERNAL_HTTP_PORT and CLICKHOUSE_EXTERNAL_NATIVE_PORT, default 8123/9000. An external setup that gave its port through CLICKHOUSE_HTTP_PORT sets the EXTERNAL pair instead. - The collector's own variable is CLICKHOUSE_PORT now, so no container reads CLICKHOUSE_NATIVE_PORT as a dial port. - check_compose renders four cases (defaults, publish moved, external host, external host and ports) and asserts every dialler and the publish. The old engine line fails it. - .env.example, README, configuration.md and developing.md say which ports are safe to move. configuration.md also stops calling KAFKA_PLAINTEXT_PORT in-network, since it is a host publish. Closes #75 * fix(compose): name the loader's in-network port in the second-stack notes The second-stack section named dfe-loader:50051 as an in-network address. 50051 is the host publish; senders dial dfe-loader:6000. The EXTERNAL ClickHouse ports are also not something a second stack clears, only something it does not move.
…162) The routing claim gave the new source 45s to land a row. On a tier with a bus the row cannot land until the loader's next topic refresh subscribes to the source's new landing topic, and that refresh runs every 60s. So the claim raced it: a run whose loader subscribed 44s after the topic was made gave up three seconds later. - The window is now built from its parts: the receiver's 5s config poll, one loader topic refresh where the tier has a bus, then the same landing budget the INGEST claim gets. 125s with a bus, 65s without. A pass still stops at the first row. - The refresh is scalo-rs kafka.topic_refresh_secs. dfe-loader has no setting for it and leaves it at 60, so it is one named constant, and the SUBSCRIPTION claim's window now derives from it too. - The subscription itself cannot be waited on. The loader's only per-topic series needs a committed offset, and the loader commits once the row is in ClickHouse, which is the row the claim already polls for. - A failure on a bus tier now names the loader's subscription as the other suspect. - Tests drive the real wait on a clock that moves only when it sleeps: a row landing a whole refresh after the topic passes, a row that never lands fails within the window, and a tier with no bus does not wait out a refresh. The old 45s fails the first and third.
CI ran every helper script on ubuntu-latest's system python3, 3.12, while the house floor is 3.14. pipx also built ruff's and pytest's venvs on that interpreter, so the unit tests ran on 3.12 too. - actions/setup-python v7.0.0, SHA-pinned, installs 3.14.7 in the three jobs that run Python: Compose, Profile projection (scripts/render_profiles.py) and Dockerfile, where make builds .env with scripts/init.py before hadolint runs. - The pipx installs take the interpreter setup-python reports, so pytest's venv is 3.14 as well. - PYTHON_VERSION sits once at the top of the workflow.
HyperDX's seeded connection was a literal http://clickhouse:8123, so pointing CLICKHOUSE_HOST somewhere else left it querying a container that is not running. - DEFAULT_CONNECTIONS takes CLICKHOUSE_HOST and CLICKHOUSE_EXTERNAL_HTTP_PORT, the same keys dfe-engine and dfe-hunt-runner dial since #161. - CLICKHOUSE_SECURE makes it https. Compose cannot turn a flag into a scheme, so resolve_profile.py exports DFE_CLICKHOUSE_RESOLVED_SCHEME, https on exactly the values the engine reads as TLS on (true, 1, yes). - Only header-dev reads DEFAULT_CONNECTIONS. Under oidc-proxy, the default, the fork asks dfe-engine for each team's connection and the engine builds it from the same keys, scheme included. - check_compose holds HyperDX to the URL in every ClickHouse case, plus a new TLS case. The old literal fails three of them.
POST's schema wait read the engine on its host port, localhost:8003. With DFE_ENGINE_API_EXTERNAL=false that port is unpublished, so the wait timed out after 300s and POST failed on a stack that was fine. - /readyz and /api/v1/system/schema now go through _api_get_json, the docker-exec transport every other engine call in POST already uses. No host port, moved or missing, decides the answer. - _over_network matches the engine's origin rather than only /api/v1, so /readyz takes that transport too. - The schema tests now serve their answers in-network and fail if anything dials the host. Putting the host read back fails eight of them.
#166) * fix(config): start vrl's consumer group at the log start dfe-transform-vrl defaults source.auto_offset_reset to latest, so a group's first join skips every record already on the topic. The filebeat and main instances now set earliest, and so does the idle kafka.yaml base, because the engine deep-merges a bound source over it and the key carries through. * fix(config): start an engine-bound vector source's group at the log start
…hen a goal needs it (#167) * fix(make): read .env only when a goal needs it `-include .env` sat beside a rule that builds .env, and make builds a missing include before any goal runs. So `make help`, every check-* target and each CI runner minted the seven generated secrets and env/ as a side effect. The include now reads .env only when it exists. env-files and infra depend on .env instead, so the start goals still mint it before their first compose call. check_compose renders in memory the .env `make init` would write, the contract-emit test removes the env/ it creates, and the Dockerfile job no longer sets up Python. * fix(apply): remove the container of a deleted source When the engine deletes a source it drops the instance from its index and removes the rendered config. The next make run regenerates docker-compose.instances.yml without that service, so its container became an orphan of the compose project, crash-looping on "config file not found". `make apply` only warned about it. Generated services now carry a dfe.instance-of label, and apply-services runs `instances.py --prune` with the same compose flags as its `up`. It stops and removes every labelled container whose service the fragment no longer declares. It is not `--remove-orphans`, because that also removes any container a compose file outside the apply chain declares, such as an extra e2e overlay. An unlabelled container is never touched, including committed services that share an instance's name prefix, like dfe-transform-vrl-filebeat.
…rl instance takes the e2e name (#168) * fix(apply): a bare make apply removes a deleted source's container The engine names no command after a delete, and make apply refused an empty SERVICES, so a deleted source's container only went when someone applied an unrelated service. With no SERVICES, apply now re-resolves the profile and prunes, and starts nothing. A named apply runs exactly the commands it did. * fix(compose): rename the static vrl filebeat instance off the generated name dfe-engine names a source's instance <app>-<source>, so a source named filebeat on dfe-transform-vrl came out as dfe-transform-vrl-filebeat, the committed service's own name, and Compose merged the two. The committed service is now dfe-transform-e2e-vrl-filebeat, the shape the vector and elastic fixtures already use. Its env files, config variables and consumer group keep their names. The group is the one the engine gives a filebeat source, so the two split the topic rather than each reading all of it. check-compose now fails on any committed service a generated instance could be named.
* fix(compose): read the custom env files the engine writes
dfe-engine writes an app's extraEnv to env/<service>.custom.env, named for the Compose service (env/dfe-loader.custom.env), and each instance of a per-source app to env/<app>-<source>.custom.env. Compose read env/loader.custom.env and its siblings, names the engine never wrote, and a generated instance read no file of its own. So a key set through the console reached no container.
Every app the engine renders now reads its own file last, dfe-transform-elastic included. Each generated instance reads its app's file and then its own, so a per-source key beats the app's. The two static e2e services drop their custom entries, since the engine writes nothing for a service it does not render.
check-compose now renders the stack with one generated instance per app, un-interpolated so the env_file lists survive. It fails on a rendered service that does not read its own file last, and on any custom file a service reads that the engine does not write for it.
* fix(compose): read custom env reads off their effect on Compose 2.38
The CI runner's Compose 2.38 prints no env_file under any config flag and rejects --no-interpolate on this file's ${VAR:-default} volumes, so the custom env check could not resolve there.
It now renders a copy of the compose file in a scratch directory whose env/ holds a sentinel per env file, and reads which files each service took from its environment. Nothing is written into the checkout's env/. The copy is needed because Compose resolves an extended service's paths against the file declaring it.
… Compose too (#170) profiles.md and architecture.md still said Compose runs one of each app and a second source needs Kubernetes. scripts/instances.py generates a service per line of the engine's instances index, and make apply starts it.
.env carries every secret init generates, and every writer opened it under the umask, so a checkout with umask 002 left it 664 -- readable by any local user on the host. All five writers now go through one helper that creates the file at its mode and reasserts it on rewrite: .env at 0640, since a daemon install shares it through the checkout's group, and the backup and access summary at 0600 as before. make init also closes an existing .env to others without touching its owner or group bits.
The engine reads DFE_ADMIN_LINKS since hyperi-io/dfe-engine#598 and Compose was handing it nothing. resolve_profile.py now builds the list in one place, from the footprint that starts Kafbat and HyperDX plus the Makefile's KAFBAT_GATED and HYPERDX_GATED, which make passes in as --unpublished. So a UI is listed only when it runs AND its port is published. Each url is DFE_EXTERNAL_ORIGIN (or the per-surface override compose already builds that UI's URLs from) plus the host port. Each probe_url is the in-network address. ClickHouse Play, dfe-ui and the engine API stay out: Play is an operator port on DFE_BIND_HOST, and the other two are the product. Tests cover the transport, footprint, gate, auth and origin matrix, and run the real Makefile in a scratch copy through docker compose config, so the JSON is shown to reach the engine service byte for byte. Nothing is started. Part of hyperi-io/dfe-engine#594.
…ment (#173) Tests that drive resolve_profile.main() redirected .profile.mk but not the instances fragment, so a run on a live checkout deleted its docker-compose.instances.yml (instances.write with nothing to write removes the file). Measured on main d8ee7d1: a planted fragment was gone after make check-tests. An autouse fixture now points both generated files at tmp_path for every test; with it the planted fragment survives unchanged and 609 pass.
The comment in config/receiver/grpc.yaml said an engine-rendered destinations.loader overrides the default loader address. The engine stopped writing that entry in dfe-engine#441, so it now says what does: an overlay setting loader.grpc_endpoint. Comment only.
dfe-ui's OTel exporter has no gRPC transport, and it was handed the collector's gRPC port 4317, so every span it sent was dropped. resolve_profile now also exports the endpoint's HTTP twin on 4318, and dfe-ui takes that with the http/protobuf protocol. The bundled collector already listens on 4318, and the other services stay on gRPC.
The e2e suite never exercised a rule or a hunt, and never started dfe-hunt-runner either, though the core footprint names it. It does now. A rules-and-hunts test creates a rule and a one-minute hunt through the engine API, sends three events that must match and three near misses that must not, queues a run, and asserts dfe.detection holds exactly the matching rows by matched_uuid, each once, under the rule and severity it was given. The hunt and the rule are deleted and asserted gone. A profile with the core footprint off skips with a reason. Proved on a local stack pinned to 2.2.0-rc.14: 12 passed. A rule widened to every failed login fails on the near miss it catches.
post's hunt claim now queues its run with POST /hunts/{name}/run and waits for the hunt's last_run to pass its last event, instead of waiting out the next cron fire. It also judges the result exactly: one failed root login it must detect and two near misses it must not (alice's failed login, a root success), under their own marker so the console claim still counts only the ingest rows.
The rule SQL, request bodies, reads, timings and the verdict all live in scripts/_detection.py now, and the rules-and-hunts e2e calls the same pieces instead of carrying its own copies.
Stale statements gone: run-now answering 501, log_buffer shifting the window (the pinned runner never reads it), and GET /hunts/status `running` counting leases (it counts runner heartbeats, so post now checks it up front like the e2e). The hunt claim no longer needs to run ahead of the others, so that ordering comment and its test go too.
* fix: pin the container prefix in tests and the POST claim count test_contract_emit.py and test_catalogue_emit.py hardcode container names under the default DFE_CONTAINER_PREFIX. A checkout whose .env sets a second stack's prefix reaches check_compose._check_env() through os.environ and renders a prefixed name, failing seven assertions. Each file now pins the prefix to empty for its own module, independent of the caller's environment. docs/observability.md said make post makes "up to five claims"; post.py's claims dict carries eight (ingest, hunts, subscription, self-monitoring, observability, console, idle apps, routing). * fix(docs): list all eight claims make post makes The sentence said eight but the list named five. Subscription, Idle apps and Routing are added, and Hunts now says what the claim checks since the run-now change: exactly the matching events, none of the near misses.
dfe-engine #627 reads each app's live metric manifest from DFE_SERVICES_METRICS_MANIFEST_URL and fetches nothing while it is empty, which it was here.
- The engine service sets http://{service}:9090/metrics/manifest. Every app serves the manifest on its container's 9090 and every service sits on the one default network, so the service name resolves whatever host port publishes it.
- test_metrics_manifest.py holds that to the rendered model: the address names each app's service key and the port its healthcheck probes, on a network the engine shares.
- docs/configuration.md says where the address comes from.
Proved on a throwaway local slim stack running an engine built from #627 (38a22bcd): POST /api/v1/service-surfaces/<app>/metrics/refresh answered refreshed: true for dfe-loader (140 metrics), dfe-receiver (77) and dfe-archiver (110). The pinned engine, v1.22.10, predates #627 and ignores the variable.
Nothing drove Create Rule on the Search page, so rc.14 shipped with it saving HyperDX's export template as a rule: no table, SQL ClickHouse refuses, and a hunt that could never run. The new search-rule-hunt test takes the analyst's path in a browser. It adds HyperDX to the stack and completes the setup wizard's organisation and first user through the API. Then it signs in to the console, searches main for failed logins held to the run's marker, narrows to root in the side panel and presses Create Rule. The stored rule has to scan dfe.main with both filters in its WHERE and no SQL errors. A hunt over it is added and triggered from the console's own form, and dfe.detection has to hold exactly the matching events. Each filter alone lets a near miss through. The side panel only offers values rows already hold, so the events go in once before the search, and those seeded near misses are judged too. The verdict takes the rule's own severity, and the rules-and-hunts test shares the pickup, trigger and verdict steps. It needs Playwright and Chrome; DFE_E2E_HEADED=1 shows the browser and DFE_E2E_SHOTS_DIR keeps the step screenshots.
service_profiles.yaml's projected slim and single blocks had drifted from dfe-infra's app catalogue: slim carried dfe-archiver, which apps.yaml no longer defaults into the docker-slim tier, and single was missing dfe-fetcher, which apps.yaml defaults into docker-single. Re-rendered with make render-profiles against dfe-infra main (8eeaa39f5). make check passes (687 tests).
A fetcher instance had no writable cursor directory: it fell back to its read-only config mount, so every restart re-fetched the lookback window and re-delivered records. Each fetcher now keeps its cursors in its own directory on a persistent fetcher-cursors volume, made writable by dlq-init and relocated with DFE_DATA_ROOT like the other stateful volumes. Fixes #182
fix(fetcher): keep cursors across restarts
The evaluating page showed one demo. One paragraph under the demo names the cyber, theatre and telco packs, their compose tags and what each streams. It says they share the demo's ports.
Co-authored-by: hypersec-ci-bot[bot] <246025198+hypersec-ci-bot[bot]@users.noreply.github.com> Co-authored-by: Derek <53845375+catinspace-au@users.noreply.github.com>
* fix(stack): refuse a prerelease stack pin on a release v1.0.5 and v1.0.6 installed stack 2.2.2-rc.5. The bot's rc pin PRs had merged. The GA pin PR had not. The release paths and make check-hardfail read the pin through release_pin. release_pin refuses a prerelease unless DFE_DOCKER_PRERELEASE_PIN=1 is set on a development branch. * fix(release): fail the release naming step on a prerelease pin The stack was read inside an echo, so a refused pin left the step green and the image built with no stack. The pin is read on its own line first. The refusal message says CI refuses the pin on every branch. The two scripts that read it each have a failing-pin test.
Four sentences in docs/developing.md and docs/observability.md replace shape with the committed setting, Kubernetes metric names, where services push, and the `postroute` sources the run sweeps.
Removes 6 stale sentences in docs, docstrings and script comments: "used to" histories, a roadmap note and a "currently". Cuts 12 comment lines from the CI workflow and Makefile, including a restated hadolint version. Keeps a short explanation beside a config value.
The README and the evaluating page each add one line on memory. The default stack idles at about 2.5 GB of RAM. Its limits are sized for a 16 GB box. Idle memory is rounded to the nearest 0.5 GB so small stack changes do not change the figure.
`make ci` sets its services and compose fragments at parse time. Its `down` runs `demo-stop`. That sub-make removes the pack's sources and the instances fragment. `up -d` then fails because the fragment is gone. `quickstart.py over-demo` runs first in `ci` and blocks it while a demo pack is streaming. It tells the reader to run `make demo-stop`, `make up`, then `make demo`. Closes #376.
Generated credentials go from 24 to 32 characters and signing keys from 48 to 64. Both use only letters and digits, giving about 190 and 381 bits. This matches what the Kubernetes deploy generates. The oauth2-proxy cookie secret takes 32 characters, which oauth2-proxy decodes to a 24-byte key. The tests read the lengths from init.py. The published compose job checks the new lengths.
The published compose no longer ships a fixed admin password. secrets-init generates a password of 32 letters and digits. On the published file, `run --rm secrets-init login` prints the name and password. quickstart.json adds this command as the second line in every up, demo and pack list. DFE_ADMIN_NAME and DFE_ADMIN_PASSWORD, when set, replace the minted login. secrets-init refuses a password the engine would refuse, without printing it. The check allows references to DFE_ADMIN_NAME and DFE_ADMIN_PASSWORD only in secrets-init's environment.
A Docker install could not take syslog or flow from outside the Compose network because the receiver published only 8080 and 9090. Six port keys now publish syslog over UDP, TCP and TLS, and NetFlow, IPFIX and sFlow over UDP. They use the same make fragment as the Beats and HEC keys. Each stays unpublished until its key is set. The configuration doc names the receiver settings that turn the listeners on. It also says rootless Docker cannot bind a host port under 1024.
An external pusher could reach the receiver only over plain HTTP on a Docker install. Under DFE_PROXY_TLS, DFE_RECEIVER_HTTPS_PORT now publishes a dfe-proxy listener on 8443 with the console's certificate. It forwards to the receiver in the Compose network. With TLS off, make rejects DFE_RECEIVER_HTTPS_PORT. DFE_FETCHER_INPUT_DIR mounts a host directory read only at /mnt/input, so a file source can run. The configuration doc explains how to turn each receiver listener on under --config. It also explains that the receiver's ip_filter sees only the proxy on that port.
The engine needs DFE_ENGINE_STACK_IMAGES to match running image digests against published stack manifests. This identifies which stack a Docker install runs. The dfe-engine service builds DFE_ENGINE_STACK_IMAGES from the same interpolations as its image lines. The variable lists the ten dfe-* images with their digests. check_compose asserts that the list matches the images Compose runs. The dev overlay clears DFE_ENGINE_STACK_IMAGES when it sets an image to :local. Local builds then show DFE_STACK_VERSION instead of a published stack.
Fix 81 incorrect rows out of 164 in the README and docs/. The README and docs/ match the code, compose files and sibling repos. Corrections cover: - The loader's rpc name. - make post failing make dev and ci. - Who creates the otel tables. - A receiver gRPC env key that does nothing under --config. operating.md and architecture.md are split by reader into auth-profile, broker, secrets-and-storage, self-test and seeding pages.
- post.py expects dfe-ui's metrics only when DFE_UI_VERSION is v1.7.40 or later. v1.7.40 is the first release that pushes them. Tests cover older, unset or local pins. - post.py and the e2e otel message name dfe-engine, which creates the otel tables. - check_compose.py keeps the 2.0 CPU floor. Its comment says scalo-rs polls Kafka on tokio's blocking pool. - deployment.example.yaml sets hyperdx_enabled and otel_enabled true, as every profile does. - PROVENANCE.md places tests/fixtures/unencumbered/ in the dfe-transform-elastic repo. - The dial's top-level key is `platform` (docker-vm or k8s). render_dial.py rejects `substrate:` and says to use `platform:`. It reports a missing `platform:` key. dfe-infra #1099 renames the key to `platform`.
Python upgrades happen across every repo together. The suite uses 3.14. Renovate #397 proposes upgrading this image alone to 3.15.2. allowedVersions <3.15 keeps the base image on 3.14 patches until that upgrade.
Self-update left a full set of images from the old pin after each update. The VM disk reached 92%. - Before the re-pin, self-update lists the images docker compose resolves to across all profiles. - Removal waits for make ci and its self test to pass. - Self-update runs docker image rm on listed images the new stack no longer names and no container uses. - Images pulled by hand are not touched. - A failed update removes nothing. - A removal error logs a warning and the update still succeeds.
The docs correct 25 claims to match the code. The fetcher and transforms never run on the loader's gRPC port. The auth profile starts by stating that it is off by default. CI starts a stack. Ten diagrams pass the diagram checker on Mermaid 12.3.0. Comment lines are removed from the Makefile, ci.yml, service_profiles.yaml, docker-compose.yml and three more files. The files parse to the same data as before.
make test-resilience could not pause the next hop before a kill, so the killed hop never held records. No tests ran with acknowledgements off to show that a kill can lose records. Tests can pause a service before the signal. They check loss as forbidden, required, unreliable or tolerated. They read the receiver's disk spool, the dead-letter files and the archive. Tests kill the loader and receiver on gRPC and Kafka. They also test graceful stops and kills with acknowledgements off.
The rebuilt branch has the old kill tests ported onto main. Its tree replaces the old branch's tree.
The spool replay check reads zstd frames. Python 3.12 has no compression.zstd. The spool replay check therefore found no record on the VM's spool. test_e2e.py pins zstandard. With the loader frozen, the receiver has no gRPC data to lose. This makes the test with acknowledgements off unreliable.
The suite generates the loader config under .tmp, so the checks never saw a loader directory. Every Kafka outage test failed with no subscriber. A killed loader had no known group. The checks recognise a loader- config as the loader config. The loader writes dead letters as DFE events, so the dead-letter reader looks for markers in each line.
The archiver commits only when it rolls a file, every 300 s by default. The transform tests on Kafka waited on its group and never started the outage. They run the 10 s roll config that kafka-outage runs. A killed archiver with no group in its config commits as dfe-archiver.
The receiver cuts its in-flight requests on SIGTERM when its next hop is frozen (dfe-receiver#278). A test can name a known_issue. Its failures report as one skip citing the issue. A clean run fails so the key is removed.
The archive fallback passed "stop" as a string, so compose ran it letter by letter. settle_known_issue ignores only an unanswered request or a lost record. A failed unpause or start still fails the run. docs/checks.md names the three hops that have an acks-off control.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
make test-resiliencekills each hop mid-stream with its next hop frozen. It counts every record answered 2xx. Acks-off controls must lose records. The three controls cover loader on gRPC, receiver on gRPC and receiver on Kafka.The VRL transform, archiver and loader on Kafka have none.
Harness in
scripts/_outage.pyandscripts/test_e2e.py:outage.pausefreezes a service so the killed hop still holds records. A graceful stop sends SIGTERM before the frozen service resumes.lossisforbidden,required,unreliableortolerated. Duplicates are counted per phase but do not fail tests.spool,poisonwithdead_letters,archiveandexpect_refusalsread the receiver's disk spool, refused records, the archive and 503s.Kill and SIGTERM cases for the loader, receiver, VRL transform and archiver are in
tests/e2e/e2e-tests.yaml.Results on the dfe-docker VM with its running images and non-default ports:
unreliable.No coverage: transform kills on gRPC, transform-vector, transform-elastic and the two-archiver case. Main has no profile for them.
The receiver cuts its in-flight requests on SIGTERM when its next hop is frozen (dfe-receiver#278). The two
receiver-sigterm-*cases haveknown_issueand report as a skip until the receiver bug is fixed.