Skip to content

fix: make apply exists, and a fresh kafka stack passes its self test - #155

Closed
catinspace-au wants to merge 1 commit into
mainfrom
fix/apply-target-and-subscription-probe
Closed

catinspace-au wants to merge 1 commit into
mainfrom
fix/apply-target-and-subscription-probe

Conversation

@catinspace-au

Copy link
Copy Markdown
Contributor

A public 2.2.0-rc.14 single stack stopped at first use in two places. Both fixes are the ones #143 carries, taken on their own because #143 conflicts with main (66 files, 4356 lines).

make apply did not exist. After a source deploy the engine (v1.22.8, appmgmt/appconfig.py APPLY_COMMAND) tells a Compose operator recreate required: make apply SERVICES=dfe-transform-vrl-<source>. Main had no such target, so the new per-source transform never got a container, and the only way to start one was make up, which runs down first. make apply re-resolves the profile, starts what it adds or changes with no down, and restarts the services it is given so a rewritten mounted config is read. The SERVICES name check moves into a services-valid target: the parse that ran it could still hold the .profile.mk from before the engine declared the instance, so make ci SERVICES=<new instance> errored on a name that was valid.

make up failed its self test on every fresh kafka stack. The subscription claim waits for rdkafka_topic_partition_consumer_lag on the loader. librdkafka reports that lag as -1 until the group commits an offset, and scalo only exports a lag it can state, so a stack that has carried no traffic never shows a topic, however well subscribed. Measured on the rc.14 stack: the broker showed all 12 main_land partitions assigned to the loader, the loader's /metrics showed no lag sample, and the sample appeared the moment a topic had a committed offset. Each attempt that sees no topic now sends one probe record through the ingest edge first.

Checks: make check-python, check-tests (376 passed), check-compose, check-hardfail, check-docs all pass. Three new tests cover the probe.

Live proof on the rc.14 single stack is in the comments below.

Two things stopped a public 2.2.0-rc.14 single stack at first use.

The engine tells a Compose operator to run make apply SERVICES=<service> after a source deploy, and this repo had no apply target. A new per-source transform therefore never got a container, and the only way to start one was make up, which takes the whole stack down first. make apply re-resolves the profile, starts what it adds or changes without a down, and restarts the services it is given so a rewritten mounted config is read. The SERVICES name check moves into a services-valid target, because the parse that ran it could still hold the profile from before the engine declared the instance.

make up on the single tier failed its self test on every fresh stack. The subscription claim waits for the loader's per-topic lag metric, which librdkafka reports as -1 until the group commits an offset, and scalo exports only a lag it can state, so a stack that has carried no traffic never shows one. Each attempt that sees no topic now sends one probe record through the ingest edge first.

Both are the same change dfe-docker #143 carries, taken on their own because that branch conflicts with main.

Signed-off-by: Derek <derek@hyperi.io>
@catinspace-au

Copy link
Copy Markdown
Contributor Author

Live proof, on the docker daemon VM, from a fresh clone of this branch and the public dfe-infra fix/stack-cut-2.2.0-rc.14 (0890a0b), images pulled anonymously with an empty DOCKER_CONFIG:

make up DFE_PROFILE=single on fresh volumes, first boot:

post: Waiting for dfe-loader to be fetching a topic at http://localhost:9091, sending probe records tagged post-2053180-9544e31d-probe
post:   attempt 1: topics fetched = none
post:   attempt 2: topics fetched = none
post:   attempt 3: topics fetched = main_land
post: PASS  dfe-loader is fetching main_land
...
post: 8/8 claim(s) asserted

The same stack from main failed at attempt 31 with FAIL dfe-loader is fetching no topic within 150s.

make apply on the engine's hint after a source deploy, with the .profile.mk still from before the instance existed:

make apply DFE_PROFILE=single SERVICES=dfe-transform-vrl-fbf5d4a68e
docker compose -f docker-compose.yml -f docker-compose.instances.yml -f docker-compose.container-logs.yml ... up -d dfe-transform-vrl-fbf5d4a68e
 Container dfe-transform-vrl-fbf5d4a68e Created
 Container dfe-transform-vrl-fbf5d4a68e Started
... restart --no-deps dfe-transform-vrl-fbf5d4a68e

The source acceptance then reported the instance up after 9s, 60 corpus records posted, 62 rows landed (the 2 over are its routing probes), 21 transformed, and the Observe search found them.

Still open: scripts/test_source.py passes --restart-exec "docker restart", which cannot create a container, so the runner's own restart row stays failed until it runs make apply instead.

@catinspace-au

Copy link
Copy Markdown
Contributor Author

Closing as superseded. Two fixes landed for the same two defects. #153 moves the loader-subscription claim after the injected events land, which passes a fresh Kafka stack without leaving probe rows in the operator's default table. #156 adds make apply, refuses an empty SERVICES, and checks names in a sub-make against the re-resolved profile. Both were proven from zero on a single stack: POST 8/8, and three per-source containers created and healthy.

@catinspace-au
catinspace-au deleted the fix/apply-target-and-subscription-probe branch September 27, 2026 06:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant