Repository navigation
fix: make apply exists, and a fresh kafka stack passes its self test - #155
catinspace-au wants to merge 1 commit into
Conversation
Two things stopped a public 2.2.0-rc.14 single stack at first use. The engine tells a Compose operator to run make apply SERVICES=<service> after a source deploy, and this repo had no apply target. A new per-source transform therefore never got a container, and the only way to start one was make up, which takes the whole stack down first. make apply re-resolves the profile, starts what it adds or changes without a down, and restarts the services it is given so a rewritten mounted config is read. The SERVICES name check moves into a services-valid target, because the parse that ran it could still hold the profile from before the engine declared the instance. make up on the single tier failed its self test on every fresh stack. The subscription claim waits for the loader's per-topic lag metric, which librdkafka reports as -1 until the group commits an offset, and scalo exports only a lag it can state, so a stack that has carried no traffic never shows one. Each attempt that sees no topic now sends one probe record through the ingest edge first. Both are the same change dfe-docker #143 carries, taken on their own because that branch conflicts with main. Signed-off-by: Derek <derek@hyperi.io>
|
Live proof, on the docker daemon VM, from a fresh clone of this branch and the public dfe-infra
The same stack from main failed at attempt 31 with
The source acceptance then reported the instance up after 9s, 60 corpus records posted, 62 rows landed (the 2 over are its routing probes), 21 transformed, and the Observe search found them. Still open: |
|
Closing as superseded. Two fixes landed for the same two defects. #153 moves the loader-subscription claim after the injected events land, which passes a fresh Kafka stack without leaving probe rows in the operator's default table. #156 adds make apply, refuses an empty SERVICES, and checks names in a sub-make against the re-resolved profile. Both were proven from zero on a single stack: POST 8/8, and three per-source containers created and healthy. |
A public 2.2.0-rc.14
singlestack stopped at first use in two places. Both fixes are the ones #143 carries, taken on their own because #143 conflicts with main (66 files, 4356 lines).make applydid not exist. After a source deploy the engine (v1.22.8,appmgmt/appconfig.pyAPPLY_COMMAND) tells a Compose operatorrecreate required: make apply SERVICES=dfe-transform-vrl-<source>. Main had no such target, so the new per-source transform never got a container, and the only way to start one wasmake up, which runsdownfirst.make applyre-resolves the profile, starts what it adds or changes with nodown, and restarts the services it is given so a rewritten mounted config is read. TheSERVICESname check moves into aservices-validtarget: the parse that ran it could still hold the.profile.mkfrom before the engine declared the instance, somake ci SERVICES=<new instance>errored on a name that was valid.make upfailed its self test on every fresh kafka stack. The subscription claim waits forrdkafka_topic_partition_consumer_lagon the loader. librdkafka reports that lag as -1 until the group commits an offset, and scalo only exports a lag it can state, so a stack that has carried no traffic never shows a topic, however well subscribed. Measured on the rc.14 stack: the broker showed all 12main_landpartitions assigned to the loader, the loader's/metricsshowed no lag sample, and the sample appeared the moment a topic had a committed offset. Each attempt that sees no topic now sends one probe record through the ingest edge first.Checks:
make check-python,check-tests(376 passed),check-compose,check-hardfail,check-docsall pass. Three new tests cover the probe.Live proof on the rc.14
singlestack is in the comments below.