Repository navigation
fix(compose): the worker's store init now runs on every up, after the core - #344
Merged
Merged
Conversation
… core The v2.5.0 clean-install gate caught a real fresh-install bug: the worker container crash-looped with "probe write: attempt to write a readonly database" and the stack never reported healthy. Root cause: the worker container is not profile-gated — it always runs to report health (only execution is gated by ATF_WORKER_ENABLED) — but the worker-store-init that grants the worker's group write access to atf.db was behind profiles: ["worker"], so a stock `docker compose up` never ran it. The worker (uid 100) then probed a root-owned atf.db, fail-closed on purpose (#285), restart-looped, and its /ops endpoint — which the API's health payload probes — never came up: status degraded, gate failed. Two ordering holes are closed, not one: 1. The init lost its profile: it runs on every `compose up` now. 2. The init waits for the core to be healthy, and the worker waits for the init to complete. On a fresh volume atf.db does not exist until the core creates it — an init that ran earlier would chgrp an empty directory and leave the newly created file root-owned, reintroducing the same crash loop on the first boot. Idempotency is unchanged: chgrp/chmod on an already-prepared store is a no-op, so re-runs after a core restart stay safe, and the postgres dependency keeps its `required: false` profile gate.
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
added 3 commits
November 7, 2026 02:00
The real gate only runs on v* tags, which is how the v2.5.0 clean install failed after the tag was cut. This is release.yml's clean-install job, copied verbatim, triggered by PRs that touch the compose stack. Deleted once the fix has shipped.
…ooping The second layer the clean-install gate exposed after the store-init fix: with ATF_CORE_RPC_TOKEN empty the worker exited at startup with "internal RPC token is empty: refusing an unauthenticated client" — even when ATF_WORKER_ENABLED=0. The container then restart-looped, its /ops endpoint (probed by the API's health payload) never served, the stack reported degraded, and the release gate failed again. The compose contract always said refusal is for an ENABLED worker: "Empty + enabled = refuse to start, on purpose. Until then the Python Core keeps its queue loop and this worker only reports health." The code deviated by constructing the gateway client unconditionally and treating its refusal as fatal regardless of the switch. The disabled path now boots with the health-only telegraph.Gateway (internal/telegraph/unconfigured): every operation fails closed with TRANSPORT_UNCONFIGURED — non-transient by construction, the refusal carries the startup reason verbatim and names the operation that was never attempted — so a latent caller gets an honest failure instead of a nil-pointer panic. An ENABLED worker with an empty token still refuses to start: fail closed, on purpose.
Its purpose is served: the gate ran on this PR, caught the regressions before any tag, and is now green (3m43s). The permanent home for this check is release.yml's clean-install job on every v* tag; keeping the copy would run a full second compose stack on every master PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
The v2.5.0 clean-install gate (release run 42301129426) caught a real fresh-install bug:
The worker container crash-looped on startup, its
/opsendpoint (probed by the API's health payload) never came up, the stack reporteddegraded, and the release gate failed. A stockdocker compose upinstall shipped with a dead worker — this is exactly what the clean-install gate exists to catch.Root cause — a profile/ordering mismatch
workercontainer is not profile-gated: it always runs to report health (only execution is gated byATF_WORKER_ENABLED).worker-store-init— the one-shot thatchgrp 101+g+ws the sharedatf.dbfor the worker's non-root user — was behindprofiles: ["worker"], so a stockupnever ran it.atf.db, fail-closed on purpose (per fix(worker): refuse to start on an unwritable store; mount the shared db at a worker-owned path #285's "refuse to start on an unwritable store"), and restart-looped.The fix — two ordering holes, not one
compose upnow.atf.dbdoes not exist until the core creates it — an init that ran earlier wouldchgrpan empty directory and leave the newly created file root-owned, reintroducing the same crash loop on first boot.Idempotency is unchanged (re-runs after a core restart stay safe); postgres keeps its
required: falseprofile gate.Verification
docker compose configclean..env.example+ATF_DISABLE_BOT=1,compose up -d --build, and the same health-poll assertion — to be confirmed green before merge (result posted in this PR).