Repository navigation
fix(xtask): stop orphaned nss units on service stop - #48
Merged
Merged
Conversation
stop_service left nss@N to its role agent. When the agent is killed or its 5s stop timeout hits, the unit outlives the stage teardown, the next init formats around it, and the fresh agent adopts it because the health check only compares the journal uuid. The stale NSS still holds the previous layout's single-node metadata volume, so stopping that BSS node panics it mid-test (state log write quorum W=1 N=1), which is what failed bss-node-failure on the x86_64 CI runner. Once no agent runs, any nss@N is an orphan, so stop it too. The CI debug artifact now keeps 50k journal lines so the teardown window before a failure is captured.
thomas-fractalbits
added a commit
that referenced
this pull request
Oct 10, 2026
fix(xtask): stop orphaned nss units on service stop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
stop_servicenow stops any lingeringnss@Nunit after the role agents are stopped. The agent normally stops its NSS on shutdown, but when it is killed or hits its 5s stop timeout the unit survives the stage teardown, the nextinit_serviceformats around it, and the fresh agent adopts it since the health check only compares the journal uuid.Root cause of the x86_64 CI failure (run 35305969572)
The
s3 api tests (Etcd backend)stage runs with one BSS node; its NSS loaded a single-node metadata volume group. That NSS outlived the stage teardown and was adopted by thebss-node-failurestage's role agent. When the test stopped bss-0, the stale NSS's state-log persist failed withwrite quorum failed: ok_ops=0, W=1, N=1and panicked; the agent SIGKILLed and restarted it, and the degraded puts failed during the ~15s restart window. It is timing dependent, which is why arm64 and the previous x86_64 run passed.Reproduced locally by SIGKILLing the role agents and running
service stop:nss@0stayed active before the change and is stopped after it.