[ryuzdev]

Root cause

Intermittent write cgroup.subtree_control: EBUSY at runner boot in the multi lane (CI run 39767096081, job 119785212884, PR #93).

The drain+enable retry loop added in #68 never fired: write_file wrapped failures with anyhow!("write {name} in {dir}: {e}"), formatting the io::Error into a string and destroying the error chain. is_busy (and is_gone) classify off error.chain() looking for a matching raw_os_error — unreachable through a formatted message. So the first orchestrator healthcheck exec (compose.yaml probes /ready every 10 s; docker exec joins the container cgroup by design) landing between is_drained and the subtree_control write bailed the boot. The /ready detail matches exactly: enabling /sys/fs/cgroup: ... EBUSY with no describe() suffix is the non-busy bail path, i.e. is_busy returned false on an EBUSY. Unit tests hand-built anyhow!(io::Error) directly, never through write_file, which is why they stayed green.

Fix (deterministic)

• write_file (plus the two other cgroup read sites) keeps the OS cause with with_context; is_busy/is_gone now classify production errors.
• By construction, the only possible squatters are external: cgroups::setup runs in main before the first child spawn, and the drain moves the runner itself into the leaf, so everything the runner forks afterward inherits the leaf. The bounded retry (50 x 100 ms) rides out transient external execs; a persistent squatter still degrades loudly past the budget. The retry is justified, not a loophole: the race is external (orchestrator healthchecks) and unavoidable from inside the container.
• Extracted drain_and_enable (verbatim loop, injected enable/settle seams + proc_root) for the barrier-driven test.
• Security-review hardening: a fleet that exits between spawn and its cgroup move now SIGKILLs its tree (orphaned workers kept the group and the port alive uncapped).

Verification

• cgroup_write_errors_keep_their_os_cause: fails pre-fix (chain erased, None vs EISDIR), passes now.
• busy_enable_retries_past_a_spawn_racing_init: fake cgroupfs + fake procfs, squatter thread pinned into the drain-to-enable window by barriers (no sleeps); proves one contested enable retries and the racer drains next round.
• cargo fmt --check, cargo clippy --locked --all-targets -- -D warnings, full cargo test --locked (590 passed), cargo build --release --locked, ts-rs bindings unchanged. bun run check skipped: Rust-only diff, no node_modules in this env.
• CI: release suite via the e2e label (this PR), watched to completion.

Security review

Independent security-reviewer pass over the diff (commit c465faf plus follow-up §1/§2 fixes in this branch): no blockers. Findings, all non-blockers:

1. Gone arm skipped the group kill its own log claims (orphaned worker survives uncapped) — fixed here (sigkill_tree(pid) in the is_gone arm, ESRCH-safe per its contract).
2. Two more format-erasing map_err sites (drain read, enable read-back) — fixed here (with_context).
3. drain_to_leaf swallows permanent denials as exited-mid-drain (burns the 5 s budget, still loud) — declined for this PR: boot-timing behavior change needing its own test; tracked as follow-up.

Coordination: F2 runs in parallel; this branch touches cgroups.rs only for the boot-enable path plus the notes above. If F2's root cause lands in the same file for the same reason I will say so; no shared edits so far.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ryuzdev]

Parent review: root cause and fix accepted.

• The bug: write_file formatted the io::Error into its message, so is_busy/is_gone (which classify off error.chain()) never fired.
• The fix: with_context keeps the OS error on the chain.
• The retry: bounded, and justified because the only racers left are external: orchestrator healthcheck execs that join the container cgroup.
• Tests: the chain-preserving test fails before the fix, and the barrier-driven contested-enable test needs no sleeps.

I squashed and rebased onto 519ecc5 and moved the CHANGELOG entry to changelog.d/multi-cgroup-boot.md. Rust fmt, clippy and 590 tests pass.

The multi-lane worker-errors.spec.ts:166 failure (method "" vs "GET") is a separate flake, G (error occurrence stored before its access-log line is tailed). A dedicated agent is on it, and it isn't caused by this diff. Labelled security-reviewed with auto-merge on.