fix(codebuild): sweep only build containers whose owning process is gone - #2526
Merged
Merged
Conversation
At startup CodeBuild removed every `fakecloud-codebuild` container whose `fakecloud-instance` label was not the current process -- including those of other fakecloud processes that are still running and sharing the daemon. Any second server starting (parallel e2e test servers, side-by-side installs) killed the first server's in-flight build container, failing the build in PROVISIONING with "failed to create /codebuild/build: " (the `docker exec` into a container that had just been removed). The codebuild_real_execution e2e tests hit exactly that when run in parallel. The shared startup reaper already got this right by checking whether the owning PID is alive. Move that check to fakecloud_core::container_net (`pid_alive` plus `owned_by_dead_process`) and use it from both, so a container is swept only when its owner is neither this process nor alive. Also: ec2_instance_runtime's "container should be running" assertion now reports the container's status, exit code, OOM flag, error, start/finish time and logs, so its remaining intermittent failure names its cause. Tests: owned_by_dead_process unit test; a new e2e starts a second server while a build runs and asserts the build still succeeds (fails with the build FAILED under the old sweep).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With #2520's retried-failure output, #2523's E2E run captured why
codebuild_real_execution::cross_phase_shell_state_persistsstill failed intermittently after #2519:That is the
docker exec mkdirinto the build container right afterdocker run -d-- the container had already been removed.Cause
At startup, CodeBuild swept leaked build containers: every
fakecloud-codebuildcontainer whosefakecloud-instancelabel was not the current process. That includes containers of other fakecloud processes that are still running on the same daemon. When a second server starts -- parallel e2e test servers, or two installs side by side -- it removes the first server's in-flight build container.The shared startup reaper (
fakecloud-server/src/reaper.rs) already does this correctly: it removes an object only when its owning PID is dead. CodeBuild's own sweep skipped that check.Reproduced with a new e2e: start a build on one server, start a second server while it runs. Old sweep: the first build ends
FAILED. New sweep:SUCCEEDED.Fix
pid_alivemoves from the server reaper tofakecloud_core::container_net, plusowned_by_dead_process(label, is_alive): an object is orphaned only when itsfakecloud-<pid>owner is neither this process nor alive, and an unparseable label is never treated as orphaned.libcbecomes acfg(unix)dependency offakecloud-core(already in the lockfile).Also, test-only:
ec2_instance_runtime'scontainer should be runningassertion now reports the container's status, exit code, OOM flag, error, start/finish time and logs. That test also failed once in #2523 (container present but stopped, not a pull failure) and the cause isn't identified yet; the next occurrence will name it.Surfaces
No API, SDK, docs, conformance or count change: startup cleanup behavior only.
Test plan
owned_by_dead_processunit test (live other owner kept, dead owner swept, self kept, unparseable labels kept) and apid_aliveprobe test infakecloud-core; the reaper's existing tests still pass against the moved function.another_server_starting_does_not_kill_a_running_build: passes with the fix; with the old sweep restored the build endsFAILED.cargo clippy -p fakecloud-core -p fakecloud-codebuild -p fakecloud --all-targets -D warnings, e2e test clippy, fmt clean;fakecloud-codebuildlib 67/67.Summary by cubic
Fixes CodeBuild's startup sweep to only remove build containers whose owning fakecloud process is dead. Previously, any container not owned by the current process was removed, so a second server starting on the same daemon would delete another server's in-flight build and fail it in PROVISIONING.
Bug Fixes
pid_alive+owned_by_dead_process) intofakecloud_core::container_netand reuses it in both the server reaper and CodeBuild's sweep.ec2_instance_runtime"container should be running" assertion to include container state and logs on failure.Written for commit 774b459. Summary will update on new commits.