fix(bootstrap): stream image push through temp file to prevent OOM#700
Merged
fix(bootstrap): stream image push through temp file to prevent OOM#700
Conversation
The image push pipeline buffered the entire Docker image tar 3x in memory (export, tar wrap, Bytes copy), causing OOM kills for images over ~1-2 GB. Replace the in-memory pipeline with a temp-file + streaming upload: export to a NamedTempFile, then stream the outer tar (header, 8 MiB file chunks, footer) directly into upload_to_container via body_try_stream. Peak memory drops from ~3x image size to ~8 MiB constant. Also adds incremental export progress reporting every 100 MiB.
johntmyers
approved these changes
Mar 31, 2026
Contributor
|
Great! Thank you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix unbounded memory allocation during
openshell sandbox create --fromimage push that causes OOM kills on images >1-2 GB. The image push pipeline now streams through a temp file with ~8 MiB constant memory usage instead of buffering 3x the image size in RAM.Related Issue
Closes #602
Closes #655
Changes
crates/openshell-bootstrap/src/push.rs: Rewrote the image push pipeline:collect_export()→export_to_tempfile(): Streamsdocker.export_images()directly to aNamedTempFileinstead of collecting intoVec<u8>wrap_in_tar()+upload_archive()→streaming_tar_upload(): Returns aStreamthat yields the outer tar (512-byte header → 8 MiB file chunks → padding/EOF blocks) usingasync_stream::try_stream!bollard::body_try_stream()instead ofbody_full(), streaming directly intoupload_to_container()crates/openshell-bootstrap/Cargo.toml: Promotedtempfilefrom dev-dependency to dependency; addedasync-streamMemory profile comparison
Testing
mise run pre-commitpasses (lint, format, license headers, tests)openshell-bootstrapunit tests passChecklist