Repository navigation
fix(aws): stop s3fs caching missing FC/kernel/envd objects - #3510
Closed
tomassrnka wants to merge 2 commits into
Closed
tomassrnka wants to merge 2 commits into
tomassrnka wants to merge 2 commits into
Conversation
The four s3fs mounts passed "-o enable_noobj_cache", which is the s3fs default rather than an override, so negative lookups were cached for stat_cache_expire (900s). A host that looked for a Firecracker/kernel/envd version before it was uploaded kept returning ENOENT for 15 minutes after the upload landed, and the only workaround was to unmount and remount. Pass "-o disable_noobj_cache" instead so a missed lookup is retried against S3 on the next access. Positive stat cache entries are unaffected.
PR SummaryLow Risk Overview Reviewed by Cursor Bugbot for commit ed2d6bb. Bugbot is set up for automated code reviews on this repo. Configure here. |
sitole
reviewed
Aug 3, 2026
Comment on lines
+78
to
+85
| # | ||
| # disable_noobj_cache: s3fs caches "this object does not exist" answers for | ||
| # stat_cache_expire (900s by default). These buckets are written out-of-band - | ||
| # a new Firecracker/kernel/envd version is uploaded after a host has already | ||
| # looked for it - so a cached miss makes the freshly uploaded object invisible | ||
| # on that host until the entry expires or the mount is recreated. | ||
| # Note that "enable_noobj_cache" is the s3fs default, so passing it was a no-op; | ||
| # "disable_noobj_cache" is the option that actually turns the negative cache off. |
sitole
approved these changes
Aug 3, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jakubno
force-pushed
the
fix/g1-s3fs-noobj
branch
from
August 3, 2026 13:33
63effc2 to
ed2d6bb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear: https://linear.app/e2b/issue/EN-386
Broken: on AWS BYOC hosts, an FC version (or kernel/envd build) uploaded to S3 after a host had already tried to read it stayed invisible on that host — sandboxes kept failing; only known workaround was unmount → clean dir → remount.
Root cause: all four s3fs mounts in
start-client.shpass-o enable_noobj_cache, which is a no-op — negative ("does not exist") caching is the s3fs default and the real toggle is-o disable_noobj_cache. Every missed lookup is remembered forstat_cache_expire(default 900 s), and these buckets are written out-of-band, so a cached miss outlives the upload by up to 15 minutes.Fix: replace (not append — later flag wins)
enable_noobj_cachewithdisable_noobj_cacheon the four mounts.Repro (MinIO + s3fs 1.93, the Ubuntu 24.04 base-AMI version, exact production flags): miss → upload → read: production flags not found for >100 s; flag removed, identical (confirms the no-op);
stat_cache_expire=20visible after ~40 s (confirms the expiry is the bound);disable_noobj_cachevisible after ~1 s. With no prior lookup the object is always visible immediately, isolating the negative cache as the cause.Verification: the
disable_noobj_cacherun above is the passing case.shellcheck -S warningreports only the pre-existing SC2155 on line 43.