Skip to content

fix(execd): stage isolated-session file writes so a failed write keeps the old file - #2476

Closed
JasonBuildAI wants to merge 6 commits into
opensandbox-group:mainfrom
JasonBuildAI:fix/execd-isolated-upload-atomic-7d3f
Closed

JasonBuildAI wants to merge 6 commits into
opensandbox-group:mainfrom
JasonBuildAI:fix/execd-isolated-upload-atomic-7d3f

Conversation

@JasonBuildAI

@JasonBuildAI JasonBuildAI commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #2475

Summary

Isolated-session file writes are the only remaining way to lose a file in execd: session.files.write_file / write_files (all five SDKs) and POST /v1/isolated/session/{id}/files/upload opened the destination with O_TRUNC and streamed into it. A write that failed partway — ENOSPC, an I/O error, a short read, execd killed mid-copy — left the file it was replacing empty or half-written, and created an empty destination where none existed. Isolated sessions are the documented mechanism for RL rollouts and batch code grading, where the harness seeds each task through this API and the workspace upper directory holds the only copy.

This is the defect class #2432 reported and #2433 fixed for the ordinary /files/upload route. #2433 explicitly left this path out of scope, and docs/architecture/data-plane/execd.md recorded the asymmetry. All three vfs.FS implementations behind the session files API had it, and the same code serves files/replace through WriteFile:

  • merged_view.go — the overlay upper directory
  • bind_view.go — every binds destination
  • confined_view.go — the strict profile's private /tmp

The fix stages the bytes in a private temp file next to the destination, syncs it, and renames it into place, so a partial write leaves the destination untouched. One shared helper serves both the plain and the os.Root-confined views, so the confined view keeps its confinement: the final-component symlink walk goes through root.Lstat/root.Readlink, and a planted link cannot redirect the rename outside the tree.

Revisions

397301d — first revision. It closed the O_EXCL staging descriptor and reopened its pathname.

e0c61629 — addresses three review findings on that revision:

  • The exclusive descriptor is held end to end. Reopening gave a workload that can write the destination directory — the ordinary case for a writable bind or a user-owned subdirectory of the strict /tmp — a window to replace the staging entry with a symlink and redirect the payload. It also made a new 0444/0000 destination fail, because the second O_WRONLY open applies the final mode. The descriptor now carries the write, the metadata changes and the rename, and a pre-rename os.SameFile check rejects a swapped entry instead of renaming it into place.
  • The destination's write permission is probed. A rename only needs a writable directory, so an existing destination is opened O_WRONLY before staging, matching writeUploadStream. A read-only file inside an execd-writable directory is rejected as before, and a destination directory that is not writable keeps working through the in-place fallback the ordinary route already has.
  • Ownership is applied before the mode. Every write previously chowned the committed path after the helper returned, and a chown clears setuid/setgid even when the IDs are unchanged, so replacing a 06755 file produced 0755. The staging inode is chowned first and its mode restored after.

06fd3013 — addresses an independent review of e0c61629:

  • rw listings leaked staging files. rw mode builds a view with LowerDir == UpperDir, so the lower pass of ReadDir and the lower walk of Search returned entries the upper pass filters — the sentence this branch had added to the docs was false. Both lower paths now filter staging names, but only when the lower directory is the upper one, so a genuine overlay lower layer (read-only, so a matching name there is a real user file) is untouched.
  • The controller filtered the wrong prefix — isUploadStagingName matches .osb-upload-, not the .osb-staging- this path creates. It now checks both via an exported isolation.IsStagingName.
  • Cleanup could delete a session-owned inode. The deferred removal was unconditional on the write-error path, so a workload that renamed its own file onto the staging name lost it. The staging inode identity is now captured at create time and removal runs only while the name still refers to it — the invariant the code already documented, applied to both exit paths.

77a3d042 — addresses a second independent review of 06fd3013, which found the cleanup fix itself still reachable:

  • The cleanup ownership check ran after Close. The open descriptor is the only thing pinning the staging inode; closing it first frees the inode number, so the allocator can hand it to a file the session creates in the gap and os.SameFile then matches — execd deletes the session's file, which is the defect the check exists to prevent. The check now runs while the descriptor is still open.
  • createStagingFile's Stat-failure branch removed the staging name unconditionally, the same bug in a new place. It now leaves the entry alone; listings and the read paths hide it anyway.
  • The read paths still served staging names. Hiding a temp file from listings does not help if Open/ReadFile/Stat reach it by name — all three views did. They now reject the reserved final component, and the prefix is documented as reserved.
  • Error text leaked the temp path. Every staged-write error now names the destination the caller asked for, since the controller returns err.Error() verbatim.
  • The rw lower-pass filter is unconditional now. Gating it on LowerDir == UpperDir made the view and the controller disagree about a genuine user file that matched the pattern, and added a path-comparison edge case (trailing slash, symlink vs resolved, case on Windows). The prefix is reserved in both passes instead.

225a1715 — addresses a third independent review of 77a3d042:

  • The staging prefix was only reserved on read. Rename and Remove still reached a staging file: a session could rename an in-flight upload to a user-visible name and download the partial bytes, or unlink it to cut the upload short. The check is now part of each view's path resolution (MergedView.safePath, ConfinedView.rel, BindView.resolve), so every operation rejects it; the write paths keep a raw resolver because a staged write resolves its own destination, never a staging name.
  • Three error paths still named the temp file, despite the previous commit claiming otherwise: the create failure, the swapped-entry error, and a rename failure — which surfaces as *os.LinkError and so slipped past the errors.As on *os.PathError. All three now name the destination.
  • The rename-failure path could not honour the ownership check because the descriptor was already closed. POSIX can unlink an open file, so the close moved after the rename and the check now runs with the inode still pinned; Windows cannot, so it closes first and keeps the narrower window, documented in staging_cleanup_windows.go.
  • The docs claimed a symlinked destination is written through for every view. True for the binds and confined views, false for the workspace views, which reject it as they always have. The sentence now says which is which.

814336e4 — takes the three nits from a fourth verification pass, which found the change converged:

  • No test drove a real rename failure, so moving the POSIX close back before the rename would have left the suite green. TestStagedWriteRenameFailureKeepsTheInodePinned now fails the rename through a recording filesystem and asserts both the message and the descriptor-pinning.
  • Windows double-closed the descriptor on the rename-failure path; the flag is now set whenever the pre-rename close succeeds.
  • The docs said no file API reaches a staging name, which is false for a session's own symlink pointing at one. The sentence now states what the guard covers.

Behavior preserved from the in-place path:

  • The destination's mode is restored before the rename, including setuid/setgid/sticky.
  • A symlinked destination is still written through for the bind and confined views, which mirror the namespace; the merged view keeps rejecting symlinks as before.
  • A non-regular destination (device, FIFO) is rejected rather than replaced by a regular file.
  • Staging names are hidden from ReadDir and Search in every mode, so an in-flight or crashed write is not returned as a user file.

Known limits, stated rather than papered over:

  • .osb-staging- is a reserved name prefix: hidden from files/search and directories/list, and rejected as the path argument of every file API — read, stat, rename, remove. A user file that matches it is unreachable through the API, and a session's own symlink pointing at one can still be read through, which exposes only that session's in-flight upload.
  • verifyStagingEntry narrows the swap window to two adjacent syscalls but cannot close it; os/*os.Root has no portable linkat/O_TMPFILE primitive.
  • The rename installs a new inode: an existing hardlink keeps the old content, and an open reader in an rw session keeps the old bytes.
  • The in-place fallback still truncates on partial failure, and its writability probe only fires when execd is unprivileged — the shipped image runs execd as root, where the probe never fires.
  • On Windows the cleanup closes the staging descriptor before the ownership check, because Windows cannot unlink an open file. Isolated sessions are a Linux-runtime feature, so that path only keeps the package correct.
  • The ordinary /directories/list and /files/search routes know only the .osb-upload- prefix, and /files/download and /files/info filter neither. Reaching a staging name there needs the host upper path, which the API does not return — latent rather than proven safe.
  • A parent-directory RemoveAll/Rename can still cut a concurrent upload short, which is a legitimate target rather than a guard hole.
  • A staging file orphaned by a killed session, or by a Close() error, occupies space until the workspace is cleaned up.
  • A bind whose own source directory is named like a staging file is unlistable at its root; contrived, and worth fixing only if that resolver is reworked.

Follow-up worth doing: stage into an execd-owned 0700 directory on the same filesystem instead of next to the destination. Staging into a directory the session can write is the single choice that forces the swap window, the inode-recycling check, the reserved prefix, and the listing filters; moving it would delete most of staged_write.go's careful parts. It changes where temp files live on the host and touches workspace lifecycle, so it does not belong in this PR.

This is a follow-up to #2433 and completes the guarantee it introduced.

Testing

  • Not run (explain why)
  • Unit tests
  • Integration tests
  • e2e / manual verification (driver-level reproduction, no live sandbox)

The two core regressions in merged_view_atomic_write_test.go are driven with a reader that yields some bytes and then fails. On unmodified main they fail exactly as the issue reports:

--- FAIL: TestMergedView_WriteFileReaderFailureKeepsTheReplacedFile
    expected: "previous content\n"
    actual  : "partial"
--- FAIL: TestMergedView_WriteFileReaderFailureLeavesNoNewFile
    Expected error with "file does not exist" in chain but got nil.

Each fix has a regression that fails on the revision before it: TestStagedWriteCreatesRestrictiveMode (0444/0000 new destinations — this, not the swap test, is what pins descriptor retention), TestStagedWriteRejectsNonWritableDestination, TestMergedView_WriteFilePreservesSetuid, TestMergedView_StagingNamesHiddenInRWMode, TestStagedWriteFailureDoesNotDeleteASwappedEntry, and TestStagedWriteInPlaceFallback (baseline behavior, not a regression test). TestStagedWriteRejectsSwappedStagingEntry guards the SameFile check; it also passes on 397301d, so it is not cited as the descriptor proof. TestStagedWriteCleanupDoesNotTrustARecycledInode fails if the cleanup close is moved back before the ownership check, which is how the ordering hole is kept shut; TestViewsRejectStagingNamesOnMutate covers Rename/Remove. The three mid-write hook readers are now one hookedReader.

Checks:

Check Result
go build ./... (execd) pass
go vet ./pkg/isolation/ ./pkg/web/controller/ clean
GOOS=linux, GOOS=darwin, GOOS=windows build clean
gofmt -l on the changed files clean
go test ./pkg/... on Linux, package results vs. the base commit identical
go test ./pkg/isolation/ in a golang:1.25.9 container, POSIX tests also as uid 65534 pass
cd docs && pnpm docs:build build complete

Verification note: the POSIX-only cases skip on this Windows host and were run in a golang:1.25.9 container, the permission-sensitive ones additionally with setpriv --reuid=65534. The full staged-write set (19 tests) passes as both root and uid 65534. One mutation check was run to prove the ordering test is load-bearing: moving the Close back before the ownership check in the cleanup defer makes TestStagedWriteCleanupDoesNotTrustARecycledInode fail. The failures that remain under ./pkg/... depend on the uid: as root they are TestBuildArgv_Setpriv, TestBuildArgv_SegmentOrder, TestOOMSmoke and TestRunInIsolatedSessionBackground_RejectsUnwritableWorkspace; as an unprivileged user the two TestBuildArgv cases and TestRunInIsolatedSessionBackground_* pass, and two pkg/runtime credential tests fail for lack of a passwd entry. Every one of them fails identically on the base commit — they need bwrap, cgroup v2, or a real account, none of which this container has.

Not run: macOS, live bwrap sessions, make golint, make multi-build, the sudo CI steps, and the Kind/e2e suites.

Breaking Changes

  • None
  • Yes (describe impact and migration path)

The wire contract is unchanged and no error status code changed. The staged path needs write permission on the containing directory, which the ordinary upload route already documents; a directory that is not writable falls back to the in-place write, so no previously-succeeding write starts failing.

Checklist

  • Linked Issue or clearly described motivation
  • Added/updated docs (if needed)
  • Added/updated tests (if needed)
  • Security impact considered — the confined view's symlink walk stays inside os.Root; the staging descriptor is never reopened by name; a swapped staging entry is rejected rather than renamed, and cleanup never removes an entry it no longer owns
  • Backward compatibility considered

AI disclosure: this change was drafted with AI assistance at my request, under my direction and review.

…s the old file

Isolated-session file writes opened the destination with O_TRUNC and streamed
into it, so a write that failed partway - ENOSPC, an I/O error, a short read,
execd killed mid-copy - left the file it was replacing empty or half-written,
and created an empty destination where none existed.

This is the same defect opensandbox-group#2432 reported and opensandbox-group#2433 fixed for the ordinary
/files/upload route, which explicitly left this path out of scope; the execd
docs recorded the asymmetry. All three session views had it: the overlay upper
directory (MergedView), binds destinations (BindView), and the strict profile's
private /tmp (ConfinedView), reached by files/upload and files/replace.

Stage the bytes in a private temp file next to the destination, sync it, and
rename it into place, so a partial write leaves the destination untouched.
Keep the destination's mode (including setuid/setgid/sticky) and, for the
root-confined view, resolve a final-component symlink through os.Root so a
planted link cannot redirect the rename outside the confined tree. Hide the
staging names from ReadDir and Search so an in-flight or crashed write is not
returned as a user file. Reject a non-regular destination, which a rename
cannot replace.

Fixes opensandbox-group#2475. Follow-up to opensandbox-group#2433.
@github-actions github-actions Bot added component/execd documentation Improvements or additions to documentation size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 29, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T08:11:48.211332Z 397301d PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

JasonBuildAI added a commit to JasonBuildAI/OpenSandbox that referenced this pull request Oct 29, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 397301d264

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

}
file, err := fsys.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, perm)
if err == nil {
_ = file.Close()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep the exclusive staging descriptor open

When the sandbox workload can mutate the destination directory (for example, a writable bind or a user-owned subdirectory of strict /tmp), closing the O_EXCL file here and reopening its pathname in stagedWrite lets the workload observe the create and replace it with a symlink before execd writes, redirecting the payload to a different file and potentially leaving that link at the requested destination. It also makes an unprivileged execd fail to create a new 0444/0000 file because the second O_WRONLY open applies the final mode, whereas the old os.WriteFile kept the creation descriptor. Return and retain the original descriptor through the write and metadata steps.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e0c6162. createStagingFile now returns the open O_EXCL descriptor and stagedWrite keeps it through the write, the chown/chmod and the rename, so the payload never goes through a pathname a workload can swap. Both halves you named are addressed:

  • The staging entry is re-checked against the descriptor with os.SameFile immediately before the rename. If the session replaced the name with a symlink, the write fails instead of renaming the link (or its target) into place, and the swapped entry is deliberately left alone on cleanup so execd does not delete whatever the session put there.
  • A new destination with a restrictive mode works again: the create descriptor carries the umask-masked final mode, and the second O_WRONLY open that applied the final mode unconditionally is gone. TestStagedWriteCreatesRestrictiveMode covers 0444 and 0000, and TestStagedWriteKeepsTheExclusiveDescriptor swaps the staging entry mid-copy and asserts the decoy is untouched and the destination is never created.

Comment on lines +119 to +121
existing, statErr := fsys.Stat(target)
switch {
case statErr == nil && !existing.Mode().IsRegular():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve destination write-permission checks

When execd runs unprivileged and an existing regular destination is mode 0444 inside an execd-writable directory, this branch accepts it and replacement proceeds by sibling rename, so isolated files/upload or files/replace succeeds even though the prior OpenFile(O_WRONLY|...) path rejected the write. The ordinary writeUploadStream path explicitly probes existing targets with O_WRONLY for this reason; stagedWrite should preserve the same check before staging.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e0c6162. stagedWrite now probes an existing destination with a plain O_WRONLY open before staging anything, matching what writeUploadStream does in pkg/web/controller/filesystem_upload.go, so a 0444 file inside an execd-writable directory is rejected exactly as the in-place path rejected it. The error is the raw os.PathError from the probe, so callers still see os.ErrPermission.

I also carried over the ordinary route's companion behavior: when the containing directory is not writable (0555 with a writable file in it), the write falls back to the in-place path rather than newly failing, and that fallback is documented on both routes. TestStagedWriteRejectsNonWritableDestination asserts the rejection and that the file survives; it skips under root, where the permission check does not bite.

if statErr == nil {
// Restore the destination mode before the rename; writing and chown
// both clear setuid/setgid, so this must happen after the copy.
if err := fsys.Chmod(stagingPath, existing.Mode()&destinationModeMask); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply ownership before restoring special mode bits

Since every successful WriteFile/WriteFileReader in BindView, ConfinedView, and MergedView calls Chown after this helper returns, Linux strips setuid/setgid again—even when the IDs stay the same—so replacing a 06755 file produces 0755. Chown the staging inode first, then restore its mode and sync it before rename, rather than chowning the committed path; otherwise the newly documented mode-preservation guarantee is false.

AGENTS.md reference: AGENTS.md:L46-L46

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e0c6162, and you were right that the documented guarantee was false as written. Ownership is now applied to the staging inode inside stagedWrite, before the mode is restored: Chown first, then Chmod with the destination's destinationModeMask bits, then Sync, then the rename. The three views no longer chown the committed path, so nothing clears setuid/setgid after the mode is set.

TestMergedView_WriteFilePreservesSetuid replaces a 06755 file through MergedView.WriteFile and asserts both bits survive; it skips under Windows and root, since neither can observe the clearing. The in-place fallback applies the same ordering.

Addresses the review findings on the staged isolated-session write.

The exclusive-create descriptor is now held from creation through the
write, the metadata changes and the rename. Closing it and reopening the
staging pathname let a workload that can write the destination directory
(the ordinary case for a writable bind or a user-owned subdirectory of
the strict /tmp) replace the staging entry with a symlink and redirect
the payload; it also made a new 0444/0000 destination fail, because the
second O_WRONLY open applies the final mode. A pre-rename SameFile check
between the name and the open descriptor rejects a swapped entry instead
of renaming it into place, and a swapped entry is left alone on cleanup.

An existing destination is probed with O_WRONLY before staging, so a
read-only file inside an execd-writable directory is still rejected the
way the in-place open rejected it; a rename would otherwise only need a
writable directory. A destination directory that is not writable keeps
working through the in-place fallback the ordinary upload route already
has, and that fallback is now documented on both routes.

Ownership is applied to the staging inode before the mode is restored,
because every session view chowned the committed path after the helper
returned, and a chown clears setuid/setgid even when the IDs are
unchanged. Replacing a 06755 file used to produce 0755; it now keeps its
bits.

Tests cover each finding: a swapped staging entry, a new 0444/0000
destination, a non-writable destination, and a setuid/setgid
replacement.
Follow-up to an independent review of the staged write.

rw mode built a view with LowerDir == UpperDir, so the lower pass of
ReadDir and the lower walk of Search returned the staging entries the
upper pass filters. An in-flight upload to an rw session was visible and
downloadable through files/search and directories/list, and a killed
daemon left a permanent artifact - which falsified the sentence this
branch had just added to the docs. Both lower paths now filter staging
names, but only when the lower directory is the upper one: a genuine
overlay lower layer is read-only, so a file there that merely matches
the pattern is a real user file.

The isolated-session controller filtered isUploadStagingName, which
matches the ordinary route's .osb-upload- prefix, so it never caught the
.osb-staging- names this path creates. It now checks both, through an
exported isolation.IsStagingName.

Cleanup no longer deletes a session-owned inode. It removed the staging
name unconditionally on the write-error path, so a workload that renamed
its own file onto that name lost it. The staging inode identity is
captured at create time and the deferred removal runs only while the
name still refers to it, which applies the invariant the code already
documented to both exit paths.

Also narrows two claims to what the code does: verifyStagingEntry
narrows the swap window rather than closing it (the rename is a
separate syscall), and a staging-path error now names the destination
the caller asked for instead of the temp file.

Tests: rw-mode listing and search hiding, the cleanup-did-not-delete
case, the in-place fallback including its documented truncation
tradeoff, and the swapped-entry test renamed to TestStagedWrite-
RejectsSwappedStagingEntry and extended to assert cleanup leaves the
session's entry alone - it passes on the first revision too, so it was
never the descriptor-retention proof; TestStagedWriteCreatesRestrictive-
Mode is.
@github-actions github-actions Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 29, 2026
@JasonBuildAI

Copy link
Copy Markdown
Contributor Author

Fixed in 06fd301. You are right on both counts, and this was the strongest finding: rw mode builds the view with LowerDir == UpperDir (pkg/runtime/isolated_session_ctrl.go:1031-1036), so the lower pass of ReadDir and the lower walk of Search returned the .osb-staging- entries the upper pass filters. An in-flight upload to an rw session was listed, searchable, Stat-able and downloadable, and a killed daemon left a permanent artifact. The sentence I had just added to docs/guides/isolation-sessions.md was therefore false as written; it is now true.

Both lower paths filter staging names, gated on MergedView.hideLowerStaging(), which returns true only when the lower directory is the upper one. Filtering a genuine overlay lower layer would hide a user file that merely matches the pattern, since that layer is read-only and a staging file can never appear there.

Your second half is also correct and separate: the controller was filtering isUploadStagingName, which matches the ordinary route's .osb-upload- prefix, so it never caught .osb-staging-. Both isolated_session_files.go:116 and :543 now check isUploadStagingName(n) || isolation.IsStagingName(n), with IsStagingName exported from the isolation package as the belt-and-braces layer for views the controller cannot reason about.

TestMergedView_StagingNamesHiddenInRWMode builds the production shape (NewMergedView(resolved, resolved, WorkspaceRW, ...)) and asserts both the listing and the search; it fails on e0c6162.

@JasonBuildAI

Copy link
Copy Markdown
Contributor Author

Second independent-review finding, also fixed in 06fd301: cleanup could delete a session-owned inode.

The invariant was documented one line above the code that violated it. The deferred removal ran fsys.Remove(staging.name) unconditionally, so a session that renamed its own file onto the staging name lost it when write() returned an error — the ordinary partial-failure path, which is exactly when the race is easiest to hit.

The staging inode identity is now captured at create time (stagingFile.created, from file.Stat() on the O_EXCL descriptor) and cleanup removes the name only while ownsEntry still matches it via os.SameFile. verifyStagingEntry uses the same predicate, so the pre-rename check and the cleanup path share one definition. Taking the identity at create time rather than re-statting later is what makes this work once the descriptor is closed.

TestStagedWriteFailureDoesNotDeleteASwappedEntry races a rename onto the staging name and then fails the read, and asserts the session's bytes survive. It fails on e0c6162.

@JasonBuildAI

Copy link
Copy Markdown
Contributor Author

Smaller points from the same independent review, folded in:

  • verifyStagingEntry's comment now says it narrows the swap window rather than closing it. The rename is a separate syscall, and os/*os.Root has no portable linkat/O_TMPFILE primitive, so two adjacent syscalls is the floor here. Reviewers should not read it as a closed hole.
  • A staging-file creation error is wrapped with the destination the caller asked for, so the 500 body no longer names .osb-staging-<hex>.
  • docs/guides/isolation-sessions.md now states that the rename installs a new inode: an existing hardlink keeps the old content, and a process already holding the file open in an rw session keeps reading the bytes it opened.
  • The "4 pre-existing failures" claim in the description was wrong — the set is uid-dependent. As root it is TestBuildArgv_Setpriv, TestBuildArgv_SegmentOrder, TestOOMSmoke and TestRunInIsolatedSessionBackground_RejectsUnwritableWorkspace; as an unprivileged user the two TestBuildArgv cases and TestRunInIsolatedSessionBackground_* pass and two pkg/runtime credential tests fail instead. All of them fail identically on the base commit.
  • TestStagedWriteKeepsTheExclusiveDescriptor is renamed TestStagedWriteRejectsSwappedStagingEntry, with a comment stating that it passes on 397301d too and that TestStagedWriteCreatesRestrictiveMode is the real descriptor-retention proof. It also now asserts cleanup left the session's entry alone.

Not changed, matching your own non-blocking call: the hideStagingEntries slice aliasing (every caller passes a freshly allocated slice; the doc comment now says so) and the reserved-name namespace squatting.

One question the review raised that I want to flag rather than silently decide: P2's severity. You are right that the probe is dead code in the shipped image, because components/execd/Dockerfile has no USER and execd runs as root — as root the old in-place open accepted a 0444 file too. The probe therefore only bites for a rootless/dev deployment. I have kept it, because it is what keeps the non-root behavior identical to the old code and it costs one open/close per replacing write, but I have stopped implying it guards the documented deployment.

Also worth recording from your rev1 analysis: O_EXCL-then-reopen was not just a same-directory redirect. execd runs as root, so following a planted symlink on reopen is arbitrary write as root, reachable by any workload in a directory it can write. That raises the case for the descriptor retention above "P1 as written" — but it is moot now that the pathname is never reopened.

Second independent review of the staged write found the cleanup fix itself
still reachable, plus three smaller gaps.

The deferred cleanup checked ownership after closing the descriptor, and the
open descriptor was the only thing pinning the staging inode. Closing it first
frees the inode number, so the allocator can hand it to a file the session
creates in the gap and os.SameFile then matches - execd deletes the session's
file, the very defect the check was added for. The check now runs while the
descriptor is still open.

createStagingFile's Stat-failure branch removed the staging name
unconditionally, reintroducing the same bug in a new place; it now leaves the
entry alone, which costs nothing because listings and the read paths hide it.

Hiding a staging file from listings is not enough if a client can still reach
it by name: MergedView, BindView and ConfinedView all served Open, ReadFile and
Stat on a staging path. All three now reject the reserved final component, so
the temp file is unreadable end to end and the prefix is documented as
reserved.

Every staged-write error now names the destination the caller asked for,
instead of leaking the .osb-staging- temp path the controller returns verbatim
in its 500 body.

The rw-mode lower-pass filter no longer depends on LowerDir == UpperDir: the
prefix is reserved, and gating the filter on a path comparison made the
controller's filter and the view's filter disagree about a genuine user file
that happened to match the pattern. Filtering unconditionally in both passes
makes the two layers agree and removes a path-comparison edge case
(trailing slash, symlink vs resolved, case on Windows).

Tests: cleanup ordering (fails when the close is moved back before the check),
a session file swapped onto the staging name during a failed write, error text
that must not contain the prefix, and Stat/ReadFile/Open rejection on all
three views. The in-place fallback test is labelled as baseline behavior
rather than a regression test, since it passes before and after.
@JasonBuildAI

Copy link
Copy Markdown
Contributor Author

A second independent review of 06fd3013 found that my cleanup fix was itself still reachable. All of it is fixed in 77a3d042, and I want the details on record because the first review's issue was "cleanup deletes a session file" and my fix left a narrower path to the same outcome.

The check ran after Close. The defer was

_ = staging.file.Close()
if staging.ownsEntry(fsys) { _ = fsys.Remove(staging.name) }

The open descriptor is the only thing pinning the staging inode. Closing it first frees the inode number, so the allocator can hand that number to a file the session creates before the Lstat inside ownsEntry, and os.SameFile then matches a file that is not ours. Reordered so the check and the unlink happen while the descriptor is still open.

TestStagedWriteCleanupDoesNotTrustARecycledInode records whether a live descriptor could still stat its own inode at the moment of the unlink. I mutation-tested it: moving the Close back before the check makes it fail, so the test is load-bearing rather than decorative.

createStagingFile's Stat-failure branch did file.Close() then fsys.Remove(path) unconditionally — the same bug, one function up, introduced by my own commit. It now leaves the entry in place; an orphaned staging file is hidden from listings and unreadable through the API, which is cheaper than risking a session's file.

The read paths. You are right that hiding a temp file from listings is not a guarantee if Open, ReadFile and Stat still reach it by name. MergedView, BindView and ConfinedView now all reject a staging final component, so the file is unreadable end to end, and docs/guides/isolation-sessions.md documents .osb-staging- as a reserved prefix. TestViewsRejectStagingNamesOnRead covers all three calls.

Error hygiene. You were right that #4 was cosmetic: my wrap added the destination but kept the inner *PathError path, and write/sync/rename errors were raw. writeError now rewrites the *PathError.Path to the destination, so no path in the message contains the prefix. TestStagedWriteErrorsDoNotNameTheStagingFile asserts both directions.

The hideLowerStaging gate is gone. Your N3 was the decisive argument: gating the lower-pass filter on LowerDir == UpperDir made the view hide a look-alike user file in a genuine overlay but the controller hide it anyway, so the view's rationale was dead code and the two layers disagreed. Worse, the gate itself was a string comparison that could go wrong on a trailing slash, a symlinked path versus its resolved target, or Windows case/separators — and a false negative there is a leak. The prefix is reserved, both passes filter unconditionally, and the controller filter now agrees with the views by construction.

On the severity calibration I agree with you and not with the first review: #1 is P3 rather than P2 (a session seeing its own in-flight upload through its own authenticated view crosses no boundary), #2 is P3/P4 defense-in-depth on top of #1 rather than a second P2, and #3 stays P3. I also accept that execd running as root does not elevate the rev1 descriptor issue above P1 — the redirect lands in a directory the session can already write, with content the session supplied, so it protects the session from itself rather than the host from the sandbox. The genuine escape vectors are the os.Root confinement and the resolved-path allowlist, neither of which touches the staging fd. The first review's reasoning there was the weak part, even though the descriptor fix was worth making.

Two things I have deliberately left open, both flagged rather than silently decided:

  • The ordinary /directories/list and /files/search routes (filesystem.go:310,383, filesystem_windows.go:307,380) still filter only .osb-upload-. They take arbitrary host paths, and a session workspace or bind source is a host path, so whether a client can reach a .osb-staging- name there depends on what the deployment exposes. I did not confirm end-to-end reachability, and I would rather not widen this PR's blast radius into the ordinary route on a suspicion. If you think it is reachable on a supported deployment, say so and I will add the prefix there too — it is a two-line change.
  • A staging file left by a killed rw session on a persisted workspace is visible to a later ro or overlay session on the same workspace, because those views have no upper directory to filter. That is by design rather than an oversight, but it means the hiding guarantee is scoped to the session that owns the staging file, and the doc sentence now lives in the section for that session.

Verification for this revision: full staged-write set (19 tests) passes in a golang:1.25.9 container as root and as uid 65534; go test ./pkg/... failure set still identical to the base commit; go build, go vet across linux/darwin/windows clean; gofmt clean; pnpm docs:build complete. Also, per your note on TestStagedWriteInPlaceFallback, its comment now labels it baseline behavior rather than a regression test, since it passes both before and after.

Third independent review found the previous commit's claims still incomplete.

The staging prefix was only rejected on read, so Rename and Remove still
reached a staging file: a session could rename an in-flight upload to a
user-visible name and download the partial bytes, or unlink it to cut the
upload short. The check is now part of each view's path resolution -
MergedView.safePath, ConfinedView.rel, BindView.resolve - so every
operation on every view rejects it, and the write paths keep a raw
resolver that bypasses it because a staged write resolves its own
destination and never a staging name.

Three error paths still named the temp file despite the previous commit
claiming otherwise: the create failure, the swapped-entry error, and a
rename failure, which surfaces as *os.LinkError and so slipped past the
errors.As on *os.PathError. All three now name the destination.

The rename-failure path could not honour the ownership check, because the
descriptor was already closed by then. POSIX can unlink an open file, so
the close moved after the rename and the check now runs with the inode
still pinned; Windows cannot, so it closes first and keeps the narrower
window, documented in staging_cleanup_windows.go.

The docs claimed a symlinked destination is written through for every
view. That is true for the binds and confined views and false for the
workspace views, which reject it as they always have; the sentence now
says which is which.

Tests: Rename/Remove rejection, and the error-message test now covers the
rename *os.LinkError path. The three mid-write hook readers collapse into
one hookedReader, and the test that only priced cleanup outcome is renamed
to say so, with TestStagedWriteCleanupDoesNotTrustARecycledInode cited as
the ordering regression.

Still open, and stated in the PR rather than fixed here: the ordinary
/directories/list and /files/search routes know only the .osb-upload-
prefix, and /files/download and /files/info filter neither. Reaching a
staging name there needs the host upper path, which the API does not
return.
@JasonBuildAI

Copy link
Copy Markdown
Contributor Author

Third independent review, and it caught that my previous commit overclaimed in exactly the way the first two rounds warned about. Fixed in 225a1715.

Rename and Remove still reached staging files. I had only rejected the reserved prefix on the read paths, so a session could rename an in-flight upload to a user-visible name and download the partial bytes, or unlink it to cut a concurrent upload short. That directly falsified the doc sentence I had just written. The check now lives in each view's path resolution — MergedView.safePath, ConfinedView.rel, BindView.resolve — so every operation on every view goes through it. The write methods use a raw resolver (resolveRel/rawRel/rawResolve) that bypasses it, because a staged write resolves the user's destination and never a staging name. TestViewsRejectStagingNamesOnMutate covers both operations and asserts the staging file survives a rejected remove.

Three error paths still named the temp file, which makes my previous commit message wrong. The create failure, the swapped-entry error, and — the one that mattered — a rename failure, which returns *os.LinkError: my errors.As on *os.PathError never fired for it, so the raw rename /…/.osb-staging-… … text went to the client. writeError now handles *os.LinkError too, and the test covers it rather than only a synthetic *PathError.

The rename-failure path could not keep the ownership check honest. The descriptor was closed before the rename, so when the rename failed, cleanup ran with the inode unpinned — the exact ordering hole the previous round identified, surviving on one path. POSIX can unlink a file that is still open, so the close moved after the rename and the check now runs with the inode pinned on every POSIX failure path. Windows cannot unlink an open file, so staging_cleanup_windows.go closes first and keeps the narrower window; isolated sessions are a Linux-runtime feature, so that tradeoff is documented rather than engineered around.

The doc claim about symlinks was wrong for the workspace views. "A symlinked destination is still written through" holds for the binds and confined views and not for MergedView, which rejects a symlinked destination exactly as it did before this PR — you verified that and I had carried the sentence over from the original description. The sentence now says which view does which.

On your nits: the three mid-write hook readers are now one hookedReader; the cleanup-outcome test is renamed TestStagedWriteCleanupKeepsASessionFileOnTheStagingName with a comment pointing at TestStagedWriteCleanupDoesNotTrustARecycledInode as the ordering regression, since you were right that it passes under the mutation; and TestStagedWriteInPlaceFallback is labelled in the PR as a regression test against 397301d2 rather than as baseline behavior, which is what my commit message wrongly called it. The duplicate blank-identifier var and its orphan comment are gone.

I have also corrected the PR description's open-items list, dropping the claim that a staging file left by a killed rw session is visible to a later ro/overlay session — you verified that is no longer true at HEAD — and keeping these, now stated as scoped rather than as guarantees:

  1. .osb-staging- is reserved: a user file matching it is not listed, not readable, not renameable to a visible name, and not removable through the file APIs. There is no migration path for a pre-existing file that matches.
  2. The swap window is narrowed to two syscalls, not closed — no portable O_TMPFILE/linkat in os/*os.Root.
  3. A replacement installs a new inode, so hardlinks keep the old content and an open reader in an rw session keeps the old bytes.
  4. The probe in stagedWrite only fires for an unprivileged execd; the shipped image runs execd as root. The in-place fallback therefore does not truncate for the documented deployment, and the docs now say the fallback is a non-root case.
  5. The ordinary /files/search and /directories/list routes filter only .osb-upload-, and /files/download and /files/info filter neither. Reaching a staging name needs the host upper path, which the API does not return, so this is latent rather than exploitable — but unproven either way.
  6. A staging file orphaned by a killed session occupies space until workspace cleanup.

On the last thing in your report I want to flag rather than quietly accept: you are right that staged_write.go now reimplements uploadTempPattern / createUploadTemp / writeUploadStream / writeUploadInPlace / uploadWriteError from filesystem_upload.go. I did not collapse them here because the two live behind different path models — the ordinary route writes to absolute host paths with no root confinement and no per-view ownership, while this one has to go through os.Root and apply a session uid/gid before the mode — and merging them would mean widening this PR into the ordinary route, which the previous rounds argued against. If a maintainer wants one staging engine, the honest shape is a shared helper in a package both can import, with the view-specific ownership and confinement passed in; I would rather do that as a follow-up than smuggle a refactor of #2433's code into a bug-fix PR. Say the word if you think that judgment is wrong.

Verification for this revision: all staged-write tests pass in a golang:1.25.9 container as root and as uid 65534; go test ./pkg/... failure set identical to the base commit; go build for linux/darwin/windows clean; go vet clean; gofmt clean on the changed files; pnpm docs:build complete. Windows native pkg/isolation failure set is byte-identical to the base commit, which is how I caught that a straight POSIX-ordering fix broke staging cleanup on Windows and needed the platform split.

Fourth review found three nits in the previous commit, all addressed here.

The rename-failure cleanup had no test that drove a real rename error, so
moving the POSIX close back before the rename would have left the suite
green. TestStagedWriteRenameFailureKeepsTheInodePinned now fails the
rename through a recording filesystem and asserts both that the message
names the destination and that the descriptor could still stat its own
inode at unlink.

Windows set descriptorClosed only after a failed close, so the pre-rename
close left the flag clear and cleanup closed the descriptor a second time.
The flag is now set whenever that close succeeds.

The docs said no file API reaches a staging name, which is true of the
path argument and false of a session's own symlink pointing at one. The
sentence now says what the guard covers and notes that the symlink case
only exposes that session's own in-flight upload.
@JasonBuildAI

Copy link
Copy Markdown
Contributor Author

Fourth and final verification pass. It converged: all four round-3 defects are fixed, no method on any view is unguarded, and the guard now sits in each view's single resolver rather than in each method — so future methods inherit it. That structural change is why I think this is now closed rather than patched again.

814336e4 takes the three nits that pass raised:

  • No test drove a real rename failure, so moving the POSIX close back before the rename would have left the suite green. TestStagedWriteRenameFailureKeepsTheInodePinned now fails the rename through a recording filesystem and asserts both that the message names the destination and that the descriptor could still stat its own inode at unlink. That is the check you asked for; it is the only follow-up I considered blocking.
  • Windows double-closed the descriptor on the rename-failure path: the pre-rename close succeeded without setting descriptorClosed, so the deferred cleanup closed it again. Harmless (ErrClosed, ignored), but now the flag is set whenever that close succeeds.
  • The docs said no file API reaches a staging name. True of the path argument, false of a session's own symlink pointing at one — which you verified reads through on BindView, ConfinedView, and MergedView's lower/overlay path. The sentence now says what the guard covers and notes that the symlink case only exposes that session's own in-flight upload, which is the same trust domain.

Items I am deliberately leaving as a follow-up rather than expanding this PR, in the order you ranked them:

  1. Move the staging file into an execd-owned 0700 per-workspace directory. I agree this is the design that ends the class: staging into a directory the session can write is the single choice that forces the swap window, the inode-recycling check, the reserved prefix, the listing/search filters, and the symlink-target hole. Doing it here would change where temp files live on the host and touch the workspace lifecycle, which is beyond a bug fix for a data-loss regression. I have noted it in the PR.
  2. The BindView guard uses filepath.Base of an absolute host path, so a bind whose source directory is named .osb-staging-<16 hex> becomes unlistable at its root while relative child paths still work. Contrived, and inconsistent with ConfinedView, which guards the relative name. Worth fixing if anyone ever reworks that resolver.
  3. denyStagingName hardcodes Op: "open", so a rejected Remove reads open …: file does not exist. Cosmetic; I did not want another round on an error string.
  4. Parent-directory RemoveAll/Rename can still cut a concurrent upload short. That is not a guard hole — the parent is a legitimate target — but it belongs in the docs' caveats rather than nowhere.
  5. Any Close() error orphans the staging entry on both platforms, because descriptorClosed short-circuits cleanup. Orphan only: hidden from listings, rejected by the API, occupies space until workspace cleanup.

Your convergence judgment is the part I most agree with, so I am quoting it into the PR rather than paraphrasing it away: the original data-loss fix took about a fifth of the current lines, and rounds two through four defended against a session's own client racing its own upload. That defense was warranted — the deletion bug was real — but the honest summary is that this change is larger than the defect and the right resolution is a follow-up that removes the need for most of it.

I have also added to the PR's open-items list, from your report: the symlink-target read-through, the BindView root edge case, the parent-directory race, the Windows double-close (now fixed) and Close()-error orphan, and the absence of a real-rename regression test (now fixed).

Verification for this revision: all staged-write tests pass in a golang:1.25.9 container as root and as uid 65534; go test ./pkg/... failure set is a subset of the base commit's; go build linux/darwin/windows and go vet clean; gofmt clean on every changed file; Windows native pkg/isolation failure set byte-identical to base; pnpm docs:build complete.

I am not going to run a fifth review round unless you want one — the guard is now structural rather than per-method, which is the property that made the last three rounds keep finding another method, and the remaining items are a deliberate follow-up rather than unknowns.

@JasonBuildAI

Copy link
Copy Markdown
Contributor Author

Closing in favour of #2478.

The code here is correct — it went through three adversarial review rounds and the fourth verification pass confirmed all findings fixed, with the staging-name guard moved from per-method into each view's single path resolver so it cannot be forgotten again. The problem is this thread: seven review comments across four revisions document my own mistakes being fixed in public, which makes the change harder to review than the code deserves.

#2478 carries the same final code as a single commit, rebased onto current upstream/main (97031ea8), with the review findings folded into the PR description instead of the comment history: the descriptor-retention rationale, the write-permission probe and in-place fallback, the chown-before-chmod ordering, the cleanup-ownership rule and why POSIX and Windows differ, the reserved prefix and where the guard lives, plus a known-limits section and the follow-up worth doing separately.

No functional difference from 814336e4 here. Closing rather than force-pushing a single commit onto this branch so the reviewers' existing comments stay attached to the context in which they were made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/execd documentation Improvements or additions to documentation size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] isolated-session file writes truncate the file they replace when the write fails partway

1 participant