Skip to content

feat(sdk)!: pause takes force and returns whether the sandbox was paused and whether forcefully - #1983

Open
tomassrnka wants to merge 3 commits into
mainfrom
feat/pause-force
Open

tomassrnka wants to merge 3 commits into
mainfrom
feat/pause-force

Conversation

@tomassrnka

@tomassrnka tomassrnka commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Summary

pause() takes a new force option and returns a result object instead of a boolean, in both SDKs, following the API's new pause contract: the node may refuse a pause (503, error_code: "sandbox_pause_refused") and the sandbox keeps running, with the reason in the message. When the reason is an unresponsive sandbox guest agent, force takes the snapshot without the health check. A successful pause now answers 200 {"forced": boolean} instead of 204.

Depends on the API change in belt pull request 3536; the generated-files check stays red until the runtime pin carries the new spec.

What changed

Generated clients (spec/openapi.yml, JS schema, Python client)

spec/openapi.yml is never hand-edited in this repo: it is copybara-synced from e2b-dev/runtime at spec/runtime-ref, and the required Generated files check regenerates from that pin. This PR nevertheless carries the pause hunks of belt's oss/runtime/spec/openapi.yml (pull request 3536 head) hand-applied to the tracked spec so reviewers can read the contract next to the SDK code: SandboxPauseRequest.force (default false), the SandboxPauseResult schema, the endpoint's 200 and 400 responses, and the sandbox_pause_refused clause inserted into the existing Error.error_code list (belt's wording verbatim; the other codes belt's spec has gained are not pulled in). Both clients are regenerated from it with the repo's generators at their pinned versions; the regenerated diff is limited to those items.

The Generated files check stays red until the pin carries the contract. Sequence to green: (1) belt 3536 merges; (2) the belt→runtime mirror lands the spec in e2b-dev/runtime; (3) bump spec/runtime-ref to that runtime commit; (4) run make codegen, which re-fetches the spec at the pin — replacing the hand-applied hunks with the synced file — and regenerates both clients; commit the result. If the synced spec differs from the hand-applied hunks only by the unrelated changes runtime has gained meanwhile, the generated clients still change only where those land.

JavaScript SDK (e2b)

  • SandboxPauseOpts.force?: boolean, sent in the request body when set and omitted otherwise.
  • SandboxApi.pause() / Sandbox.pause() (and the deprecated betaPause) return Promise<SandboxPauseResult> with paused and forced (type exported). 200 → { paused: true, forced: body.forced }, 409 → { paused: false, forced: false }, 204 from an older API → { paused: true, forced: false }.
  • New SandboxPauseRefusedError (subclass of ServiceBusyError, statusCode 503, errorCode) for the refused pause, mapped centrally in apiErrorFromCode / handleApiError on the body's error_code (so any endpoint answering sandbox_pause_refused gets it, including errors embedded in fork results); its message is the API's verbatim, which already names the reason and what to do for it (the SDK appends nothing; the general guidance lives in the class docs). A 503 with no error_code stays ServiceBusyError; 409 is still handled at the pause call site; 404 stays SandboxNotFoundError; 400 (force on a filesystem-only pause) is SandboxError with statusCode 400. The code string is one constant, SANDBOX_PAUSE_REFUSED_ERROR_CODE, exported from the package and read by the error class, the mapping and the retry predicate; SandboxPauseResult is an interface.
  • Transport: a 503 whose body carries error_code: "sandbox_pause_refused" is not replayed by withRetry (other 503s are retried up to retries times), so a refusal surfaces after one request instead of re-running the node's health probe four times. The body is peeked at only when its media type is JSON (case-insensitive, parameters ignored), unconsumed and unlocked and the request is not already aborted, through a reader the predicate owns, capped at 8 KiB and bounded by its own inspection budget of 1 s (at least the retry delay, within the remaining request budget, so a slow refusal body is still caught and a trickling non-refusal body costs at most a second per attempt) and the request signal, waited through the injected sleep so one clock governs all waiting; a stalled or oversized body is not a refusal, the reader is cancelled so the abandoned read stops consuming the stream, the time spent peeking counts towards the delay, and the 503 is retried as before. The original body stays untouched for the caller.
  • CLI: e2b sandbox pause takes -f, --force, passes it only when given (so the SDK omits it from the request body otherwise), says when the result reports a forced pause, and prints a refused pause's API message (no stack) before exiting 1.

Python SDK (e2b)

  • pause(..., force: Optional[bool] = None) on Sandbox and AsyncSandbox (instance and class forms); unset is omitted from the payload.
  • Returns the SandboxPauseResult dataclass (paused, forced), exported from e2b, with the same mapping as the JavaScript SDK.
  • New SandboxPauseRefusedException (subclass of ServiceBusyException, status_code 503, error_code), mapped centrally in api_exception_from_code / handle_api_exception on the body's error_code, with the API's message verbatim; a non-object JSON body is treated as carrying neither field. The code string is one constant, SANDBOX_PAUSE_REFUSED_ERROR_CODE, exported from e2b. The other mappings are unchanged. The deprecated beta_pause forms take force too, matching the JavaScript betaPause.
  • Transport: RetryableTransport / AsyncRetryableTransport (which retry 503s like the JavaScript SDK) peek at a 503 body only when its media type is JSON (case-insensitive, parameters ignored) and it is not declared larger than 8 KiB, streaming it under the existing try with an 8 KiB cap and a wall-clock budget of min(max(delay, 1 s), remaining request budget) measured with the transport's monotonic clock across chunks (each chunk is also bounded by the request's read timeout as pyqwest applies it; none when the request timeout is disabled) — a chunk whose wait began within the budget is kept, only waiting for more past it gives up; a failed, timed-out, over-cap or over-budget read decides nothing and the 503 is closed and retried; with no retry budget left the 503 is the outcome, as on the JavaScript side: a body read in full is returned as is, one stopped at the byte cap or cut off by the wall-clock budget is handed on whole (peeked chunks first, then the rest of the stream — as before this PR), and a failed read re-raises its error — never a fabricated body, since the API client parses every 503 body as JSON. While budget remains, the iterator a stopped peek left is closed explicitly before the retry. A body nested too deep to parse is not a refusal; an interruption mid-peek (cancellation, KeyboardInterrupt) closes the inner response before propagating; the handed-on stream closes the inner response in a finally and tolerates a body generator a reader is still running; a non-string error_code is not a refusal rather than a TypeError; a peek cut off by the budget on the last chunk holds one complete JSON document and is treated as the body read in full (a refusal is returned at once, any other complete API error such as a busy 503 follows the normal path), the tail left unread; only a body that does not yet parse is handed on. A handed-on stream's live tail is bounded by the request deadline (a chunk whose wait would begin past it is not waited for; the inner response is closed and ReadTimeout raised), so the API client's read ends at the deadline instead of running on chunk by chunk — and since a retry budget can only be exhausted after a peek when the clock is already at the deadline, a hand-on in practice delivers what the peek buffered and then the timeout, while a body read in full or buffered chunks that already form the refusal come back as content. A refusal is returned as a response rebuilt from the bytes read (the transport's response carries no request yet, so the handled request is attached explicitly; Content-Encoding, Transfer-Encoding and Content-Length dropped; only plain extensions such as http_version copied, never pyqwest's get_trailers bound to the closed native response), so the caller reads it as the API sent it. The pause tests over httpx.MockTransport now run through these transports with responses built the way httpx and pyqwest hand them over (no request=), which is what exposed the missing-request failure.

forced reports what the API did rather than echoing the option: the API answers forced: false when a forced request joined a normal pause already in flight. The refusal code covers more than an unresponsive guest agent (belt also emits it for a busy node and for a forced pause it cannot pin); the message carries the reason, and the docs on every method, overload and error class say so, with the reboot caveat for a forced snapshot (its memory may not resume; onResume: 'reboot' / on_resume="reboot" recovers the disk state).

Migration

pause() returned true when the sandbox got paused and false when it was already paused. Read paused from the result where that boolean was read:

// before
if (await sandbox.pause()) { ... }
// after
if ((await sandbox.pause()).paused) { ... }
# before
if sandbox.pause(): ...
# after
if sandbox.pause().paused: ...

A truthiness check on the result object is always true, so every call site that branched on the old boolean must be updated. Changeset: major for e2b (JavaScript) and @e2b/python-sdk, and — because @e2b/code-interpreter and @e2b/desktop export * from 'e2b' and subclass its Sandbox, and e2b-code-interpreter / e2b-desktop subclass the Python Sandbox — major for those four as well (.changeset/config.json has updateInternalDependencies: patch, which would otherwise ship the breaking pause() in a patch of theirs); @e2b/cli minor. The two Python dependents' e2b bound is raised from <3.0.0 to <4.0.0 in their pyproject.toml (locks regenerated with uv lock; they still resolve the workspace e2b) so they keep resolving against e2b 3.x. Verified with pnpm changeset status --verbose.

Verification

  • JavaScript: pnpm build (tsc --noEmit + tsdown), oxlint, prettier; vitest unit project on tests/sandbox/pause.test.ts, tests/sandbox/apiDefaults.test.ts, tests/sandbox/lifecycleRequest.test.ts, tests/api/handleApiError.test.ts, tests/retry.test.ts (msw-mocked API: force omitted/true/false/with mode in the body; 200 with forced true and false; forced reported false for a forced request that joined a normal pause; 204 compatibility; 409; coded refusal → SandboxPauseRefusedError, still instanceof ServiceBusyError, after exactly one request; the busy-node refusal; 503 with no error_code → ServiceBusyError; 400 with the status; 404; at the transport, a coded 503 answered after one request with its body intact, a 503 with another code still retried, a 503 whose body stalls or drips retried after the delay with the peek's reader cancelled, a refusal arriving within the delay still short-circuiting, and an oversized body treated as a plain 503).
  • Python: ruff check, ruff format --check, ty check; pytest on tests/shared/sandbox/test_pause.py, tests/shared/sandbox/test_api_defaults.py, tests/test_api_exception.py, tests/test_rate_limit_retry_transport.py, same cases for sync and async, plus tests that go through the generated client's real response parsing over an httpx.MockTransport (200 with forced true and false, the 204 older-API path, a coded refusal), and a JSON-list 503 body still mapping to ServiceBusyException.
  • CLI: pnpm typecheck, oxlint, prettier; vitest on tests/commands/sandbox/pause.test.ts (default, --force / -f, forced-but-not-forced, already paused).
  • The live-sandbox suites were not run; the wire shapes come from the belt spec and handler.

@tomassrnka

Copy link
Copy Markdown
Member Author

API counterpart: https://github.com/e2b-dev/belt/pull/3536 (the pause contract: refusal when the guest agent is unresponsive with the sandbox left running, the force override, the 200 result).

@cla-bot cla-bot Bot added the cla-signed label Oct 14, 2026
@changeset-bot

changeset-bot Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 891dec7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 7 packages
Name Type
e2b Major
@e2b/python-sdk Major
@e2b/code-interpreter Major
@e2b/desktop Major
@e2b/code-interpreter-python Major
@e2b/desktop-python Major
@e2b/cli Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cursor

cursor Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

PR Summary

High Risk
Breaking pause() return type and semantics across JS/Python SDKs and subclassing packages; retry-layer body peeking on 503 affects all API calls that can return pause refusals.

Overview
pause() is a breaking change in the JS and Python SDKs (and dependents): it returns SandboxPauseResult with paused and forced instead of a boolean, accepts force to snapshot without a guest-agent health check, and maps API refusals (503, error_code: sandbox_pause_refused) to SandboxPauseRefusedError / SandboxPauseRefusedException. Transport retries peek JSON 503 bodies and do not replay that refusal code. The OpenAPI spec adds force, SandboxPauseResult, and 200 pause responses. e2b sandbox pause adds -f/--force and prints refusal messages from ServiceBusyError. Python wrapper packages bump e2b to <4.0.0; changesets mark major SDK releases and a minor CLI bump.

Reviewed by Cursor Bugbot for commit 891dec7. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Package Artifacts

Built from dbda15a. Download artifacts from this workflow run.

JS SDK (e2b@2.56.1-feat-pause-force.0, with @e2b/dockerfile-utils@0.1.2-feat-pause-force.0):

npm install ./e2b-dockerfile-utils-0.1.2-feat-pause-force.0.tgz ./e2b-2.56.1-feat-pause-force.0.tgz

CLI (@e2b/cli@2.22.1-feat-pause-force.0):

npm install ./e2b-cli-2.22.1-feat-pause-force.0.tgz

Code Interpreter JS SDK (@e2b/code-interpreter@2.9.1-feat-pause-force.0):

npm install ./e2b-code-interpreter-2.9.1-feat-pause-force.0.tgz

Desktop JS SDK (@e2b/desktop@2.5.1-feat-pause-force.0):

npm install ./e2b-desktop-2.5.1-feat-pause-force.0.tgz

Python SDK (e2b==2.56.0+feat.pause.force, with e2b-dockerfile-utils==0.1.1+feat.pause.force):

pip install ./e2b_dockerfile_utils-0.1.1+feat.pause.force-py3-none-any.whl ./e2b-2.56.0+feat.pause.force-py3-none-any.whl

Code Interpreter Python SDK (e2b-code-interpreter==2.11.0+feat.pause.force):

pip install ./e2b_code_interpreter-2.11.0+feat.pause.force-py3-none-any.whl

Desktop Python SDK (e2b-desktop==2.7.0+feat.pause.force):

pip install ./e2b_desktop-2.7.0+feat.pause.force-py3-none-any.whl

devin-ai-integration[bot]

This comment was marked as resolved.

@tomassrnka

Copy link
Copy Markdown
Member Author

On the red checks at ed2b77a: Generated files is red by design until spec/runtime-ref carries the new contract. The four Staging SDK jobs fail on the template build timeout and the run-command-as-missing-user message in both SDKs, which the three most recent completed PR runs on this repository (38414259158, 38411904011, 38391379064) fail identically, so they predate this branch. The one difference from baseline is Production Python's paginate-paused-sandboxes test (one paused sandbox listed instead of two right after two pauses); its async twin passed in the same job and this branch does not touch listing or the production pause contract, which still answers 204 and is mapped to paused true; I have re-run the failed jobs to discriminate a listing flake from a cause.

@tomassrnka

Copy link
Copy Markdown
Member Author

Re-run at fa5a195: the Production legs are green, including the auto-resume and file-watch tests that failed once, so those were flakes; what stays red is Generated files (by design until the runtime pin moves) and the four Staging legs failing on the pre-existing template-build and run-command tests.

@tomassrnka
tomassrnka marked this pull request as ready for review October 14, 2026 20:30
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-14T20:32:25.533230Z fa5a195 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fa5a19561d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread spec/openapi.yml
Comment on lines +1160 to +1162
force:
type: boolean
default: false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Advance the source pin before changing the synced spec

Because spec/runtime-ref is unchanged, the Generated files workflow will run make codegen, re-fetch this file from the old pinned runtime commit, and report the resulting spec/client changes as uncommitted, so every PR containing this commit deterministically fails that required check. Wait for the upstream runtime spec, bump the pin, and regenerate rather than applying the pause schema directly.

AGENTS.md reference: AGENTS.md:L4-L4

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and this is the known sequencing constraint rather than an oversight: the pause contract lands in belt 3536 first, mirrors to runtime, then this PR bumps spec/runtime-ref and runs make codegen, which replaces the hand-applied hunks with the generated ones and turns Generated files green. The hunks are kept in the meantime so reviewers can read the contract the SDK code is built against; the PR body states the sequence and the PR does not merge before it. Leaving this thread open as the tracker until the pin moves.

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings marked 🟡 are optional suggestions and need no follow-up push.

Comment thread spec/openapi.yml
Comment on lines +1160 to +1177
force:
type: boolean
default: false
description: >-
Take the pause even when the sandbox guest agent is unresponsive.
Applies to memory pauses only. The memory snapshot may not resume;
a reboot resume recovers the disk state. Defaults to false.

SandboxPauseResult:
type: object
required:
- forced
properties:
forced:
type: boolean
description: >-
Whether the pause was taken with the force override, skipping the
guest agent health check.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The next make codegen at the current pins will silently undo the pause contract in both generated clients. CLAUDE.md forbids editing spec/ by hand. This hunk adds force, SandboxPauseResult and the 200/400 responses, but leaves spec/runtime-ref unchanged. Codegen re-fetches the spec at that pinned commit, which does not have these hunks. Fix: keep spec/openapi.yml and the generated files byte-equal to codegen at the pinned refs. Land this once runtime-ref carries the belt change, or bump the pin in this PR. [also at: spec/openapi.yml:3338 - Maintainers get a hand-edited API spec that the next sync will overwrite, which breaks the root CLAUDE.md rule against editing spec/ by hand.]

Why this was flagged

spec/runtime-ref still pins 90ad3bdbb1a4ad78161463dc90b66b4f8303897b. Per CLAUDE.md, make codegen re-fetches spec/openapi.yml at that commit before generating, and the PR description says that spec does not have the pause hunks. So the next contributor who runs codegen for an unrelated spec change drops SandboxPauseRequest.force, SandboxPauseResult and the pause 200 response from spec/openapi.yml, schema.gen.ts and post_sandboxes_sandbox_id_pause.py. In JS, res.data?.forced at packages/js-sdk/src/sandbox/sandboxApi.ts:1678 then no longer typechecks. In Python, the regenerated client goes back to parsing only 204, so a 200 pause loses its body and forced comes back as False. The PR author says the generated-files check stays red until the pin moves. That confirms the drift; it does not prevent it. The base has no such mismatch.

Verification: The diff hand-edits spec/openapi.yml, but spec/runtime-ref is not touched and still reads 90ad3bdbb1a4ad78161463dc90b66b4f8303897b. The Makefile defines codegen: fetch-specs, which overwrites spec/openapi.yml with the copy at the pinned commit. So the next make codegen at this pin drops force, SandboxPauseResult and the 200 response from spec/openapi.yml and packages/js-sdk/src/api/schema.gen.ts.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and this is the known sequencing constraint rather than an oversight: the pause contract lands in belt 3536 first, mirrors to runtime, then this PR bumps spec/runtime-ref and runs make codegen, which replaces the hand-applied hunks with the generated ones and turns Generated files green. The hunks are kept in the meantime so reviewers can read the contract the SDK code is built against; the PR body states the sequence and the PR does not merge before it. Leaving this thread open as the tracker until the pin moves.

Comment thread .changeset/pause-force.md
Comment thread packages/python-sdk/e2b/api/__init__.py
Comment thread packages/python-sdk/e2b/retry.py Outdated
@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed 10864ed after a full adversarial pass: the error message is the API's verbatim with the guidance moved to the class docs; Python tests now run the generated client's real response parsing for 200, 204 and the refusal; the CLI gains --force and reports a forced pause; the error-code constant is exported from both package roots; the Python transports read only 503 bodies and tolerate a non-dict body; the JS retry predicate bounds the body read by the retry budget. The spec hunks stay hand-applied for readability; the PR body states the sequence that turns Generated files green (belt 3536 merges, the runtime mirror lands, bump spec/runtime-ref, make codegen).

devin-ai-integration[bot]

This comment was marked as resolved.

@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed 1b4eff5: the JS retry predicate owns its reader, caps the peek at 8 KiB and bounds it by the retry delay; the Python transports read a 503 body only when small and inside the retry's error handling; the CLI prints a refused pause as one line; the changeset marks the four re-exporting dependents major. Both regressions from the last pass have tests that fail on the previous head.

devin-ai-integration[bot]

This comment was marked as resolved.

@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed d575ecb: the dripping-body test source is cancel-safe, so the JS unit run exits clean in every runtime again; the peek returns not-a-refusal before touching a consumed, locked or missing body, a non-JSON body or an already-aborted request; the untouched-body assertions sit on a response that was actually peeked; the peek waits through the injected sleep with a 50 ms floor; Python subtracts the body-read time from the following sleep and re-checks the deadline.

devin-ai-integration[bot]

This comment was marked as resolved.

@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed cdeda30 for the three findings on the retry peek: a bounded stream read in Python with no length-header dependency, an inspection budget independent of the backoff in JS, and a case-insensitive media-type check in both.

devin-ai-integration[bot]

This comment was marked as resolved.

@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed 6396fcb: a 503 whose body read fails and whose budget is then exhausted is returned to the caller as the JS side does, with tests sync and async; the CLI passes force only when the flag is given, following the SDK's omit path.

cursor[bot]

This comment was marked as resolved.

@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed 504f271. The last pass caught two Python regressions that the fakes had masked: the rebuilt refusal response read a request httpx attaches only later, and an empty-body 503 broke the generated parser. Both fixed; the pause tests now run through the SDK's own retry transport over fakes shaped like the production transport (no pre-attached request), and the previous head fails that test with the runtime error while this one passes. Also in: the wall-clock cap on the Python peek, the rebuilt response dropping transfer-encoding and the trailers extension, and the retry docs stating the inspection budget.

@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed 9a7e507: the Python lint format failure fixed, the exhausted-budget outcomes distinguished (a complete body is returned, an over-cap body is handed on whole through a resumed stream so the caller still reads the refusal, a budget cut-off raises the retry timeout, a read error re-raises), each tested sync, async and through Sandbox.pause; the peek closes its generator explicitly when it gives up.

@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed 4dc4424: a peek cut off by the wall-clock budget at an exhausted retry budget is handed on whole through the resumed stream, as the over-cap case and the JS side do; the leftover iterator is closed deterministically on retry and on stream close; one rebuild path; tests cover the live-iterator replay, the header stripping on the stream form, the async complete body and the over-cap refusal through Sandbox.pause.

devin-ai-integration[bot]

This comment was marked as resolved.

@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed 6eba48d: the Python peek treats a RecursionError from a pathological JSON body as not-a-refusal, closes the inner response on any BaseException out of the peek (cancellation tested), and the resumed stream closes the inner response in a finally; over-cap fixtures prove the byte-cap branch on its own; one rebuild path.

@tomassrnka
tomassrnka force-pushed the feat/pause-force branch 2 times, most recently from 6eba48d to 7f66599 Compare October 14, 2026 23:11
@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed 7f66599: a non-string error_code is treated as not-a-refusal; a running sync generator's close is tolerated; buffered chunks that already form the refusal are returned without pulling the tail; the handed-on tail is bounded by the request deadline. Consequence stated in the body: a refusal body still trickling at the request deadline ends in the read timeout rather than being read past the deadline.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread packages/python-sdk/e2b/retry.py
@tomassrnka

Copy link
Copy Markdown
Member Author

Final adversarial pass at 7f66599 scores this 9/10, mergeable once the known spec sequence runs (belt 3536 merges, the runtime mirror lands, bump spec/runtime-ref, make codegen replaces the hand-applied hunks and turns Generated files green). Verified on a clean box: lint, format, type check, the non-live Python suite, the full JS unit project with zero unhandled errors, CLI build and tests, changeset plan (six majors, CLI minor), and the refusal, cut-off, over-cap, cancellation and cross-close paths against real httpx. Two notes for a later tidy, neither blocking: the resumed-stream classes are reachable only in a one-clock-read window now that the tail is deadline-bounded, so they could collapse to replay-then-timeout; and a trickling 503 at the deadline surfaces as httpx's ReadTimeout rather than the SDK's TimeoutException, which is the API client's pre-existing contract.

…esult

What changed: spec/openapi.yml takes the pause contract from belt pull
request 3536 (head b65468a1a7, oss/runtime/spec/openapi.yml):
SandboxPauseRequest gains an optional boolean `force` (default false), a
successful pause answers 200 with the SandboxPauseResult object
`{"forced": boolean}` instead of 204, the endpoint documents 400 (force
on a filesystem-only pause), and the sandbox_pause_refused clause is
inserted into the existing Error.error_code list with belt's sentence
verbatim; the other codes belt's spec has gained are not pulled in. The
JavaScript schema and the Python client are regenerated from it with the
repo's generators (redocly 2.54.3 + openapi-typescript,
e2b-openapi-python-client 0.26.2 + ruff), at the pins in
codegen.Dockerfile and the python-sdk codegen group.

Why: the node may refuse a pause (503 with error_code
sandbox_pause_refused) and keep the sandbox running, and lifts the
guest-agent health check only when the caller passes force; the SDKs
need the field, the result type and the code to expose that.

Only the pause hunks are taken: belt's spec is ahead of the runtime pin
in spec/runtime-ref by unrelated changes (egress proxy TLS, pauseMode,
admin sandbox lookup, idempotency keys). spec/openapi.yml is normally
copybara-synced from e2b-dev/runtime at that pin, so the generated-files
check stays red until runtime-ref moves to a runtime commit that carries
this contract; bump the pin and re-run `make codegen` then.

Verification: the regenerated diff is limited to the pause endpoint,
SandboxPauseRequest.force, the new SandboxPauseResult model and the
Error.error_code clause; `pnpm build` (tsc --noEmit) and `ruff check` /
`ruff format --check` / `ty check` pass.
…ndboxPauseRefusedError for a refused pause

What changed: `SandboxPauseOpts.force` is sent in the pause request body
when set (omitted otherwise, like `memory`). `SandboxApi.pause()`,
`Sandbox.pause()` and the deprecated `betaPause()` return
`Promise<SandboxPauseResult>` (an interface `{ paused, forced }`,
exported) instead of a boolean: 200 answers
`{ paused: true, forced: body.forced }`, 409 (already paused)
`{ paused: false, forced: false }`, and a 204 from an older API
`{ paused: true, forced: false }`. The CLI's `sandbox pause` reads
`paused` from the result, takes `-f, --force` (passed only when given,
so the SDK omits it from the body otherwise), and says when the result
reports a forced pause.

A 503 whose body carries `error_code: "sandbox_pause_refused"` is raised
as the new `SandboxPauseRefusedError` (exported; `statusCode` 503,
`errorCode`, subclass of `ServiceBusyError` so existing handlers keep
working). The mapping lives in `apiErrorFromCode` / `handleApiError`,
keyed on the body's `error_code` and passed through for errors embedded
in fork results too, so any endpoint answering the code gets the typed
error; the pause call site keeps only the 409 → `paused: false` case.
The message is the API's verbatim: belt's messages already name the
reason and what to do for it (retry, retry without force, or pause with
force and the reboot caveat), so the SDK appends nothing and the
general guidance lives in the class docs. A 503 without the code stays
a `ServiceBusyError`, 404 stays `SandboxNotFoundError`, and 400 (force
on a filesystem-only pause) is the generic `SandboxError` carrying the
status. The code string is one constant,
`SANDBOX_PAUSE_REFUSED_ERROR_CODE`, exported from the package and read
by the error class, the mapping and the retry predicate.

The transport no longer replays a 503 carrying that code: `withRetry`
peeks at a 503 body and returns the response when its `error_code` is
non-retryable, so a refusal surfaces after one request instead of
re-running the node's health probe and restore cycle on each of the
retries. The peek runs only for a body whose media type is JSON
(lower-cased, parameters dropped), unconsumed and unlocked, on a
request not already aborted (anything else is "not a refusal" and
retried as before, instead of a synchronous TypeError failing the
request); it reads a clone through a reader it owns, capped at 8 KiB
and bounded by its own inspection budget of max(retry delay, 1 s)
within the remaining request budget — so a slow refusal body is still
caught when the first backoff is 50 ms, and a trickling non-refusal
body costs at most a second per attempt — and by the request signal,
waited through the injected sleep so RetryDependencies.sleep governs
all waiting (clamped so a huge budget cannot overflow setTimeout); when it stops early the reader is
cancelled so the abandoned read stops consuming the stream, the time
spent counts towards the delay, the deadline is re-checked, and the
503 is retried as before, so a stalled or trickling body cannot eat
the retry budget. The caller's body stays untouched. Other 503s are
retried as before. The CLI prints a refused
pause's API message (no stack) before exiting 1.

The docs on the option, the methods and the error class describe the
refusal as the node refusing the pause with the sandbox kept running and
the reason in the message (belt emits the code for an unresponsive guest
agent, a busy node and a forced pause it cannot pin), and present force
as the answer for the unresponsive-guest reason only, with the reboot
caveat.

Why: the API now refuses a memory pause when the sandbox guest agent is
unresponsive and keeps the sandbox running; callers need to tell that
apart from a busy node (retry) and need the override to snapshot
anyway, accepting that the memory may not resume (a resume with
`onResume: 'reboot'` recovers the disk state). The result object is a
breaking change, decided by the operator: `forced` is informative, not
an echo of the option, since the API reports `forced: false` when a
forced request joined a normal pause already in flight, and a boolean
cannot carry it. Migration: read `paused` where the boolean was read.

Verification: unit tests (msw-mocked API) cover force omitted / true /
false / with mode in the body; 200 with forced false and true; forced
reported false for a forced request that joined a normal pause; 204
compatibility; 409; the instance method; the coded refusal mapped to
SandboxPauseRefusedError (still instanceof ServiceBusyError) with the
API message verbatim, after exactly one request; the busy-node
refusal; a 503 with no error_code staying ServiceBusyError; 400
carrying the status; 404. handleApiError tests cover the central
mapping on the code regardless of endpoint and a 503 with another code
staying ServiceBusyError. The transport tests assert a coded 503 is
returned after one request with its body intact, a 503 with another
code is still retried, a stalled body is retried after each attempt's
inspection budget and delay (sleeps 1000/50, 1000/100, 1000/200 ms on
the stub clock, four requests), a refusal arriving within the delay and
one arriving after the first backoff but within the inspection budget
both short-circuit, a mixed-case Content-Type is matched by media type, a dripping body (cancel-safe source) is
retried with the peek's reader cancelled (the retried sources
cancelled, no further pulls past the tee's high-water mark; the
untouched-body assertions sit on the peeked-and-returned refusal), an
oversized body is a plain 503, and a consumed body, a locked body and
an already-aborted request are not peeked at. The whole js-sdk unit
project runs with zero unhandled errors; its only failures without an
API key are the live-sandbox suites. CLI tests cover the default, --force /
-f, a forced request the API did not force, a refused pause printing
the API message and exiting 1, and an already paused sandbox. `pnpm
build`, oxlint, prettier, the CLI typecheck and lint, and the unit
suites pass (249 js-sdk tests across the five touched files, 5 CLI
tests).
…; SandboxPauseRefusedException for a refused pause

What changed: `Sandbox.pause()` and `AsyncSandbox.pause()` (instance and
class forms, and the deprecated `beta_pause` forms, matching the
JavaScript `betaPause`) take `force: Optional[bool] = None`, sent in
the pause request body when set and omitted otherwise, like `mode`, and
return the `SandboxPauseResult` dataclass (`paused`, `forced`),
exported from `e2b` under its own section, instead of a bool: 200 gives
`paused=True, forced=body.forced`, 409 (already paused)
`paused=False, forced=False`, and a 204 from an older API
`paused=True, forced=False`.

A 503 whose body carries `error_code: "sandbox_pause_refused"` is raised
as the new `SandboxPauseRefusedException` (exported; `status_code` 503,
the `error_code` attribute, subclass of `ServiceBusyException` so
existing handlers keep working). The mapping lives in
`api_exception_from_code` / `handle_api_exception`, keyed on the body's
`error_code` and passed through for errors embedded in fork results
too, so any endpoint answering the code gets the typed exception; a
non-object JSON body (a list) carries neither field and maps as before;
the pause call site keeps only the 409 → `paused=False` case. The
message is the API's verbatim: belt's messages already name the reason
and what to do for it, so the SDK appends nothing and the general
guidance lives in the class docstring. A 503 without the code stays a
`ServiceBusyException`, 404 raises `SandboxNotFoundException`, and 400
(force on a filesystem-only pause) is the generic `SandboxException`
carrying the status. The code string is one constant,
`SANDBOX_PAUSE_REFUSED_ERROR_CODE`, exported from `e2b` and read by the
exception class, the mapping and the retry predicate.

`RetryableTransport` / `AsyncRetryableTransport`, which retry 503s the
same way the JavaScript transport does, peek at the body only for a 503
whose media type is JSON (lower-cased, parameters dropped) and that is
not declared larger than 8 KiB, streaming it through iter_bytes /
aiter_bytes under the existing try with that cap and a wall-clock
budget of min(max(delay, 1 s), remaining request budget) measured with
the transport's monotonic clock across chunks (each chunk is also
bounded by the request's read timeout as pyqwest applies it, none when
the request timeout is disabled; a chunk whose wait began within the
budget is kept, only waiting for more past it gives up), so a chunked
or HTTP/2 refusal without Content-Length is read and a trickling body
cannot hold the loop: a read that fails, times out, exceeds the cap or
outlives the budget decides nothing and the 503 is closed and retried,
where an earlier draft let the exception escape after one request; the
time the read took is subtracted from the following sleep and the
deadline re-checked, as the JavaScript side does; with no retry budget
left, the 503 is the outcome as on the JavaScript side: a body read in
full is returned as is, one stopped at the byte cap or cut off by the
peek budget is handed on whole (the peeked chunks first, then the rest
of the stream, through a resumed stream whose close releases the body
iterator and the original response), and a failed read re-raises its
error, never a fabricated body (the API client parses every 503 body
as JSON). While budget remains, the iterator a stopped peek left is
closed explicitly before the retry, so nothing is left to garbage
collection; the rebuilt response is constructed once, with httpx taking
the stream over the content when both are given. A body nested too deep
for json.loads (RecursionError, not ValueError) is not a refusal; a
BaseException out of the peek (cancellation, KeyboardInterrupt) closes
the inner response before propagating, so the connection is not leaked;
the resumed stream closes the inner response in a finally and tolerates
a body generator a reader is still running (RuntimeError for an async
generator, ValueError for a sync one; left to that reader). An
``error_code`` that is not a string (a list or object from a proxy's or
a buggy server's body) is not a refusal rather than a TypeError out of
the transport. A peek cut off by the budget on what turns out to be the
last chunk holds one complete JSON document: it is treated as the body
read in full (a refusal is returned at once; any other complete API
error follows the normal path, retried while budget remains and
returned whole when it is exhausted) and the tail is left unread; only
a body that does not yet parse is handed on. Without this a busy 503
whose body completed in the cut-off chunk surfaced as ReadTimeout
instead of ServiceBusyException. The handed-on stream's live tail is bounded by the
request deadline: a chunk whose wait would begin past it is not waited
for, the inner response is closed and ReadTimeout is raised, so the
API client's read ends at the deadline instead of running on chunk by
chunk past the request timeout. Because the peek's time counts against
the retry delay, a budget exhausted after a peek means the clock is at
the deadline, so in practice an over-cap or cut-off body handed on at
that point yields what the peek buffered and then the timeout; a body
read in full, or buffered chunks that already form the refusal, are
returned as content and never reach the tail. The two readers'
cross-close tests therefore build the resumed streams directly with the
deadline ahead.
A refusal is returned after one request as a response rebuilt from the
bytes read: the transport's response carries no request yet (httpx
attaches it after the transport returns; pyqwest's arrive without one),
so the handled request is attached explicitly, Content-Encoding,
Transfer-Encoding and Content-Length are dropped (the content is
decoded and complete; httpx sets the length), and only plain
extensions such as http_version are copied, never pyqwest's
get_trailers bound to the closed native response. The earlier draft
read response.request and raised RuntimeError on the production
transport for every refused pause; the pause tests missed it because
their fakes attached request= and bypassed the retry layer.

`@e2b/code-interpreter`, `@e2b/desktop`, `@e2b/code-interpreter-python`
and `@e2b/desktop-python` take the same major in the changeset: the
JavaScript two re-export `e2b` and subclass its Sandbox, the Python two
subclass the Python Sandbox, and `updateInternalDependencies: patch`
would otherwise ship the breaking `pause()` in a patch of theirs. The
two Python dependents' `e2b` bound is raised from `<3.0.0` to `<4.0.0`
so they keep resolving against e2b 3.x (their locks are unchanged: they
resolve the workspace e2b). `pnpm changeset status --verbose` lists
exactly those six majors and the CLI minor.

Docstrings on every overload and on the exception describe the refusal
as the node refusing the pause with the sandbox kept running and the
reason in the message, and present force as the answer for the
unresponsive-guest reason only, with the reboot caveat. The two live
tests that asserted the old bool read `.paused`. The changeset is major
for both SDKs (minor for the CLI) with the migration note.

Why: the API now refuses a memory pause when the sandbox guest agent is
unresponsive and keeps the sandbox running; callers need to tell that
apart from a busy node (retry) and need the override to snapshot
anyway, accepting that the memory may not resume (a resume with
`on_resume="reboot"` recovers the disk state). The result object is a
breaking change, decided by the operator: `forced` is informative, not
an echo of the option (the API reports `forced=False` when a forced
request joined a normal pause already in flight). `force` is
`Optional[bool]` rather than `bool = False` so an unset flag is omitted
from the payload (#1749: no SDK-side defaults in API request bodies).
The generated 200 model is imported as `SandboxPauseResultAPI` to keep
the public dataclass name.

Verification: unit tests with the generated client mocked cover force
omitted / true / false / with mode in the body; 200 with forced False
and True; forced reported False for a forced request that joined a
normal pause; 204 compatibility; 409; the coded refusal raising
SandboxPauseRefusedException (a ServiceBusyException) with the API
message verbatim; the busy-node refusal; a 503 with no error_code
staying ServiceBusyException; 400 carrying the status; 404, for sync
and async. Tests over an httpx.MockTransport-backed ApiClient /
AsyncApiClient now run through RetryableTransport /
AsyncRetryableTransport with responses built as httpx and pyqwest hand
them over (no request=): the refusal-through-real-parsing tests fail
on the earlier draft with RuntimeError and pass here; they also cover
200 with forced True and False, the 204 older-API path (parsed is
None) and, through Sandbox.pause, a refusal body read that times out
with no budget left surfacing as the timeout. The transport fakes lost
their request= too. The transport tests assert a coded 503 is returned
after one request with its body intact, a chunked refusal without
Content-Length is returned after one request with its body readable
(sync and async), an oversized chunked body and a non-JSON body are
retried, a trickling body is retried once the peek budget is spent
(sync and async), a complete but slow body read with no budget left is
returned with its body (sync and async), an over-cap body with no
budget left surfaces the 503 whole with its third chunk read only by
the resumed stream and no length or transfer-encoding header (sync and
async, and through Sandbox.pause as SandboxPauseRefusedException), a
read cut off by the budget with no budget left is handed on whole too
(sync and async, and through Sandbox.pause as the typed refusal), a
3000-deep JSON body is retried (sync and async), cancelling the peek
while it awaits a chunk closes the inner response exactly once, and
closing the handed-on response while a reader awaits its live tail
closes the inner exactly once (an asyncio task, and a thread inside
iter_bytes), a list-valued error_code is retried (sync and async), and
a cut-off peek whose buffered chunks form the refusal returns it
without retrying and without pulling the tail (sync and async, and
through Sandbox.pause and AsyncSandbox.pause), a busy 503 whose
complete body lands in the cut-off chunk is retried while budget
remains and surfaces as ServiceBusyException through Sandbox.pause
and AsyncSandbox.pause when none is left, the over-cap deadline cut is
covered through AsyncSandbox.pause too, a handed-on tail is cut at the deadline with
ReadTimeout and the inner closed once (transport sync and async, and
through Sandbox.pause for both the cut-off and the over-cap case), a failed read with no budget left re-raises (sync and
async), a mixed-case Content-Type is matched by media type,
the read time counts towards the delay, the rebuilt refusal drops the
transfer headers and native extensions and carries the request, a 503
with another code is still retried, sync and async. `ruff check`,
`ruff format --check`, `ty check` and the non-live pytest suite pass (670 tests); `make lint` exits 0 and a second `ruff format` leaves
every file unchanged.
@tomassrnka

Copy link
Copy Markdown
Member Author

Pushed 891dec7 for the last point: a complete API error body that lands in the cut-off chunk is surfaced as that error rather than a read timeout; async twins of the over-cap and cut-off pause tests added. On the two production legs that went red at 7f66599: the JS node leg failed an env-var test with a bare fetch failure and the code-interpreter Python leg a context removal with a connection reset, both live-sandbox network errors on paths this change does not touch.

@tomassrnka

Copy link
Copy Markdown
Member Author

Confirming pass at 891dec7: the 9/10 holds. The complete-JSON shortcut is prefix-safe for object bodies (the API error schema is always an object), the 8 KiB bound precedes parsing, and the busy 503 takes the ordinary retry or return-whole path. Three docstring sites still describe the earlier hand-on-whole wording and the shortcut could require a dict rather than any JSON value; both are tidy-ups for a later pass, not blockers.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant