Repository navigation
fix(runtime-core): release a disposed VM's buffered sidecar events - #2065
Merged
Merged
Conversation
The host event buffer kept every event no waiter claimed, so each VM left its unclaimed lifecycle events (disposing, disposed) behind. A long-lived sidecar overflowed its 4096-event buffer after about 2048 create and dispose cycles and then failed every call. When the sidecar reports a VM disposed, release that VM's buffered events instead of buffering more. Also export SidecarEventBufferOverflow from @rivet-dev/agentos-core and secure-exec.
|
This PR was not deployed automatically as @eersnington does not have access to the Railway project. In order to get automatic PR deploys, please add @eersnington to your workspace on Railway. |
Contributor
📟 Pip-Boy ·
|
| package (linux-x64-gnu) | @latest |
base | this PR | Δ vs base | packages |
|---|---|---|---|---|---|
@rivet-dev/agentos-core |
458.2 MB | 458.2 MB | 458.2 MB | +1.1 KB (+0.0%) | 121 (0) |
`@rivet-dev/agentos-core` details
| platform | base | this PR | Δ |
|---|---|---|---|
| linux-x64-gnu | 458.2 MB | 458.2 MB | +1.1 KB (+0.0%) |
| linux-arm64-gnu | 455.4 MB | 455.4 MB | +1.1 KB (+0.0%) |
| darwin-arm64 | 434.7 MB | 434.7 MB | +1.1 KB (+0.0%) |
| darwin-x64 | 444.2 MB | 444.2 MB | +1.1 KB (+0.0%) |
| win32-x64 | 314.6 MB | 314.6 MB | +1.1 KB (+0.0%) |
Entrypoints (own code, minified; third-party imports excluded)
| entrypoint | base | this PR | Δ | imports | exports |
|---|---|---|---|---|---|
. |
291.3 KB | 291.6 KB | +324 B (+0.1%) | — | +SidecarEventBufferOverflow |
All 6 packages · linux-x64-gnu
| package (linux-x64-gnu) | @latest |
base | this PR | Δ vs base | packages |
|---|---|---|---|---|---|
@rivet-dev/agentos-core |
458.2 MB | 458.2 MB | 458.2 MB | +1.1 KB (+0.0%) | 121 (0) |
@rivet-dev/agentos-node-pty |
7.6 KB | 18.0 KB | 18.0 KB | 0 | 1 (0) |
@rivet-dev/agentos-posix |
253.6 MB | 253.6 MB | 253.6 MB | +909 B (+0.0%) | 7 (0) |
@rivet-dev/agentos-runtime-core |
253.6 MB | 253.6 MB | 253.6 MB | +909 B (+0.0%) | 6 (0) |
@rivet-dev/agentos-sandbox |
495.5 MB | 495.5 MB | 495.5 MB | +1.1 KB (+0.0%) | 177 (0) |
@rivet-dev/agentos-toolchain |
130.8 KB | 141.2 KB | 141.2 KB | 0 | 2 (0) |
🧹 Rust dependencies
Skipped: this PR doesn't change any Rust code or dependencies.
🦀 Binaries
Skipped: this PR doesn't change any Rust code.
⏱ Timing
| job | wall time | slowest step |
|---|---|---|
zizmor |
8s | checkout · 2s |
changes |
29s | checkout · 25s |
footprint |
2m01s | Pack this commit · 41s |
⏭ self-test |
— | skipped |
⏭ hygiene |
— | skipped |
⏭ native · ${{ matrix.platform }} |
— | skipped |
report |
17s | setup-node · 4s |
🔭 Full CI trace for this commit
One OpenTelemetry trace of CI and Pip-Boy for 04f0b6e: every job and step, with the sizes above attached. Run after CI finishes.
pnpm install --frozen-lockfile --filter 'pipboy...'
pnpm --filter pipboy pipboy trace --sha 04f0b6e8d0e5b261fac9be356eb74c0780a119fcAdd --otlp <url> to send it to any OTLP/HTTP collector.
Agent instruction: From a rivet-dev/agentos checkout, run the commands above, then read the trace file the last command prints. List the slowest jobs and steps and explain where CI time went.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2052.
A long-lived sidecar, including the default shared one, failed every call with
SidecarEventBufferOverflowafter about 2048 VM create/dispose cycles.vm_lifecycledisposinganddisposedafter the VM's event loop has stopped, so no waiter claims them. The client buffered them forever: 4096 / 2 = 2048 VMs.disposedevent arrives unclaimed, the buffer now releases every buffered event for that VM. A waiter that is already registered still receives the event.SidecarEventBufferOverflowis exported from@rivet-dev/agentos-coreandsecure-exec.main