Skip to content

fix(runtime-core): release a disposed VM's buffered sidecar events - #2065

Merged
eersnington merged 1 commit into
mainfrom
fix/event-buffer-disposed-vm
Oct 21, 2026
Merged

eersnington merged 1 commit into
mainfrom
fix/event-buffer-disposed-vm

Conversation

@eersnington

Copy link
Copy Markdown
Member

Fixes #2052.

A long-lived sidecar, including the default shared one, failed every call with SidecarEventBufferOverflow after about 2048 VM create/dispose cycles.

  • Each dispose emits vm_lifecycle disposing and disposed after the VM's event loop has stopped, so no waiter claims them. The client buffered them forever: 4096 / 2 = 2048 VMs.
  • When a VM's final disposed event arrives unclaimed, the buffer now releases every buffered event for that VM. A waiter that is already registered still receives the event.
  • SidecarEventBufferOverflow is exported from @rivet-dev/agentos-core and secure-exec.
  • The Rust client is not affected: it fans events out through a bounded broadcast channel.
2100 create → evaluate → dispose cycles, one sidecar main This PR
Result overflow at cycle 2048 all 2100 succeed

The host event buffer kept every event no waiter claimed, so each VM left its unclaimed lifecycle events (disposing, disposed) behind. A long-lived sidecar overflowed its 4096-event buffer after about 2048 create and dispose cycles and then failed every call. When the sidecar reports a VM disposed, release that VM's buffered events instead of buffering more. Also export SidecarEventBufferOverflow from @rivet-dev/agentos-core and secure-exec.
@railway-app

railway-app Bot commented Oct 21, 2026

Copy link
Copy Markdown

This PR was not deployed automatically as @eersnington does not have access to the Railway project.

In order to get automatic PR deploys, please add @eersnington to your workspace on Railway.

@github-actions

Copy link
Copy Markdown
Contributor

📟 Pip-Boy · 04f0b6e

📦 Packages · fresh npm install resolve, no lockfile · this PR vs base

package (linux-x64-gnu) @latest base this PR Δ vs base packages
@rivet-dev/agentos-core 458.2 MB 458.2 MB 458.2 MB +1.1 KB (+0.0%) 121 (0)
`@rivet-dev/agentos-core` details
platform base this PR Δ
linux-x64-gnu 458.2 MB 458.2 MB +1.1 KB (+0.0%)
linux-arm64-gnu 455.4 MB 455.4 MB +1.1 KB (+0.0%)
darwin-arm64 434.7 MB 434.7 MB +1.1 KB (+0.0%)
darwin-x64 444.2 MB 444.2 MB +1.1 KB (+0.0%)
win32-x64 314.6 MB 314.6 MB +1.1 KB (+0.0%)

Entrypoints (own code, minified; third-party imports excluded)

entrypoint base this PR Δ imports exports
. 291.3 KB 291.6 KB +324 B (+0.1%) — +SidecarEventBufferOverflow
All 6 packages · linux-x64-gnu
package (linux-x64-gnu) @latest base this PR Δ vs base packages
@rivet-dev/agentos-core 458.2 MB 458.2 MB 458.2 MB +1.1 KB (+0.0%) 121 (0)
@rivet-dev/agentos-node-pty 7.6 KB 18.0 KB 18.0 KB 0 1 (0)
@rivet-dev/agentos-posix 253.6 MB 253.6 MB 253.6 MB +909 B (+0.0%) 7 (0)
@rivet-dev/agentos-runtime-core 253.6 MB 253.6 MB 253.6 MB +909 B (+0.0%) 6 (0)
@rivet-dev/agentos-sandbox 495.5 MB 495.5 MB 495.5 MB +1.1 KB (+0.0%) 177 (0)
@rivet-dev/agentos-toolchain 130.8 KB 141.2 KB 141.2 KB 0 2 (0)

🧹 Rust dependencies

Skipped: this PR doesn't change any Rust code or dependencies.

🦀 Binaries

Skipped: this PR doesn't change any Rust code.

⏱ Timing

job wall time slowest step
zizmor 8s checkout · 2s
changes 29s checkout · 25s
footprint 2m01s Pack this commit · 41s
⏭ self-test — skipped
⏭ hygiene — skipped
⏭ native · ${{ matrix.platform }} — skipped
report 17s setup-node · 4s
🔭 Full CI trace for this commit

One OpenTelemetry trace of CI and Pip-Boy for 04f0b6e: every job and step, with the sizes above attached. Run after CI finishes.

pnpm install --frozen-lockfile --filter 'pipboy...'
pnpm --filter pipboy pipboy trace --sha 04f0b6e8d0e5b261fac9be356eb74c0780a119fc

Add --otlp <url> to send it to any OTLP/HTTP collector.

Agent instruction: From a rivet-dev/agentos checkout, run the commands above, then read the trace file the last command prints. List the slowest jobs and steps and explain where CI time went.

@eersnington
eersnington merged commit 1536575 into main Oct 21, 2026
15 checks passed
@eersnington
eersnington deleted the fix/event-buffer-disposed-vm branch October 21, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explicit sidecar stops accepting VMs after ~2048 create then dispose cycles "SidecarEventBufferOverflow"

1 participant