Skip to content

fix(runtime): admit resource reservations all-or-nothing - #2084

Merged
eersnington merged 1 commit into
mainfrom
fix/ledger-atomic-admission
Oct 28, 2026
Merged

eersnington merged 1 commit into
mainfrom
fix/ledger-atomic-admission

Conversation

@eersnington

Copy link
Copy Markdown
Member

Fixes #2083. Reported and diagnosed in #2083.

On a 1-CPU host, a guest HTTPS download larger than the VM's buffer limits made the sidecar unresponsive, and the client killed it after 30 s. Every VM on the sidecar was lost.

  • ResourceLedger::reserve charged the process ledger first and undid the charge when the VM ledger was full. The undo called notify_one(), which stores a permit when nobody is waiting yet.
  • reserve_when_available then consumed its own permit, retried, failed, and stored a new one. It never returned Pending, so on one runtime worker the task that frees VM capacity never ran. The waiter here is AsyncCompletionSender::send (crates/native-sidecar/src/state.rs).
  • CapabilityRegistry::reserve had the same loop on a single ledger: it took the capability and ready-handle counts, failed on Sockets, and released them.
flowchart LR
    R["reserve / CapabilityRegistry::reserve / HTTP/2 admission"] --> A["ResourceLedger::reserve_all"]
    A --> L["lock process ledger, then VM ledger"]
    L --> C{"all requests fit<br/>in every scope?"}
    C -- no --> E["LimitError: no charge, no notify"]
    C -- yes --> OK["charge all, return Reservations"]
Loading
  • New ResourceLedger::reserve_all(&[(ResourceClass, usize)]) checks every request in every scope under ordered locks, then charges. A rejected admission changes nothing, so a waiter can no longer wake itself.
  • reserve calls reserve_all with one request. CapabilityRegistry::reserve and reserve_http2_resources admit their counts in one call.
  • Locks are always taken root first. The helpers called while locked (maybe_warn, observe_usage) take no ledger lock.
  • A near-limit warning no longer fires for a reservation that is then rejected.
17 MB guest fetch, Linux main This PR
1 CPU (taskset -c 0) 0/3, sidecar unresponsive 3/3
16 CPUs 3/3 3/3

A rejected reservation charged the process ledger, then undid that charge
when the VM ledger was full. The undo stored a notify_one permit, so
reserve_when_available woke itself and retried without ever returning
Pending. With one runtime worker the task that frees VM capacity never
ran, and a large guest HTTPS download made the sidecar unresponsive.

ResourceLedger::reserve_all checks every request in every scope under
ordered locks and charges only when all fit, so a rejected admission has
no side effects. reserve, CapabilityRegistry::reserve, and HTTP/2 stream
admission use it.

Fixes #2083.
@railway-app

railway-app Bot commented Oct 28, 2026

Copy link
Copy Markdown

This PR was not deployed automatically as @eersnington does not have access to the Railway project.

In order to get automatic PR deploys, please add @eersnington to your workspace on Railway.

@github-actions

Copy link
Copy Markdown
Contributor

📟 Pip-Boy · 339e226

📦 Packages

Skipped: this PR doesn't change any published package.

🧹 Rust dependencies · cargo deny + cargo shear

check new in this PR fixed existing on base
advisories ✅ 0 0 28
licenses ✅ 0 0 0
sources ✅ 0 0 0
bans ✅ 0 0 0
unused ✅ 0 0 16

🦀 Binaries · publish Dockerfiles, --release

Base: main@3c60c9a, the merge base.

artifact npm @latest base this PR Δ vs base
agentos-native-sidecar linux-x64-gnu 143.8 MB 143.8 MB 143.8 MB −1.6 KB (−0.0%)

⏱ Timing

job wall time slowest step
zizmor 10s checkout · 4s
changes 37s checkout · 32s
native · linux-x64-gnu 14m52s Build with the publish Dockerfile · 13m47s
hygiene 1m01s checkout · 25s
⏭ footprint — skipped
⏭ self-test — skipped
report 29s Write the comment and run summary · 10s
🔭 Full CI trace for this commit

One OpenTelemetry trace of CI and Pip-Boy for 339e226: every job and step, with the sizes above attached. Run after CI finishes.

pnpm install --frozen-lockfile --filter 'pipboy...'
pnpm --filter pipboy pipboy trace --sha 339e2269e330d1acebb1dcea00e37184dec570fb

Add --otlp <url> to send it to any OTLP/HTTP collector.

Agent instruction: From a rivet-dev/agentos checkout, run the commands above, then read the trace file the last command prints. List the slowest jobs and steps and explain where CI time went.

@eersnington
eersnington merged commit b815ecd into main Oct 28, 2026
14 of 15 checks passed
@eersnington
eersnington deleted the fix/ledger-atomic-admission branch October 28, 2026 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sidecar hangs on 1-vCPU hosts: ResourceLedger::reserve_when_available busy-loops on its own rollback

1 participant