Track sandbox resource limit bugs: provision records a --memory-mib that OpenVMM refuses, so the sandbox never starts #733

Bug-hunt tracking issue for sandbox resource limits: --memory-max and --pids-max, which the host passes on the kernel command line, and which the guest's init writes into the container cgroup.

Baseline: origin/dev at 6f0facefd035a9ca9271c51436b9686874326272. Every permalink below uses this SHA.

Hunt: time-boxed, started 2026-10-21 16:21 -03:00. A first pass on the host VMM supervisor ended early and wrongly reported no bugs in #720, which is now closed. The hunt resumed and stopped at 16:49, when this bug was confirmed. This issue records the finding and the coverage of both passes.

Findings

1. The host accepts limits the guest kernel can't store: --memory-max wraps modulo 2^64, and a large --pids-max fails the boot

Severity: Medium for --memory-max, because the wrong limit is applied silently. Low for --pids-max, which fails loudly, but late.

Where

• The host only requires a positive integer, and Int has no upper bound:
  • nvx-rs/src/host/sandbox/mod.rs#L242-L246
  • CLI parsing: nvx-rs/src/host/cli/sandbox/mod.rs#L378-L379
  • The documented contract: #L134-L138.
• The host passes both values through verbatim, with pids_max + 1: nvx-rs/src/host/sandbox/mod.rs#L394-L399.
• The guest's init writes them unchecked into the container cgroup: guest/common/nvx-init-agent#L110-L119.
• A golden fixture already expects a value the guest kernel always refuses, nvx_pids_max=100000000000000000000: nvx-rs/tests/fixtures/sandbox.json#L1320.

Root cause, in the Linux cgroup v2 parsers

• memory.max goes through page_counter_memparse() → memparse() → simple_strtoull(), which ignores overflow. simple_strntoull() drops the KSTRTOX_OVERFLOW flag, so the byte count wraps modulo 2^64.
• pids.max uses kstrtoll() and refuses limit >= PIDS_MAX, which is PID_MAX_LIMIT + 1 = 4,194,305 on x86-64. Because the host adds one, the largest --pids-max the guest can apply is 4,194,303.

Trigger: nvx-rs sandbox run|create … --memory-max N, or the equivalent library settings, with N ≥ 2^64 (18446744073709551616), or --pids-max M with M ≥ 4194304.

Expected: The host refuses a limit the guest can't apply, or the guest refuses it before the workload runs. It must not apply a different limit.

Actual

• --memory-max 18446744073709551616 (2^64) becomes memory.max = 0, and the workload is OOM-killed as soon as it runs.
• 2^64 + 512 MiB becomes 512 MiB, and 10^20 becomes 7766279631452241920.
• --pids-max 4194304 passes the host, then the guest's init stops the boot with fatal "invalid container process limit".

Reproduction

1. Host half, with the nvx-rs CLI built from this SHA on Windows. --dry-run with placeholder OpenVMM, kernel, and initramfs files exits 0 and puts both values on the guest kernel command line:

   > nvx-rs sandbox run --layer distro,<root>\distro.erofs,11111111-1111-1111-1111-111111111111 --scratch <root>\scratch.ext4 --entrypoint /bin/server --memory-max 18446744073709551616 --pids-max 4194304 --memory-mib 512 --hypervisor whp --dry-run
   exit=0
   … nvx_memory_max=18446744073709551616 nvx_pids_max=4194305 …
   

2. Guest half: the same echo VALUE > writes as nvx-init-agent, as root, into a throwaway cgroup v2 child on WSL2's Linux 6.18.33. NVX pins Linux 6.18.38 in SOURCE-MANIFEST.json.

   memory.max <- 268435456 : accepted, reads back 268435456
   memory.max <- 18446744073709551615 : accepted, reads back max
   memory.max <- 18446744073709551616 : accepted, reads back 0
   memory.max <- 18446744073709551617 : accepted, reads back 0
   memory.max <- 18446744074246422528 : accepted, reads back 536870912
   memory.max <- 100000000000000000000 : accepted, reads back 7766279631452241920
   pids.max <- 65 : accepted, reads back 65
   pids.max <- 4194304 : accepted, reads back 4194304
   pids.max <- 4194305 : write rejected: sh: 8: echo: echo: I/O error
   pids.max <- 100000000000000000001 : write rejected: sh: 8: echo: echo: I/O error
   

3. A process placed in a cgroup whose memory.max received 2^64:

   memory.max reads back: 0
   Killed
   workload exit status: 137
   memory.events: max 35 oom 1 oom_kill 1
   

Suggested fix

• Host (nvx-rs/src/host/sandbox/mod.rs):
  • Refuse memory_max above i64::MAX (9223372036854775807). The kernel stores at most PAGE_COUNTER_MAX pages and saturates larger in-range values to max.
  • Refuse pids_max above 4194303, for example with --pids-max must be at most 4194303.
  • Add tests for 2^64 and 4194304.
  • Update the sandbox.json goldens that expect nvx_pids_max=100000000000000000000.
• Guest, as defense in depth (guest/common/nvx-init-agent):
  • Require decimal digits within those bounds before writing.
  • After writing, read the limit back and fatal on a mismatch. The kernel rounds memory.max down to whole pages, so compare against the page-rounded value.

Explored and verified (no bug)

Second pass, 16:34–16:49

1. Guest image keys and VMM-test selection (ci: key the guest images and VMM tests on the guest binaries' sources #710: SOURCE_PATTERN, OPENVMM_VMM_TEST_INPUT_PATHS, _check_guest_build_inputs).
   • 24 edge paths gave the intended results for both the classifier and the identity, including host.rs, hostile.rs, hosts/, tests.rs, tests/, loopback.rs, the binaries, build.rs, and .cargo/.
   • The guest build's --no-default-features --features guest keeps artifacts.json out of build.rs's watch set, so the source identity covers everything the build reads.
2. Persisting performance results after a history reset (performance: keep results measured before a history reset out of the series #658: _reset_dimensions_since, _without_reset_rows).
   • Traced cases: resets merged from a PR branch, a measurement at the reset commit itself, deleted history files, and side branches.
   • The persist action re-fetches dev on each of its three attempts, and the persist job checks out with fetch-depth: 0.
3. NVXS host connection (HostConnection::send and write): an explorer suggested that a send timeout which wrote nothing should leave the connection usable.
   • It can't: HostSession::send has already advanced the sequence number and credit, and write_all may have written part of the record.
   • So ending the connection is required. See "Checked, not filed".
4. Host mapping payloads (encode_maps_payloads): the header is counted once per payload, oversized entries are refused, and the count is patched on flush. Also checked: the time-probe option diagnostics (nvx-rs: report the time probe's invalid options with one method #681).

First pass, 16:21–16:33, recorded in #720
5. Host VMM supervisor (#656): launch, commit and abandon, Vmm::stop, reattach_with, and store recovery (recover, end_unidentified).

• Every failing step ends the launch.
• An unrecorded VMM is killed through its handle first.
• Runtime records are cleared only for the same launch.

6. Windows process waits (ProcessHandle::wait) and the runtime-adapter Decoder::push (tests: serve the control session in the runtime adapter's guest agent #640).
7. command-line-limit scenario (run_command_line_limit, openvmm: refuse kernel command lines that the guest kernel would truncate #661): the padding arithmetic is correct, and the marker extraction is line-anchored.
8. OpenVMM transcript streaming (OpenvmmProcess.close, microvm tests: stream OpenVMM transcripts to disk as they are read #659): the drain runs only after the reader stops.
9. Windows Azure metadata snapshot (ci: recover Windows runners after VMSS image upgrades #669): the real bootstrap functions, run in Windows PowerShell 5.1, keep dataDisks a JSON array with one disk too. Also checked: benchmarkRunnerName matching and the CI event classification (ci: run the OpenVMM suites on dev pushes only when their inputs change #660).

Checked, not filed

These are low impact, by design, or unreachable today.

• sandbox exec --timeout (default 60 s) bounds the whole exec exchange, including the workload's run time (Session::exec).
  • A workload that runs longer than --timeout fails on the host, even with --exec-timeout-ms 0.
  • This faithfully ports the deleted Python ControlSession.exec ("The response timeout bounds the whole exchange"), so it's a usability trap, not a regression. The help text could say so.
• The HostConnection docs are ambiguous. They say a send that "times out before it wrote anything" leaves the connection usable, but any write_all timeout ends it (nvxs.rs#L260-L271). That behavior is correct; only the wording is unclear.
• LaunchedProcess::terminate and wait_timeout can wait for INFINITE on Windows. A timeout of exactly 4,294,967,295 ms maps to INFINITE (windows.rs#L313-L338). Every caller passes at most 10 s.
• OpenvmmProcess transcript write errors (_drain_available, _record). A write error can mask the original failure and leave a gap in the transcript. It is only reachable on log I/O failure.
• Release manifest checks compare with == (_manifest_mismatches). So "format": true or 1.0 passes as format 1. This predates release: name the mismatched fields in Ubuntu artifact manifest errors #664, and the build itself writes these manifests.

Validation limits

• No end-to-end sandbox boot. The hunt ran on a Windows 11 host without KVM or MSHV, and without built OpenVMM or guest artifacts for WHP. Each half of the finding was reproduced separately:
  • The host half with the nvx-rs CLI built from this SHA, through --dry-run.
  • The guest half with the same cgroup writes on WSL2's Linux 6.18.33, not the pinned 6.18.38 guest kernel.
• Tests. The nvx-rs tests were built from this SHA (cargo test --locked --all-features --no-run), but only the CLI binary was exercised. The validate-nvx Python suite ran on this host with Python 3.14, which the docs allow ("3.10 or newer"): 1,098 tests in 692 s, with 18 failures, 5 errors, and 37 skipped. The one failure in the captured output is environmental: a test_setup_azure shell script's awk can't open a Windows temporary path (awk: fatal: cannot open file 'C:/Users/…/grub.cfg'), most likely because bash resolves to WSL on this host. The other failures weren't triaged, because the hunt had already stopped at its confirmed bug. CI's Windows job runs the same suite.
• Out of scope: the openvmm/ submodule, guest-only Rust code beyond reading, and areas that earlier bug-hunt issues cover (Track release source collection bugs: Ubuntu output junction deletes target files #374–Track host telemetry bugs: Azure metadata trickles bypass timeout #514).