Benchmark: characterize 1,000-microVM egress and shared-vsock saturation #4626

Agent pickup triage

Category: ready. This classifies the next task, not completion or a verified current-source pass.

Next bounded step: Start with the reproducible authenticated-session load harness and resource instrumentation; obtain explicit approval before billable hosts or uploads, and keep real-VM evidence distinct.

Handoff evidence: comment. PR states at triage: none in the inspected cross-references.

Read the linked handoff and coordinate any active owner before implementation. Preserve the original acceptance criteria below; this triage grants no spending, credentials, publication, host reconfiguration, or landing authority.

Parent: #4607

Related to #4622 and #4635. This issue measures the current NIC-less FlowMux/host-egress path; #4635 separately measures mailbox behavior using compatible host-resource instrumentation.

Parent decisions

MVM must measure the 1,000-workload contention claim before adding a guest NIC, another virtio-vsock device, vnet/HeatNet, or another network data plane. Workloads remain NIC-less for this benchmark. Existing machine-control, telemetry, network-flow, and other typed services retain their separate vsock ports and connections; the benchmark determines whether they still contend at the shared virtio-vsock device, host endpoint, kernel, NAT/conntrack, or physical NIC.

Questions this benchmark must answer

1. At what concurrency and offered load does aggregate outbound throughput stop scaling?
2. Which resource reaches its knee first: guest CPU, VMM/vsock queues or credits, host endpoint CPU, file descriptors, sockets, DNS, NAT/conntrack, host kernel networking, or the physical NIC?
3. Does sustained egress traffic increase machine-control or telemetry latency despite their separate typed vsock connections?
4. Are tenants and workloads scheduled fairly, or can a small set of producers starve the rest?
5. Does using more connections change throughput or isolation without adding physical capacity?
6. Is a host P2P overlay such as Iroh material to the bottleneck, or is it outside the measured guest-to-host egress path?

Measurement matrix

Run each workload shape at 1, 10, 100, 250, 500, and 1,000 concurrent workload identities:

• idle connected sessions;
• one simultaneous request per workload;
• sustained one in-flight request per workload;
• sustained eight in-flight requests per workload;
• a single shared destination;
• 100 evenly selected destinations;
• mixed traffic: egress plus telemetry plus periodic machine-control probes;
• one abusive workload at its admitted limit while the remaining workloads send low-rate traffic.

Use bounded request and response bodies of 1 KiB, 64 KiB, and 1 MiB. Redirects remain disabled. DNS, TLS setup, connection reuse, and direct-versus-relayed host P2P legs are reported separately rather than blended.

The first stage may use an authenticated transport/session load harness to reach 1,000 identities deterministically. The final stage must repeat the relevant knee-point and 1,000-identity cases with real microVMs on a host sized for the run. Any billable remote host or tracked-checkout upload requires the repository's explicit operator authorization before execution.

Required metrics

Record at least:

• request throughput and goodput;
• p50, p95, and p99 request latency;
• machine-control and telemetry p50, p95, and p99 latency during egress load;
• connection success, refusal, timeout, reset, and retry counts;
• per-workload throughput distribution and starvation;
• guest, VMM, endpoint, and host CPU;
• memory and file-descriptor use;
• vsock queue depth, credit stalls, and bytes by typed service;
• host socket, DNS, NAT, and conntrack pressure where applicable;
• physical NIC throughput, packets, drops, and retransmissions;
• direct and relayed P2P path classification when that scenario is enabled.

No captured artifact may contain credentials, payload contents, or customer data.

Implementation constraints

• Run Linux-specific and Firecracker operations only inside the builder VM unless the repository-approved KVM test-host path is explicitly authorized.
• Run native HVF measurements on macOS under the documented exception.
• Arm observers before load begins and use events for owned process completion; do not sleep-poll.
• Use one reproducible workload generator, fixed seeds, machine-readable results, and enough repetitions to report variance.
• Measure the current architecture first. Do not add a guest NIC or second network path as part of the baseline.
• Keep telemetry and control traffic independently attributable by their existing service/port identities.

Acceptance criteria

• Machine-readable results cover the complete concurrency and workload matrix, or a recorded resource limit identifies the exact first unreachable case and the measured reason.
• The result identifies the first saturation point and attributes it to a measured resource.
• Control and telemetry tail latency under egress saturation are compared with their idle baselines.
• Fairness and noisy-neighbor behavior are quantified.
• Shared-device contention is distinguished from physical-NIC saturation.
• The 1,000-identity transport result and the real-microVM validation result are reported separately.
• Direct and relayed Iroh paths, if included, are reported separately from ordinary Internet egress.
• A recommendation states whether the next change should be scheduling/quotas, endpoint sharding, connection changes, host networking changes, an optional P2P overlay, or no architectural change.
• The evidence is sufficient to update Research: evaluate per-VM vsock NIC adapters for vnet/HeatNet and SpiralCore #4607 without assuming that additional virtual adapters increase physical NIC capacity.