Repository navigation
consomme: answer guest UDP to host loopback from the gateway - #137
Merged
Merged
Conversation
Part of microsoft/nvx#560. Consomme translates guest UDP to the gateway onto host loopback, and presents host-local peers to the guest at virtual addresses, such as 10.0.0.254 and fd00::a00:ff for 10.0.0.2/24, so that the guest replies through its NIC. It applied that presentation to every datagram that a guest flow's host socket received, including the replies of the host service that the guest had addressed at the gateway. The reply to a datagram that the guest sent to 10.0.0.1 therefore came from 10.0.0.254, and the reply to one sent to fd00::a00:1 from fd00::a00:ff. An unconnected guest socket accepted the reply from the unexpected address, but a connected one, which accepts datagrams only from the address and port to which it is connected, dropped it, so connected UDP to a host-loopback service timed out over IPv4 and IPv6. Have each guest UDP flow remember, by host address, the peers that the guest addresses at another address than the host address that its datagrams reach, such as the gateway, and answer the datagrams of those peers from the address that the guest addressed. A datagram that the guest sends to the host address itself forgets the peer. A peer expires with the flow's UDP timeout unless datagrams keep refreshing it, and each flow remembers at most 64 peers, forgetting expired peers and then the oldest, so a guest cannot grow the table without bound. Host-local peers that the guest did not address, such as the host clients of published listeners, still appear at virtual addresses, and the egress policy's admission of replies is unchanged. Cover with tests the replies of host services on IPv4 and IPv6 loopback that the guest reaches at the gateway, from one guest port that alternates between two services as an unconnected socket may, the presentation of another host peer at a virtual address and the guest's reply there, and the expiry and bound of the remembered peers. Name the hop limit of the frames that the tests build, and update the Guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ppenna
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 16, 2026
Fixes #560. Pin nanvix/openvmm#137 at eae7da6f314ec0eacb98da8091dbd557972757ce, one commit on c6d97c52. OpenVMM answered the guest's UDP to host-loopback services, which the guest reaches at its gateway, from the addresses at which it presents other host peers: 10.0.0.254 instead of 10.0.0.1, and fd00::a00:ff instead of fd00::a00:1, for 10.0.0.2/24. Unconnected guest sockets accepted those replies, but connected ones, which accept datagrams only from the peer to which they are connected, dropped them, so connected UDP to host loopback timed out over IPv4 and IPv6 on every backend. Replies on the guest's UDP flows now come from the address and port that the guest addressed, while host peers that the guest did not address keep their virtual addresses. Extend the allow phase of the host-loopback-policy microVM scenario: under its egress default of deny, with rules that admit only these datagrams, the guest's connected nc exchanges a datagram with host UDP services on IPv4 and IPv6 loopback through its gateway. The phase fails with the previous pin. Describe the source of these replies in the run guide and the network design, and index the new pin in the roadmap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Part of microsoft/nvx#393. When the VM worker stopped, LoadedVm::run tore down the partition, removed the vmbus server, and dropped the VM without stopping its state units, so device workers that run on their own threads kept running while teardown invalidated the guest memory mapping. netvsp's queue workers read the control pages of their vmbus rings through locked pages, which point into that mapping, and the VA mapper does not implement page locking, so locked pages do not keep the mapping in place. A queue worker polled after the invalidation therefore faulted outside trycopy, whose signal handler aborted the process without a message. ttrpc::test_ttrpc_interface hit this intermittently on Linux/MSHV runners after a guest power-off: a core dump from one shows the fault in IncomingRing::verify_interrupts_unmasked on a netvsp queue thread, tens of milliseconds after the Consomme endpoint of that queue saw its port request channel close. Stop the state units first, as the restart path already does, so that the device workers stop before the partition and guest memory go away. On bare-metal Linux/MSHV, delaying the queue worker after the port request channel closes aborts OpenVMM without this change; with it, the stop waits for the worker and the VM tears down cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ppenna
added a commit
to microsoft/nvx
that referenced
this pull request
Oct 16, 2026
Fixes #560. Part of #393. Pin nanvix/openvmm#137 at 5ab8a84621524f1777b2cb5e65a70dab42c8b453, two commits on c6d97c52. OpenVMM answered the guest's UDP to host-loopback services, which the guest reaches at its gateway, from the addresses at which it presents other host peers: 10.0.0.254 instead of 10.0.0.1, and fd00::a00:ff instead of fd00::a00:1, for 10.0.0.2/24. Unconnected guest sockets accepted those replies, but connected ones, which accept datagrams only from the peer to which they are connected, dropped them, so connected UDP to host loopback timed out over IPv4 and IPv6 on every backend. Replies on the guest's UDP flows now come from the address and port that the guest addressed, while host peers that the guest did not address keep their virtual addresses. OpenVMM also tore down the partition and guest memory without stopping the VM's devices, so a netvsp queue worker that ran during teardown could read its vmbus ring from guest memory that was already unmapped, and OpenVMM aborted. ttrpc::test_ttrpc_interface failed this way on Linux/MSHV, including in this change's first CI run. The second commit stops the devices first. Extend the allow phase of the host-loopback-policy microVM scenario: under its egress default of deny, with rules that admit only these datagrams, the guest's connected nc exchanges a datagram with host UDP services on IPv4 and IPv6 loopback through its gateway. The phase fails with the previous pin. Describe the source of these replies in the run guide and the network design, and index both OpenVMM commits in the roadmap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ppenna
marked this pull request as ready for review
October 16, 2026 21:10
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
6 open issues
self.pause().await;. Ifpause()is the… · Newudp_lenis ausizebut is cast tou16for the UDP length without validation; if… · Newudp_lenis ausizebut is cast tou16for the UDP length without validation; if… · Newexchange_udpruns inside async tests but uses blockingstd::net::UdpSocket::recv_from. Even… · Newhostis normalized viaunscoped(host), butaddressis stored and compared without… · Newhostis normalized viaunscoped(host), butaddressis stored and compared without… · New
What changed in this PR
Fixes Consomme UDP source-address translation for guest-connected sockets targeting host loopback via the gateway, and hardens OpenVMM teardown ordering to prevent device threads from touching unmapped guest memory.
Changes:
- Track per-flow “translated peers” so host-loopback UDP replies egress with the gateway address/port the guest addressed.
- Add regression/unit tests for UDP gateway/loopback behavior (IPv4/IPv6), translated-peer expiry, and bounding.
- Pause/stop device workers earlier during VM teardown to avoid teardown races and crashes.
| File | Description |
|---|---|
| vm/devices/net/net_consomme/consomme/src/udp.rs | Adds translated-peer tracking and uses it when emitting UDP replies to the guest. |
| vm/devices/net/net_consomme/consomme/src/udp/tests/translated_peers.rs | New unit tests covering translated-peer update/expiry/bounding behavior. |
| vm/devices/net/net_consomme/consomme/src/tests.rs | Adds UDP frame build/parse helpers and an end-to-end UDP gateway reply test harness. |
| vm/devices/net/net_consomme/consomme/src/tests/static_ipv4.rs | Adds IPv4 regression test for gateway UDP replies coming from the gateway address. |
| vm/devices/net/net_consomme/consomme/src/tests/static_ipv6.rs | Adds IPv6 regression test for gateway UDP replies coming from the gateway address. |
| openvmm/openvmm_core/src/worker/dispatch.rs | Pauses VM/device work prior to tearing down partition/guest memory. |
| Guide/src/reference/backends/consomme.md | Documents gateway-sourced replies for connected UDP flows. |
Review effort: Lite
Comment on lines
+4540
to
+4544
| // Stop the devices before tearing down the partition and guest | ||
| // memory. Device workers run on their own threads and access guest | ||
| // memory through direct mappings (e.g., vmbus ring control pages), so | ||
| // a worker that is still running can touch memory that teardown has | ||
| // already unmapped, which aborts the process. |
Comment on lines
+610
to
+611
| let udp_len = UDP_HEADER_LEN + payload.len(); | ||
| let mut frame = vec![0u8; ETHERNET_HEADER_LEN + ip_header_len + udp_len]; |
Comment on lines
+643
to
+648
| let mut udp = UdpPacket::new_unchecked(udp_buffer); | ||
| udp.set_src_port(src.port()); | ||
| udp.set_dst_port(dst.port()); | ||
| udp.set_len(udp_len as u16); | ||
| udp.payload_mut().copy_from_slice(payload); | ||
| udp.fill_checksum(&src.ip().into(), &dst.ip().into()); |
Comment on lines
+721
to
+728
| consomme | ||
| .access(client) | ||
| .send(&frame, &ChecksumState::NONE) | ||
| .unwrap(); | ||
| let mut request = [0; 16]; | ||
| let (len, flow) = host.recv_from(&mut request).unwrap(); | ||
| assert_eq!(&request[..len], b"guest"); | ||
| host.send_to(b"host:guest", flow).unwrap(); |
Comment on lines
+313
to
+319
| let host = unscoped(host); | ||
| if address == host { | ||
| if !self.translated_peers.is_empty() { | ||
| self.translated_peers.remove(&host); | ||
| } | ||
| return; | ||
| } |
Comment on lines
+338
to
+339
| self.translated_peers | ||
| .insert(host, TranslatedPeer { address, seen: now }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The first commit fixes microsoft/nvx#560. The second fixes the root cause of microsoft/nvx#393, an intermittent abort that failed microsoft/nvx#583's first CI run.
consomme: answer guest UDP to host loopback from the gateway
Guest-initiated connected UDP to a host-loopback service timed out over IPv4 and IPv6 (microsoft/nvx#560). Consomme translates guest UDP to the gateway onto host loopback, but
UdpConnection::poll_connpassed every datagram that a flow's host socket received throughtranslate_remote_address, which presents host-local peers at virtual addresses. The host service's reply therefore came from10.0.0.254orfd00::a00:ffinstead of the gateway that the guest addressed,10.0.0.1orfd00::a00:1for10.0.0.2/24. Unconnected guest sockets accepted it; connected ones, which accept datagrams only from the peer to which they are connected, dropped it. TCP was unaffected.UdpConnectionremembers, by host address, the peers that the guest addresses at another address than the one that its datagrams reach, such as the gateway, andpoll_connanswers their datagrams from the address that the guest addressedMAX_TRANSLATED_PEERS), forgetting expired peers and then the oldest, so a guest cannot grow the table without boundopenvmm_core: stop the devices before tearing down the VM
On Linux/MSHV CI runners,
ttrpc::test_ttrpc_interfaceintermittently failed withchannel closedfromTeardownVm, because OpenVMM aborted while its VM worker stopped after a guest power-off (microsoft/nvx#393). The runner kept the core dump of the latest abort (SIGABRT). Symbolized against a build of the same source and toolchain, it shows a SIGSEGV on a netvsp queue thread inIncomingRing::verify_interrupts_unmasked, reached fromNetChannel::main_loopthroughReadHalf::poll_ready, whichtrycopy's signal handler turned into an abort without a message.WorkerRpc::Stop,LoadedVm::runtore down the partition, removed the vmbus server, and dropped the VM without stopping its state units, so device workers on their own threads kept running while teardown invalidated the guest memory mappingLockedPages, which point into that mapping;VaMapperdoes not implementlock_gpns, so locked pages do not keep the mapping in placeLoadedVm::runnow stops the state units first, as the restart path already does, so that the device workers stop before the partition and guest memory go awayPart of microsoft/nvx#560 and microsoft/nvx#393.
Revision
5ab8a84621524f1777b2cb5e65a70dab42c8b453, two SSH-signed commits directly onmainatc6d97c529986c0b3b1bf1fd7f00406654cdeee67:eae7da6f3(consomme) and5ab8a8462(openvmm_core)Validation
eae7da6f3On Windows, at that commit:
cargo clippy --all-targetsforconsommeandnet_consomme: no lintscargo doc --no-depswith-D warningsfor the same packagescargo nextest run --profile agentfor the same packages: 200 passedcargo xtask fmt --fix: cleanOn Linux (Azure Linux 3.0) with Rust 1.95.0, at that commit:
cargo clippy --all-targetsforconsommeandnet_consomme: no lintscargo nextest run --profile agentfor the same packages: 213 passed, including the Unix-only UDP testsOn bare-metal Linux/KVM, Linux/MSHV, and Windows/WHP hosts, with OpenVMM built from this change before its test-only hop-limit refactor (so with the same production code), against the released build of
c6d97c52(NVXv0.1.0-dev.36fc8aedcb39), Alpine guest on10.0.0.2/24with host loopback allowed:c6d97c52, unconnected replies came from10.0.0.254andfd00::a00:ffand connected sockets timed out; with this change, all four replies came from the gatewayhost-loopback-policymicroVM scenario, extended with connected UDP exchanges through the gateway: fails withc6d97c52and passes with this change on all threevirtio-net,directional-network-policy,private-network-ingress,ipv6-egress-policy,l3-l4-egress-policy,l3-l4-egress-protocols,l3-l4-egress-port-ranges,network-snapshot, andendpoint-policy-snapshotpass on all three5ab8a8462At the head, for
openvmm_core, on Windows and on Linux (Azure Linux 3.0) with Rust 1.95.0:cargo check,cargo clippy --all-targets,cargo doc --no-deps,cargo nextest run --profile agent(106 passed), andcargo xtask fmt --fix: clean.On bare-metal Linux/MSHV, with
cargo xflowey vmm-tests-run --release --ci-profilefrom a tree identical to the head (treea533cd46):ttrpc::test_ttrpc_interfacein 8 concurrent loops; a temporary, environment-gated delay on the netvsp queue thread right after its Consomme endpoint's port request channel closes, where the CI runner's thread stalled, reproduces it (1/1), with the same faulting frame in the corettrpc::test_ttrpc_interfacepasses 200/200 in 8 concurrent loops; and 32/32 of the ttrpc, Linux direct-boot, andx86_64_exclusiveVMM tests passNot run for this commit: KVM, WHP, VMM tests that need VHD images or the
x86_64-unknown-uefitarget, and a natural reproduction on a virtual-machine runner; microsoft/nvx#583's CI runs the VMM tests on all three backends.