ci: host-loopback port-pair retry exhausts its attempts inside a Windows TCP port exclusion (flaky NVX CLI / Windows) #473

Summary

The dev push run for e054b2a (#471) failed in NVX CLI / Windows because two host-loopback unit tests could not allocate a port number that was free for both TCP and UDP. _bind_tcp_udp_listener_pair() takes a UDP port by binding to port 0, binds TCP to the same number, and retries up to 16 times when the TCP bind fails. On Windows, back-to-back UDP binds return consecutive ports, so the 16 attempts probe 16 neighboring ports. Windows keeps separate port-exclusion tables for TCP and UDP, usually reserved in 100-port blocks. When the first UDP port falls inside a block that is excluded only for TCP, every attempt fails with WinError 10013.

This is an intermittent test-harness defect that depends on the runner's port exclusions. #471 did not cause it: it does not touch this code, and the job passed both on the previous dev head and on #471's final PR run. The code is unchanged on the current dev head. The same helper also backs the real host-loopback-policy microVM scenario, which the WHP microVM job runs on the self-hosted Windows runners.

Incident

• Workflow: CI run 38057714971, attempt 1, push to dev.
• Head: e054b2af7249f0533bdb5b88968324dc779dcc22, the merge of setup: read tool versions from one canonical manifest #471. Push runs on dev do not cancel each other, so this run was not superseded.
• Failed job: NVX CLI / Windows, step Validate NVX CLI (python -m unittest … scripts/test_microvm_tests.py …). 2 of 891 tests failed, and 105 were skipped.
• Runner: GitHub-hosted windows-latest, Windows Server 2025 10.0.26100, image windows-2025-vs2026 20260925.250.1, Python 3.12.10.
• Downstream: the failure skipped Platform / Windows / WHP / Virtual machine, NVX microVM tests / Windows / WHP, and its Debug kernel variant. Required status check failed, and Publish development release and Persist performance baseline were skipped. All other build and test jobs passed.

Decisive output:

FAIL: test_host_loopback_scenario_detects_udp_proxy_leak (scripts.test_microvm_tests.MicrovmTests.test_host_loopback_scenario_detects_udp_proxy_leak)
  File "D:\a\nvx\nvx\scripts\nvx_tools\microvm_tests.py", line 3189, in _bind_tcp_udp_listener_pair
    tcp_listener.bind(("0.0.0.0", port))
PermissionError: [WinError 10013] An attempt was made to access a socket in a way forbidden by its access permissions
The above exception was the direct cause of the following exception:
RuntimeError: failed to allocate a port available to both TCP and UDP after 16 attempts
During handling of the above exception, another exception occurred:
  File "D:\a\nvx\nvx\scripts\test_microvm_tests.py", line 2515, in test_host_loopback_scenario_detects_udp_proxy_leak
AssertionError: "reached a host UDP service" does not match "failed to allocate a port available to both TCP and UDP after 16 attempts"

test_host_loopback_scenario_requires_observed_udp_control failed the same way at L2539, where it expects UDP positive control. Each test finished within about 6 ms, so the TCP binds were rejected immediately and nothing timed out.

History

I checked the logs of every failed Windows or WHP job in the 690 CI runs created since 2026-09-23, including earlier attempts of rerun runs. Two failed with WinError 10013 in a host-loopback test:

The earlier incident predates the retry. At that time, the test took a port from TCP and bound UDP to the same number. #197 replaced that with the current UDP-first retry. This run is the first failure since that change, and no WHP microVM job in the window failed this way. Logs for 4 jobs of one PR run from 2026-09-23 (35898808624) are no longer available.

Root cause

Permalinks point to e054b2a. These files are byte-identical on the current dev head, 218ad09.

1. Same-number pairs are required. run_host_loopback_policy() needs a TCP listener and a UDP listener on the same port, for both the general service and the proxy. The guest probes UDP on the proxy's port to prove that --host-loopback deny blocks it (L3216-L3227).
2. Allocation. _bind_tcp_udp_listener_pair() binds UDP to 127.0.0.1:0 and then TCP to 0.0.0.0:<port>. On OSError, it closes both sockets and tries again, at most HOST_LOOPBACK_PORT_BIND_ATTEMPTS (16) times (L3175-L3203, L204).
3. Windows excludes ports per protocol. netsh interface ipv4 show excludedportrange protocol=tcp and protocol=udp list separate tables. Hyper-V, WinNAT, and similar services typically reserve these ports in 100-port blocks. A bind to an excluded port fails with WSAEACCES (WinError 10013). The UDP allocator skips UDP exclusions, but it knows nothing about TCP exclusions.
4. Back-to-back UDP binds return consecutive ports. The 16 attempts run within milliseconds, so they probe 16 consecutive ports. If the first port lands in a TCP-only block with at least 16 ports left, every attempt fails. On a Windows 11 workstation, UDP binds less than a second apart returned consecutive ports and skipped exactly the UDP-excluded range 57626–57725. After a pause of about 5 s, binds resumed from an unrelated port, so whether the walk starts inside a block varies from run to run.
5. The regression test models only an isolated conflict. test_host_loopback_listener_pair_retries_protocol_port_conflict fails the first TCP bind and lets the second succeed (L2394-L2425). It therefore cannot detect a contiguous exclusion.

The CI log does not record the runner's exclusion tables, so the block cannot be identified for this job. The reproduction below shows that this mechanism produces the exact CI signature.

Reproduction

I used Windows 11 Enterprise 10.0.26300 with Python 3.14.7 and a clean checkout of e054b2a. No hypervisor is needed.

• Control: the two failing tests and the mocked retry test pass when the next UDP port is not in a TCP-only exclusion.
• The script below finds a TCP-only exclusion ahead of the UDP allocator and advances the allocator to the port just before it with back-to-back binds. It then calls the real helper and runs the two CI tests. This script and an equivalent longer version targeted the blocks 54609–54708 and 58912–59011. Each time, the helper failed after exactly 16 attempts; against 58912–59011 it consumed ports 58912–58927. Both tests then failed with the CI assertion messages.

import re
import socket
import subprocess
import sys
import unittest

sys.path.insert(0, "scripts")
from nvx_tools import microvm_tests  # noqa: E402


def exclusions(protocol):
    output = subprocess.run(
        ["netsh", "interface", "ipv4", "show", "excludedportrange", f"protocol={protocol}"],
        capture_output=True, text=True, check=True,
    ).stdout
    return [(int(a), int(b)) for a, b in re.findall(r"^\s*(\d+)\s+(\d+)", output, re.M)]


def next_udp_port():
    with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as udp:
        udp.bind(("127.0.0.1", 0))
        return udp.getsockname()[1]


udp_excluded = exclusions("udp")
port = next_udp_port()
# Pick a TCP-only exclusion of at least 64 ports ahead of the UDP allocator.
start, end = next(
    (a, b) for a, b in exclusions("tcp")
    if a > port + 1 and b - a >= 63
    and not any(x <= p <= y for x, y in udp_excluded for p in range(a - 1, a + 64))
)
while port < start - 1:  # back-to-back binds return consecutive ports
    port = next_udp_port()
assert port == start - 1, f"allocator jumped to {port}; rerun"
print(f"next UDP port {start} is in TCP-only exclusion {start}-{end}")
try:
    microvm_tests._bind_tcp_udp_listener_pair(5.0, 1.0)
except RuntimeError as error:
    print(f"{error}; cause: {error.__cause__!r}")
unittest.main(
    module="test_microvm_tests",
    argv=[
        "repro",
        "MicrovmTests.test_host_loopback_scenario_detects_udp_proxy_leak",
        "MicrovmTests.test_host_loopback_scenario_requires_observed_udp_control",
    ],
    exit=False,
)

Output from the second run:

next UDP port 54609 is in TCP-only exclusion 54609-54708
failed to allocate a port available to both TCP and UDP after 16 attempts; cause: PermissionError(13, 'An attempt was made to access a socket in a way forbidden by its access permissions', None, 10013, None)
FAIL: test_host_loopback_scenario_detects_udp_proxy_leak (test_microvm_tests.MicrovmTests.test_host_loopback_scenario_detects_udp_proxy_leak)
AssertionError: "reached a host UDP service" does not match "failed to allocate a port available to both TCP and UDP after 16 attempts"
FAIL: test_host_loopback_scenario_requires_observed_udp_control (test_microvm_tests.MicrovmTests.test_host_loopback_scenario_requires_observed_udp_control)
AssertionError: "UDP positive control" does not match "failed to allocate a port available to both TCP and UDP after 16 attempts"
FAILED (failures=2)

Impact

• e054b2a has a red required check and no development release or persisted performance history. The next dev head, 218ad09 (doc: name every guest the validation suites boot #472, documentation only), passed in run 38059091171, as expected for an intermittent failure.
• When this job fails, the Windows/WHP workload jobs for that head are skipped, so the head loses all WHP coverage, not only one unit test.
• host-loopback-policy is one of the default MICROVM_TEST_SCENARIOS (L104), and the WHP microVM job runs test-microvm without --scenario (run-nvx-microvm-tests.yml#L103-L110). For example, it ran on azure-windows-2 in run 38024829470. The real scenario therefore uses the same helper on Hyper-V hosts and can fail the same way before it boots a guest, although it has not done so in the window above.
• No product code is affected, because the helper is part of the test harness.

Suggested fix

• Choose each candidate port independently, for example uniformly at random from the dynamic range (49152–65535 by default), and bind UDP 127.0.0.1:<port> and TCP 0.0.0.0:<port>, retrying on OSError. Each attempt then fails only in proportion to the share of excluded or busy ports, so a contiguous block can no longer exhaust the attempts.
• Include the ports tried and the last error in the RuntimeError so that the next occurrence identifies the block.
• Replace the single-conflict model in test_host_loopback_listener_pair_retries_protocol_port_conflict with one in which every port of a block longer than the attempt budget is excluded for TCP. The test should assert that allocation still succeeds and that partial pairs are closed.
• Raising HOST_LOOPBACK_PORT_BIND_ATTEMPTS alone is not enough, because exclusion blocks are usually 100 ports long and can be adjacent.

Related work

• Make host-loopback port allocation protocol-safe #197 introduced the UDP-first retry after run 35802625855.