UC reliability over TB4/USB4: hybrid completion, retransmit budget, measured loss classes (stacked on #75)

[afrog33k]

UC reliability over TB4/USB4: hybrid completion, retransmit budget, and the measured loss classes

Stacked on #75 — commits 36a47f1/2b65af3 (the NHI throttle + its compat probe) are #75's; review those there. This PR's own content is the 6 commits after them. Builds clean on both ends of our leg: Fedora 6.18 (stock NHI tree) and Asahi 7.1.13-usb4gpu (Apple NHI tree).

The problem

Two hosts — Apple M2 Ultra, Asahi 7.1.13-usb4gpu (Apple NHI) and AMD Strix Halo, Fedora 6.18-rc7 (stock PCI NHI) — joined by one 20 Gb/s TB4 cable. After the throttle fix, every sustained UC run froze permanently: the sender's last send completed, the receiver had posted exactly one fewer reply, no RNR / duplicate / error anywhere, empty dmesg on both ends.

What the counters said (module debugfs, per-QP)

loss class	measured rate	symptom
stranded RX completion (frame DMA'd, wire-ACKed, IRQ never signalled)	~1/280 frames under load	permanent freeze, clean counters
true wire loss	~1/1M frames	recoverable by retransmit, costs ~5.09 s
orphan fragment (tail before vanished head)	~1/2M messages	previously fatal

The design

• Hybrid UC completion (tbv-uc-local-completion): the WC fires at local TX drain (UC semantics) but the send context stays pending until the peer ACK; a reap retransmits drained-but-unacked sends. UC sends are solicited unconditionally and the receiver ACKs at message completion, so an ACK is a real delivery receipt.
• tbv-rx-supp-poll: the driver's own supplemental RX poll (1 ms delay / 16 ms window) reaps frames whose completion interrupt was lost. It existed but was hard-disabled out of an Apple-backend reorder concern; now enabled native-only behind a module param (rx_supp_poll=-1). One 1M-iteration run logged 18,334 rescues — the stranded-completion class is two orders of magnitude more common than previously estimated.
• Sendq slot leak: every UC send leaked a slot, releasing at DRAIN (E4) instead of ACK — a depth-1 QP (ibv_uc_pingpong's normal shape) dies at max_send_wr with zero counter evidence.
• UC RNR budget: non-retryable sends froze in rnr_waiting forever; they now retry under TBV_UC_RNR_MAX_RETRIES=7 (module policy — UC retry_cnt=0 must not mean "zero wire retries"). This intersects Default data QPs to infinite RNR retries #71 (infinite RNR retries as default): we'd rather keep the budget explicit and bounded; happy to converge either way.
• Watchdog vs retransmit race: the receiver's 5.00 s active-msg watchdog errored 90 ms before the sender's 5.09 s retransmit was due, and its error-ACK killed the sender's QP too. UC rx timeout now out-waits the sender's full budget (×8).
• Orphan fragments buffer through the reorder machinery instead of erroring the connection.
• rc-reorder + hot-poll: per-frame completion was the bandwidth ceiling; reordered completion polling gives 3.4× at 256 KiB and takes the 2 B floor from 22 to 14 µs.

Everything new sits behind module params and is gated to the native backend; the Apple compatibility backend is untouched.

Measured acceptance (both ends checksummed)

measurement	result
1M UC pingpong, 4096 B fragmented	rc=0 both ends, 96.19 µs/iter, 681 Mbit/s, 0 retransmits, 0 errors
64 B × 10M runs	converged, floor 57.64 µs/iter
llama.cpp RPC over RDMA	3/3 runs, warm floor 0.693 s
16 MiB byte-verified echo	identical, 1.049 Gbit/s (per-frame-CPU-bound — documented as the open gap)
MLX ring all_sum, both GPUs	CORRECT

Limitations, stated

Bandwidth at large messages is per-frame-completion-CPU-bound (bigger native frames / striping next). A true wire loss still costs the full retransmit stall — a selective fragment NAK would shrink 5.09 s to ~200 µs. Wire budgets are module policy, not negotiated. Validated on exactly one leg topology (Apple NHI ↔ stock NHI, Asahi 7.1.13 ↔ Fedora 6.18).

[hellasbot]

Hydra: passed

Head 536c4a64e47c · Evaluation #166198 · Hydra jobset

All 22 builds passed.