Skip to content

HV inline fast path has no retransmission — a single dropped frame past guest eth0 wedges the flow #486

Description

@AprilNEA

The residual gap from #451 (its fix covered only the bridge-polled path): both inline owners — arcbox-net-inject's readv InlineConn and splicetcp::direct_rx's tokio ConnSink — cap the honored window at 256 KiB but have no retransmission. The 2026-07 cross-repo audit re-confirmed it.

Why window gating alone is not enough

inline_conn.rs claims "the inject path is lossless — so staying inside [the window] means no gap can form". That holds only to guest eth0. For container flows the TCP endpoint is inside the container netns, and #451 established that the guest-internal bridge → veth → container-netns backlog drops frames under burst before TCP buffering — the advertised window bounds the container socket's buffer, not the veth hop. One dropped frame = permanent hole = wedged flow (and #451 measured window gating alone making it worse: every drop became a full-window deadlock).

Today the exposure is limited: inline is HV-only in practice and gated on peer_mss ≥ 1460. But it blocks #262 (promote all established flows to inline) and matters for #250 (HV as default backend).

Design (assessed 2026-07-21)

The zero-copy read (socket → guest descriptor buffer directly) is fundamentally at odds with retransmission, which needs the sent bytes host-side. Options considered:

  • Bound in-flight to a "lossless" budget — no such budget exists past eth0 (above).
  • Only promote root-netns flows to inline — the 4-tuple can't distinguish veth-backed destinations.
  • Tee-on-send (chosen): after each zero-copy read into the guest buffer, memcpy the payload into a per-flow retransmit ring (bounded by the same 256 KiB honored-window cap) shared with the bridge's FastPathConn (Arc<Mutex<…>> beside the existing shared atomics). poll_fast_path — which the shared arcbox_net loop already drives for inline flows (fix(net): add a zero-window persist probe to the download fast path #477 relies on this) — runs its existing dup-ACK/RTO machinery against the shared ring and re-emits via the normal frame path (emit_data_frames → guest_tx), draining the ring as guest_acked advances. The extra memcpy is noise next to the syscall+datapath cost at single-digit Gbps; the 1-syscall hot loop is untouched.

Applies identically to both owners (inject.rs, direct_rx.rs). Acceptance: the #450 network-workload suite (W1–W14) green on the HV backend, which currently cannot pass with inline flows under burst loss.

Refs: #451 (bridge-path fix + residual-gap note), #262 (blocked), #250, common/splicetcp/src/tcp_bridge/fast_path.rs (RTO machinery), virt/arcbox-net-inject/src/inline_conn.rs ("lossless" comment to correct).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions