Repository navigation
fix: Tell clients when a VIP packet is too big for the fabric - #753
Conversation
The gateway's load balancer adds a 40-byte outer header to every packet it forwards to a backend. When that made a client's packet too big for the route, the FIB lookup refused it, and the datapath counted a fib_frag_needed drop and discarded it. The client never heard why, so its path MTU discovery never adapted and every large packet to the VIP vanished. edge_lb now answers with a rate-limited ICMPv6 Packet Too Big, ported from the egress shard's (#669). It is sourced from the VIP, the address the client was talking to and the one announced publicly, rather than the node's SRv6 address, which upstream source filtering may drop. It reports the route's MTU less the 40 bytes of encapsulation and quotes the client's IPv6 header and first 8 transport bytes, read straight from the frame, where the client's packet is still intact behind the header push_outer_header wrote. Nothing is copied per packet. The packet still counts as a fib_frag_needed drop against the VIP. Messages draw on a per-CPU token bucket with the shard's limits, 1000 a second and a burst of 100, and a refusal counts under a new drop reason, icmp_rate_limited. edge_return still drops a reply too big for its route without a Packet Too Big; the architecture doc's Known Constraints entry now says so. New datapath tests run in a namespace with a 1500-byte fabric link and check the message's addresses, MTU, quote and checksum, and the rate-limited path. Fixes #700 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fabric route under 1320 bytes cannot carry even a minimum-MTU IPv6 packet once the gateway adds its 40-byte outer header. A Packet Too Big would then report less than 1280, which no IPv6 sender goes below (RFC 8200 section 5), so its next 1280-byte packet would fail the same way and the message would only spend a rate-limit token. edge_lb now drops such a packet, counted as fib_frag_needed as before, without sending one. A new test covers a 1300-byte fabric link, and the gateway architecture doc's Known Constraints entry says so. Related to #700 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Second review pass on d74eb50: pr-rereviewer said merge (the Packet Too Big fix is present and adds no regressions), and pr-conventions-reviewer said merge. Remaining items are nits: commit subject length, and the gateway architecture doc overlapping with #752 and #738 at different hunks. Whether the change ships with the next gateway release was raised as a decision item. It does, because merges deploy to staging and releases to production. The first pass on ccc0642 split: pr-conventions-reviewer said merge, pr-adversary said hold. The split was settled in d74eb50, which sends no Packet Too Big below 1280 and adds a test. The tail-growth warning and the adjust_tail counter nit were accepted as they are. The edgeprog package ran as root at this head. All 15 tests passed, including the verifier load test. No fix commits were made in this phase. CI: all 14 checks pass at d74eb50. Unit Tests (root) failed once on TestPMTU_VethPodLearnsPathMTU, an unrelated flaky PMTU test outside this change, and passed on re-run. Auto-merge is enabled with the merge method, the only one the ruleset allows. The ruleset requires one approving review and code owner review, and dismisses stale approvals on push, so the PR waits for a human approval. |
2 participants
Summary
When the gateway's 40-byte encapsulation made a client's packet too big for the route, the gateway dropped it silently, so the client's path MTU discovery never adapted and large packets to a VIP vanished.
The gateway now sends a rate-limited ICMPv6 Packet Too Big from the VIP, with an MTU that leaves room for the encapsulation, and still counts the drop.
Replies from backends that are too big for their route still drop without a message, as the architecture doc now notes.
Test plan
Fixes #700
🤖 Generated with Claude Code