mmap: publish the completion queue to userspace

[afrog33k]

Every post_send and every poll_cq on this driver is a uverbs ioctl. Measured on a two-machine pingpong with perf and strace: ~96 ioctls per iteration, and the poll loop is where nearly all of them come from — the receiver issues ~95 polls per round trip, while the data path itself needs one ioctl per message. This publishes the completion queue to userspace so the provider can poll it by reading memory.

Design

The per-CQ buffer becomes one vmalloc_user() allocation: page 0 is a shared header, the kernel's ib_wc ring follows, then the ib_uverbs_wc ring the provider reads. A new .mmap op exposes it through rdma_user_mmap_entry_insert(), and .mmap_free releases it.

The ring carries struct ib_uverbs_wc — the uapi wire format — not the kernel's struct ib_wc (which holds a struct ib_qp * and is not a wire format) and not userspace's struct ibv_wc. tbv_cqe_from_wc() translates on push, tbv_poll_cq() translates back.

Occupancy is a pair of monotonic totals with one writer each: produced (kernel) and consumed (consumer). The obvious design — publish head/tail and derive occupancy — cannot distinguish full from empty at cqe == 1. That case is not hypothetical: ib_send_lat creates its send CQ with capacity 1 (TX depth 1), so (tail + 1) % cqe == head is a tautology (x % 1 == 0), and the CQ declared overflow before storing a single completion.

Overflow is a monotonic ovf_seq, not a sticky flag, and the consumer latches its own copy. A latched flag makes the CQ permanently dead, and perftest's do { } while (ne == 0) poll loop cannot escape it.

create_cq returns a vendor response tail (struct tbv_uresp_create_cq) with the mmap offset, cqe count, shm ABI and map_len. The kernel fills it only when the provider's outbuf is large enough, so an older provider sees nothing and never mmaps, and a newer provider on an older kernel reads zeros from its own calloc and falls back to ioctls.

On the userspace side, create_cq maps the buffer when offered and polls it lock-free — acquire load of the tail, read the entries, release store of the head. When the kernel offers nothing, or the map fails, the cq's mmap state stays null and every call falls through to the existing ibv_cmd_* path, byte-identical with the current provider.

Measured

Two machines, one Thunderbolt 4 cable, ib_send_lat, interleaved A/B, 6 rounds, both arms md5-verified on install, and every run asserting its own iteration count in the same output as the number taken from it (a failed run otherwise reports a flatteringly small syscall count):

arm	ioctls / iteration
current provider	9.14
this branch	2.02

The residual 2.02 is exactly post_send + post_recv, one each.

What this does not do

Throughput and latency do not improve. Across a queue-pair sweep (1–8) and a message-size sweep (64 KiB–1 MiB) the difference sits inside the run-to-run spread, and at q=4 it is nil. The syscall path was never the throughput limit on this hardware. This change removes CPU from the completion path and is the prerequisite for a lockless one; it should be judged on that, not on a bandwidth claim it cannot support.

Build

• kernel half compiles clean against an out-of-tree 6.18 tree (make -j8 in kernel/, no errors).
• proto/tbv_cq_shm.h is included by both sides. Both are Linux and both take the fixed-width types from <linux/types.h> — note that a userspace branch declaring its own __u32/__aligned_u64 does not compile against rdma-core, which already defines them; that was the first build failure of this header.

🤖 Generated with Claude Code