Two-PC Linux setup: native HELLO/READY negotiates and registers usb4_rdma0/rocep* at kernel level, but ibverbs provider never claims it (no driver_id, PCI ID mismatch?)

[angeleitec-lang]

Setup

Two AMD Strix Halo boxes (Ryzen AI Max+ 395, 128GB unified memory each), connected directly via a USB4 cable between their native AMD USB4 host routers (1022:158d / 1022:158e, "Strix/Strix Halo PCIe USB4 Bridge" + "Strix Halo USB4 Host Router"). Ubuntu 26.04, kernel 7.0.0-29-generic. Goal: use this alongside ds4's --tensor-parallel --transport rdma, since our other RDMA link (Mellanox ConnectX-3 Pro in native InfiniBand mode) doesn't expose an IPv4-mapped GID that ds4's RDMA connection setup needs (LID-addressed fabric, not RoCE).

What worked

• thunderbolt-ibverbs-dkms_0.3.4_all.deb builds and signs cleanly via DKMS against this kernel (needed Secure Boot MOK enrollment on both machines — worth a mention in the README, took us a couple of reboots to get the MOK password to register right at the pre-boot MOK Manager screen).
• The usb4-rdma-provider_0.3.4.noble_amd64.deb (Ubuntu 24.04 build) installs a provider .so tagged -rdmav34, but our system's ibverbs-providers (rdma-core 61.0, Ubuntu 26.04) expects ABI v59 (every other provider in /usr/lib/x86_64-linux-gnu/libibverbs/ is -rdmav59.so). libibverbs warns couldn't load driver 'libusb4_rdma-rdmav59.so' and silently skips it. The generic, non-codename usb4-rdma-provider_0.3.4_amd64.deb does ship -rdmav59.so and matches — might be worth making that the default recommendation for non-LTS/newer Ubuntu, or bumping the noble build's target ABI.
• Loading the module bare (modprobe thunderbolt_ibverbs) leaves Thunderbolt service binding disabled in dmesg and no negotiation happens. Had to pass:

  modprobe thunderbolt_ibverbs profile=linux_perf bind_services=1 allocate_rings=1 start_rings=1 negotiate_native=1 enable_tunnels=1 register_verbs=1
  

  on both ends before any HELLO/HELLO_ACK/READY exchange happened. This isn't mentioned in the top-level README (only found it by asking an assistant to dig through the repo) — might be worth calling out as required, not optional, for the two-PC-Linux case.

What's still broken

With those params, both sides negotiate successfully:

thunderbolt_ibverbs: native HELLO negotiated route=0x2 rail=0x0 remote_out=9 remote_tx=2 remote_rx=2 attempt=1
thunderbolt_ibverbs: enabled tunnel route=0x2 rail=0x0 out_hop=9 remote_out_hop=9 tx_hop=2 rx_hop=2
thunderbolt_ibverbs: native READY received route=0x2 rail=0x0
thunderbolt_ibverbs: registered native ib_device usb4_rdma0 rail=peer1/0 domain=0 guid=0200544257524253

...on both machines, and a new device shows up in /sys/class/infiniband/ (rocep200s0f5) plus a fresh /dev/infiniband/uverbs1 + umad2. But ibv_devices / ibv_devinfo never list it — the provider library loads (init is called, confirmed via LD_DEBUG=libs) but doesn't claim the device.

Checked /sys/class/infiniband_verbs/uverbs1/device/driver_id — doesn't exist. Modern rdma-core prefers matching providers via that sysfs attribute; without it, it presumably falls back to a PCI vendor/device table baked into the provider .so. uverbs1/device/modalias reports pci:v00001022d0000158Dsv00001022sd0000158Dbc0Csc03i40 (the AMD USB4 host router's real PCI ID) — worth checking whether the provider's fallback matching table actually includes 1022:158d/158e, or expects something else (maybe a synthetic ID tied to the emulated rocep* device rather than the physical USB4 controller?).

Happy to grab more debug output (RDMAV_HUGEPAGES_SAFE, provider tracing, whatever's useful) if that'd help narrow it down — this got us most of the way there, just missing the last userspace hop.

There's also an unrelated thunderbolt 0000:c8:00.5: invalid hop: -1 / probe with driver thunderbolt_ibverbs failed with error -12 right before the successful native registration on both machines — looked benign (registration succeeds right after) but flagging in case it's a symptom of something.

[angeleitec-lang]

Found it — root cause is in usb4_rdma_match_device() in userspace/usb4_rdma/usb4_rdma.c.

The kernel intentionally varies the node GUID per rail (ibdev.c):

dev->base.node_guid =
    cpu_to_be64(0x0200544256524253ULL +
                ((u64)backend << 56) +
                ((u64)rail->peer->peer_id << 24) +
                rail->rail_id);

but the userspace matcher checks for exact equality against the unmodified base constant:

static bool usb4_rdma_match_device(struct verbs_sysfs_dev *sysfs)
{
	return sysfs->match ||
	       sysfs->node_guid == USB4_RDMA_NODE_GUID ||
	       sysfs->node_guid == USB4_APPLE_NODE_GUID;
}

So it only matches peer_id=0, rail_id=0, backend=0, which never happens on a real link (in my case peer_id=1 shifted one byte, 0x...56... → 0x...57..., exactly what I saw in ibv_devinfo/dmesg). The name-based match (VERBS_NAME_MATCH("usb4_rdma", ...)) doesn't save it either, since the device gets renamed by udev to ropceN/rocepBUSs0fN on Linux (as your own comment above the match function anticipates).

Fix — mask off the variable bits (backend in the top byte, peer_id/rail_id in the bottom 32 bits) and only compare the stable middle identity bytes:

 #define USB4_RDMA_NODE_GUID  0x0200544256524253ULL
 #define USB4_APPLE_NODE_GUID 0x0200544256524254ULL
+/*
+ * The kernel encodes rail identity (backend<<56 | peer_id<<24 | rail_id)
+ * into the low/high bits of the node GUID so each lane gets a distinct
+ * GID (RCCL keys discovery on this). Exact-match against the base
+ * constant therefore only matches peer_id=0/rail_id=0/backend=0, which
+ * never happens on a real 2-peer link. Mask off the variable bits and
+ * compare only the stable middle identity bytes instead.
+ */
+#define USB4_RDMA_GUID_MASK  0x00FFFFFF00000000ULL

 static bool usb4_rdma_match_device(struct verbs_sysfs_dev *sysfs)
 {
 	return sysfs->match ||
-	       sysfs->node_guid == USB4_RDMA_NODE_GUID ||
-	       sysfs->node_guid == USB4_APPLE_NODE_GUID;
+	       (sysfs->node_guid & USB4_RDMA_GUID_MASK) ==
+	       (USB4_RDMA_NODE_GUID & USB4_RDMA_GUID_MASK) ||
+	       (sysfs->node_guid & USB4_RDMA_GUID_MASK) ==
+	       (USB4_APPLE_NODE_GUID & USB4_RDMA_GUID_MASK);
 }

Rebuilt the full patched rdma-core (your two patches + this one, against upstream v62.0, same recipe as packaging/test-rdma-patches.sh) and dropped the resulting libusb4_rdma-rdmav59.so in on both machines — ibv_devices now lists the device (rocep200s0f5 in our case) on both ends:

    device          	   node GUID
    ------          	----------------
    ibp196s0        	e0071bffff799c10
    rocep200s0f5    	0200544257524253

ibv_devinfo -d rocep200s0f5 still fails to query (Failed to query device props) — I'm assuming that's expected given the "Tier 1 skeleton" comment at the top of the file (query_device stubbed to -ENOSYS), not something I looked further into. Just wanted to get the matching bug documented since it silently drops the device with zero indication of why otherwise — took some digging to trace the exact-match-vs-encoded-GUID mismatch. Happy to send a PR with the mask fix if useful.

(Context: this got us this far while chasing ds4's --tensor-parallel --transport rdma on two Strix Halo boxes over USB4 as an alternative to our Mellanox link — landed on --rdma-device-forced Mellanox in the end since that path is fully implemented in ds4, but figured the matching bug here was worth reporting regardless since it's a real, silent bug independent of that.)

[angeleitec-lang]

One more data point, in case it saves someone a debugging cycle: even with the GUID-mask fix above (device now enumerates fine in ibv_devices), ibv_devinfo -d rocep200s0f5 still fails with Failed to query device props, and a real application (ds4's --tensor-parallel --transport rdma) that calls ibv_query_port() before opening a QP gets back a nonzero return and treats the device as having no active port:

ds4-server: tp rdma: no device with an active port (); is the peer up and rdma_ctl enabled on both machines?

So query_device/query_port look unimplemented on the kernel uverbs side (consistent with the "Tier 1 skeleton" comment at the top of usb4_rdma.c — enumeration/match works, but the data-plane ioctls a real app needs before it can even open a QP don't). Ran a side-by-side benchmark against our Mellanox RDMA link on the same model (DeepSeek-V4-Flash, tensor-parallel 50/50): Mellanox completed end-to-end at ~17.3 tok/s decode, USB4 never got past the query_port step so there's no comparable number for it yet. Not filing this as a separate bug since it's presumably just "not implemented yet" rather than broken, but figured it's useful context for anyone hitting the same "device shows up but nothing else works" wall after the matching fix.

[angeleitec-lang]

Pinpointed exactly where the Failed to query device props comes from — it's not query_device itself, it's the very first ioctl (ibv_open_device() → ibv_cmd_get_context()), which fails before query_device is even reached:

openat(AT_FDCWD, "/dev/infiniband/uverbs1", O_RDWR|O_CLOEXEC) = 3
ioctl(3, RDMA_VERBS_IOCTL, "\x28\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...") = -1 ENOSPC (No space left on device)
Failed to query device props

Header: length 0x28 (40 bytes), command id 0x0001 (UVERBS_METHOD_GET_CONTEXT presumably) — so the GET_CONTEXT ioctl itself is what's rejected with ENOSPC, before any query_port/query_device call ever gets made. Given tbv_alloc_ucontext() in kernel/ibdev.c doesn't write any output attributes at all (just bumps a refcount), and ENOSPC from the ioctl multiplexer is the generic "output attribute buffer too small vs. what the driver/core declared" error, this smells like an attribute-size mismatch somewhere in how GET_CONTEXT's response is declared for this device (possibly num_comp_vectors/response-length handling, or an ABI-version mismatch between the userspace ibv_cmd_get_context() v55+ form — noted in your own CMakeLists.txt as needing an extra fd_arr param — and what the kernel side expects for this method).

Diffed the installed kernel/ibdev.c (v0.3.4, from the .deb) against current main — nothing in the git history since the release looks related to GET_CONTEXT/ucontext/num_comp_vectors, so this doesn't look fixed upstream yet either.

Stopping here rather than guessing further at the uverbs ioctl object framework internals — figured the exact failing syscall + raw bytes would be more useful to whoever knows this driver's attribute declarations than me poking at it blind. Environment for reproduction: kernel 7.0.0-29-generic, thunderbolt-ibverbs-dkms + usb4-rdma-provider both v0.3.4, two Strix Halo boxes over USB4.