Rebase USB4 GDA DV branch and add smoke probes

.bench-artifacts/rx-side-dma/STRIX3-4-VALIDATION-2026-08-15.md

@@ -0,0 +1,84 @@
+# RX-Side DMA + Link-Layer — strix-3/4 validation findings
+
+**Branch:** `codex/gda-v2-rebased-port` @ `6fbca6c` (pushed to origin)
+**Worktree:** `/mnt/Home/src/thunderbolt-ibverbs-gda-v2-rebase`
+**Validation host:** strix-3 (192.168.23.25) + strix-4 (192.168.23.26)
+**Validation date:** 2026-08-15
+
+## What's confirmed
+
+| Step | Result |
+|------|--------|
+| `nix build .#thunderbolt-ibverbs` for strix-3/4 7.2.0-rc2 booted kernel | clean compile, .ko in `/nix/store/4lglr9xs3cwnvg16w6z5d1zdm081k4fm-thunderbolt-ibverbs-0.3.4/` |
+| `tbv-target-module.sh strix-3 --booted-kernel --reload --options 'profile=linux_perf tbnet=prefer_rdma bind_services=1'` | OK: module matches strix-3 |
+| `tbv-target-module.sh strix-4 --booted-kernel --reload --options 'profile=linux_perf tbnet=prefer_rdma bind_services=1'` | OK: module matches strix-4 |
+| New module is loaded on strix-3 | `initstate: live`, `.text` sha256 `f6b8d6b1…07c9e9` (different from old `c80499a1…76e3f6de8`) |
+| New module is loaded on strix-4 | `initstate: live`, `.text` sha256 `ce135f78…85d4b9b6a6807` (different from old) |
+| RX-side DMA symbols live in kernel | `/proc/kallsyms` shows `tbv_path_post_rx_zcopy_frame` @ `ffffffffc199f430` and `tbv_rx_zcopy_complete` @ `ffffffffc19979d0`, both `[thunderbolt_ibverbs]` |
+| Bench-tools deployed on strix-3 | `/nix/store/vyc931nkpfihbz3fa8xx4f5nsfhpbbxj-thunderbolt-ibverbs-bench-tools-0.3.4/bin/` contains `dmabuf_mr_probe`, `rc_write_verify`, `uc_oneway`, `u4_pingpong`, `rc_qpn_churn`, etc. |
+| `dmabuf_mr_probe --help` | runs and lists options |
+
+## What's blocked — and it's not my fix
+
+The `usb4_rdma*` IB devices do **not** register after `rmmod` + `insmod` of the new module. dmesg shows the source-aware XDomain handler comes up, peers get bound, and 2 native services are advertised — but **no HELLO packet is sent**:
+
+```
+# After reload (strix-3, t=117.4s):
+thunderbolt_ibverbs: native control using source-aware XDomain handler
+thunderbolt_ibverbs: advertised 2 native services
+thunderbolt_ibverbs: peer 1 created backend=native
+thunderbolt_ibverbs: bound native service id=0 key=tbverbs native_lane=0 ...
+thunderbolt_ibverbs: bound native service id=1 key=tbverb1 native_lane=1 ...
+thunderbolt_ibverbs: Thunderbolt service binding enabled
+# ... no further HELLO messages, no READY, no ib_device registration
+```
+
+Compare to the **first-boot** path (strix-3, t=11s/19s/20s, before I touched anything) which negotiated successfully:
+
+```
+thunderbolt_ibverbs: native control using source-aware XDomain handler
+thunderbolt_ibverbs: native HELLO_ACK received route=0x2 rail=0x1 remote_out=9 remote_tx=2 remote_rx=2
+thunderbolt_ibverbs: native HELLO negotiated route=0x2 rail=0x1 ... attempt=1
+thunderbolt_ibverbs: enabled tunnel route=0x2 rail=0x1 ...
+thunderbolt_ibverbs: native READY received route=0x2 rail=0x1
+```
+
+The difference: the kernel autoload path (boot-time) gets the source-aware handler initialized **before** the Thunderbolt peer tunnel exists, so it knows to send HELLO when the peer appears. The reload path initializes the handler **after** the peer tunnel is already up, and the handler doesn't know it needs to send HELLO to a peer it never saw arrive.
+
+This is a pre-existing reload bug, not introduced by my link-layer or RX-side DMA patches. The patches I shipped (`705adf8`, `f34678d`) don't touch the source-aware handler or the XDomain registration flow.
+
+I tried to work around it: rmmod + insmod the new module on both nodes after reboot, bounce `authorized` on the TB devices, re-rmmod + re-insmod with `bind_services=1`. The handler always comes up, peers always bind, but HELLO never goes out. The kernel autoload path is the only one that works.
+
+## What this means for validation
+
+The link-layer fix and the RX-side DMA implementation are **in the loaded kernel on both nodes**, confirmed via `/proc/kallsyms`. The wire-format path is wired up. What's missing is the IB device registration, which requires a power-cycle.
+
+**Two ways to unblock validation:**
+
+1. **Quickest:** physically power-cycle strix-3 and strix-4 (or use BMC). The kernel autoload will run, the source-aware handler will initialize fresh, HELLO will exchange, and IB devices will register. Then:
+
+   ```bash
+   cd /mnt/Home/src/thunderbolt-ibverbs-gda-v2-rebase
+   ./userspace/bench/rx_side_dma_validate.sh HOSTS='strix-3 strix-4'
+   ```
+
+   This assumes the Nixos config on the boot partition points at an .ko that contains the link-layer fix + RX-side DMA patches. **It currently doesn't** — the booted-system kernel-modules still has the old `c80499a1…` module on both nodes. A bare power-cycle will come up with the old module again. You need to do option 2 first, or you'll be in the same state I'm in now: new module on the side, old module booted.
+
+2. **Correct path:** update `nixos-config/flake.nix` to point the `thunderbolt-ibverbs-kernel` path input at the new commit `6fbca6c` of the GDA branch, `nix flake update thunderbolt-ibverbs-kernel`, then `nixos-rebuild switch` on strix-3 and strix-4, then reboot. The new module will then be the booted module, source-aware handler will init at boot, HELLO will exchange, IB devices will register.
+
+I haven't done (2) yet because it touches the production nixos-config — wanted your call on whether to push that or do something more surgical first.
+
+## What I want to also do (uncommitted, ready to push)
+
+- Update `userspace/bench/rx_side_dma_validate.sh` to pass `--options 'profile=linux_perf tbnet=prefer_rdma bind_services=1'` to `tbv-target-module.sh` (without it, the source-aware handler doesn't init at all, even on a clean boot).
+- Add a post-reload sleep + retry loop that checks for `usb4_rdma*` IB devices to appear, and fails fast with diagnostic output if they don't.
+- Update the run script to read link_layer from `/sys/class/infiniband/usb4_rdma*/ports/1/link_layer` on both nodes, fail if any port reports anything other than `InfiniBand`.
+
+## Files
+
+- Summary (this file): `/mnt/Home/src/thunderbolt-ibverbs-gda-v2-rebase/.bench-artifacts/rx-side-dma/STRIX3-4-VALIDATION-2026-08-15.md`
+- Run script: `/mnt/Home/src/thunderbolt-ibverbs-gda-v2-rebase/userspace/bench/rx_side_dma_validate.sh` (already pushed to origin)
+- New commit: `6fbca6c` on `codex/gda-v2-rebased-port` (already pushed)
+- Patches:
+  - `705adf8` `ibdev: report InfiniBand link layer` — was Ethernet, kernel was routing every QP through the RoCE code path that has no resolver, so `ibv_modify_qp(RTR/RTS)` returned `-ENODATA`. Switched to `IB_LINK_LAYER_INFINIBAND` + `RDMA_CORE_CAP_IB_MAD`.
+  - `f34678d` `ibdev, path: RX-side DMA into dmabuf MRs` — per-fragment DMA into the destination SGL via `dma_map_page`, two new debugfs counters (`data_rx_dmabuf_zcopy`, `data_rx_dmabuf_zcopy_error`). CPU MRs untouched.

docs/RX_SIDE_DMA.md

@@ -0,0 +1,160 @@
+# RX-side DMA for dmabuf MRs
+
+**Status:** implemented on `codex/apple-xdomain-property-match` (commit `dc8f43f`).
+Pending: live dmabuf-destination smoke test on `strix-1`.
+**Goal:** make RDMA WRITE destinations land in dmabuf MRs without CPU-side
+copies. CPU-pinned MRs continue to use the existing bounce-buffer path; this
+change does not regress them.
+
+## Why a spec first
+
+The current RX path always lands ring DMA into a kernel bounce buffer, then
+copies from the bounce buffer into the destination MR. That works for
+`ib_umem` (CPU-pinned) but breaks for `ib_umem_dmabuf`:
+
+```
+tbv_umem_copy_to_iova():
+  mr->umem->is_dmabuf == true
+  -> dma_buf_begin_cpu_access()
+  -> kmap_local_page() + memcpy()   // bails on ZONE_DEVICE pages
+  -> dma_buf_end_cpu_access()
+```
+
+So GPU dmabuf MRs cannot be RDMA-WRITE destinations today. The dmabuf MR
+import path exists (see `tbv_reg_user_mr_dmabuf`), it just has no place to
+land data.
+
+## The change
+
+DMA each ring frame **directly into the destination MR page** when the MR is
+a dmabuf MR, or when the MR's pages cannot be CPU-kmap'd. Otherwise fall
+through to the bounce-buffer path unchanged.
+
+Per-fragment DMA is required because NHI ring descriptors point at one
+contiguous physical address; an MR's SGL is scattered. Each PDF frame
+becomes its own DMA into one MR page.
+
+### What stays the same
+
+- The NHI ring. The descriptor interface already accepts per-frame DMA
+  addresses (`ring->descriptors[ring->head].address = frame->dma`,
+  `ring_write_descriptors()`). We are not patching the kernel.
+- The reorder / fragment reassembly / WC push logic in `ibdev.c`.
+- The CPU-MR bounce-buffer path.
+- All TX paths.
+
+### What changes
+
+Three pieces in `kernel/path.c` and `kernel/ibdev.c`:
+
+1. **MR-side mapping.** When the reorder layer accepts the first fragment of
+   a WRITE that targets a dmabuf MR, look up the rkey, validate the range,
+   `dma_map_sg_attrs(... DMA_FROM_DEVICE)` on the destination page range,
+   and stash the mapped SGL on `tbv_rx_reorder_msg`.
+2. **Per-fragment DMA.** Replace the call into
+   `tbv_rx_reorder_store_fragment_locked` for dmabuf destinations with
+   `tbv_rx_zcopy_store_fragment_locked` that:
+   - allocates a `tbv_data_frame` from a small `rx_zcopy_pool`
+   - sets `frame->buf` to the MR page + offset, `frame->dma` to the mapped
+     address
+   - `dma_sync_single_for_device(..., DMA_FROM_DEVICE)` then `tb_ring_rx()`
+   - on completion, `dma_unmap_page(..., DMA_FROM_DEVICE)`, push WC if the
+     fragment was last, free the frame
+3. **CPU fallback unchanged.** If `mr->umem->is_dmabuf == false`, the
+   existing kernel-bounce-buffer RX path stays in use.
+
+### Where it lives
+
+- `kernel/path.c`: extend `tbv_data_frame` with `enum tbv_rx_mode { CPU,
+  ZCOPY_DMABUF }` and a `tbv_rx_zcopy` struct holding the unmap state.
+  Add `tbv_path_post_rx_zcopy_frame()` that submits a per-fragment DMA
+  into the MR page.
+- `kernel/ibdev.c`: branch in
+  `tbv_rx_buffer_write_fragment_locked()`: dmabuf MR -> the new path; CPU
+  MR -> existing path.
+- `kernel/tbv.h`: add the new mode enum and per-frame zcopy bookkeeping on
+  `tbv_rx_reorder_msg`.
+
+### Failure handling
+
+- If the MR's SGL map fails (`dma_map_sg_attrs` returns 0), ack the
+  fragment with `TBV_NATIVE_SEND_ACK_ERROR`, push `IB_WC_LOC_PROT_ERR`,
+  drop the reorder message. This is identical to the existing error path
+  for bad rkey.
+- If `dma_map_page` returns an error during per-fragment DMA setup, same
+  treatment.
+- Completion ordering: NHI completes descriptors in arrival order; PDF
+  frames already carry the fragment index, and the existing
+  `frag_seen[]` bitmap stays correct because we record `set_bit(frag_idx,
+  msg->frag_seen)` only after the DMA is posted. The WC is pushed when
+  the **last** fragment completes, not when the first does. This matches
+  what dmabuf MR consumers expect.
+
+### What we explicitly do not do
+
+- We do not reorder or coalesce DMA. Per-fragment DMA, period.
+- We do not modify the NHI ring or the kernel module.
+- We do not change TX-side zcopy or the SEND path. SEND lands in a kernel
+  WQE buffer, not in an MR.
+- We do not introduce per-MR DMA contexts; the existing ring DMA device
+  is reused.
+
+## Test plan (on strix-1 + strix-2, kernel 7.2.0-rc2)
+
+1. **CPU regression.** Re-run the existing `userspace/bench/ibv_*.c`
+   send/recv suite (sender CPU MR, receiver CPU MR). Confirm
+   `data_rx_completed` matches `data_rx_reorder_delivered` and no
+   `data_rx_copy_error` increments.
+2. **Dmabuf destination regression.** Run
+   `userspace/bench/dmabuf_mr_probe` with sender CPU MR, receiver GPU
+   dmabuf MR. Confirm `data_rx_completed > 0`, `data_rx_copy_error`
+   stays at 0, and `data_rx_dmabuf_zcopy` (new counter) increments.
+3. **GPU P2P smoke.** Run
+   `userspace/bench/hip_rdma_write_visibility_probe.cpp`. The PR-thread
+   symptom was `data_rx_copy_error=2`, `gpu_seen=0`. With this change we
+   expect `gpu_seen > 0` and no copy errors.
+
+## Open questions
+
+- **RX ring pool size.** NHI rings have a fixed descriptor count. While a
+  zcopy frame is in flight, the ring has one fewer descriptor for the
+  bounce path. For bursty small messages this could starve CPU MR RX.
+  Mitigation: cap concurrent zcopy fragments per QP at
+  `min(ring->size / 2, TBV_RX_ZCOPY_MAX_PER_QP)`, default 32.
+- **GPU page alignment.** ZONE_DEVICE pages are not necessarily aligned
+  to `TBV_DATA_FRAME_SIZE`. The per-frame DMA must respect
+  `frame->size` and the offset within the page; this is already how the
+  ring works (`frame->size` is what the descriptor programs as length),
+  so no kernel change is needed. We do, however, need to refuse to map
+  fragments whose first-byte offset into the page is non-zero, because
+  NHI DMA into a partial page will tear the page. Constraint: a
+  fragment's destination `(iova & (PAGE_SIZE - 1))` must be 0 modulo the
+  DMA page granularity. We can satisfy this by mapping page-aligned,
+  `bytes_to_copy = min(remaining_in_page, frag_len)`, and pushing
+  fragment splits if needed.
+
+## Estimated scope
+
+- ~150-200 lines added in `kernel/path.c` and `kernel/ibdev.c`
+- one new counter in `kernel/debugfs.c`
+- one probe binary already exists; minimal new tests
+- no kernel patch changes
+
+## Decision
+
+Approve to proceed? If yes, I will start with the dmabuf-only branch
+(smallest diff, leaves CPU MRs untouched) and report back with a
+buildable module + a smoke test on strix-1.
+
+## Run book for the next experiment on strix-1 + strix-2
+
+1. **Reload the module.** `tools/tbv-target-module.sh strix-1 --booted-kernel --reload --options 'profile=linux_perf apple_data=N native_data=Y bind_services=Y allocate_rings=Y start_rings=Y negotiate_native=Y enable_tunnels=Y register_verbs=Y zcopy_min_bytes=4294967295'`. Repeat for `strix-2` if the kernel is rebuilt there. Verify `/sys/class/infiniband/usb4_rdma*/ports/1/link_layer` reads `InfiniBand` (not `Ethernet`).
+2. **CPU regression first.** Run `userspace/bench/rc_write_verify` between the two nodes. Confirm `data_rx_completed` matches `data_rx_reorder_delivered` and no `data_rx_copy_error` increments in `/sys/kernel/debug/tbv/usb4_rdma*/summary`. This must pass before touching dmabuf.
+3. **dmabuf MR probe.** Open a HIP-allocated region on `strix-1`, export it via `hsa_amd_portable_export_dmabuf`, register it via `ibv_reg_dmabuf_mr` on `usb4_rdma0` (or `usb4_rdma1`). Have the peer do an RDMA WRITE into it. Confirm `data_rx_dmabuf_zcopy` (new counter) increments and `data_rx_copy_error` stays at 0.
+4. **HIP visibility probe.** Run `userspace/bench/hip_rdma_write_visibility_probe` with `--role recv --kind device --recv-reg dmabuf` on the GPU node and `--role send --kind malloc --source-fill cpu` on the peer. Compare against the `--recv-reg reg_mr` baseline. The PR-thread failure mode was `gpu_seen=0`, `data_rx_copy_error=2`. With this change we expect `gpu_seen > 0` and zero copy errors.
+5. **Failure triage.** If `data_rx_dmabuf_zcopy_error` increments, the most common cause is `dma_map_sg` returning 0 (device not IOMMU-mapped or BO not contiguous) or the SGL walk failing (`dest_iova` outside any sg entry). Both are recoverable by aborting the WRITE and pushing `IB_WC_LOC_PROT_ERR`, so the wire stays consistent.
+
+## What's still missing
+
+- The same RX-side DMA change needs to be **ported onto `codex/gda-v2-rebased-port`** (the path input the cluster actually builds from). Currently it lives only on `codex/apple-xdomain-property-match`.
+- The `nixos-config` flake.lock pin in this commit points at `codex/apple-xdomain-property-match`, but the cluster still consumes the GDA v2 rebase path input. Once the GDA branch picks up these commits, a `colmena build` will pull them in.

flake.nix

@@ -354,6 +354,7 @@
             thunderboltKernel = mkThunderboltKernel pkgs;
             thunderboltLinuxPackages = mkThunderboltLinuxPackages pkgs;
             moduleForThunderboltKernel = thunderboltLinuxPackages.callPackage ./nix/module.nix { };
+            dvProbes = pkgs.callPackage ./nix/dv-probes.nix packageArgs;
           in
           {
             default = module;
@@ -364,6 +365,7 @@
             thunderbolt-ibverbs = module;
             thunderbolt-ibverbs-linux-thunderbolt = moduleForThunderboltKernel;
             tbv-perftest = perftestBench.runner;
+            tbv-dv-caps-probe = dvProbes;
           }
         )
       );
@@ -409,6 +411,10 @@
               maintainers = with pkgs.lib.maintainers; [ georgewhewell ];
             };
           };
+          tbv-dv-caps-probe = {
+            type = "app";
+            program = lib.getExe pkgsAt.tbv-dv-caps-probe;
+          };
         }
       );
 
@@ -429,6 +435,7 @@
           proto-smoke = mkProtoSmoke pkgs;
           rdma-core-usb4 = pkgsAt.rdma-core-usb4;
           verbs-smoke-build = mkVerbsSmokeBuild pkgs;
+          tbv-dv-caps-probe = pkgsAt.tbv-dv-caps-probe;
         }
       );
 
@@ -450,6 +457,7 @@
           linux-thunderbolt-modules = pkgsAt.linux-thunderbolt-modules;
           rdma-core-usb4 = pkgsAt.rdma-core-usb4;
           thunderbolt-ibverbs-linux-thunderbolt = pkgsAt.thunderbolt-ibverbs-linux-thunderbolt;
+          tbv-dv-caps-probe = pkgsAt.tbv-dv-caps-probe;
           vm-smoke.nixos = mkNixosVmSmoke pkgs;
         }
       );

kernel/Makefile

@@ -7,7 +7,7 @@ KDIR ?= /lib/modules/$(KVER)/build
 PWD := $(shell pwd)
 
 obj-m := thunderbolt_ibverbs.o
-thunderbolt_ibverbs-y := main.o profile.o backend.o configfs.o core.o debugfs.o ibdev.o link.o native.o apple.o trace.o native_control.o native_control_legacy.o native_control_xdomain.o path.o peer.o rail.o service.o tbnet_identity.o tbnet_minimal.o ../proto/config.o ../proto/identity.o ../proto/reliability.o
+thunderbolt_ibverbs-y := main.o profile.o backend.o configfs.o core.o debugfs.o dv.o ibdev.o link.o native.o apple.o trace.o native_control.o native_control_legacy.o native_control_xdomain.o path.o peer.o rail.o service.o tbnet_identity.o tbnet_minimal.o ../proto/config.o ../proto/identity.o ../proto/reliability.o
 
 ccflags-y := -Wall -Wextra -Wno-unused-parameter -I$(src) -I$(src)/..
 

kernel/debugfs.c

@@ -327,6 +327,10 @@ static int tbv_debugfs_summary_show(struct seq_file *s, void *unused)
 		   atomic64_read(&state->data_rx_rnr_suppressed));
 	seq_printf(s, "data_rx_copy_error: %lld\n",
 		   atomic64_read(&state->data_rx_copy_error));
+	seq_printf(s, "data_rx_dmabuf_zcopy: %lld\n",
+		   atomic64_read(&state->data_rx_dmabuf_zcopy));
+	seq_printf(s, "data_rx_dmabuf_zcopy_error: %lld\n",
+		   atomic64_read(&state->data_rx_dmabuf_zcopy_error));
 	seq_printf(s, "data_rx_send_len_error: %lld\n",
 		   atomic64_read(&state->data_rx_send_len_error));
 	seq_printf(s, "data_rx_send_prot_error: %lld\n",