OdinLink-Five

Thunderbolt 5 RDMA for Linux — kernel driver, libibverbs provider, NCCL/RCCL plugins

OdinLink turns a Thunderbolt cable into a high-speed RDMA interconnect between machines. It provides the full ibv_* verbs API so any verbs-aware application (NCCL, MPI, PyTorch DDP) can use Thunderbolt DMA without code changes.

80 Gbps  ·  sub-µs latency  ·  zero-copy GPU  ·  standard ibv_verbs API

Progress

Layer	Component	Status
🟢	Kernel module (`odl_tb5.ko`)	NHI ring DMA, XDomain handshake, loopback mode
🟢	Userspace library (`libodl_tb5.so`)	C API, stream I/O, mmap, DMA-buf
🟢	Verbs provider (`libodl_tb5_verbs.so`)	`ibv_open_device`, `ibv_reg_dmabuf_mr`, QP/CQ lifecycle
🟢	rdma-core plugin (`libodl_tb5-rdmav59.so`)	Auto-discovered by `ibv_devinfo`
🟢	Async I/O	`poll()` + `O_NONBLOCK` ioctls end-to-end
🟢	No-cable testing	`loopback=1` module param + mock library
🟢	NCCL verbs transport	NCCL's built-in `NCCL_NET_PLUGIN=IB` transport auto-discovers ODL via `ibv_get_device_list`
🟡	NCCL custom plugin	DMA-buf zero-copy path (legacy, use verbs transport instead)
🟡	Async DMA-buf	Needs callback-based cleanup — stream path is already async via poll()

Quick Start

Build & Run

sudo dnf install gcc make cmake kernel-devel pkgconfig rdma-core-devel libibverbs-utils
git clone https://github.com/hetrag/OdinLink-Five.git
cd OdinLink-Five && mkdir build && cd build
cmake .. -DBUILD_VERBS=ON
make -j$(nproc) odl_tb5_verbs odl_tb5_verbs_provider
make -j$(nproc) driver            # builds driver/odl_tb5.ko (kernel module is an opt-in target)

# Test without cable:
sudo insmod driver/odl_tb5.ko loopback=1
ibv_devinfo                     # needs a kernel uverbs device (see docs/INSTALL.md)
build/verbs/tests/test_verbs_basic

Point-to-Point (two machines)

# Machine A:
sudo insmod driver/odl_tb5.ko
build/cli/odl_tb5_cli --server --device 0

# Machine B:
sudo insmod driver/odl_tb5.ko
build/cli/odl_tb5_cli --client --device 0 --test bandwidth

Full install guide → docs/INSTALL.md

Architecture

┌────────────────────────────────────────────────────┐
│  Application (NCCL, MPI, PyTorch, ibv_* API)       │
├────────────────────────────────────────────────────┤
│             libibverbs (libibverbs.so.1)            │
├────────────────────────────────────────────────────┤
│  libodl_tb5-rdmav59.so  (verbs provider plugin)    │
├────────────────────────────────────────────────────┤
│  libodl_tb5.so  (OdinLink C API)                   │
├────────────────────────────────────────────────────┤
│  odl_tb5.ko  (kernel module — NHI DMA)             │
├────────────────────────────────────────────────────┤
│  Thunderbolt 5 NHI DMA Engine                       │
└────────────────────────────────────────────────────┘

Components

Component	Binary	Description
Kernel driver	`odl_tb5.ko`	NHI ring DMA, XDomain handshake, char device
Library	`libodl_tb5.so`	C API wrapping ioctls, streams, mmap
Verbs provider	`libodl_tb5_verbs.so`	Standalone `ibv_*` via symbol interposition
Verbs plugin	`libodl_tb5-rdmav59.so`	rdma-core provider plugin (`ibv_devinfo`)
NCCL plugin	`libnccl-net-ODL_TB5.so`	NVIDIA GPU collectives
RCCL plugin	`librccl_net_odl_tb5.so`	AMD GPU collectives
CLI tool	`odl_tb5_cli`	Bandwidth, latency, jitter, MIMO tests
Loopback module	`loopback=1` param	Fake peer for no-cable testing
Mock library	`libodl_tb5_mock.so`	LD_PRELOAD simulation (no kernel needed)

GPU and daemon/tray → `docs/GPU.md`, `docs/INSTALL.md`

Verbs API Coverage

Operation	Status	Notes
`ibv_open_device`	✅	Symbol interposition + rdma-core plugin
`ibv_query_device`	✅	Attributes from peer info
`ibv_query_port`	✅	Port state from peer connection
`ibv_alloc_pd` / `ibv_dealloc_pd`	✅	Protection domains
`ibv_reg_mr` / `ibv_dereg_mr`	✅	Host memory registration
`ibv_reg_dmabuf_mr`	✅	Zero-copy GPU memory (Linux DMA-buf)
`ibv_create_cq` / `ibv_destroy_cq`	✅	Eventfd-based completion queues
`ibv_poll_cq` / `ibv_req_notify_cq`	✅	Poll + eventfd notification
`ibv_create_qp` / `ibv_destroy_qp`	✅	RC QP → stream mapping
`ibv_modify_qp`	✅	RESET → INIT → RTR → RTS
`ibv_post_send`	✅	Async via workqueue + poll()
`ibv_post_recv`	✅	Non-blocking via poll()
`ibv_query_qp`	✅	State + capabilities

Async I/O Model

ibv_post_send(qp, wr, NULL)
    │
    ▼  (non-blocking, returns immediately)
Enqueue WR → per-QP submission queue
    │
    ▼  (worker thread)
poll(fd, POLLOUT)  ← kernel signals TX readiness
    │
    ▼
ioctl(STREAM_SEND) ← O_NONBLOCK, never blocks
    │
    ├── -EAGAIN → re-queue WR, poll again
    └── success → post struct ibv_wc → CQ → eventfd

Testing Without Hardware

# Option 1: kernel loopback (real module, fake peer)
sudo insmod driver/odl_tb5.ko loopback=1
build/verbs/tests/test_verbs_basic

# Option 2: user-space mock (no kernel module at all)
mkfifo /dev/odl_tb5_0
LD_PRELOAD=build/verbs/tests/libodl_tb5_mock.so \
  LD_LIBRARY_PATH=build/verbs:build/lib \
  build/verbs/tests/test_verbs_mock_loopback

Smoke Tests (with hardware)

Verbose logging is essential for diagnosing failures. Use the provided helper script which captures everything automatically:

# Full smoke test suite — all logs go to smoke-test-<timestamp>/:
./scripts/smoke-test.sh

# Run only the verbs provider test:
./scripts/smoke-test.sh -t verbs

# Bandwidth test (two machines, machine A first):
sudo ./scripts/smoke-test.sh -t bandwidth -m server   # Machine A
sudo ./scripts/smoke-test.sh -t bandwidth -m client   # Machine B

# Custom output directory:
./scripts/smoke-test.sh -o /tmp/odl-debug

Verbose logging is also available manually:

# Watch kernel driver logs (run in a separate terminal):
sudo dmesg -w | grep odl_tb5

# Trace all verbs calls — set level 1–5 (5 = most verbose):
export ODL_VERBS_DEBUG=5

Manual smoke test steps

1. Kernel module + device node

sudo insmod driver/odl_tb5.ko
sudo chmod 666 /dev/odl_tb5_0    # allow non-root access
ls -l /dev/odl_tb5_0             # appears only when a TB5 peer is connected

2. Full test suite (3 suites: device, lib API, plugin)

build/tests/odl_tb5_test

3. Verbs provider lifecycle test

build/verbs/tests/test_verbs_basic

Exercises: device discovery, context open, PD/MR/CQ/QP lifecycle, post_send/post_recv.

4. End-to-end bandwidth (two machines)

# Machine A:
build/cli/odl_tb5_cli --server --device 0

# Machine B (wait for server to be ready):
build/cli/odl_tb5_cli --client --device 0 --test bandwidth

5. ibv_devinfo discovery

ibv_devinfo     # should list an odl_tb5 device

Module Parameters

Param	Default	What it does
`e2e=0`	1 (on)	Disables end-to-end flow control handshake. Only needed for old TB3 controllers that choke on E2E. TB4/TB5 leave this alone.
`loopback=1`	0 (off)	Creates fake devices with no cable — data loops back inside your own machine. For testing without a peer.
`protocol=1`	0 (OdinLink)	Switches to Apple's protocol ID (0xFA57) so macOS peers can discover OdinLink. For Mac↔Linux only.
`odl_ring_size=512`	4096	DMA packet slots per ring (each slot = 4 KB). Larger = smoother bursts, more RAM. Lower it if `probe` fails with `-ENOMEM` (the default 4096 allocates four 16 MB contiguous DMA buffers, which can fail on fragmented systems).

# Examples:
sudo insmod driver/odl_tb5.ko                  # TB4/TB5, default everything
sudo insmod driver/odl_tb5.ko e2e=0            # old TB3 controller
sudo insmod driver/odl_tb5.ko loopback=1        # no cable, just testing
sudo insmod driver/odl_tb5.ko protocol=1        # talk to macOS
sudo insmod driver/odl_tb5.ko odl_ring_size=512 # DMA alloc fails with -ENOMEM at 4096

Debug

export ODL_VERBS_DEBUG=5     # Trace all verbs calls
sudo dmesg -w | grep odl_tb5 # Kernel driver logs

Troubleshooting → docs/TROUBLESHOOTING.md

Cross-Platform: macOS

Apple ships libthunderboltrdma.dylib + libibverbs on macOS 26.5, but the kernel extension is a stub — IORDMAFamily is not shipped. Mac Thunderbolt RDMA is not currently functional. OdinLink is the only working implementation. See COMPAT.md.

Repository

Resource	Link
Install guide	`docs/INSTALL.md`
GPU / NCCL / RCCL	`docs/GPU.md`
Troubleshooting	`docs/TROUBLESHOOTING.md`
Packaging / .deb	`docs/PACKAGING.md`
Agent instructions	`AGENTS.md`
Verbs provider manual	`verbs/VERBS_PROVIDER.md`
Cross-platform compat	`COMPAT.md`

License

Kernel driver (odl_tb5.ko): GPL v2
All userspace: MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
assets		assets
bridge		bridge
cli		cli
daemon		daemon
docs		docs
driver		driver
lib		lib
nccl		nccl
packaging		packaging
rccl		rccl
scripts		scripts
tests		tests
third_party		third_party
tray		tray
verbs		verbs
.dockerignore		.dockerignore
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CMakeLists.txt		CMakeLists.txt
COMPAT.md		COMPAT.md
Dockerfile.test		Dockerfile.test
ODL_TB5_RCCL_PLAN.md		ODL_TB5_RCCL_PLAN.md
README.md		README.md
docker-compose.yml		docker-compose.yml

Navigation Menu

Repository files navigation

OdinLink-Five

Progress

Quick Start

Build & Run

Point-to-Point (two machines)

Architecture

Components

GPU and daemon/tray → `docs/GPU.md`, `docs/INSTALL.md`

Verbs API Coverage

Async I/O Model

Testing Without Hardware

Smoke Tests (with hardware)

Manual smoke test steps

Module Parameters

Debug

Cross-Platform: macOS

Repository

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

OdinLink-Five

Progress

Quick Start

Build & Run

Point-to-Point (two machines)

Architecture

Components

GPU and daemon/tray → docs/GPU.md, docs/INSTALL.md

Verbs API Coverage

Async I/O Model

Testing Without Hardware

Smoke Tests (with hardware)

Manual smoke test steps

Module Parameters

Debug

Cross-Platform: macOS

Repository

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

GPU and daemon/tray → `docs/GPU.md`, `docs/INSTALL.md`