A Full-Stack Open-Source High-Performance RDMA Hardware and Software Implementation, Focused on AI Applications, Committed to Surpassing Commercial Solutions
English | 简体中文
Ideal for Researchers, University Students, and RDMA Beginners
With the rapid development of AI large model technology, the high-performance networking provided by RDMA has become a critical component of AI infrastructure. However, RDMA technology born in the last century is difficult to fully adapt to today's AI computing scenarios: existing commercial RDMA solutions are constrained by legacy compatibility burdens and struggle to iterate quickly to meet AI demands; RDMA network cards as closed-source black-box products form a significant contradiction with the open-source AI large model software ecosystem. At a time when hardware-software co-optimization is increasingly important, black-box RDMA network cards have become a key bottleneck restricting global optimization of AI systems.
To address these issues, Pazhou Laboratory (Huangpu) and Datanlord jointly initiated the Open-RDMA open-source project. We deeply recognize that relying solely on a single research institution's strength is insufficient to complete the development and debugging of a full-stack system, let alone change the existing industry landscape. Therefore, we chose a full-stack open-source approach, starting from the dimensions of academic research and talent cultivation, leveraging open-source community power to lower the technical barrier of this specialized RDMA field, cultivate RDMA technical talent, enhance the brand influence of the Open-RDMA project community, and gradually expand Open-RDMA's industry influence in the entire RDMA field.
Whether you are a practitioner or student in hardware (FPGA, ASIC), software information technology (driver development, training/inference frameworks, communication protocols), or algorithms (GPU Kernel), you can find a technical direction aligned with your field in the Open-RDMA open-source project.
Break the Black Box, Master the Network | Complete Open-Source RDMA Technology Stack from RTL to Drivers
🌟 Star Us · 📚 Learning Resources · 🚀 Quick Start · 🤝 Contributing
OpenRDMA is a GPU-friendly RDMA design: the GPU submits only a short instruction, the NIC builds the full request, and data is transferred through the Multipath Reliable Connection (MRC) protocol. The figure below compares how OpenRDMA and existing RDMA solutions submit communication requests:
- GPUDirect RDMA (GDR), orange path: The CPU builds and submits the request.
- InfiniBand GPUDirect Async (IBGDA), blue path: The GPU builds and submits the request.
- OpenRDMA, dark-blue path: The GPU sends only a minimal 8-byte Doorbell to select a preconfigured operation; the NIC parses it and builds the full request.
GDR, IBGDA, and OpenRDMA control paths for submitting communication requests
Multipath Reliable Connection (MRC) is an open network protocol jointly developed by OpenAI, AMD, Broadcom, Microsoft, and NVIDIA. It allows packets from the same QP to use multiple Ethernet paths and handles out-of-order delivery, loss recovery, congestion, and path failures. OpenRDMA draws on these mechanisms to provide reliable multipath transport.
|
|
From RTL hardware design to Linux user-space drivers, every line of code is transparent and public |
Inspired by the MRC protocol, Ethernet-based with hardware-software co-design for ultra-low latency |
|
FPGA-based, freely optimized for AI clusters, GPU communication, and other scenarios—no vendor lock-in |
From verification testing to AI-assisted development tools, accelerating AI infrastructure iteration through AI technology |
|
Researchers can quickly customize experimental platforms based on this project to test novel congestion control algorithms and communication protocols |
All materials publicly available with introductory guides, helping more newcomers become RDMA experts |
The OpenRDMA-flow M0–M11 architecture: the TX front end (M0–M3), the closed TX → RX → Retx loop, and the M11 → M3 TRAFFIC_RETRY reinjection path.
Click the figure for the interactive version: zoom, node focus, upstream/downstream path tracing, light/dark themes and export.
|
Hardware RTL code for the open-rdma project |
Driver code including user-space and kernel-space drivers |
Setting up an environment from hardware simulation to software driver debugging and application development is no easy task, so we provide an out-of-the-box development environment. |
|
Language Server for Bluespec SystemVerilog, providing a smooth development experience for BSV projects. |
An AI Agent-driven RTL verification framework launched by Beijing Institute of Open Source Chip, we added support for Cocotb test environment. |
PCIe behavioral and simulation models based on Cocotb, we added Altera RTile support. |
- A Practical Tutorial on PCIe for Total Beginners on Windows
- Although using Windows as an example, most content is platform-agnostic.
- PCIe Deep Dive (Chinese)
New to the project? Start with open-rdma-dev-env:
an out-of-the-box container that bundles the whole toolchain — the Bluespec compiler, the Rust
toolchain, rdma-core and the Open-RDMA sources — and boots a QEMU virtual machine for driver work.
Prerequisites: a Linux host with Docker and Docker Compose installed.
# 1. Clone the development environment
git clone https://github.com/open-rdma/open-rdma-dev-env.git
cd open-rdma-dev-env
# 2. One-click start: builds the image and boots the VM inside the container
./quick-start.sh
# 3. Log into the VM as root:root, then run the baseline send/recv test
cd /root/open-rdma/open-rdma-driver && ./tests/base_test/scripts/test_send_recv_sim.shCtrl+B, D detaches from the VM session and leaves it running in the background. To re-attach:
docker compose exec open-rdma tmux attach-session -t open-rdma📖 Detailed installation, manual startup and troubleshooting
SSH key setup, switching image registries, building the image locally, proxy configuration, the FAQ
and the image design notes all live in the
open-rdma-dev-env README.
We welcome all forms of contributions!
- 🐛 Report a Bug
- 💡 Request a Feature
- 📖 Improve documentation
- 🔧 Submit a Pull Request
See the Contributing Guide for the full workflow and commit conventions, and the 🎓 student onboarding path if you are new to RDMA and are wondering where to start.
- 💬 GitHub Discussions - Questions and discussions
This project is licensed under the Apache 2.0 License.