Hi! I used your native Windows port as the foundation of a TP2 experiment and wanted to (a) report that it carried a much heavier engine than the single-GPU one cleanly, and (b) share the result that came out of it, since both may be useful to you.
What I built on your port. Your compatibility layer was cherry-picked into the TP2 fork (wamansou/ninfer-tp2-1m, itself a fork of Neroued/ninfer) and became the windows-tp2 branch here: https://github.com/ivanov84/ninfer-windows-tp2. To be explicit about attribution: the Windows build system (CMake/vcpkg), the CreateFileW/MapViewOfFile artifact reader, the load-progress/console-log ports, the winsock media acquire, and the MSVC-safe TMA descriptors and plan-move bodies are your work тАФ the per-commit mapping (8e7f596, 965c67f, 0785bd3, 9bc46ce, 48035f7+d7cc655+b408783) is documented in docs/PROVENANCE.md in the fork, and the NOTICE carries the Apache-2.0 attribution. What I added on top is a communication transport, described below.
Why your port mattered. The TP2 engine issues ~128 cross-device allreduces per decode round. On the test machine (2x RTX 5060 Ti 16 GB, WDDM, cudaDeviceCanAccessPeer == 0, GPU0 CPU-slot / GPU1 chipset Gen3 x4) the staged fallback cost ~277 us per 10 KiB reduction тАФ ~58% of the MTP0 token, 16.37 tok/s. The fixed cost was the staging choreography (4 driver-staged copies + a cross-device event chain per collective), not the ~3.3 us of useful payload time.
The addition: a pinned-host peer mailbox. Both devices exchange partial sums directly through a pinned write-back host slab via small kernels (publish -> fence -> release flag -> poll peer flag -> read peer partial -> local sum), captured inside the existing cross-device CUDA graph; the staged path is retained as the fallback for eager execution and oversized payloads, and NINFER_TP2_MAILBOX=0 restores the old behavior.
Result (single request, greedy, thinking off, engine-committed decode tok/s, Qwen3.8-27B NVFP4, CUDA graphs on):
| Config |
staged |
mailbox |
| MTP0, 512 tok |
16.37 |
35.77 |
| MTP3, 512 tok |
32.58 |
66.8 |
| MTP4, 512 tok |
- |
68.5-68.9 |
| MTP4, 2048 tok |
- |
76.65 (workload-dependent) |
| Correctness: exact BF16 sum equality vs the staged path, bit-identical repeated runs across 512/1024/2048-token generations, zero graph failures, zero fallback events in the final long run. One of your port details directly survived this: the CUDA 12.8 ptxas shared-memory overflow on the GQA decode kernel was hit here too and dodged the same way (CUDA 13.1). |
|
|
| So: your port runs the two-device engine (cross-device CUDA graph capture and all) natively on Windows 11 without WSL2, and on this machine it is what made the whole experiment possible. Thank you for publishing it. |
|
|
| I did not PR any of this to your repo (our changes live on the TP2 fork, and the port layer is yours) тАФ but if anything in the mailbox transport or the benchmark data is interesting for your fork, happy to coordinate. |
|
|
| Details: docs/windows-peer-mailbox.md (transport writeup) and benchmarks/windows-tp2-benchmarks.md (methodology + numbers) in the fork above. |
|
|
This issue was posted by an AI agent (OpenHands) on behalf of Dmitriy Ivanov.
Hi! I used your native Windows port as the foundation of a TP2 experiment and wanted to (a) report that it carried a much heavier engine than the single-GPU one cleanly, and (b) share the result that came out of it, since both may be useful to you.
What I built on your port. Your compatibility layer was cherry-picked into the TP2 fork (wamansou/ninfer-tp2-1m, itself a fork of Neroued/ninfer) and became the
windows-tp2branch here: https://github.com/ivanov84/ninfer-windows-tp2. To be explicit about attribution: the Windows build system (CMake/vcpkg), theCreateFileW/MapViewOfFileartifact reader, the load-progress/console-log ports, the winsock media acquire, and the MSVC-safe TMA descriptors and plan-move bodies are your work тАФ the per-commit mapping (8e7f596, 965c67f, 0785bd3, 9bc46ce, 48035f7+d7cc655+b408783) is documented in docs/PROVENANCE.md in the fork, and the NOTICE carries the Apache-2.0 attribution. What I added on top is a communication transport, described below.Why your port mattered. The TP2 engine issues ~128 cross-device allreduces per decode round. On the test machine (2x RTX 5060 Ti 16 GB, WDDM,
cudaDeviceCanAccessPeer == 0, GPU0 CPU-slot / GPU1 chipset Gen3 x4) the staged fallback cost ~277 us per 10 KiB reduction тАФ ~58% of the MTP0 token, 16.37 tok/s. The fixed cost was the staging choreography (4 driver-staged copies + a cross-device event chain per collective), not the ~3.3 us of useful payload time.The addition: a pinned-host peer mailbox. Both devices exchange partial sums directly through a pinned write-back host slab via small kernels (publish -> fence -> release flag -> poll peer flag -> read peer partial -> local sum), captured inside the existing cross-device CUDA graph; the staged path is retained as the fallback for eager execution and oversized payloads, and
NINFER_TP2_MAILBOX=0restores the old behavior.Result (single request, greedy, thinking off, engine-committed decode tok/s, Qwen3.8-27B NVFP4, CUDA graphs on):
This issue was posted by an AI agent (OpenHands) on behalf of Dmitriy Ivanov.