add direct ccu alltoall - #71
Conversation
# Conflicts: # tests/comm/unit/test_tilexr_source_guards.cpp
Refactor CCU as internal collective backend
- Apply direct-CCU-only and resource window defaults for P2P CCU copy smoke runs. - Forward RA context registration mode and lower-layer WQE defaults to rank processes. - Accept skipped non-receiver ranks while requiring at least one passing receiver result.
Initialize the CCU backend behind an explicit environment gate and keep comm initialization degradable when direct CCU is unavailable. Track direct CCU runtime init failures per device, add RA ctx EID selection for resource-window registration, and make P2P smoke submit active-rank only. Reclaim allocator resources on LIFO receipt release and cover the lifecycle, runner, allocator, and smoke probe paths with focused tests.
Add direct CCU signal/wait program generation and wire it through the internal backend, smoke probe, and runner paths. Harden direct CCU runtime/session lifecycle, lower-layer ownership tracking, driver mission status helpers, and endpoint route handling for repeated launches. Align P2P CCU copy with hcomm address semantics by using original device VAs in microcode while routing lower-layer channels through imported segment VAs, with regression coverage and documentation.
Add the source CKE prelude before SyncCke in direct CCU signal/wait programs so wait-only peers observe the posted CKE instead of timing out. Update signal/wait program coverage to assert the hcomm-style source CKE initialization sequence for signal-only and signal-and-wait roles.
Add a TileXR-owned direct CCU alltoall MVP for two-rank validation without introducing HCCL or hcomm private APIs. Implement the alltoall program builder with 4KB memory slices, 32KB blocks, 64-block 2MB transfers, rank-aware copy phases, and validation for invalid slice or rank inputs. Wire the direct alltoall install path through the CCU planner and orchestrator, including route-scoped remote memory overrides so the data copy route targets the peer buffer while sync routes keep using the CCU resource window. Harden lower-layer planning for multi-route endpoint snapshots by expanding the PFE jetty window and preserving distinct channel, CKE, and jetty ownership for pre-sync, data, and post-sync resources. Add a host-phased alltoall smoke mode that reuses the proven P2P CCU RemoteToLocal copy path: both ranks prepare each phase, only the active rank submits the CCU copy mission, and host ready/done files serialize phase0 and phase1 with bounded stream synchronization. Cover the new behavior with alltoall program tests, orchestrator and planner source checks, backend boundary tests, lower-layer plan regressions, and smoke-runner assertions for timeout-safe execution. Verified on the remote CCU environment with unit tests, tile-comm rebuild, hcomm/HCCL dependency guard, and a device 0/1 2MB alltoall smoke run passing on both ranks with zero mismatches.
Add TileXR-owned Direct CCU alltoall refinements for the 2-rank P2P smoke path, including HCCL-style PreSync/PostSync masks, planner/orchestrator prepare plumbing, bounded stream synchronization, SyncXn ping smoke coverage, and phase-scoped submit ready/done gates for multi-phase host submission. Document the code changes, server environment requirements, and validated 141.62.24.62 smoke results in docs/CCU_DIRECT_ALLTOALL_CHANGE_RECORD.md. Verification: git diff --check; python -m unittest tests.ccu.test_tilexr_ccu_alltoall_program tests.ccu.test_tilexr_ccu_direct_orchestrator tests.ccu.test_tilexr_ccu_direct_smoke_probe tests.ccu.test_tilexr_ccu_lower_layer_plan_builder.
# Conflicts: # src/comm/tilexr_comm.cpp # src/comm/tilexr_comm.h # tests/comm/unit/test_tilexr_source_guards.cpp
**Summary** Generalize direct CCU AllToAll mesh planning and microcode generation from a fixed 4-rank shape to runtime rank sizes from 2 through 64, including grouped completion CKE masks and dynamic instruction/resource sizing. Align peer route exchange, channel-strided remote XN allocation, shared verified jetty use, and HCCL topology transport selection for multi-rank execution while preserving the 2-rank long-mission path. Update the hardware smoke runner and CCU source checks for runtime rank counts, payload-derived instruction capacity, RA retry handling, and generalized mesh contracts. **Validation** git diff --check Built tile-comm on 141.61.50.31 with /home/pkg/b110/cann-9.1.0 and CMake 3.31.12. Ran 2-rank 2 MB long-mission CCU AllToAll on devices 6,7: both ranks passed with zero mismatches. Ran 4-rank 2 MB mesh CCU AllToAll on devices 0,1,2,3: all four ranks passed with zero mismatches and stable resources.
c412025 to
01d358d
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ce9951b5ca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "-I", | ||
| str(COMM_DIR), | ||
| str(PROBE_SOURCE), | ||
| str(CCU_DIRECT_RUNTIME_SOURCE), |
There was a problem hiding this comment.
When test_probe_compiles_against_tilexr_owned_ccu_ra_chain runs with a local C++ compiler, linking fails because tilexr_ccu_direct_runtime.cpp now calls TileXRCcuResolvePeerEidRoutes, but this standalone source list omits tilexr_ccu_topology.cpp. The focused CCU suite therefore fails with an undefined reference before the basic-info probe can run.
AGENTS.md reference: AGENTS.md:L37-L37
Useful? React with 👍 / 👎.
| if (finalRet == 0) { | ||
| const int checkRet = CheckAllToAllState(&alltoall); |
There was a problem hiding this comment.
When TILEXR_CCU_DIRECT_SMOKE_ALLTOALL=1 is used with the runner's default TILEXR_CCU_DIRECT_SMOKE_SUBMIT=0, RunAllToAllCopyPhase only prepares the missions and never modifies the receive buffer, but this unconditional check compares that sentinel buffer with peer data and returns error 14. Consequently the advertised prepare-only all-to-all smoke path always fails; validate device data only after submission, or require submission before entering this mode.
Useful? React with 👍 / 👎.
| auto peers = spec.peers; | ||
| std::sort(peers.begin(), peers.end(), [](const TileXRCcuAllToAllMeshPeerSpec& lhs, | ||
| const TileXRCcuAllToAllMeshPeerSpec& rhs) { | ||
| return lhs.peerRank < rhs.peerRank; | ||
| }); |
There was a problem hiding this comment.
When a mesh spec with more than 16 peers supplies peers in a non-rank order, validation associates each route's copyCompletionCke with its original ordinal, but this sort changes the ordinal used to generate completion masks. A peer moved across a 16-peer boundary then posts its completion bit to the old group's CKE while the generated wait checks the new group's CKE, so an otherwise accepted program can wait indefinitely. Either require sorted input before validating group assignments or reassign completion CKEs after this sort.
Useful? React with 👍 / 👎.
|
此 PR 已由可逐层审查的 Draft PR 栈替代：#72 core microcode planning -> #73 HCCP driver adapters -> #74 install planning -> #75 runtime engine -> #76 collective backend -> #77 signal wait -> #78 two-rank AllToAll -> #79 four-rank mesh -> #80 rank generalization -> #81 CI cleanup。请按 #72 到 #81 的顺序审查和合并；#80 的 Host Checks 已覆盖前序累计代码。 |
2 participants
Summary
Align peer route exchange, channel-strided remote XN allocation, shared verified jetty use, and HCCL topology transport selection for multi-rank execution while preserving the 2-rank long-mission path.
Update the hardware smoke runner and CCU source checks for runtime rank counts, payload-derived instruction capacity, RA retry handling, and generalized mesh contracts.
Validation
git diff --check -- src/comm/ccu tests/ccuexport TILEXR_RUN_CCU_DIRECT_SMOKE_PROBE=1 \ export TILEXR_CCU_SMOKE_DEVICES=0,1,2,3 \ export TILEXR_CCU_RANK_SIZE=4 \ export TILEXR_CCU_DIRECT_SMOKE_ALLTOALL=1 \ export TILEXR_CCU_DIRECT_SMOKE_ALLTOALL_MESH=1 \ export TILEXR_CCU_DIRECT_SMOKE_SUBMIT=1 \ export TILEXR_CCU_ALLTOALL_BYTES=2097152 \ export TILEXR_CCU_ALLTOALL_MEM_SLICE_PER_LOOP=8 \ export TILEXR_CCU_ALLTOALL_LOOP_COUNT=1 \ export TILEXR_CCU_SMOKE_TIMEOUT=240 \ export TILEXR_CCU_DIRECT_SMOKE_READY_TIMEOUT_MS=30000 \ export TILEXR_CCU_DIRECT_SUBMIT_TIMEOUT=12000bash tests/ccu/run_tilexr_ccu_direct_smoke.sh