feat(ccu): generalize alltoall mesh ranks - #80
Conversation
hechangcheng2303
left a comment
There was a problem hiding this comment.
发现一个 rank 泛化后的高风险问题:mesh peer 的 completion CKE 分组校验和实际 program 生成使用的 peer 顺序不一致。
ValidateMeshSpec 里按调用方传入的 spec.peers 原始顺序检查:
peer.route.copyCompletionCke != spec.remoteCompletionCkes[ordinal / TILEXR_CCU_CKE_MASK_BITS]但 TileXRCcuBuildAllToAllMeshProgram 后面会先复制并按 peerRank 排序:
auto peers = spec.peers;
std::sort(peers.begin(), peers.end(), ...);然后 copy signal 使用排序后的 ordinal:
peer.route.copyCompletionCke,
static_cast<uint16_t>(1U << (ordinal % TILEXR_CCU_CKE_MASK_BITS))超过 16 个 peer 时这会变成实质问题:如果调用方传入的 peers 不是按 peerRank 排序的,某些 peer 在校验阶段属于 CKE group N,但排序后实际发 signal 时可能落到另一个 group 的 bit 位;后续 grouped wait 仍按 remoteCompletionCkes[group] 等待,可能导致 wait 等不到对应完成信号,运行时卡住。
这个问题在 4-rank/8-rank 不明显,因为所有 peer 都在同一个 16-bit completion CKE 组里。现有 64-rank 测试也是按 peerRank 顺序构造 peers,所以排序前后 ordinal 没变,没覆盖到这个场景。
建议:要么在 ValidateMeshSpec 内也先按 peerRank 排序后再检查 completion CKE 分组,要么把 copyCompletionCke 的分配统一放到排序后的构建路径里,不依赖调用方原始 peer 顺序。建议补一个 18+ rank、peers 故意乱序的测试。
| const uint32_t count = pfeId > TILEXR_CCU_HCOMM_MAX_INNER_FE_ID ? | ||
| TILEXR_CCU_HCOMM_OUTER_FE_JETTY_NUM : | ||
| TILEXR_CCU_HCOMM_INNER_FE_JETTY_NUM; | ||
| if (peerOrdinal >= count || start + peerOrdinal >= 128U) { |
There was a problem hiding this comment.
peerOrdinal is the global ordinal across all peers, but count is capacity within one PFE partition. This guarantees endpoint creation fails for rank sizes above 37 (and above 24 when the selected route is on an inner PFE), even though this PR accepts up to 64 ranks; it also wastes slots when peers are spread across PFEs. Please allocate an ordinal per PFE/partition instead of comparing the global peer ordinal to per-PFE capacity.
Summary
Stack
9 of 9. Depends on
codex/ccu-stack-08-alltoall-mesh.Validation
git diff --check origin/main...HEADTileXR source guard checks passedHardware boundary
No fresh NPU run was performed from this macOS host. #71 records prior successful 2-rank 2 MB and 4-rank 2 MB hardware runs; rank sizes above 4 remain unvalidated on the data plane.