Skip to content

feat(ccu): generalize alltoall mesh ranks - #80

Draft
Kur0x wants to merge 4 commits into
codex/ccu-stack-08-alltoall-meshfrom
codex/ccu-stack-09-rank-generalization
Draft

feat(ccu): generalize alltoall mesh ranks#80
Kur0x wants to merge 4 commits into
codex/ccu-stack-08-alltoall-meshfrom
codex/ccu-stack-09-rank-generalization

Conversation

@Kur0x

@Kur0x Kur0x commented Jul 28, 2026

Copy link
Copy Markdown
Member

Summary

  • generalize mesh planning and program generation to runtime rank sizes 2 through 64
  • add grouped completion masks, dynamic instruction/resource sizing, and channel-strided remote XN handling
  • preserve two-rank long-mission timing isolation and update generalized source contracts

Stack

9 of 9. Depends on codex/ccu-stack-08-alltoall-mesh.

Validation

  • git diff --check origin/main...HEAD
  • 142 generalized AllToAll/backend/orchestrator/lower-layer tests passed; 13 skipped
  • 19 signal-wait, microcode, and memory-program tests passed
  • compiled and ran the TileXR source guard: TileXR source guard checks passed

Hardware boundary

No fresh NPU run was performed from this macOS host. #71 records prior successful 2-rank 2 MB and 4-rank 2 MB hardware runs; rank sizes above 4 remain unvalidated on the data plane.

@hechangcheng2303 hechangcheng2303 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

发现一个 rank 泛化后的高风险问题:mesh peer 的 completion CKE 分组校验和实际 program 生成使用的 peer 顺序不一致。

ValidateMeshSpec 里按调用方传入的 spec.peers 原始顺序检查:

peer.route.copyCompletionCke != spec.remoteCompletionCkes[ordinal / TILEXR_CCU_CKE_MASK_BITS]

TileXRCcuBuildAllToAllMeshProgram 后面会先复制并按 peerRank 排序:

auto peers = spec.peers;
std::sort(peers.begin(), peers.end(), ...);

然后 copy signal 使用排序后的 ordinal:

peer.route.copyCompletionCke,
static_cast<uint16_t>(1U << (ordinal % TILEXR_CCU_CKE_MASK_BITS))

超过 16 个 peer 时这会变成实质问题:如果调用方传入的 peers 不是按 peerRank 排序的,某些 peer 在校验阶段属于 CKE group N,但排序后实际发 signal 时可能落到另一个 group 的 bit 位;后续 grouped wait 仍按 remoteCompletionCkes[group] 等待,可能导致 wait 等不到对应完成信号,运行时卡住。

这个问题在 4-rank/8-rank 不明显,因为所有 peer 都在同一个 16-bit completion CKE 组里。现有 64-rank 测试也是按 peerRank 顺序构造 peers,所以排序前后 ordinal 没变,没覆盖到这个场景。

建议:要么在 ValidateMeshSpec 内也先按 peerRank 排序后再检查 completion CKE 分组,要么把 copyCompletionCke 的分配统一放到排序后的构建路径里,不依赖调用方原始 peer 顺序。建议补一个 18+ rank、peers 故意乱序的测试。

const uint32_t count = pfeId > TILEXR_CCU_HCOMM_MAX_INNER_FE_ID ?
TILEXR_CCU_HCOMM_OUTER_FE_JETTY_NUM :
TILEXR_CCU_HCOMM_INNER_FE_JETTY_NUM;
if (peerOrdinal >= count || start + peerOrdinal >= 128U) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

peerOrdinal is the global ordinal across all peers, but count is capacity within one PFE partition. This guarantees endpoint creation fails for rank sizes above 37 (and above 24 when the selected route is on an inner PFE), even though this PR accepts up to 64 ranks; it also wastes slots when peers are spread across PFEs. Please allocate an ordinal per PFE/partition instead of comparing the global peer ordinal to per-PFE capacity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants