diff --git a/docs/moonep/COMBINE_V2_MULTIHOST_PERF.md b/docs/moonep/COMBINE_V2_MULTIHOST_PERF.md new file mode 100644 index 0000000..03d5369 --- /dev/null +++ b/docs/moonep/COMBINE_V2_MULTIHOST_PERF.md @@ -0,0 +1,574 @@ +# MoonEP Combine V2 多机通信性能测试指导 + +本文说明如何在多台 Ascend 服务器上构建、同步和运行 MoonEP Combine V2 +单算子通信性能测试。流程不依赖 MPI,由主服务器按 hostfile 顺序通过 SSH +并发拉起所有 rank。 + +## 1. 当前测试范围 + +当前 benchmark 固定以下算子参数: + +| 参数 | 当前值 | 是否可免编译修改 | +| --- | --- | --- | +| `BS` | 默认 `128` | 是,使用 `--bs` 或 `--bs-list` | +| `K` | `16` | 否,当前写在 benchmark C++ 中 | +| `H` | `3584` | 否,当前写在 benchmark C++ 中 | +| 专家总数 | 默认 `64` | 是,使用 `--experts` 或 `-Experts` | +| dtype | `BF16` | 否,当前写在 benchmark C++ 中 | +| 每台服务器 rank 数 | `1-8` | 是,由 hostfile 配置 | + +`--bs-list` 会在一次进程任务中覆盖多个 BS 点位。TileXR/UDMA 只初始化一次, +workspace 按列表中的最大 BS 申请并注册一次,所有点位复用该 workspace。 + +### 1.1 性能结果展示约束 + +正式性能报告展示以下两组结果: + +- `avg_ms`:先对每个 rank 除去 warmup 后的全部计时 iteration 求平均,得到每卡 + 耗时,再对所有卡的耗时求算术平均。 +- `max_ms`:所有卡耗时中的最大值,即最慢卡在全部计时 iteration 上的平均耗时。 +- `avg_alg_bw_GBps`:完整逻辑通信数据量除以 `avg_ms`。 +- `max_alg_bw_GBps`:完整逻辑通信数据量除以 `max_ms`。 + +不统计或展示单轮样本的 `p50/min/max`。这里的 `max_ms` 是每卡平均耗时中的最大值, +不是 80 轮单次耗时的最大值。 + +BF16 Combine V2 的单 rank 逻辑通信数据量和算法带宽统一按以下公式计算: + +```text +data_bytes = BS * K * H * 2 +avg_alg_bw_GBps = data_bytes / avg_ms / 1e6 +max_alg_bw_GBps = data_bytes / max_ms / 1e6 +``` + +同 rank、同服务器跨卡和跨服务器的数据均计入 `data_bytes`,不再分别统计卡内、 +跨卡或跨机比例。算法带宽使用单 rank 完整逻辑数据量,不额外乘 world size。 +`GB/s` 使用十进制单位 `1 GB = 10^9 bytes`。 + +## 2. 脚本职责 + +| 文件 | 执行位置 | 职责 | +| --- | --- | --- | +| `tools/moonep/run_combine_v2_perf_cluster.sh` | 主服务器 | 完整入口:远端编译、平铺同步、启动测试 | +| `tools/moonep/build_combine_v2_perf.sh` | 主服务器 | 编译 benchmark,整理 `bin/` 和 `lib64/`,检查 RPATH 和 MPI 依赖 | +| `tools/moonep/sync_combine_v2_perf_runtime.sh` | 主服务器 | 主服务器直接 rsync 到 hostfile 中的每台服务器并校验 SHA-256 | +| `tools/moonep/run_combine_v2_perf_multihost.sh` | 主服务器 | NPU 预检、rank 映射、SSH 拉起、日志校验和性能聚合 | +| `tests/moonep_combine_v2/demo/tilexr_moonep_combine_v2_hardware_probe.cpp` | 编译产物 | 单 rank benchmark、TCP barrier、正确性检查和 ACL Event 计时 | + +同步是平铺模式:主服务器直接同步每个目标节点,不允许计算节点继续向其他 +节点分层转发。 + +## 3. 远端目录 + +建议在 `/home/h00580772` 下为任务建立独立根目录: + +```text +/home/h00580772/tilexr_combine_v2/ +|-- source/ # Mutagen 只同步到主服务器 +|-- build/ # 只在主服务器编译 +|-- install/ # 主服务器向全部测试节点平铺同步 +|-- logs/ # 控制日志、逐 rank 日志、NPU 快照 +`-- hostfile +``` + +本文中的已验证 16P 环境使用: + +```text +/home/h00580772/tilexr_combine_v2_16p_cdf2b01_20260811 +``` + +## 4. 环境前提 + +开始前确认: + +1. 本地已安装 `mutagen`、`scp` 和 `ssh`。 +2. 本地能够免密 SSH 登录主服务器,并可用 `scp` 上传临时 bash 入口到测试目录。 +3. 主服务器能够免密 SSH 登录 hostfile 中的每一台服务器。 +4. 所有服务器存在兼容的 CANN 和驱动;CANN 路径按第 4.2 节的环境规则选择, + 不要求所有环境统一使用 B150。 +5. 主服务器和计算节点安装了 `bash`、`rsync`、`sha256sum`、`timeout`、 + `ss` 和 `npu-smi`。 +6. 主服务器使用 Bash 4.3 或更高版本,以支持 `wait -n`。 + +计算节点不需要安装 MPI。 + +### 4.1 环境信息来源与名称解析 + +测试节点必须从以下两份本地环境清单选择,二者是互相独立的环境: + +| 环境 | 权威信息来源 | 节点选择规则 | +| --- | --- | --- | +| 512P 环境 | `D:\3_codex\512P环境信息.txt` | 用户指定“`xx柜`”或“`xx柜 CPUy`”时,按该文件中的柜号和 CPU 编号解析 IP | +| A10-64P 环境 | `D:\3_codex\A10-64P环境.txt` | 文件按行给出独立服务器 IP;该环境没有“柜号/CPU 编号”语义 | + +例如,`0号柜 CPU1` 必须从 512P 环境解析为 `141.61.53.150`。A10-64P +环境中的 `141.61.49.226`、`141.61.49.223` 等服务器不属于任何 `xx柜`。 +不得根据 A10-64P 文件的行号或排列顺序推导柜号、CPU 编号,也不得把两个环境的 +节点混入同一个 hostfile,除非用户明确要求跨环境测试。 + +柜号不保证连续。新增或调整测试规模时应重新读取对应源文件,不在脚本或文档中 +凭历史结果猜测 IP。 + +### 4.2 CANN 版本选择约束 + +CANN 版本约束按测试环境生效,不是全局约束: + +| 测试环境 | 编译与运行 CANN 约束 | +| --- | --- | +| 512P 环境 9 号柜、15 号柜 | 必须使用 `/home/pkg/910_B150/cann-9.1.0` 编译,并在 launcher 中通过 `--cann-path` 使用同一路径运行 | +| A10-64P 环境,包括 `141.61.49.223` | 不要求使用 B150;使用目标服务器实际安装且与编译产物兼容的 CANN,例如 223 当前可使用 `/home/pkg/b131/cann-9.1.0` | +| 其他 512P 柜号 | 不自动套用 9/15 号柜的 B150 约束;测试前检查目标柜的实际安装路径,并显式传入 `-CannPath` 或 `--cann-path` | + +不得因为某次任务要求 9/15 号柜使用 B150,就把该要求扩展到 +`141.61.49.223`、`141.61.49.226` 等 A10-64P 节点。反过来,也不得在 9/15 +号柜沿用 A10-64P 的 b131/b061 路径。主服务器编译时使用的 CANN 必须与 +hostfile 中运行节点可用的运行时和 ABI 匹配;若不匹配,应重新编译并重新平铺 +同步产物,不能直接复用旧 `install/`。 + +## 5. Hostfile 与 rank 映射 + +hostfile 每行格式为 `IP:slots`,空行和以 `#` 开头的行会被忽略: + +```text +141.61.49.226:8 +141.61.49.223:8 +``` + +规则如下: + +- 第一台服务器承担 rank 0、bootstrap、编译和任务控制。 +- hostfile 顺序决定 global rank 顺序,不根据 hostname 重新排序。 +- `global_rank = 前面节点的 slots 总和 + local_rank`。 +- `device = local_rank`。 +- 每个 host 只能出现一次,单机 slots 必须为 `1-8`。 +- world size 必须为 `2-8`、`16`、`32`、`64` 或 `128`;这是 Combine V2 + 调度支持的 rank 集合。 +- 专家总数必须能被 world size 整除。`ExpNum=256, world size=128` 表示每个 + rank 对应 2 个专家。 +- 每个 BS 必须能被 world size 整除。 + +每台服务器使用 8 个 rank 时,当前直接支持: + +| 节点数 | world size | 是否支持 | +| --- | --- | --- | +| 1 | 8P | 是 | +| 2 | 16P | 是 | +| 3 | 24P | 否,24 不能整除 64 | +| 4 | 32P | 是 | +| 8 | 64P | 是 | +| 16 | 128P | 是 | + +扩容时只需要在 hostfile 中按期望 rank 顺序列出对应服务器。64P 使用 8 台, +128P 使用 16 台;每台均为 8 个 rank: + +```text +PRIMARY_IP:8 +WORKER_1_IP:8 +WORKER_2_IP:8 +WORKER_3_IP:8 +WORKER_4_IP:8 +WORKER_5_IP:8 +WORKER_6_IP:8 +WORKER_7_IP:8 +``` + +## 6. 创建 Mutagen 会话 + +每个主服务器和远端根目录使用独立会话,不要重定向已有的无关会话: + +```bash +mutagen sync create \ + --name tilexr-combine-v2-64p \ + --mode one-way-replica \ + --ignore-vcs \ + --ignore build \ + --ignore "build_*" \ + --ignore install \ + --compression zstandard \ + "D:\3_codex\TileXR-PR-DEBUG\TileXR" \ + "root@PRIMARY_IP:/home/h00580772/tilexr_combine_v2/source" +``` + +运行前由本地手动 flush 源码会话: + +```bash +mutagen sync flush tilexr-combine-v2-64p +``` + +源码只通过 Mutagen 同步到主服务器,不同步到计算节点。测试入口不再封装 +Mutagen;需要先确认会话目标是 `@:/source`。 + +## 7. 完整流程 + +### 7.1 16P 单 BS + +先在本地仓库根目录将服务器端入口上传到主服务器测试目录,再通过 SSH 在主服务器上执行: + +```bash +scp tools/moonep/run_combine_v2_perf_cluster.sh \ + root@141.61.49.226:/home/h00580772/tilexr_combine_v2_16p_cdf2b01_20260811/run_combine_v2_perf_cluster.sh + +ssh root@141.61.49.226 \ + "bash /home/h00580772/tilexr_combine_v2_16p_cdf2b01_20260811/run_combine_v2_perf_cluster.sh \ + --remote-root /home/h00580772/tilexr_combine_v2_16p_cdf2b01_20260811 \ + --bs 128" +``` + +该命令依次执行: + +1. 在主服务器配置和增量编译 benchmark。 +2. 检查可执行文件没有 MPI 依赖和 build 目录 RPATH。 +3. 主服务器将 `install/` 直接同步到全部节点。 +4. 每台服务器校验相同的 SHA-256 manifest。 +5. 对全部服务器执行 NPU 占用预检。 +6. 主服务器通过 SSH 并发拉起所有 rank。 +7. 收集逐 rank 结果并输出全局聚合性能。 + +### 7.2 多 BS 批量测试 + +```bash +scp tools/moonep/run_combine_v2_perf_cluster.sh \ + root@PRIMARY_IP:/home/h00580772/tilexr_combine_v2/run_combine_v2_perf_cluster.sh + +ssh root@PRIMARY_IP \ + "bash /home/h00580772/tilexr_combine_v2/run_combine_v2_perf_cluster.sh \ + --remote-root /home/h00580772/tilexr_combine_v2 \ + --bs-list 128,256,512,1024,8192 \ + --experts 64 \ + --warmup 20 \ + --iterations 80" +``` + +`--bs` 与 `--bs-list` 互斥。没有指定时默认测试 `BS=128`。 + +当 `PRIMARY_IP` 和 hostfile 属于 9 号柜或 15 号柜时,上述命令必须额外传入: + +```bash +--cann-path /home/pkg/910_B150/cann-9.1.0 +``` + +在 `141.61.49.223` 等 A10-64P 节点上不要机械添加该参数,应按第 4.2 节使用 +该节点实际可用且与产物匹配的 CANN。 + +### 7.3 常用 Bash 参数 + +| 参数 | 默认值 | 说明 | +| --- | --- | --- | +| `--remote-root` | 必填 | 远端任务根目录 | +| `--hostfile` | `/hostfile` | 远端 hostfile 绝对路径 | +| `--source-dir` | `/source` | 远端源码目录 | +| `--build-dir` | `/build` | 远端构建目录 | +| `--install-dir` | `/install` | 远端运行产物目录 | +| `--cann-path` | 当前 `ASCEND_HOME_PATH` 或 toolkit latest | CANN 根目录;9/15 号柜必须显式改为 `/home/pkg/910_B150/cann-9.1.0`,223 不强制 B150 | +| `--bs` | `128` | 单个 BS 点位 | +| `--bs-list` | 空 | 逗号分隔的多个 BS 点位 | +| `--warmup` | `20` | 每个 BS 的预热次数 | +| `--iterations` | `80` | 每个 BS 的计时次数,也称 loop 数 | +| `--experts` | `64` | 专家总数,必须能被 world size 整除 | +| `--comm-domain` | `141` | Shared-QP 通信域 | +| `--comm-port` | `10067` | rank 0 bootstrap 端口 | +| `--wait-seconds` | `120` | NPU 最大等待时间 | +| `--retry-seconds` | `15` | NPU 占用重试间隔 | +| `--rank-timeout` | `600` | 每个 rank 的任务超时 | +| `--build-jobs` | `nproc` | 编译并发度 | +| `--log-file` | 自动生成 | 主服务器控制日志绝对路径 | + +## 8. 免编译快速测试 + +如果只修改 BS、warmup、iterations 或通信端口,并且 `install/` 已经同步到 +hostfile 中的全部服务器,可以跳过 Mutagen、编译和产物同步,直接在主服务器 +执行 launcher: + +```bash +bash /home/h00580772/tilexr_combine_v2_16p_cdf2b01_20260811/source/tools/moonep/run_combine_v2_perf_multihost.sh \ + --hostfile /home/h00580772/tilexr_combine_v2_16p_cdf2b01_20260811/hostfile \ + --install-dir /home/h00580772/tilexr_combine_v2_16p_cdf2b01_20260811/install \ + --cann-path /home/pkg/b061/cann-9.1.T560 \ + --ssh-user root \ + --bs-list 128,256,512,8192 \ + --experts 64 \ + --warmup 20 \ + --iterations 80 \ + --comm-id 141.61.49.226:10067 \ + --wait-seconds 120 \ + --retry-seconds 15 \ + --timeout 600 +``` + +服务器端完整入口默认会执行一次增量 build;源码未变化时通常很快。若要严格跳过 +编译,可以向完整入口传入 `--skip-build`,或直接使用上述服务器端 launcher。 + +上述免编译示例针对 A10-64P 的 226/223 环境,因此使用 b061。若目标是 9/15 +号柜,`--cann-path` 必须改为 `/home/pkg/910_B150/cann-9.1.0`,并确认现有 +`install/` 本身也是用该 B150 路径编译;否则不能跳过编译和产物同步。 + +以下变化不能直接复用旧产物: + +- 修改 C++ benchmark、算子 Host、Kernel、CMake 或头文件。 +- 修改当前硬编码的 `H/K/dtype`。 +- 更换不兼容的 CANN、驱动、SoC 或 ABI。 + +## 9. 手工分步执行 + +需要排查某个阶段时,可以按以下顺序执行。 + +### 9.1 本地同步源码 + +```bash +mutagen sync flush tilexr-combine-v2-16p-cdf2b01-226 +``` + +源码只通过 Mutagen 同步到主服务器,不同步到计算节点。 + +### 9.2 主服务器编译 + +```bash +bash ${REMOTE_ROOT}/source/tools/moonep/build_combine_v2_perf.sh \ + --source-dir ${REMOTE_ROOT}/source \ + --build-dir ${REMOTE_ROOT}/build \ + --install-dir ${REMOTE_ROOT}/install \ + --cann-path /home/pkg/b061/cann-9.1.T560 \ + --jobs 16 +``` + +上例的 b061 路径适用于已有兼容环境,不是全局强制值。在 9/15 号柜执行时,必须 +将编译参数改为: + +```bash +--cann-path /home/pkg/910_B150/cann-9.1.0 +``` + +在 `141.61.49.223` 执行时不要求 B150,应传入该服务器实际可用且与运行产物 +匹配的路径,例如: + +```bash +--cann-path /home/pkg/b131/cann-9.1.0 +``` + +构建成功后,运行产物位于: + +```text +${REMOTE_ROOT}/install/bin/tilexr_moonep_combine_v2_perf +${REMOTE_ROOT}/install/lib64/ +``` + +### 9.3 主服务器平铺同步 + +```bash +bash ${REMOTE_ROOT}/source/tools/moonep/sync_combine_v2_perf_runtime.sh \ + --hostfile ${REMOTE_ROOT}/hostfile \ + --install-dir ${REMOTE_ROOT}/install \ + --ssh-user root +``` + +成功标志是 hostfile 中每台服务器均输出: + +```text +: SHA256 verified +``` + +### 9.4 主服务器启动测试 + +使用第 8 节的 launcher 命令。launcher 不调用 `mpirun`,也不检查任何 MPI +安装目录。 + +## 10. 新增服务器或更换主服务器 + +### 10.1 新增计算节点 + +1. 在主服务器的 hostfile 中按期望 rank 顺序增加 `IP:slots`。 +2. 配置主服务器到新节点的免密 SSH。 +3. 确认新节点的 CANN、驱动和工具路径满足第 4 节要求。 +4. 重新运行平铺同步或服务器端完整入口。 +5. launcher 会自动把新节点加入 NPU 预检和 rank 映射。 + +只增加节点且代码不变时不要求重新编译,但必须把当前 `install/` 同步到新节点。 + +### 10.2 更换主服务器 + +1. 创建指向新主服务器 `/source` 的新 Mutagen 会话。 +2. 将 `-PrimaryHost` 改为 hostfile 第一项。 +3. 确认新主服务器存在 hostfile,并能免密 SSH 到全部节点。 +4. flush 新的 Mutagen 会话,并通过 `scp` 上传入口后用 `ssh` 运行服务器端完整入口。 + +## 11. NPU 占用规则 + +正式启动前,每轮都会对 hostfile 中所有服务器执行 `npu-smi info`,原始输出 +保存到: + +```text +.npu_preflight/attempt_NN/.log +``` + +判定规则: + +- 没有 NPU 进程:允许启动。 +- 只有进程名以 `tilexr_` 开头:允许多任务共用 NPU。 +- Python 或其他通信测试进程:阻塞启动。 +- 阻塞后默认每 15 秒重新检查。 +- 120 秒仍未空闲:退出码为 `75`,不启动任何 rank。 +- SSH、`npu-smi` 或进程表解析失败:立即停止。 +- 不终止任何无关任务。 + +每个 rank 使用本任务专属 PID 文件和 `job_id`。中断、失败或超时时只清理本次 +任务,不使用全局 `pkill`。 + +## 12. 通信端口 + +默认 bootstrap 地址为: + +```text +TILEXR_COMM_ID=:10067 +``` + +TCP barrier 默认使用 bootstrap 端口加 97,即默认 `10164`。启动前会检查两个 +端口是否已监听;任一端口被占用都会停止任务。并发运行多个 TileXR 测试时, +必须为每个任务设置不同的 `-CommPort` 或 `--comm-id`。 + +## 13. 日志与结果口径 + +默认日志位于主服务器 `/logs/`。每次测试包含: + +```text + # 控制日志和全部 rank 输出 +.ranks/rank_0000.log # rank 0 原始日志 +.ranks/rank_0001.log # rank 1 原始日志 +.ranks/rank_averages.tsv # 每个 rank 的计时样本平均值 +.npu_preflight/ # 每轮 NPU 原始快照 +``` + +rank 映射示例: + +```text +RANK_MAP rank=8 host=141.61.49.223 local_rank=0 device=0 +``` + +单 rank 样本: + +```text +COMBINE_V2_SAMPLE bs=128 iteration=0 rank=8 elapsed_ms=0.412345 +``` + +脚本最终聚合结果只包含正式指标: + +```text +COMBINE_V2_PERF bs=128 k=16 h=3584 experts=64 dtype=bf16 ranks=16 iterations=80 avg_ms=0.410000 avg_alg_bw_GBps=35.805034 max_ms=0.450000 max_alg_bw_GBps=32.622364 correctness=passed +``` + +聚合方法: + +1. 每个 rank 使用 ACL Event 测量单次算子时间,warmup 不进入样本。 +2. 每个 rank 对自己的全部计时 iteration 求平均,得到本卡耗时。 +3. 对全部卡的耗时求算术平均得到 `avg_ms`,取最大值得到 `max_ms`。 +4. 用 `BS * K * H * dtype_bytes` 分别除以 `avg_ms` 和 `max_ms` 计算两项等效 + 算法带宽,所有路由数据统一计为通信数据量。 +5. 每个 BS 必须具备 `world size` 条 `COMBINE_V2_RANK_PERF` 通过记录,并且 + 每个 rank 必须具备完整的计时样本,否则脚本返回失败。 + +正确性检查读取 `activeOutputOffset` 指向的当前 scratch epoch,验证 BF16 路由行 +来自预期 source rank。该地址表示 Combine V2 原始通信输出,不是 TopK reduce +后的最终输出张量。 + +## 14. 已验证结果 + +### 14.1 512P 环境单机 8P + +2026-08-11 在 512P 环境 2 号柜第一台服务器完成单机 8P 验证。节点按 +`D:\3_codex\512P环境信息.txt` 解析为: + +```text +CPU1 141.61.52.35 +``` + +测试参数为 `BS=128, K=16, H=3584, experts=64, BF16, warmup=20, +iterations=80`。结果按“每卡 80 轮平均,再对所有卡取平均值和最大值”的正式 +口径统计: + +| BS | `avg_ms` | `avg_alg_bw_GBps` | `max_ms` | `max_alg_bw_GBps` | +| ---: | ---: | ---: | ---: | ---: | +| 128 | 0.187879 | 78.135892 | 0.195413 | 75.123272 | + +8 个 rank 正确性全部通过,共包含 640 个计时样本。正式日志: + +```text +/home/h00580772/tilexr_combine_v2_8p_cab2_cpu1_logs/combine_v2_8p_bs128_w20_i80_20260811.log +``` + +### 14.2 A10-64P 环境历史 16P 结果 + +2026-08-11 在以下两台服务器上完成 16P、多 BS 验证: + +```text +141.61.49.226:8 +141.61.49.223:8 +``` + +该记录生成于默认次数调整之前,测试参数为 +`K=16, H=3584, experts=64, BF16, warmup=5, iterations=20`。当前正式测试 +默认值已经调整为 `warmup=20, iterations(loop)=80`。下表采用旧的“逐轮最慢 +rank 再求平均”口径,仅保留为历史记录,不能与 14.1 的当前正式口径直接比较。 + +| BS | `avg_ms` | `alg_bw_GBps` | +| ---: | ---: | ---: | +| 128 | 0.532375 | 27.574668 | +| 256 | 0.723498 | 40.580800 | +| 512 | 0.882888 | 66.509292 | +| 1024 | 2.202714 | 53.316278 | +| 2048 | 3.892452 | 60.342690 | +| 4096 | 7.777167 | 60.402721 | +| 8192 | 15.758240 | 59.621131 | + +正式日志: + +```text +/home/h00580772/tilexr_combine_v2_16p_cdf2b01_20260811/logs/combine_v2_16p_bs128_8192_batch_20260811.log +``` + +该结果包含 112 条 rank 正确性通过记录和 2240 条计时样本。测试基于当前工作树, +其中 Combine V2 kernel 的 `kEnableSafetyChecks=false`。 + +### 14.3 512P 环境 0/2 号柜 128P 结果 + +2026-08-11 在 512P 环境 0 号柜和 2 号柜共 16 台服务器上完成 128P 单点测试, +每台服务器使用 8 张卡。测试参数为 +`BS=8192, K=16, H=3584, experts=256, BF16, warmup=20, iterations=80`。 + +单 rank 的完整逻辑通信数据量为: + +```text +8192 * 16 * 3584 * 2 = 939,524,096 bytes +``` + +结果按“每卡 80 轮平均,再对 128 张卡取平均值和最大值”的正式口径统计: + +| BS | `avg_ms` | `avg_alg_bw_GBps` | `max_ms` | `max_alg_bw_GBps` | +| ---: | ---: | ---: | ---: | ---: | +| 8192 | 20.588683 | 45.633035 | 20.930525 | 44.887747 | + +128 个 rank 的正确性全部通过,共包含 10240 个计时样本;每个 rank 均具备完整的 +80 个计时样本。最慢卡为 rank 1,其 80 轮平均耗时为 `20.930525 ms`。 + +全部性能测量和正确性检查完成后,最终 TCP barrier 有 36 个 rank 失败,因此 +launcher 返回码为 1,未输出正常的 `COMBINE_V2_PERF` 聚合行。上表由原始 +`COMBINE_V2_SAMPLE` 日志按第 13 节口径重新聚合,可用于本次性能结果;该异常仅 +表示测试结束阶段的 barrier/清理未正常完成,不表示计时样本或正确性检查失败。 + +正式日志: + +```text +/home/h00580772/tilexr_combine_v2_128p_cab0_2_logs/combine_v2_128p_bs8192_exp256_w20_i80_20260811.log +``` + +## 15. 常见失败 + +| 现象 | 检查项 | +| --- | --- | +| `remote runtime validation failed` | 新节点的 install、CANN、`timeout` 或 `ss` 是否存在 | +| `unsupported Combine V2 world size` | 调整 hostfile,使 rank 总数为 `2-8/16/32/64/128` | +| `expert count ... must be divisible by world size` | 调整 `--experts` 或 hostfile rank 总数 | +| `batch size ... must be divisible by world size` | 调整 BS 或 rank 总数 | +| `bootstrap or barrier port is already listening` | 更换 `-CommPort` | +| `NPU preflight timed out` | 查看 `.npu_preflight/`,等待非 TileXR 任务结束 | +| `a rank launcher failed` | 查看 `.ranks/rank_NNNN.log` | +| `rank logs do not contain...` | 检查缺失的 rank 正确性或 iteration 样本 | +| SHA-256 校验失败 | 重新从主服务器执行平铺同步,不要直接在 worker 修改 install | diff --git a/docs/plans/2026-08-11-combine-v2-step0-detailed-profiling.md b/docs/plans/2026-08-11-combine-v2-step0-detailed-profiling.md new file mode 100644 index 0000000..5f152bc --- /dev/null +++ b/docs/plans/2026-08-11-combine-v2-step0-detailed-profiling.md @@ -0,0 +1,41 @@ +# Combine V2 Step0 Detailed Profiling Plan + +## Goal + +Expand MoonEP Combine V2 profiling so the Step0 send interval reports named, +accumulated costs for selection, self-copy, and remote WQE submission while +preserving the existing cumulative timestamps and failure diagnostics. + +## Scope + +- Version the internal profile record and add eight cycle counters. +- Instrument the profiling-only kernel path with non-overlapping counters for + selection load/select, self route/copy, and remote route/descriptor/build/ + publish work. +- Copy and print the counters as named microsecond fields in the hardware probe. +- Update layout and source-contract tests for the new record contract. +- Build and run the existing focused tests, then build and validate the profile + artifact on the approved 198/226 16P environment using Mutagen source sync. + +## Non-Goals + +- Do not optimize the kernel in this change. +- Do not change the public Combine V2 launch ABI or result calculations. +- Do not use detailed-profile timings as the final non-profile performance result. + +## Implementation Order + +1. Extend `combine_v2_profile.h` with profile v2 metric indices and capacity. +2. Add profiling-only metric accumulation to the kernel and serialize it. +3. Extend the hardware probe sample validation and named output. +4. Update focused unit expectations and add source-contract coverage. +5. Run local/static tests, compile remotely, sync the runtime from the primary + server, and run the 16P BS=8192 profile case. + +## Risks And Verification + +Per-row cycle reads perturb profile builds, so counters are compiled out when +profiling is disabled and are used only for attribution. Profile record size is +an internal workspace ABI; host, kernel, and probe must be rebuilt and deployed +together. Verification must cover record size/layout, profile source wiring, +target compilation, record validation, correctness, and named metric output. diff --git a/docs/plans/2026-08-11-moonep-combine-v2-128p-exp256.md b/docs/plans/2026-08-11-moonep-combine-v2-128p-exp256.md new file mode 100644 index 0000000..020982f --- /dev/null +++ b/docs/plans/2026-08-11-moonep-combine-v2-128p-exp256.md @@ -0,0 +1,45 @@ +# MoonEP Combine V2 128P ExpNum=256 Execution Plan + +## Goal + +Run the Combine V2 single-operator benchmark on cabinets 0 and 2 with 128 +ranks, `BS=8192`, `K=16`, `H=3584`, BF16, and 256 experts. Report the +average and maximum of the 128 per-rank 80-iteration means, with equivalent +bandwidth for both times. + +## Scope + +- Add a runtime expert-count option to the C++ benchmark and Bash launchers. +- Accept the operator-supported world sizes through 128 ranks. +- Compile on cabinet 2 CPU1 (`141.61.52.35`). +- Flat-sync runtime artifacts from that server to all 16 cabinet 0/2 hosts. +- Launch ranks through direct SSH without MPI. + +The expert count is benchmark metadata and a sharding contract. It is not an +argument to the Combine V2 operator and does not change the bandwidth payload, +which remains `BS * K * H * sizeof(BF16)` per rank. + +## Implementation + +1. Update `tests/moonep_combine_v2/demo/tilexr_moonep_combine_v2_hardware_probe.cpp` + to parse `--experts`, validate `experts % world_size == 0`, and report the + configured value. +2. Update `tools/moonep/run_combine_v2_perf_multihost.sh` to accept and forward + `--experts`, validate 128P, and emit the configured value in final results. +3. Update `tools/moonep/run_combine_v2_perf_cluster.sh` and the performance + guide with the same interface and semantics. +4. Use the cabinet mapping from `D:\3_codex\512P环境信息.txt` to construct the + 16-host, 128-rank hostfile in cabinet 0 then cabinet 2 order. +5. Sync source with Mutagen, build and stage the non-MPI runtime on + `141.61.52.35`, then flat-sync artifacts to every host. +6. Run `warmup=20`, `iterations=80`, and preserve the full per-rank logs. + +## Verification + +- Source/build checks pass and the benchmark has no MPI dependency. +- Runtime SHA256 verification passes on all 16 hosts. +- NPU preflight follows the 15-second retry and 120-second maximum wait rule. +- The log contains 128 passing rank records and 10,240 timed samples. +- Final output contains `experts=256`, `avg_ms`, `avg_alg_bw_GBps`, `max_ms`, + and `max_alg_bw_GBps`. +- No benchmark processes remain after completion. diff --git a/docs/plans/2026-08-12-moonep-combine-v2-self-copy.md b/docs/plans/2026-08-12-moonep-combine-v2-self-copy.md new file mode 100644 index 0000000..db1683c --- /dev/null +++ b/docs/plans/2026-08-12-moonep-combine-v2-self-copy.md @@ -0,0 +1,62 @@ +# MoonEP Combine V2 Self-Copy Implementation Plan + +Status: Implemented. B150 compile, six focused Host tests, and bounded 8P +correctness passed using `/home/pkg/910_B150/cann-9.1.0` on 9号柜 CPU1 for +BS=128 and BS=8192. + +## Goal + +Implement the approved Self-copy pipeline in +`docs/specs/2026-08-12-moonep-combine-v2-self-copy-design.md` without changing +the Planner route ABI, Remote transport protocol, or public launch interface. + +## Scope + +The implementation touches the Combine V2 kernel, its focused source guards and +Host reference tests, and the approved design/plan artifacts. It must preserve +C++14 and target CANN compatibility. + +The experimental untracked `tilexr_moonep_combine_v2_kernel_back.h` is +read-only reference material and is not part of the build or implementation. + +## Work + +1. Replace the separate oversized issue buffers with one continuous 260-entry + UB allocation. Expose logical six-port and two-port views at entries 0 and + 194, prefill only 192/64 payload templates, append up to two controls per + lane, and retain the existing per-SQ MTE3 publication and ring-wrap split. +2. Delete obsolete compare/gather/descriptor declarations, allocations, and + inactive implementation from the active kernel. +3. Allocate two dedicated 64 KiB Self relay buffers with `TQue`, while retaining + a compile-time proof that all send buffers fit in 216 KiB UB. +4. Reuse `LoadSelectionChunk()` and `SelectPeerRoutes()` in `SendSelfStep()`. + Consume each selected `RouteEntry` batch before invoking the selector again. +5. For rows at most 64 KiB, derive rows per group from aligned `rowBytes_`, cap + the group at eight rows, and overlap the next MTE2 group with the pending + MTE3 group. For larger rows, flatten routes into at-most-64-KiB tiles and use + the same two-buffer pipeline. +6. Update source guards and Host reference tests for continuous WQE offsets, + lane capacities, dynamic group sizing, legacy-code removal, RouteEntry-based + Self addressing, and synchronization ordering. +7. Run focused local Host checks. Synchronize through Mutagen and compile with + `/home/pkg/b150` on an available target server. Before any hardware run, + apply the task's process-occupancy rules. + +## Constraints + +- Every WQE remains fully constructed in UB and is copied to SQ with MTE3. +- Doorbells remain `st_dev` operations after SQ MTE3 completion. +- Self does not modify SQ/CQ/control state. +- `sourceSlotIndex` is absolute and must not receive `chunkStart` twice. +- Copy exactly `rowBytes_`; aligned UB padding is never written to GM. +- Do not reuse WQE, route, cursor, or old selection storage as Self relay. +- Do not commit, push, or remove unrelated untracked files as part of this plan. + +## Verification + +- Focused Combine V2 schedule and source-guard tests pass. +- `git diff --check` passes. +- Target kernel compiles against `/home/pkg/b150`. +- Bounded hardware correctness covers Self routes and representative dynamic + relay group sizes when a suitable target is available. +- Profiling distinguishes selector time from the complete Self copy pipeline. diff --git a/docs/specs/2026-08-12-moonep-combine-v2-self-copy-design.md b/docs/specs/2026-08-12-moonep-combine-v2-self-copy-design.md new file mode 100644 index 0000000..185a9c8 --- /dev/null +++ b/docs/specs/2026-08-12-moonep-combine-v2-self-copy-design.md @@ -0,0 +1,536 @@ +# MoonEP Combine V2 Self-Copy Pipeline Design + +Status: Implemented; B150 compile and bounded hardware correctness complete + +Baseline: `c0d40d7 perf: batch MoonEP combine v2 WQE construction` + +Target environment: Ascend 3510, CANN `/home/pkg/910_B150/cann-9.1.0` + +The environment exposes the requested B150 build as `910_B150`, not the +lowercase literal path `/home/pkg/b150`. On 2026-08-12 the implementation was +synced through Mutagen to 9号柜 CPU1 (`141.61.55.118`) and compiled with +`/home/pkg/910_B150/cann-9.1.0`. + +Completed evidence: + +- the pure AICore kernel, Combine V2 Host library, and hardware probe compile + with B150; +- all six focused Combine V2 Host tests pass in the B150 build tree; +- the send-path UB layout is compile-time fixed at 216,096 bytes, leaving + 5,088 bytes of the 216 KiB budget; +- the obsolete Self mismatch allowance is removed from the probe and launcher; +- single-machine 8P correctness passes with the B150 build and runtime on + 9号柜 CPU1 for BS=128 and BS=8192, H=3584, K=16, BF16, ExpNum=64; and +- BS=8192 exercises selector batches above 128 routes and resume processing. + +The B150 hardware checks used `warmup=0` and `iterations=1` as bounded +correctness runs. Their timing output is not a formal performance result. The +controller log is +`/home/h00580772/tilexr_selfcopy_b150/logs/selfcopy_8p_bs128_8192_correctness.log`. + +This design restores `SendSelfStep()` after the SIMT route-selection and WQE +construction optimization. It reuses the active route-selection path and adds a +dedicated double-buffered GM-to-UB-to-GM copy pipeline for locally routed rows. + +This document supersedes the following decisions in +`2026-08-12-moonep-combine-v2-simt-wqe-design.md`: + +- self-copy no longer returns immediately; +- old compare/gather/descriptor UB buffers and their dead implementation are + deleted rather than retained under `#if 0`; and +- the remote WQE issue allocation becomes one continuous 260-WQE region split + into fixed 194- and 66-WQE lane regions. + +The untracked `tilexr_moonep_combine_v2_kernel_back.h` file is an experimental +reference only. It is not an implementation source and must not be copied over +the active kernel. In particular, its reuse of old selection buffers is not +part of this design. + +## Goals + +1. Restore correct local-route movement from the registered workspace to the + current scratch epoch. +2. Reuse the current SIMT selector and its compacted `RouteEntry` output. +3. Overlap Self MTE2 reads and MTE3 writes with two dedicated 64 KiB UB relay + buffers. +4. Copy up to eight complete rows per relay batch when the runtime row size + permits it, and reduce the row count automatically for larger hidden sizes. +5. Preserve the remote 3:1 QP distribution, control-WQE ordering, CQ behavior, + and doorbell ordering. +6. Recover UB capacity by deleting obsolete compare/gather selection storage + and reducing the WQE issue capacity to the actual maximum plus controls. + +## Non-Goals + +This change does not: + +- change the Planner route encoding; +- replace route division and modulo with a packed bit-field representation; +- use UDMA WQEs, SQs, doorbells, grants, done tokens, or CQs for Self traffic; +- add a SIMT function for Self address construction; +- expose relay size or rows per copy batch as a Host API parameter; +- change the reduction algorithm; or +- reuse WQE, route, cursor, compare, or gather storage as the Self relay. + +## Existing Route Contract + +The inverse route table remains: + +```text +dstLocal[expertRecvSlot] = srcRank * NvS + token * K + topk +``` + +For an entry selected for the current peer, the active SIMT selector produces: + +```cpp +struct MoonEpCombineV2RouteEntry { + uint32_t sourceSlotIndex; + uint32_t targetSlot; +}; +``` + +For Self traffic: + +```cpp +source = workspace_ + + static_cast(route.sourceSlotIndex) * rowBytes_; +target = scratch_ + + static_cast(route.targetSlot) * rowBytes_; +``` + +`sourceSlotIndex` is already absolute. `SendSelfStep()` must not add the current +selection chunk offset a second time. `targetSlot` is already decoded by the +selector; Self must not reload `dstLocal` or repeat the modulo operation. + +Workspace input and scratch output are different GM regions. The design does +not support or require an in-place overlapping copy. + +## Reused Remote Selection Path + +The following active Remote components are reused without a second selection +implementation: + +- `dstSlotBuf_`, including its 64 KiB GM-to-UB chunk load; +- `LoadSelectionChunk()`; +- `MoonEpCombineV2SelectPeerRoutesVf()`; +- `SelectPeerRoutes()`; +- `routeEntryBuf_`; +- `threadMaxSlotIdxBuf_`; +- `selectStateBuf_`; +- the peer-lifetime `curWqeNum` initialization; +- `firstPass` and per-thread resume cursors; +- `pausedThreadCount` and repeated scans over a resident chunk; and +- the maximum of 256 selected `RouteEntry` objects per selector invocation. + +Self invokes this path with `peer == rank_`. The selector continues to perform +the existing division and modulo and writes the same route representation used +by the Remote payload builder. + +The following Remote stages are not used by Self: + +- operator- or peer-level WQE prefill; +- remote-field resolution; +- payload WQE construction; +- 3:1 QP assignment; +- SQ publication and doorbells; +- grant and done control WQEs; +- SQ head, CQ target, completion-count, and outstanding-row updates; and +- remote admission or final-CQ handling. + +`WaitInboundDone()` already treats the local source rank as ready. Self does +not publish a local done token. + +## Deleted Legacy Selection State + +The active kernel will delete the declarations, allocations, initialization, +and unreachable implementation associated with the old compare/gather path: + +- `dstRankBuf_`; +- `slotIndexBuf_`; +- `selectedIndexBuf_`; +- `compareMaskBuf_`; +- the old 4 KiB `relayBuf_`; +- `descriptorBuf_`; +- `SelectPeer()`; +- `CreateVecIndex`, `Compares`, and `GatherMask` selection code; +- the selected-index-based Self loop; and +- obsolete descriptor append code. + +These elements are deleted from the active source rather than moved to another +inactive block. The experimental back file remains outside this cleanup and is +not added to the build. + +## Continuous WQE Issue Region + +The maximum selector output remains 256 payload WQEs. The 3:1 distribution has +the following maxima: + +```text +six-port payload maximum: 192 +two-port payload maximum: 64 +``` + +Each QP lane may append at most two control WQEs: grant and done. The issue +capacities therefore become: + +```cpp +constexpr uint32_t kSixPortPayloadCapacity = 192U; +constexpr uint32_t kTwoPortPayloadCapacity = 64U; +constexpr uint32_t kControlWqesPerLane = 2U; +constexpr uint32_t kSixPortIssueCapacity = 194U; +constexpr uint32_t kTwoPortIssueCapacity = 66U; +constexpr uint32_t kTotalIssueCapacity = 260U; +``` + +One `TBuf` owns all 260 consecutive 64-byte WQEs: + +```cpp +TBuf wqeIssueBuf_; + +pipe_->InitBuffer(wqeIssueBuf_, kTotalIssueCapacity * kWqeBytes); +``` + +The physical UB layout is fixed: + +```text +entry 0 entry 259 ++--------------------------------------------------------------+ +| six-port region: 194 entries | two-port region: 66 entries | +| payload capacity 192 + 2 | payload capacity 64 + 2 | ++--------------------------------------------------------------+ +^ entry 0 ^ entry 194 +``` + +The two logical views are derived from the one allocation: + +```cpp +LocalTensor allIssue = wqeIssueBuf_.Get(); +LocalTensor sixPortIssue = allIssue; +LocalTensor twoPortIssue = + allIssue[kSixPortIssueCapacity * kWqeBytes]; +``` + +The four control capacities are reserved capacity, not fixed control indices. +For a short final batch, control WQEs are appended immediately after the actual +payload count in that lane: + +```text +lane region: [active payload][grant, if any][done][unused capacity] +``` + +This preserves a compact active prefix for each lane. The fixed boundary at +entry 194 can leave unused space between the active six-port prefix and the +two-port region in a short batch. That space is not a data gap within either +QP submission because the two regions are copied to different SQs. + +Operator and peer prefill cover the 192 and 64 payload capacities. Scalar +`AppendControlWqe()` clears and fully constructs each appended control entry. +When a short final batch overwrites an entry inside the payload-capacity part of +a region, the next peer-level prefill restores it as a payload template. + +Each lane still normally needs one MTE3 operation for its compact active +prefix. `CopyIssueToSq()` uses two MTE3 operations for that lane only when the +target SQ range wraps at the ring boundary. The continuous 260-entry UB +allocation cannot be published as one MTE3 because the lane regions target two +different SQs. + +## Send-Path UB Layout + +The proposed active send-path allocation is: + +| Buffer | Bytes | Purpose | +| --- | ---: | --- | +| `dstSlotBuf_` | 65,536 | 16,384 inverse-route entries | +| continuous `wqeIssueBuf_` | 16,640 | 260 complete WQEs, split 194/66 | +| `routeEntryBuf_` | 2,048 | Up to 256 compact Self or Remote routes | +| `threadMaxSlotIdxBuf_` | 512 | 128 selector resume cursors | +| `selectStateBuf_` | 32 | Selector atomic and batch state | +| `wqeContextBuf_` | 256 | Operator, peer, and build context | +| `selfCopyQueue_` | 131,072 | Two independent 64 KiB relay buffers | +| **Total** | **216,096** | | + +The 216 KiB budget is 221,184 bytes, leaving 5,088 bytes. Compile-time +assertions must prove the total, the 194/66 split, and the maximum payload plus +two controls per lane. + +The Self relay is a dedicated allocation. It does not alias the WQE region, +route entries, selector state, or cursor storage. This keeps the Self MTE +lifetime independent of both SIMT and Remote template restoration. + +## Self Relay and Dynamic Batch Size + +The Self copy uses two 64 KiB relay buffers: + +```cpp +constexpr uint32_t kSelfRelayHalfBytes = 64U * 1024U; +constexpr uint32_t kSelfMaxBatchRows = 8U; + +TQue selfCopyQueue_; +pipe_->InitBuffer(selfCopyQueue_, 2U, kSelfRelayHalfBytes); +``` + +`rowBytes_`, not `h_`, is the authoritative transfer size. For BF16 without +additional row padding it is `H * sizeof(bfloat16_t)`, but the copy path must +continue to work when the launch contract supplies another valid row size. + +For a row that fits in one relay half: + +```cpp +const uint64_t localRowStride = AlignUp(rowBytes_, kUbAlignBytes); +const uint32_t rowsPerBatch = static_cast(Min( + static_cast(kSelfMaxBatchRows), + static_cast(kSelfRelayHalfBytes) / localRowStride)); +``` + +The division is evaluated only after checking `rowBytes_ != 0` and proving +that `AlignUp` cannot overflow. `rowsPerBatch` is a computed value and need not +be a power of two. It is capped at eight to avoid creating an unnecessarily +large group of non-contiguous DMA commands for small rows. + +Examples for BF16 rows are: + +| H | `rowBytes_` | Computed rows in 64 KiB | Active batch rows | +| ---: | ---: | ---: | ---: | +| 3,584 | 7,168 | 9 | 8 | +| 4,096 | 8,192 | 8 | 8 | +| 7,168 | 14,336 | 4 | 4 | +| 8,192 | 16,384 | 4 | 4 | +| 16,384 | 32,768 | 2 | 2 | +| 32,768 | 65,536 | 1 | 1 | + +Each row occupies `localRowStride` bytes in UB, but MTE2 and MTE3 copy exactly +`rowBytes_` logical bytes. Padding between relay rows is not written back to GM. + +## Oversized-Row Fallback + +If `localRowStride > kSelfRelayHalfBytes`, a complete row cannot reside in one +relay half and `rowsPerBatch` would be zero. The implementation must not reject +the shape merely for that reason and must not return to the old serialized +4 KiB helper. + +Instead, flatten each selected row into ordered copy tasks: + +```text +(route index, row byte offset, tile bytes) +``` + +where: + +```cpp +tileBytes = Min(rowBytes_ - rowOffset, kSelfRelayHalfBytes); +``` + +The source and destination for a tile retain the same byte offset: + +```cpp +source = workspace_ + sourceSlotIndex * rowBytes_ + rowOffset; +target = scratch_ + targetSlot * rowBytes_ + rowOffset; +``` + +The same two-buffer pipeline processes these tiles. All tiles for a route must +be copied, but no ordering relationship is required between different target +rows beyond completion before `SendSelfStep()` returns. + +## Copy Helper Responsibilities + +The old per-row `CopyBytesGmToGm()` helper is not the top-level Self copy +interface. The new helper consumes the current compacted route batch: + +```cpp +__aicore__ inline bool CopySelfRouteBatch(uint32_t selectedCount); +``` + +Its responsibilities are: + +1. Read `selectedCount` entries from `routeEntryBuf_`. +2. Compute the dynamic row grouping or oversized-row tile sequence. +3. Issue GM-to-UB copies into one relay buffer. +4. Publish the filled relay buffer to the MTE3 consumer. +5. While MTE3 drains the pending buffer, issue the next MTE2 group into the + other buffer. +6. Wait before reusing a buffer whose MTE3 is still active. +7. Drain the last pending MTE3 before returning. + +No Self-specific SIMT address-builder is introduced initially. The MTE commands +are issued by scalar code, and each route needs only two address multiply-adds. +An extra SIMT launch and cross-model barrier would need measured evidence before +being added. + +## MTE2/MTE3 Pipeline + +For complete rows, one buffer group contains up to `rowsPerBatch` non-contiguous +rows. The logical schedule is: + +```text +relay 0: [MTE2 group 0] [MTE3 group 0] +relay 1: [MTE2 group 1] [MTE3 group 1] +relay 0: [MTE2 group 2] ... +``` + +Each group issues one `DataCopyPad` per active row because the route-derived GM +addresses are generally non-contiguous. Batching reduces synchronization and +buffer-turnover cost; it does not make the scatter rows one DMA range. + +The required dependencies are: + +1. MTE2 completion for a relay buffer before MTE3 reads that buffer. +2. MTE3 completion for a relay buffer before it is freed or reused by MTE2. +3. Completion of the final MTE3 before `CopySelfRouteBatch()` returns. +4. Completion of the route batch before `SelectPeerRoutes()` is invoked again + and overwrites `routeEntryBuf_`. +5. Completion of all Self copies before reduction reads `scratch_`. + +The implementation should use the target-supported `TQue` ownership and event +handoffs rather than a `PIPE_ALL` barrier per row. The existing dispatch helper +`CopyContiguousBytesGmToGmPipelined()` is a useful queue-lifetime reference, but +it cannot be called directly because Self source and destination rows are a +scatter operation. + +Exact `TQue`, `DataCopyPad`, `DataCopyExtParams`, and HardEvent usage must be +confirmed by compiling against `/home/pkg/b150`. In particular, GM-to-UB and +UB-to-GM overloads have direction-specific signatures and tail semantics; a +working overload in another CANN version is not treated as proof for b150. + +## SendSelfStep Flow + +`SendSelfStep(peer)` follows the same chunk and resume structure as Remote: + +```cpp +state->curWqeNum = 0U; + +for (each dstLocal chunk) { + LoadSelectionChunk(chunkStart, chunkElements); + + bool firstPass = true; + uint32_t pausedThreadCount = 0U; + do { + const uint32_t selectedCount = SelectPeerRoutes( + peer, chunkStart, chunkElements, + firstPass, pausedThreadCount); + + if (selectedCount != 0U) { + CopySelfRouteBatch(selectedCount); + } + firstPass = false; + } while (pausedThreadCount != 0U); +} +``` + +Unlike Remote, Self does not submit an empty final batch. There are no control +WQEs to publish when no Self routes match. + +The peer argument is expected to equal `rank_`. Safety-enabled builds should +reject or report a violation rather than silently copying routes for another +peer through the Self path. + +## Remote Submit Behavior After WQE Resize + +Remote payload construction continues to compact each lane independently. The +active prefix of a final lane submission is: + +```text +[payload entries][grant when another step exists][done] +``` + +For example, 99 selected payload WQEs are distributed according to +`sequenceBase & 3`. Depending on the phase, the split is 75/24 or 74/25. Each +lane appends its one or two controls to its own compact prefix. With no SQ wrap, +the submit therefore issues two MTE3 calls: one to each lane's SQ. It does not +issue extra MTE3 calls because of unused capacity in the continuous UB region. + +`CopyIssueToSq()` must accept the two logical tensor views and copy only the +active count. Its existing ring-wrap split remains unchanged. Doorbells are +rung only after all MTE3 writes for both lanes have completed. + +## Profiling + +The selection metrics retain their current meaning: + +- `SELECTION_LOAD`: GM-to-UB load of the inverse-route chunk; +- `SELECTION_SELECT`: SIMT peer filtering, division/modulo decode, route + compaction, and selector synchronization. + +The old `SELF_ROUTE_DECODE` metric must not claim that a separate scalar decode +still exists. It should either remain zero/reserved for profile ABI stability or +be renamed only with an explicit profile-version change. + +`SELF_COPY` measures the complete Self data-movement section, including scalar +address generation, MTE2, MTE3, queue/event handoffs, and final drain. If more +detail is temporarily needed during performance work, profiling may split it +into read, write, and wait components without putting timestamp collection into +the normal non-profiling build. + +## Correctness and Boundary Conditions + +The implementation must preserve these invariants: + +- `selectedCount <= 256`; +- every selected route is copied exactly once; +- no unselected or `-1` route is copied; +- each copy reads exactly `rowBytes_` from its source row; +- each copy writes exactly `rowBytes_` to its target row; +- aligned UB stride padding never reaches GM; +- row and tile offsets use 64-bit arithmetic; +- the relay buffer is not reused before its MTE3 completes; +- `routeEntryBuf_` is not overwritten while a Self copy still consumes it; +- Self never changes `issuedRows_`, lane heads, CQ targets, or completion + counts; +- each lane's payload plus controls fits 194 or 66 entries; and +- every WQE is fully constructed in UB and published through MTE3 before its + SQ doorbell. + +Planner is expected to provide the valid inverse-route mapping and unique +target ownership required by Combine V2. Safety checks continue to validate the +encoded destination range when enabled; this change does not add an expensive +hot-path duplicate-target detector. + +## Validation Plan + +Implementation acceptance requires: + +1. Compile-time assertions for the 216 KiB UB total and the continuous 260-WQE + region. +2. Unit checks for the fixed WQE offsets: six-port at entry 0 and two-port at + entry 194. +3. Unit checks for maximum payload/control counts: 192+2 and 64+2. +4. Unit checks for dynamic batch rows at representative `rowBytes_`, including + 8, 4, 2, and 1 row cases. +5. Unit checks for a row larger than 64 KiB and its final partial tile. +6. Source guards proving the obsolete compare/gather/descriptor buffers and + dead path are removed. +7. Source guards proving Self consumes `routeEntryBuf_` and does not add + `chunkStart` to `sourceSlotIndex` again. +8. Source guards for MTE2-before-MTE3, MTE3-before-buffer-reuse, final drain, + and SQ MTE3-before-doorbell ordering. +9. A target compile with `/home/pkg/b150` CANN. +10. Focused hardware correctness for no Self routes, one route, short batches, + more than 128 Self routes, and multiple selector resume passes. +11. Hardware correctness for H values producing 8, 4, 2, and 1 rows per relay + group, plus an oversized-row case if the public shape contract permits it. +12. Profiling that reports selection and Self copy separately. +13. Performance comparison with the same warmup, loop count, shape, rank count, + and device placement used for the current SIMT WQE baseline. + +Correctness is required again once `SendSelfStep()` is restored; the prior +`self_only_failed` allowance is no longer an acceptable final result. + +## Review Decisions Recorded + +- Reuse the current SIMT selector and `RouteEntry` representation for Self. +- Delete old compare/gather/descriptor UB allocations and implementation. +- Allocate a dedicated two-half Self relay; do not alias other send storage. +- Use two 64 KiB relay halves. +- Compute rows per batch from relay-half bytes divided by aligned runtime row + bytes, capped at eight. +- Use the same ping-pong pipeline for an oversized row by splitting it into + tiles. +- Keep Self address generation scalar for the first implementation. +- Replace the two oversized WQE buffers with one physically continuous + 260-entry allocation. +- Split that allocation into fixed 194- and 66-entry QP regions. +- Reserve capacity for at most two control WQEs per QP and append controls + immediately after the actual payload prefix. +- Preserve separate MTE3 publication to the two SQs and the existing ring-wrap + split. +- Validate the exact data-copy and queue synchronization APIs against CANN + `/home/pkg/b150` before hardware testing. diff --git a/docs/specs/2026-08-12-moonep-combine-v2-simt-wqe-design.md b/docs/specs/2026-08-12-moonep-combine-v2-simt-wqe-design.md new file mode 100644 index 0000000..b0992f0 --- /dev/null +++ b/docs/specs/2026-08-12-moonep-combine-v2-simt-wqe-design.md @@ -0,0 +1,663 @@ +# MoonEP Combine V2 SIMT WQE Construction Design + +Status: Implemented; final `/home/pkg/b150` validation pending + +Baseline: `80f184e perf: checkpoint combine v2 with self-copy disabled` + +The implementation has passed the focused Host tests, a CANN b131 target +compile, and bounded 8P hardware runs on `141.61.49.223`. The requested b150 +compile remains pending because `/home/pkg/b150` is not installed or mounted on +that host as of 2026-08-12. The b131 evidence must not be treated as b150 proof. + +## Goal + +Reduce the WQE preparation cost in the MoonEP Combine V2 send path. Replace +the current vector selection, scalar route decoding, scalar descriptor append, +and descriptor-driven WQE construction with the following staged flow: + +1. Prefill operator-constant payload WQE fields once after send UB allocation. +2. Prefill peer- and QP-constant fields once before sending to each peer. +3. Load up to 64 KiB of `dstLocal` from GM to UB. +4. Select and compact routes for the current peer with a SIMT function. +5. Build the selected payload WQEs evenly with a second SIMT function. +6. Append grant and done control WQEs with the existing scalar path. +7. Copy the active WQEs from UB to SQ with MTE3, then ring the doorbells. + +The first implementation keeps the existing `dstLocal` encoding and keeps +self-copy disabled. It is a payload WQE preparation optimization, not a change +to the public Combine V2 interface or to the Planner forward-route ABI. + +## Scope + +The implementation will change the Combine V2 kernel send path and its focused +tests. It will: + +- introduce four SIMT functions for operator prefill, peer prefill, route + selection, and payload WQE construction; +- allocate a 64 KiB `dstLocal` UB buffer; +- allocate fixed six-port and two-port issue buffers for 256 and 128 WQEs; +- allocate a route buffer for at most 256 payload entries; +- allocate one 32-byte selection state and 128 per-thread cursors; +- preserve the existing 3:1 six-port/two-port payload distribution; +- preserve scalar construction of grant and done WQEs; +- preserve MTE3 publication to SQ and `st_dev` doorbell ordering; +- retain the replaced selection and descriptor code under `#if 0` instead of + deleting it; and +- make `SendSelfStep()` return success immediately. + +This design does not: + +- change `dstLocal` to a packed bit-field format; +- change Planner's public `dst` route encoding; +- optimize the remaining integer division and modulo; +- restore or optimize self-copy; +- change CQ generation, grant/done semantics, or QP ownership; or +- expose a new runtime or host parameter. + +## Existing Route Contract + +Planner's public forward route remains: + +```text +dst[token * K + topk] = expertRank * NvS + expertRecvSlot +``` + +Combine V2 consumes the internal inverse route table: + +```text +dstLocal[expertRecvSlot] = srcRank * NvS + token * K + topk +``` + +For a non-negative `dstLocal` entry, the Combine V2 send path continues to +decode it as: + +```cpp +peer = static_cast(encoded) / slots; +targetSlot = static_cast(encoded) % slots; +``` + +`-1` remains the invalid or padding value. `NvS` is not required to be a power +of two. In particular, existing shapes such as `NvS = 2040` must keep working. +This design therefore does not replace division and modulo with shift and mask. + +The compacted route contains: + +```cpp +struct MoonEpCombineV2RouteEntry { + uint32_t sourceSlotIndex; + uint32_t targetSlot; +}; +``` + +`sourceSlotIndex` is the absolute row in the local registered workspace that +contains the expert output. It is also the index of the selected `dstLocal` +entry. `targetSlot` is the row in the peer's receive scratch that must receive +that output, normally the peer's `token * K + topk` route index. + +The payload builder derives the addresses as: + +```cpp +localAddr = localRowBase + + static_cast(route.sourceSlotIndex) * rowBytes; +remoteAddr = remoteRowBase + + static_cast(route.targetSlot) * rowBytes; +``` + +The selector computes modulo only after an entry's decoded peer matches the +current peer. Division remains necessary for every valid entry examined for a +peer. Its cost must be measured separately after this refactor; changing the +route encoding is explicitly deferred. + +## Single Tuning Constant + +The selector threshold and both SIMT launch widths are derived from one +compile-time constant: + +```cpp +constexpr uint32_t kMoonEpCombineV2PayloadBatchRows = 128U; + +constexpr uint32_t kMoonEpCombineV2SelectorThreads = + kMoonEpCombineV2PayloadBatchRows; +constexpr uint32_t kMoonEpCombineV2BuilderThreads = + kMoonEpCombineV2PayloadBatchRows; +constexpr uint32_t kMoonEpCombineV2MaxSelectedPayloadWqes = + 2U * kMoonEpCombineV2PayloadBatchRows; +``` + +The constant is not exposed through the public API, host launch arguments, an +environment variable, or a benchmark option. A later experiment changes only +this definition from 128 to 64 or 32 and recompiles the operator. + +The active design supports values up to 128: + +| Batch rows | Selector threads | Builder threads | Maximum payload WQEs | +| ---: | ---: | ---: | ---: | +| 128 | 128 | 128 | 256 | +| 64 | 64 | 64 | 128 | +| 32 | 32 | 32 | 64 | + +The UB capacities remain allocated for the maximum 128-row configuration when +the tuning constant is reduced. Compile-time assertions must reject zero and +values above 128 and must prove that both QP lane buffers can hold the maximum +payload distribution plus two control WQEs. + +## Why Selection and WQE Construction Are Separate + +The uniform hardware probe creates destinations with: + +```cpp +targetRank = slot % world; +``` + +A selector that scans with: + +```cpp +index = threadIdx.x + iteration * selectorThreads; +``` + +has highly uneven matches for a fixed peer: + +- at 128P with 128 selector threads, one thread sees all entries for a peer; +- at 16P, eight threads see entries for a peer; and +- at 8P, sixteen threads see entries for a peer. + +Constructing a WQE immediately in the matching selector thread would therefore +make the 128P uniform case a single-thread WQE builder. The first SIMT function +only selects and compacts route information. The second assigns compacted +entries by route-buffer index, so WQE construction is balanced even when route +selection is not. + +The user-visible correctness contract permits WQEs for the same peer to be +issued in a different order. Each route must retain its original +`sourceSlotIndex` and `targetSlot`; grant and done controls must still be +published after all payload WQEs for the peer. + +## UB Layout + +The active send-path allocations are: + +| Buffer | Size at batch rows 128 | Purpose | +| --- | ---: | --- | +| `dstSlotBuf_` | 64 KiB | 16,384 `int32_t` inverse-route entries loaded from GM | +| `sixPortIssueBuf_` | 16 KiB | 256 complete 64-byte WQEs | +| `twoPortIssueBuf_` | 8 KiB | 128 complete 64-byte WQEs | +| `routeEntryBuf_` | 2 KiB | 256 `sourceSlotIndex/targetSlot` pairs | +| `threadMaxSlotIdxBuf_` | 512 B | One `uint32_t` cursor for each of 128 threads | +| `selectStateBuf_` | 32 B | Peer counter and per-invocation selector state | +| prefill/build context | implementation-sized, aligned | Operator, peer, QP, head, and row fields | + +The issue-buffer sizes are fixed capacities, not the number copied to SQ. Each +submission copies only its active payload and control WQE counts. + +The following old buffers and their active allocations are disabled but +retained under `#if 0` for comparison during this optimization: + +- `dstRankBuf_`; +- `slotIndexBuf_`; +- `selectedIndexBuf_`; +- `compareMaskBuf_`; +- `descriptorBuf_`; and +- the old relay/self-copy buffer when it has no other active consumer. + +Keeping those old allocations active while expanding the issue buffers would +exceed the current 216 KiB UB budget. The new active allocation must retain a +compile-time total-UB assertion after all context and alignment sizes are +finalized. + +## Selection State + +The 32-byte core-local UB state is shared by the selector threads on one AI +Core. It is not shared across AI Cores: + +```cpp +struct alignas(32) MoonEpCombineV2SelectState { + uint32_t curWqeNum; + uint32_t batchBase; + uint32_t batchSelected; + uint32_t pausedThreadCount; + uint32_t reserved[4]; +}; +``` + +The exact field access may be scalar arguments plus UB words, but the active +storage must remain 32 bytes and preserve these semantics: + +- `curWqeNum` is cumulative for one peer and is cleared only before processing + that peer, not before every selector invocation or GM chunk; +- `batchBase` snapshots `curWqeNum` before one selector invocation; +- `batchSelected` is `curWqeNum - batchBase` after the SIMT function returns; +- `pausedThreadCount` is reset before each selector invocation and counts + threads that stopped at the overflow threshold while more entries remain in + their strided range. + +`curWqeNum` is the only per-route atomic counter. According to the target +`asc_atomic_add` contract, the API returns the value that existed before the +addition. The route-buffer index is therefore: + +```cpp +const uint32_t old = asc_atomic_add(&state->curWqeNum, 1U); +const uint32_t routeIndex = old - state->batchBase; +``` + +The implementation must compile this exact use against the `/home/pkg/b150` +CANN headers before relying on it. The official API reference used for this +contract is `asc_atomic_add`: + + + +## Per-Thread Resume Cursors + +`threadMaxSlotIdxBuf_` contains 128 `uint32_t` entries. Active selector thread +`tid` exclusively owns entry `tid`, so updating the entry does not require an +atomic max. A thread keeps its last examined absolute `dstLocal` index in a +register and writes it to UB once before returning from the selector. + +The cursor is the last examined slot, not only the last matching slot. This is +required to prevent rescanning non-matching entries and to ensure that a resumed +thread neither skips nor duplicates an entry. + +For the first invocation over a newly loaded chunk: + +```cpp +index = chunkStart + threadIdx.x; +``` + +Every active thread writes this first candidate to its cursor before the first +invocation returns, even when the candidate is already at or beyond the chunk +end. This initializes cursor entries for short tail chunks in which some +threads have no valid slot to examine. Such an out-of-range cursor remains +exhausted on later invocations over the same chunk. + +For later invocations over the same chunk: + +```cpp +index = threadMaxSlotIdx[threadIdx.x] + + kMoonEpCombineV2SelectorThreads; +``` + +Each thread then advances by `kMoonEpCombineV2SelectorThreads`. The first-pass +flag is explicit; the implementation must not manufacture an initial cursor by +subtracting the stride from an unsigned `chunkStart`, because the first chunk +would underflow. + +The cursor and `sourceSlotIndex` are absolute GM indices. `dstSlots` points to +the beginning of the current UB chunk, so the selector reads the encoded value +with the chunk-relative index: + +```cpp +encoded = dstSlots[index - chunkStart]; +``` + +An absolute index must never be applied directly to the 64 KiB UB buffer. + +When a thread reaches the end of its strided range, it records its last scan +position and returns. For a thread with at least one valid entry this is the last +examined index; for an initially out-of-range tail thread it is the initialized +first candidate. When a thread stops at the overflow threshold and its next +strided index is still inside the chunk, it atomically increments +`pausedThreadCount`. A nonzero `pausedThreadCount` causes another selector +invocation over the same GM-loaded UB chunk. Exhausted threads return +immediately in that invocation, while paused threads resume from their own +cursors. + +The `dstLocal` chunk remains resident in UB until every thread has exhausted +its strided range. Only then may MTE2 overwrite the buffer with the next chunk. + +## Selector Bound + +For every matching route in one invocation, a thread obtains `routeIndex` with +the atomic add, stores the complete `MoonEpCombineV2RouteEntry`, and then tests +the overflow threshold: + +```cpp +if (routeIndex >= kMoonEpCombineV2PayloadBatchRows) { + // The current route has already been stored. + // Record this thread's cursor, mark it paused if work remains, and exit. +} +``` + +Before the threshold is reached, exactly `payloadBatchRows` entries can obtain +indices `[0, payloadBatchRows - 1]`. After that point, each selector thread can +store at most one additional entry because its first index at or above the +threshold makes it exit. Therefore: + +```text +batchSelected <= payloadBatchRows + selectorThreads + == 2 * payloadBatchRows +``` + +At the default value this is at most 256 payload routes. The bound applies to +payload only; grant and done control WQEs are not included. + +The single-thread 128P uniform case usually produces 129 selected routes per +full selector invocation: the matching thread stores indices 0 through 128 and +then exits. This is expected. The 256 value is a strict capacity bound, not a +target count that every input must reach. + +## SIMT Interfaces + +The implementation uses four SIMT functions. Names may be adjusted to match +local style, but their ownership and data flow must remain as specified. + +### Operator-Level Prefill + +```cpp +__simt_vf__ __aicore__ +LAUNCH_BOUND(kMoonEpCombineV2BuilderThreads) +void MoonEpCombineV2PrefillOperatorWqesVf( + __ubuf__ uint8_t *sixPortWqes, + __ubuf__ uint8_t *twoPortWqes, + __ubuf__ const MoonEpCombineV2OperatorFields *fields); +``` + +This function runs once after the send UB buffers are initialized and fills +fields that do not change between peers or batches: + +- zero/default bits for each 64-byte WQE; +- payload opcode; +- payload flag and `nf` defaults; +- `inlineMsgLen` and `sgeNum`; +- SGE length from `rowBytes`; and +- the local SGE token ID default. + +It prefills all 256 six-port entries and all 128 two-port entries, independent +of the active payload batch constant. + +### Peer-Level Prefill + +```cpp +__simt_vf__ __aicore__ +LAUNCH_BOUND(kMoonEpCombineV2BuilderThreads) +void MoonEpCombineV2PrefillPeerWqesVf( + __ubuf__ uint8_t *sixPortWqes, + __ubuf__ uint8_t *twoPortWqes, + __ubuf__ const MoonEpCombineV2PeerFields *fields); +``` + +This function runs once at the beginning of `SendRemoteStep()` after the two +QP-specific remote memory records for the current peer have been resolved. It +fills fields that are constant for the peer and lane: + +- token enable; +- remote jetty type; +- target hint; +- TP ID; +- remote jetty or segment ID; +- remote token value; and +- remote EID low and high words. + +It also restores the operator-constant payload fields in every issue entry that +may have been overwritten by the previous peer's scalar control WQEs. In the +initial implementation the peer-level function writes those payload constants +for all fixed issue-buffer entries, rather than tracking the previous control +indices. In particular, it must restore payload `flag`, SGE length, and local +SGE token ID; updating only the remote QP fields is incorrect. + +The remote row address is not peer-constant at WQE level because it includes +`targetSlot`. SQ basic-block index, owner, and local SGE address are also +batch- or route-dependent and are left to the payload builder. + +Peer prefill covers the complete fixed issue-buffer capacities. Operator-level +prefill establishes the initial template once; peer-level prefill both applies +the new peer fields and makes every entry a valid payload template again before +reuse. + +### Route Selection + +```cpp +__simt_vf__ __aicore__ +LAUNCH_BOUND(kMoonEpCombineV2SelectorThreads) +void MoonEpCombineV2SelectPeerRoutesVf( + __ubuf__ const int32_t *dstSlots, + uint32_t chunkStart, + uint32_t chunkElements, + uint32_t peer, + uint64_t slots, + bool firstPass, + __ubuf__ MoonEpCombineV2SelectState *state, + __ubuf__ uint32_t *threadMaxSlotIdx, + __ubuf__ MoonEpCombineV2RouteEntry *routes); +``` + +The selector: + +1. obtains the first absolute index from `firstPass` or the thread cursor; +2. walks that thread's strided entries in the resident UB chunk; +3. skips `-1` entries; +4. obtains `peer` with division by `slots`; +5. skips entries for other peers; +6. obtains `targetSlot` with modulo by `slots` only for a match; +7. atomically reserves the route-buffer index; +8. stores `sourceSlotIndex` and `targetSlot` before testing the threshold; and +9. records its private cursor before returning. + +The selector does not write WQEs and does not access SQ GM. + +### Payload WQE Construction + +```cpp +__simt_vf__ __aicore__ +LAUNCH_BOUND(kMoonEpCombineV2BuilderThreads) +void MoonEpCombineV2BuildPayloadWqesVf( + __ubuf__ uint8_t *sixPortWqes, + __ubuf__ uint8_t *twoPortWqes, + __ubuf__ const MoonEpCombineV2RouteEntry *routes, + uint32_t selectedCount, + uint32_t sequenceBase, + __ubuf__ const MoonEpCombineV2BuildContext *context); +``` + +The builder assigns compacted routes as: + +```cpp +for (uint32_t route = threadIdx.x; + route < selectedCount; + route += kMoonEpCombineV2BuilderThreads) { + // Build this route's WQE. +} +``` + +Since `selectedCount <= 2 * builderThreads`, every thread builds at most two +payload WQEs. The builder uses `sequenceBase + route` to preserve the existing +3:1 six-port/two-port sequence across selector invocations and GM chunks. +`sequenceBase` is the peer-cumulative `batchBase`; it is not reset per batch. + +For each active WQE the builder writes only dynamic fields: + +- lane-local output index; +- SQ basic-block index; +- owner bit; +- remote address low and high words; and +- local SGE virtual address. + +All other payload fields come from operator and peer prefill. + +## SendRemoteStep Flow + +`SendRemoteStep(peer, step)` follows this sequence: + +1. Clear `curWqeNum` once for the peer. +2. Resolve both QP lane records for the peer. +3. Run peer-level prefill for both fixed issue buffers. +4. For each 64 KiB `dstLocal` chunk: + 1. copy the chunk from GM to `dstSlotBuf_` with MTE2; + 2. establish the MTE2-to-SIMT dependency; + 3. invoke the selector with `firstPass = true`; + 4. snapshot `batchSelected` after the selector completes; + 5. build and submit the selected payload WQEs; + 6. while `pausedThreadCount != 0`, repeat selection over the same UB chunk + with `firstPass = false`; and + 7. load the next GM chunk only after the current one is exhausted. +5. Mark only the last payload submission for the peer as final. +6. If the final selector result contains zero payload routes, still issue the + required final grant/done control-only submission. + +The implementation may avoid a look-ahead by treating a batch as final when +the selector reports no paused thread and the current chunk is the last GM +chunk. Empty non-final selector results are not submitted. + +## QP Distribution and Capacity + +Payload sequence position continues to select the two-port lane once every four +WQEs; the other three positions use the six-port lane. `sequenceBase` preserves +that phase across all batches for one peer. + +With 256 payload WQEs, the maximum active payload counts are: + +```text +six-port: 192 +two-port: 64 +``` + +The final batch appends at most two control WQEs per lane. The maximum active +counts are therefore 194 and 66, within the fixed 256 and 128 issue capacities. +The 256-payload limit does not include these controls. + +## Control WQEs and SQ Publication + +`AppendControlWqe()` already constructs a complete WQE in the supplied issue +UB buffer. It does not write directly to SQ GM. Grant and done WQEs therefore +remain scalar and are appended after the actual payload count in each lane's +existing issue buffer. + +No dedicated control-WQE UB allocation is added. A separate four-WQE region +would consume another 256 bytes and would complicate contiguous MTE3 publication +without solving a capacity problem. + +The required publication order remains: + +1. finish operator/peer/dynamic WQE writes in UB; +2. append scalar grant and done WQEs for the final batch; +3. establish the scalar/SIMT-to-MTE3 dependency; +4. copy only active six-port and two-port WQEs from UB to their SQ rings; +5. wait for MTE3-to-scalar completion; +6. update SQ head and completion-count device fields; and +7. ring both SQ doorbells with `st_dev`. + +No SIMT function may store a WQE directly into SQ GM or ring a doorbell. + +## Self Step and Retained Code + +Self-copy stays disabled during this optimization. `SendSelfStep()` becomes: + +```cpp +__aicore__ inline bool MoonEpCombineV2::SendSelfStep(uint32_t peer) +{ + (void)peer; + return true; + +#if 0 + // Previous selection, route decode, relay, and self-copy implementation. +#endif +} +``` + +This immediate return intentionally performs no self selection, route decode, +or local copy. Correctness checks that include self-routed rows are expected to +fail until self-copy is optimized separately. Performance results from this +state must be labeled as excluding self-copy. + +The old `dstRank`, `slotIndex`, `selectedIndex`, compare mask, descriptor, and +self-copy code remains in the source under `#if 0`. Its active UB allocations +must be disabled; merely making the execution path unreachable does not recover +the UB capacity. + +## Synchronization Contract + +This kernel mixes MemBase MTE operations, scalar code, and SIMT functions. The +implementation must preserve explicit handoffs between these models: + +- MTE2 completion before the selector reads `dstSlotBuf_`; +- selector completion before scalar code reads selection state or launches the + payload builder; +- operator and peer prefill completion before the builder reuses their WQEs; +- builder completion before scalar control append or MTE3 publication; +- scalar control append completion before MTE3 publication; and +- MTE3 completion before SQ head updates and doorbells. + +The initial implementation may retain the existing broad barriers around +`Simt::VF_CALL` while correctness is established. Any later narrowing of those +barriers requires target-version evidence and hardware validation. A MemBase +pipeline event must not be assumed to provide SIMT thread visibility unless the +target CANN programming model documents that handoff. + +## Failure and Boundary Handling + +The implementation must preserve or add checks for the following when safety +checks are enabled: + +- `encoded == -1` is skipped; +- decoded peer is less than `rankSize`; +- decoded target slot is less than `slots`; +- `routeIndex` is less than + `kMoonEpCombineV2MaxSelectedPayloadWqes`; +- `batchSelected` is no greater than that same limit; +- lane payload plus control count fits the fixed issue buffer; +- SQ outstanding entries remain below the existing limit; and +- the thread cursor never resumes outside the resident chunk. + +The current trusted-input performance configuration may compile these checks +out, but the bounds remain part of the design and must be covered by host-side +or reference tests. + +## Profiling + +Existing metric meanings should be retained where possible: + +- selection load measures GM-to-UB `dstLocal` movement and its handoff; +- selection select measures SIMT peer selection and route compaction; +- remote WQE build measures dynamic payload WQE construction; +- remote submit measures scalar controls, MTE3 SQ copy, and doorbells. + +Operator and peer prefill must be distinguishable during focused profiling, +either with dedicated metrics or clearly defined inclusion in prepare/send +timelines. The old scalar descriptor metric must not silently continue to claim +that the removed descriptor path is active. + +## Validation + +Before performance testing, validation will cover: + +1. Compile-time UB and lane-capacity assertions for batch rows 128. +2. A host reference model of per-thread cursor progression and atomic ticket + allocation. +3. No-match, one-match, sparse, all-match, and invalid-entry chunks. +4. The 128P uniform pattern where one selector thread owns every match. +5. Exactly 128 matches, more than 128 matches, and the 256-entry upper bound. +6. Chunk tails smaller than the selector width. +7. Multiple selector invocations without skipped or duplicate source indices. +8. Multiple 64 KiB chunks with one peer-lifetime counter and preserved QP + sequence phase. +9. Builder distribution proving at most two WQEs per thread. +10. Six-port/two-port counts and final control capacity. +11. Source guards proving WQEs are copied from UB to SQ through MTE3 before + doorbells. +12. Source guards proving self step returns immediately and old code is + retained but inactive. +13. Compilation with `/home/pkg/b150` CANN for the target 3510 build. +14. Hardware profiling against baseline commit `80f184e` with self-copy still + excluded from both measurements. + +Correctness output involving self-routed rows is not an acceptance criterion +for this intermediate state. Remote destination-address stability, bounds, +completion behavior, and absence of hangs remain required. + +The hardware probe therefore classifies output as `passed`, +`self_only_failed`, or `failed`. The temporary +`--allow-correctness-failure` mode accepts only `self_only_failed`; any mismatch +whose source rank differs from the local rank rejects the run before timing. + +## Review Decisions Recorded + +- Keep the current multiply/divide/modulo `dstLocal` encoding. +- Store `sourceSlotIndex` and decoded `targetSlot` in each route entry. +- Use one private 32-bit maximum/last-scanned slot ID for each of 128 threads. +- Do not use atomic max for per-thread cursors. +- Keep one peer-lifetime atomic WQE counter in a 32-byte UB state. +- Default the single compile-time payload batch constant to 128. +- Bound payload selection at twice that constant, initially 256. +- Keep fixed issue capacities of 256 six-port and 128 two-port WQEs. +- Treat the 256 limit as payload only; controls are additional active WQEs. +- Keep grant and done construction scalar in the existing issue buffers. +- Keep old selection/descriptor/self-copy code under `#if 0`. +- Make `SendSelfStep()` return success immediately. diff --git a/docs/specs/2026-08-13-moonep-combine-v2-full-grant-round-design.md b/docs/specs/2026-08-13-moonep-combine-v2-full-grant-round-design.md new file mode 100644 index 0000000..0b8cb2e --- /dev/null +++ b/docs/specs/2026-08-13-moonep-combine-v2-full-grant-round-design.md @@ -0,0 +1,495 @@ +# MoonEP Combine V2 Full Grant Round Design + +## Status + +Proposed for implementation. + +This design supersedes the final-step grant statements in +`2026-08-13-moonep-combine-v2-ring-schedule-design.md`. In particular, the +final step now publishes and waits for a grant, and the grant workspace grows +from seven slots to eight slots per epoch/core/lane. + +## Goal + +Make every Combine V2 data step use the same post-send synchronization +sequence: + +1. process the step data, either through remote UDMA or the local Self-copy + path, and publish the grant for the next schedule position; +2. wait for the CQ completion generated by this step when the step submitted a + completion-producing UDMA WQE; and +3. wait for the grant that admits the next schedule position. + +For a runtime schedule with `N` data steps, the kernel must publish `N` grant +rounds and wait for `N` grant rounds. In the 128P case, this changes the +protocol from seven grant publications and waits to eight publications and +waits. + +The final grant closes the schedule ring. It does not create another data +step. After the final CQ and grant waits complete, execution proceeds directly +to inbound-done processing and reduction. + +## Non-Goals + +This change does not: + +- add an extra payload, Self-copy, or data step; +- change either single-ring or bidirectional-ring peer order; +- change QP count, QP ownership, route selection, payload construction, or + reduction; +- change the public Host API or kernel launch argument ABI; +- make the schedule mode runtime-selectable; or +- use the final grant from one invocation to admit step zero of the next + invocation. Each invocation retains its own magic value. + +## Existing Problem + +The current send loop uses a pre-admission model: + +```cpp +for (uint32_t step = 0U; step < stepCount_; ++step) { + WaitAdmission(step); + SendStep(step); +} +WaitFinalCqs(); +``` + +`WaitAdmission(0)` is a no-op. For later steps it jointly waits for the +previous CQ and the grant whose token contains the current step. A remote +step's final WQE batch publishes a grant only when: + +```cpp +step + 1U < stepCount_ +``` + +Consequently, an eight-step 128P schedule publishes grants for transitions +`0 -> 1` through `6 -> 7`, but not for the terminal transition `7 -> 0`. +There are seven grant publications and seven grant waits. Final CQs are then +handled by a separate epilogue. + +The current grant workspace mirrors that behavior with seven slots per +epoch/core/lane. + +## Terminology + +The design distinguishes data steps from grant transition rounds. + +For a runtime step count `N` and completed data step `s`: + +```cpp +nextStep = (s + 1U) % N; +transitionRound = s; +``` + +`transitionRound` identifies the data step that has just completed. Its grant +admits the source scheduled for `nextStep`. + +For 128P: + +| Data step | Transition round | Grant admits | +| ---: | ---: | ---: | +| 0 | 0 | step 1 | +| 1 | 1 | step 2 | +| 2 | 2 | step 3 | +| 3 | 3 | step 4 | +| 4 | 4 | step 5 | +| 5 | 5 | step 6 | +| 6 | 6 | step 7 | +| 7 | 7 | terminal closure to step 0 | + +The last row is a synchronization closure only. No step-zero data is sent +again in the current invocation. + +## Unified Step State Machine + +The main loop becomes a post-send synchronization loop: + +```cpp +for (uint32_t step = 0U; + step < stepCount_ && succeeded; + ++step) { + const StepSubmission submission = SendStep(step); + RecordStepSendEnd(step); + + if (submission.hasCompletion) { + succeeded = WaitStepCqs(step); + } + if (succeeded) { + succeeded = WaitStepGrant(step); + } + RecordStepReadyEnd(step); +} + +if (succeeded) { + succeeded = WaitInboundDone(); +} +``` + +`SendStep(step)` includes both data processing and publication of transition +round `step`'s grant. `WaitStepGrant(step)` waits for the same transition +round on the local rank. + +`WaitFinalCqs()` is removed. Every completion required by the current +invocation is consumed inside the corresponding loop iteration. + +The decision to wait for a CQ is based on actual UDMA submission state, not on +`peer == rank_`. This distinction is necessary because Self data has no UDMA +payload, but a Self step can still submit a remote grant WQE. + +## Remote Step Behavior + +The remote path retains payload batching. When the grant successor is remote, +the final batch of every data step, including the final data step, appends two +control WQEs per lane in this order: + +1. grant WRITE to the successor source, using strong ordering; and +2. done WRITE to the current data peer, using ordered completion. + +The ordered-completion done WQE produces one CQ completion per lane and covers +all earlier WQEs submitted on that lane for the step, including the grant WQE. +The existing per-lane issue capacity already reserves two control entries, so +the maximum `194/66` issue split does not grow. + +The fixed issue-capacity helper therefore reserves the remote-successor upper +bound: + +```cpp +return finalBatch ? 2U : 0U; +``` + +It no longer gives the final data step only one control WQE. A local successor +publishes the grant directly and consumes only the done control entry in the +UDMA batch. + +Empty-payload remote steps still submit their grant and done controls in the +final batch and therefore still produce a CQ completion. + +## Self Step Behavior + +The Self path first completes its existing local copy operation. It publishes +no remote payload and no done token, because `WaitInboundDone()` already treats +the local source as ready. + +After the local copy, it must nevertheless publish the current transition +round's grant. At the final data step in 128P, the successor is generally a +remote rank in both schedule modes. The Self path therefore submits one +grant-only WQE per lane when the successor is remote. + +Because there is no following done WQE to carry completion semantics, the +grant-only WQE uses ordered completion. The step updates `submittedHead`, +`completionCount`, and `cqTarget` exactly as other completion-producing +submissions do. `WaitStepCqs(step)` then reclaims that submission before the +step waits for its inbound grant. + +Thus, "Self has no CQ" applies only to the Self data copy itself. A Self step +that publishes a grant through UDMA does have a control-submission CQ. The +kernel must never skip a CQ merely because the scheduled peer is Self. + +`StepSubmission::hasCompletion` records this behavior explicitly. Under the +full-grant protocol, remote steps and Self steps both normally set it, but the +state remains submission-derived so a future local-only control path cannot +silently break CQ accounting. + +## Grant Successor Closure + +For a source rank `source`, data step `step`, core `core`, and schedule mode +`mode`, define: + +```cpp +nextStep = (step + 1U) % stepCount; +``` + +The successor must satisfy the cyclic invariant for every step, including the +final step: + +```cpp +EffectivePeer(source, step, core, rankSize, mode) == + EffectivePeer(successor, nextStep, core, rankSize, mode); +``` + +The sender writes its grant to `successor`. The successor waits for that grant +after completing its own current data step and before it could enter +`nextStep`. + +### Single Ring + +The existing single-ring successor direction is already cyclic. Cores 0-7 use +the previous source group and cores 8-15 use the next source group. The same +mapping also satisfies final-step-to-step-zero closure. + +For rank sizes of 2P through 8P, which have one runtime data step, the +successor is the source itself and `nextStep` is zero. + +When the cyclic successor is the local rank, the kernel publishes the grant +directly into the local receive slot. It must not submit a UDMA loopback WQE, +because local-rank UDMA completion is not a supported synchronization path. +Remote data still waits for its done completion; a Self step with a local +successor has no UDMA CQ to wait and proceeds to the common grant-token wait. + +### Bidirectional Ring + +The bidirectional successor remains offset-derived, but it must compute the +next offset using wrapped `nextStep`, not the out-of-range expression +`step + 1U`: + +```cpp +currentOffset = EffectiveOffset(step, core, source); +nextOffset = EffectiveOffset(nextStep, core, source); +successorLocal = Mod(sourceLocal + currentOffset - nextOffset, halfRankCount); +``` + +The invalid same-half peer is interpreted as effective Self with offset zero. +This is especially important at the final step, where the invalid slot must +still grant the source assigned to the same target at step zero. + +The 8P bidirectional schedule has one data step. Its cyclic successor is the +source itself. Bidirectional mode for 2P through 7P continues to fall back to +the single-ring schedule. + +## Grant Addressing + +Increase the fixed maximum grant round count from seven to eight: + +```cpp +kMoonEpCombineV2GrantStepCount = kMoonEpCombineV2StepCount; // 8 +``` + +Grant slots are indexed by the completed data step, not by a one-based +admission step: + +```cpp +GrantIndex(epoch, core, lane, transitionRound) = + ((((epoch * 16U) + core) * 2U + lane) * 8U) + transitionRound; +``` + +Valid runtime transition rounds are `[0, stepCount)`. The allocation keeps a +fixed stride of eight so epochs, cores, and lanes have stable, disjoint +regions for every supported rank size. + +Relative to the old protocol, rounds zero through six retain their meaning: +they store the grants sent after data steps zero through six. Round seven is +the newly allocated terminal slot. + +## Grant Token + +No virtual step eight is encoded. The grant written after data step `step` +uses the wrapped next-step token: + +```cpp +nextStep = (step + 1U) % stepCount; +grantToken = MoonEpCombineV2Token(magic, nextStep); +``` + +For the 128P terminal transition, this is `Token(magic, 0)`. It fits the +existing three-bit step field and does not change `kMoonEpCombineV2MaxMagic`. + +The terminal token cannot be mistaken for a stale step-zero token because it +occupies transition-round slot seven, whereas the step-zero-to-step-one token +occupies slot zero. It cannot admit the next invocation because that invocation +uses a different magic value. + +Done tokens remain unchanged and continue to encode the data step on which the +remote output was written. + +## Grant Publication and Wait Contract + +For every rank, active core, lane, and runtime data step `s`: + +1. exactly one predecessor publishes + `Token(magic, (s + 1U) % stepCount)` into the receiver's grant slot for + transition round `s`; +2. the receiver polls exactly that slot after its own data/control submission + for step `s` has completed; and +3. the receiver does not execute data step `s + 1` until the wait succeeds. + +For the final step, item 3 means only that the cyclic protocol has closed. +The loop terminates immediately after the wait, so no new data send follows. + +CQ wait and grant wait remain separate functions. They execute sequentially in +the requested order: + +```text +send data and grant -> wait local submission CQ -> wait inbound grant +``` + +Timeout diagnostics record the current data step/transition round in +`failureStep`. For a grant timeout, `failureExpected` records the wrapped +next-step token, so a terminal failure is reported as transition round seven +with expected token step zero. + +## Workspace Layout + +The grant allocation changes from: + +```text +2 epochs * 16 cores * 2 lanes * 7 rounds * 512 bytes = 229376 bytes +``` + +to: + +```text +2 epochs * 16 cores * 2 lanes * 8 rounds * 512 bytes = 262144 bytes +``` + +This adds 32768 bytes to the grant region. `controlSourceOffset`, +`failureOffset`, and `outputOffset` must be recomputed from the new grant end. +No structure field or kernel argument is added. + +The registered workspace total remains the aligned result of the normal Host +layout calculation. It must not be updated by blindly adding 32768 bytes to the +old aligned total. For the current 8192/3584/16/131072 target layout, the extra +grant bytes fit within existing 2 MiB alignment slack, so the expected +`totalBytes` remains 2879389696 while the internal offsets move. The 8/3584/16/128 +layout likewise remains 4194304 bytes. Other valid shapes must be verified by +the layout builder rather than assumed to retain their old total. + +Because the registered workspace layout changes, all ranks in one invocation +must use the same rebuilt Combine V2 library. There is no supported mixed-old- +layout/new-layout execution. + +## Profiling Semantics + +The current per-step profile points are ordered as `GRANT_END` followed by +`SEND_END`, which matches the pre-admission loop. The new execution order is +the reverse. + +Retain two cumulative points per data step but redefine them in execution +order: + +1. `STEP_SEND_END`: data processing and outbound grant publication have been + submitted; +2. `STEP_READY_END`: the step CQ, when present, and inbound grant wait have + completed. + +The final standalone CQ point becomes a step-loop completion point or is kept +as an equal timestamp immediately after the final `STEP_READY_END`. The profile +record remains 384 bytes with 22 cumulative timestamps. + +Since existing numeric time-point meanings change, increment the profile +version. Consumers must reject or explicitly decode the old and new versions +rather than assigning the old `GRANT_END`/`SEND_END` labels to new records. + +## Affected Implementation + +The expected production changes are limited to: + +1. `src/moonep/combine_v2/common/combine_v2_schedule.h` + - allocate eight grant rounds; + - index grants by completed data step; + - add wrapped next-step and cyclic successor behavior; + - make final remote batches reserve grant plus done controls; and + - retain both schedule modes and their existing peer order. +2. `src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.h` + - replace pre-step `WaitAdmission()` with post-step CQ and grant waits; + - publish a grant on every remote step; + - add grant-only ordered-completion submission after Self copy; + - derive CQ waits from submission state; + - remove the standalone final CQ epilogue; and + - enter inbound-done/reduction directly after the terminal grant wait. +3. `src/moonep/combine_v2/host/combine_v2_layout.cpp` + - derive the larger grant region and downstream offsets. +4. `src/moonep/combine_v2/common/combine_v2_profile.h` + - align per-step profile names/order with the new state machine; and + - increment the profile version. + +Required unit and source-guard changes belong under +`tests/moonep_combine_v2`. The untracked backup kernel header is not an active +build source and remains outside the implementation. + +## Required Invariants + +For every supported rank size and both schedule modes: + +1. the number of data steps remains `MoonEpCombineV2StepCount(rankSize)`; +2. every data step publishes one grant per active core and lane; +3. every data step waits for one grant per active core and lane; +4. no data, payload, or Self-copy operation occurs after the final grant wait; +5. grant slots are unique across epoch, core, lane, and transition round; +6. the cyclic successor invariant holds for all steps, including the final + step; +7. single-ring and bidirectional-ring peer tables remain unchanged; +8. remote-successor final batches contain grant followed by done on both lanes, + while local-successor grants are published directly before done submission; +9. Self steps publish grant-only ordered-completion WQEs on both lanes for a + remote successor and directly publish a local-successor grant; +10. every completion-producing submission is consumed before the next step or + kernel exit; +11. done-token production and inbound-done expectations remain unchanged; and +12. all grant tokens use the current invocation's magic. + +## Validation Plan + +### Host and Schedule Tests + +Update focused tests to cover: + +- `kMoonEpCombineV2GrantStepCount == 8`; +- exact grant index anchors, including epoch stride and terminal round seven; +- exhaustive grant-index uniqueness and bounds; +- terminal token wrapping to `Token(magic, 0)` without aliasing slot zero; +- two control WQEs for the final batch of every remote data step; +- `grantBytes == 262144` and recomputed layout offsets; +- unchanged aligned totals for the current target and small layout fixtures; +- exhaustive cyclic successor checks for all runtime steps in both ring modes; +- unchanged peer coverage, Self placement, and receive-step inversion; and +- one-step schedules, especially 8P single-ring and bidirectional-ring modes. + +### Source and Compile Checks + +Source guards or compile checks must prove that: + +- no final-step conditional suppresses grant publication; +- the main loop orders send, CQ wait, then grant wait; +- the terminal grant wait occurs inside the loop and no send follows it; +- CQ waits are keyed by submission state rather than `peer != rank_`; +- Self grant-only WQEs request ordered completion; +- the standalone `WaitFinalCqs()` path is removed; and +- all peer, receive-step, and successor calls use the selected schedule mode. + +Build the Host unit targets and the CANN 9.1.0 A5/3510 kernel target. The +existing UDMA requirements still apply: WQEs are assembled in UB, copied to SQ +through MTE3, and doorbells are rung with `st_dev` only after MTE3 completion. + +### Hardware Acceptance + +The primary acceptance case is 128P in both compile-time schedule modes. +For each mode: + +1. synchronize code to the server with Mutagen; +2. run Combine V2 correctness for hidden data and route weights; +3. confirm eight data steps, eight grant publications, and eight grant waits; +4. confirm the final grant wait is followed by inbound-done/reduction rather + than another send; +5. run repeated invocations to exercise both epochs and changing magic values; +6. confirm no CQ/SQ state is left outstanding at the next invocation; and +7. compare output with the established reference path before collecting + performance data. + +When checking server NPU availability, other `tilexr_*` tasks may share the +NPU. Python processes or other communication tests cannot share it: wait 15 +seconds and recheck, and stop the task if no compatible slot becomes available +within 120 seconds. + +## Risks + +- Wrapping only the token but not the bidirectional successor sends the + terminal grant to the wrong source and deadlocks final grant waits. +- Adding the final remote grant without adding the Self grant leaves one core + per rank unable to complete the terminal round. +- Skipping CQ solely for Self can return with a grant-only WQE still + outstanding and corrupt SQ/CQ accounting on the next invocation. +- Encoding a literal step eight aliases or exceeds the existing three-bit step + token contract. Terminal closure must encode wrapped step zero in the + distinct round-seven slot. +- Expanding the grant region without recomputing downstream offsets overlaps + control, failure, or output memory. +- Reordering execution without updating profile semantics produces valid but + mislabeled timing records. + +## Acceptance Summary + +The implementation is accepted when an `N`-step schedule has exactly `N` data +iterations and `N` post-step grant handshakes, both ring modes satisfy the +cyclic successor invariant, Self and remote paths leave no completion +outstanding, and the final handshake transitions directly to finalization with +no additional data send. diff --git a/docs/specs/2026-08-13-moonep-combine-v2-ring-schedule-design.md b/docs/specs/2026-08-13-moonep-combine-v2-ring-schedule-design.md new file mode 100644 index 0000000..607a662 --- /dev/null +++ b/docs/specs/2026-08-13-moonep-combine-v2-ring-schedule-design.md @@ -0,0 +1,467 @@ +# MoonEP Combine V2 Selectable Ring Schedule Design + +Status: Implemented; hardware validation pending + +Date: 2026-08-13 + +## Goal + +Preserve the existing MoonEP Combine V2 send schedule and add a selectable +bidirectional-ring schedule. The active schedule is selected by one +compile-time parameter in the operator, so changing that parameter and +rebuilding is sufficient to switch schedules. + +The bidirectional schedule must: + +1. support 8P, 16P, 32P, 64P, and 128P; +2. keep the current active-core and step-count contracts; +3. for rank sizes of at least 16, keep cores 0-7 responsible for the absolute + lower half of ranks and cores 8-15 responsible for the absolute upper half; +4. expand around the source rank in alternating positive and negative ring + distances; +5. represent the missing Self entry with one invalid peer slot and process + that slot through the existing Self-copy path; and +6. keep peer selection, receive-step inversion, and grant propagation + mutually consistent. + +## Non-Goals + +This change does not: + +- delete or alter the existing schedule; +- change the Host API or kernel launch ABI; +- select the schedule at runtime without rebuilding; +- change the number or ownership of QPs; +- change the 3:1 payload distribution between the two QPs owned by a core; +- change grant, done-token, failure-record, or workspace layouts; +- change route selection, payload construction, Self-copy, reduction, CQ, or + doorbell implementations; or +- guarantee a physical Fullmesh or CLOS path merely from the logical send + order. Physical paths remain determined by communicator `(peer, qp)` route + configuration. + +## Existing Contracts + +The current scheduling dimensions remain unchanged: + +| Rank size | Active cores | Step count | Scheduled slots per rank | +| ---: | ---: | ---: | ---: | +| 8 | 8 | 1 | 8 | +| 16 | 16 | 1 | 16 | +| 32 | 16 | 2 | 32 | +| 64 | 16 | 4 | 64 | +| 128 | 16 | 8 | 128 | + +For rank sizes greater than 8: + +```cpp +activeCoreCount = 16U; +stepCount = rankSize / 16U; +``` + +For 8P: + +```cpp +activeCoreCount = 8U; +stepCount = 1U; +``` + +Each core continues to own two QPs: + +```cpp +sixPortQp = core; +twoPortQp = 16U + core; +``` + +The mode changes only the `(sourceRank, step, core) -> peer` mapping and the +dependent inverse and grant mappings. + +## Schedule Mode + +Add the following type to `combine_v2_schedule.h`: + +```cpp +enum MoonEpCombineV2ScheduleMode : uint32_t { + MOONEP_COMBINE_V2_SINGLE_RING = 0U, + MOONEP_COMBINE_V2_BIDIRECTIONAL_RING = 1U, +}; + +constexpr uint32_t kMoonEpCombineV2InvalidPeer = UINT32_MAX; +``` + +Add one compile-time selection point in the active Combine V2 kernel: + +```cpp +constexpr TileXRMoonEp::MoonEpCombineV2ScheduleMode + kCombineV2ScheduleMode = + TileXRMoonEp::MOONEP_COMBINE_V2_SINGLE_RING; +``` + +Changing only this initializer and rebuilding switches between the two +schedules. The default is `SINGLE_RING`; set it to +`MOONEP_COMBINE_V2_BIDIRECTIONAL_RING` to select the new order. + +All schedule helpers receive the mode explicitly. This keeps Host unit tests +able to evaluate both modes even though the active kernel mode is currently +compile-time fixed: + +```cpp +MoonEpCombineV2Peer(sourceRank, step, core, rankSize, mode); +MoonEpCombineV2ReceiveStep(destinationRank, sourceRank, rankSize, mode); +MoonEpCombineV2Successor(sourceRank, step, core, rankSize, mode); +``` + +## Existing Single-Ring Mode + +`MOONEP_COMBINE_V2_SINGLE_RING` executes the current formulas without any +behavioral change. The existing code should be moved into a clearly named +single-ring helper or retained as the default branch of each public schedule +helper. + +The following properties remain exactly as they are today: + +- rank sizes from 2P through 8P map `peer = core`; +- 16P through 128P use the existing group-based forward/backward order; +- Self is returned as a valid peer and remains in the current final step; +- the current receive-step inverse remains in use; and +- the current step-independent successor remains in use internally for this + mode, although the public successor helper gains a `step` argument shared + with the bidirectional mode. + +## Bidirectional Mode for 16P Through 128P + +For these sizes, define: + +```cpp +halfRankCount = rankSize / 2U; +stepCount = rankSize / 16U; +targetHalf = core / 8U; // 0 for core 0-7, 1 for core 8-15 +lane = core % 8U; +ordinal = step * 8U + lane; +sourceHalf = sourceRank / halfRankCount; +sourceCenter = sourceRank % halfRankCount; +``` + +`targetHalf` is absolute rather than relative to the source rank. Therefore: + +- cores 0-7 always target ranks `[0, halfRankCount)`; and +- cores 8-15 always target ranks `[halfRankCount, rankSize)`. + +For every source rank, one core group targets the source rank's own half and +the other targets the opposite half. + +### Same-Half Ring + +The group targeting `sourceHalf` excludes Self and walks alternating positive +and negative distances: + +```text ++1, -1, +2, -2, ..., +(H/2), invalid +``` + +where `H = halfRankCount`. + +For `ordinal < H - 1`: + +```cpp +distance = ordinal / 2U + 1U; +offset = (ordinal & 1U) == 0U ? distance : -distance; +``` + +For `ordinal == H - 1`, return `kMoonEpCombineV2InvalidPeer`. This slot +represents Self and is always the final lane of the final step for that +same-half core group. + +### Opposite-Half Ring + +The group targeting the opposite half covers all `H` positions: + +```text +0, +(H/2), +1, -1, +2, -2, ..., +(H/2-1), -(H/2-1) +``` + +The offset is: + +```cpp +if (ordinal == 0U) { + offset = 0; +} else if (ordinal == 1U) { + offset = H / 2U; +} else { + pairOrdinal = ordinal - 2U; + distance = pairOrdinal / 2U + 1U; + offset = (pairOrdinal & 1U) == 0U ? distance : -distance; +} +``` + +### Target Rank + +Use signed arithmetic for the offset or an explicit modular helper. Do not +cast a negative offset to `uint32_t` before applying the ring modulus. + +Conceptually: + +```cpp +targetLocal = Mod(sourceCenter + offset, H); +peer = targetHalf * H + targetLocal; +``` + +`Mod(value, H)` must return a value in `[0, H)` for both positive and negative +inputs. + +### Invalid Slot Location + +The invalid slot belongs to the absolute half containing the source rank: + +| Rank size | Final step | Lower-half source | Upper-half source | +| ---: | ---: | --- | --- | +| 16 | 0 | core 7 | core 15 | +| 32 | 1 | core 7 | core 15 | +| 64 | 3 | core 7 | core 15 | +| 128 | 7 | core 7 | core 15 | + +After replacing the one invalid slot with Self, every source rank visits every +rank exactly once. + +## Bidirectional Mode for 8P + +8P retains one step and eight active cores. It does not split into two +simultaneous half rings because there are not 16 active cores. The mode +therefore degenerates to one bidirectional ring over all eight ranks: + +```text +core 0-7: +1, -1, +2, -2, +3, -3, +4, invalid +``` + +The invalid `core 7` slot represents Self. After substituting Self, all eight +destinations are covered once. + +For 2P through 7P, bidirectional mode falls back to the existing schedule. +This avoids introducing special odd-size diameter and invalid-slot rules for +configurations that are not part of the requested optimization. + +## Effective Peer and Self Processing + +The raw schedule helper returns `kMoonEpCombineV2InvalidPeer` for the one +bidirectional Self slot. The send loop converts it to local processing: + +```cpp +const uint32_t scheduledPeer = MoonEpCombineV2Peer( + rank_, step, core_, rankSize_, kCombineV2ScheduleMode); + +if (scheduledPeer == rank_ || + scheduledPeer == TileXRMoonEp::kMoonEpCombineV2InvalidPeer) { + succeeded = SendSelfStep(rank_); +} else { + succeeded = SendRemoteStep(scheduledPeer, step); +} +``` + +Define the effective peer used by schedule reasoning as: + +```cpp +effectivePeer = scheduledPeer == kMoonEpCombineV2InvalidPeer ? + sourceRank : scheduledPeer; +``` + +The invalid slot does not generate UDMA payload, grant, done, CQ, or doorbell +work. This is safe because: + +- it occurs only in the final step; +- there is no next-step grant to publish from it; and +- `WaitInboundDone()` already treats the local source as ready without a done + token. + +`InitLaneStates()` runs before the send loop and must not pass +`kMoonEpCombineV2InvalidPeer` into UDMA lookup. Convert an invalid first +scheduled peer to `rank_` before initializing the lane state. This matters for +16P, where the invalid slot is in step zero, and for 8P. Existing communicator +construction already provides the Self/fallback queue image needed by the +current Self-assigned core. + +## Receive-Step Inversion + +Changing only the peer formula is incorrect because inbound done-token waits +derive their expected step from `MoonEpCombineV2ReceiveStep()`. + +Single-ring mode retains the existing inverse. For bidirectional mode at 16P +through 128P, define: + +```cpp +H = rankSize / 2U; +sourceHalf = sourceRank / H; +destinationHalf = destinationRank / H; +sourceLocal = sourceRank % H; +destinationLocal = destinationRank % H; +clockwise = (destinationLocal + H - sourceLocal) % H; +distance = Min(clockwise, H - clockwise); +``` + +For a destination in the same half: + +```cpp +if (distance == 0U) { + return stepCount - 1U; // Self is represented by the final invalid slot. +} +return (distance - 1U) / 4U; +``` + +For a destination in the opposite half: + +```cpp +if (distance == H / 2U) { + return 0U; +} +return distance / 4U; +``` + +The `H/2` opposite-half diameter is deliberately placed in step zero by the +opposite-half sequence. + +For 8P there are no remote step distinctions, so receive step remains zero. +For 2P through 7P, use the existing receive-step behavior. + +## Grant Successor + +The admission protocol for step `s + 1` requires the source that sent to a +given destination in step `s` to grant the source that will send to that same +destination on the same core in step `s + 1`. + +The required invariant is: + +```cpp +EffectivePeer(sourceRank, step, core) == + EffectivePeer(successor, step + 1U, core); +``` + +The current single-ring successor remains valid for single-ring mode. + +For bidirectional mode, the successor must depend on `step`. Let +`EffectiveOffset(step, core, sourceRank)` be the offset used by the selected +target half, with the invalid same-half slot interpreted as offset zero. Then: + +```cpp +H = rankSize / 2U; +sourceHalfBase = (sourceRank / H) * H; +sourceLocal = sourceRank % H; +currentOffset = EffectiveOffset(step, core, sourceRank); +nextOffset = EffectiveOffset(step + 1U, core, sourceRank); + +successorLocal = Mod( + sourceLocal + currentOffset - nextOffset, H); +successor = sourceHalfBase + successorLocal; +``` + +The successor stays in the same source half. This is required because, for a +fixed absolute target-half core group, sources from the same half share the +same offset family. + +The helper is called only when `step + 1U < stepCount`. No successor is needed +after the final step. Grant indices, grant tokens, and grant workspace sizing +remain unchanged. + +## Affected Implementation + +The intended production-code changes are limited to: + +1. `src/moonep/combine_v2/common/combine_v2_schedule.h` + - add the schedule mode and invalid-peer constant; + - retain the existing schedule as the single-ring path; + - add bidirectional peer and offset helpers; + - add the mode parameter to peer and receive-step helpers; + - add `step` and mode parameters to the successor helper; and + - implement the generalized 8P/16P/32P/64P/128P rules. +2. `src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.h` + - add the one compile-time mode parameter; + - pass it to schedule helpers; + - convert an invalid first peer before lane initialization; + - dispatch an invalid send peer to `SendSelfStep(rank_)`; and + - pass `step` to successor calculation. + +The experimental untracked +`src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel_back.h` is not +an active implementation source and is outside the change. + +No Host layout or launch changes are required. + +## Required Invariants + +For every supported rank and both schedule modes: + +1. every effective peer is in `[0, rankSize)`; +2. the complete `(step, core)` schedule covers every destination exactly once; +3. Self occurs exactly once; +4. raw single-ring peers are always valid; +5. raw bidirectional peers contain exactly one invalid slot for 8P, 16P, 32P, + 64P, and 128P; +6. bidirectional mode for 2P through 7P contains no new invalid behavior; +7. `ReceiveStep(EffectivePeer(...), sourceRank, ...) == step` for remote peers; +8. the grant successor invariant holds for every non-final step and core; +9. cores 0-7 target the lower absolute half and cores 8-15 target the upper + absolute half for rank sizes of at least 16; and +10. changing the compile-time mode does not change QP indices, workspace + bytes, launch arguments, or public ABI. + +Self is excluded from the receive-step round-trip assertion because +`WaitInboundDone()` does not wait on a local done token. Its conceptual step is +the final step in bidirectional mode. + +## Validation Plan + +Update `tests/moonep_combine_v2/unit/test_combine_v2_schedule.cpp` to evaluate +both modes independently. + +Required unit coverage: + +1. Exact peer tables for representative rank 0 and a nonzero center rank for + 8P, 32P, 64P, and 128P. +2. Exhaustive coverage and uniqueness over every source rank for 8P, 16P, + 32P, 64P, and 128P. +3. Proof that single-ring output is unchanged for every currently supported + rank size. +4. Exactly one invalid raw bidirectional peer and exactly one effective Self + peer for each optimized rank size. +5. Exhaustive receive-step inversion for every remote peer. +6. Exhaustive grant successor checks for every non-final step and core. +7. Absolute target-half ownership checks for cores 0-7 and 8-15. +8. A source guard or compile check proving every active schedule call supplies + the selected mode. +9. Focused Host unit build and test execution. +10. Target CANN kernel compilation. + +Hardware acceptance should compare both modes with identical rank placement, +shape, warmup, iteration count, communicator route configuration, and CANN +build. Correctness must pass at each tested size before performance results are +compared. Server synchronization must use Mutagen, and NPU occupancy handling +must follow the repository `AGENTS.md` rules. + +## Risks and Boundaries + +- A peer-only change would mismatch receive done-token steps and can hang. +- A peer-and-receive-only change would send grants to the wrong next source + and can hang from step one onward. +- The invalid peer must be consumed before any UDMA table lookup. +- 8P does not provide two half rings; it provides a single bidirectional ring + using the same selectable mode. +- Logical ordering does not prove physical Fullmesh/CLOS selection. Route + configuration and hardware profiling are required for that conclusion. +- Performance may differ by rank size even when schedule correctness is + identical, so both modes remain available for comparative hardware + validation. + +## Recorded Decisions + +- Preserve the current schedule as `SINGLE_RING`. +- Add `BIDIRECTIONAL_RING` without changing public APIs. +- Select the mode with one compile-time constant in the operator. +- Generalize the bidirectional schedule to 8P, 16P, 32P, 64P, and 128P. +- Fall back to the existing schedule for 2P through 7P. +- Preserve absolute lower-half/upper-half ownership for the two eight-core + groups at 16P and above. +- Represent Self with one invalid peer in the final same-half slot. +- Reuse the existing Self-copy path for that invalid slot. +- Keep QP count and ownership unchanged. +- Update receive-step inversion and grant successor together with peer order. +- Require `step` in successor calculation for bidirectional mode. +- Use `SINGLE_RING` as the compile-time default while retaining + `BIDIRECTIONAL_RING` as a one-line selectable alternative. diff --git a/src/moonep/combine_v2/common/combine_v2_profile.h b/src/moonep/combine_v2/common/combine_v2_profile.h index 5d87175..cd9123e 100644 --- a/src/moonep/combine_v2/common/combine_v2_profile.h +++ b/src/moonep/combine_v2/common/combine_v2_profile.h @@ -6,7 +6,7 @@ namespace TileXRMoonEp { constexpr uint32_t kMoonEpCombineV2ProfileMarker = 0x54584750U; // TXGP -constexpr uint16_t kMoonEpCombineV2ProfileVersion = 1U; +constexpr uint16_t kMoonEpCombineV2ProfileVersion = 3U; constexpr uint32_t kMoonEpCombineV2ProfileCyclesPerUs = 1000U; constexpr uint32_t kMoonEpCombineV2ProfileTimePointCapacity = 32U; constexpr uint32_t kMoonEpCombineV2ProfileStepCount = 8U; @@ -16,14 +16,14 @@ enum MoonEpCombineV2ProfileTimePointIndex : uint32_t { MOONEP_COMBINE_V2_TIME_INIT_BEGIN = 0U, MOONEP_COMBINE_V2_TIME_INIT_END = 1U, MOONEP_COMBINE_V2_TIME_PREPARE_END = 2U, - MOONEP_COMBINE_V2_TIME_STEP0_GRANT_END = 3U, - MOONEP_COMBINE_V2_TIME_STEP0_SEND_END = 4U, - MOONEP_COMBINE_V2_TIME_FINAL_CQ_END = - MOONEP_COMBINE_V2_TIME_STEP0_GRANT_END + + MOONEP_COMBINE_V2_TIME_STEP0_SEND_END = 3U, + MOONEP_COMBINE_V2_TIME_STEP0_READY_END = 4U, + MOONEP_COMBINE_V2_TIME_STEP_LOOP_END = + MOONEP_COMBINE_V2_TIME_STEP0_SEND_END + kMoonEpCombineV2ProfileStepCount * kMoonEpCombineV2ProfileStepPointStride, MOONEP_COMBINE_V2_TIME_INBOUND_DONE_END = - MOONEP_COMBINE_V2_TIME_FINAL_CQ_END + 1U, + MOONEP_COMBINE_V2_TIME_STEP_LOOP_END + 1U, MOONEP_COMBINE_V2_TIME_FINAL_END = MOONEP_COMBINE_V2_TIME_INBOUND_DONE_END + 1U, }; @@ -31,6 +31,19 @@ enum MoonEpCombineV2ProfileTimePointIndex : uint32_t { constexpr uint32_t kMoonEpCombineV2ProfileTimePointCount = MOONEP_COMBINE_V2_TIME_FINAL_END + 1U; +enum MoonEpCombineV2ProfileMetricIndex : uint32_t { + MOONEP_COMBINE_V2_METRIC_SELECTION_LOAD = 0U, + MOONEP_COMBINE_V2_METRIC_SELECTION_SELECT, + MOONEP_COMBINE_V2_METRIC_SELF_ROUTE_DECODE, + MOONEP_COMBINE_V2_METRIC_SELF_COPY, + MOONEP_COMBINE_V2_METRIC_REMOTE_ROUTE_DECODE, + MOONEP_COMBINE_V2_METRIC_REMOTE_DESCRIPTOR, + MOONEP_COMBINE_V2_METRIC_REMOTE_WQE_BUILD, + MOONEP_COMBINE_V2_METRIC_REMOTE_SUBMIT, +}; + +constexpr uint32_t kMoonEpCombineV2ProfileMetricCount = 8U; + enum MoonEpCombineV2ProfileDiagnosticIndex : uint32_t { MOONEP_COMBINE_V2_DIAG_FAILURE_STATUS = kMoonEpCombineV2ProfileTimePointCount, @@ -51,7 +64,10 @@ struct alignas(64) MoonEpCombineV2ProfileRecord { uint32_t core; uint32_t blockDim; uint32_t timePointCount; + uint32_t metricCount; + uint32_t reserved; int64_t timePoint[kMoonEpCombineV2ProfileTimePointCapacity]; + uint64_t metric[kMoonEpCombineV2ProfileMetricCount]; }; static_assert(kMoonEpCombineV2ProfileTimePointCount == 22U, @@ -59,7 +75,10 @@ static_assert(kMoonEpCombineV2ProfileTimePointCount == 22U, static_assert(MOONEP_COMBINE_V2_DIAG_OBSERVED < kMoonEpCombineV2ProfileTimePointCapacity, "Combine V2 diagnostics exceed the profile record"); -static_assert(sizeof(MoonEpCombineV2ProfileRecord) == 320U, +static_assert(MOONEP_COMBINE_V2_METRIC_REMOTE_SUBMIT + 1U == + kMoonEpCombineV2ProfileMetricCount, + "Combine V2 profile metric count mismatch"); +static_assert(sizeof(MoonEpCombineV2ProfileRecord) == 384U, "Combine V2 profile record ABI changed"); } // namespace TileXRMoonEp diff --git a/src/moonep/combine_v2/common/combine_v2_wqe_batch.h b/src/moonep/combine_v2/common/combine_v2_wqe_batch.h index 068d440..1d3af74 100644 --- a/src/moonep/combine_v2/common/combine_v2_wqe_batch.h +++ b/src/moonep/combine_v2/common/combine_v2_wqe_batch.h @@ -11,9 +11,80 @@ namespace TileXRMoonEp { -constexpr uint32_t kMoonEpCombineV2WqeBatchCapacity = 128U; +constexpr uint32_t kMoonEpCombineV2PayloadBatchRows = 128U; +constexpr uint32_t kMoonEpCombineV2MaxSelectorThreads = 128U; +constexpr uint32_t kMoonEpCombineV2SelectorThreads = + kMoonEpCombineV2PayloadBatchRows; +constexpr uint32_t kMoonEpCombineV2BuilderThreads = + kMoonEpCombineV2PayloadBatchRows; +constexpr uint32_t kMoonEpCombineV2MaxSelectedPayloadWqes = + 2U * kMoonEpCombineV2PayloadBatchRows; +constexpr uint32_t kMoonEpCombineV2WqeBatchCapacity = + kMoonEpCombineV2PayloadBatchRows; constexpr uint32_t kMoonEpCombineV2BatchQpCount = 2U; constexpr uint32_t kMoonEpCombineV2QpSplitPeriod = 4U; +constexpr uint32_t kMoonEpCombineV2SelfRelayHalfBytes = 64U * 1024U; +constexpr uint32_t kMoonEpCombineV2SelfMaxBatchRows = 8U; +constexpr uint32_t kMoonEpCombineV2SelfUbAlignment = 32U; + +static_assert(kMoonEpCombineV2PayloadBatchRows > 0U && + kMoonEpCombineV2PayloadBatchRows <= + kMoonEpCombineV2MaxSelectorThreads, + "Combine V2 payload batch rows must be in [1, 128]"); +static_assert(kMoonEpCombineV2SelectorThreads == + kMoonEpCombineV2BuilderThreads, + "Combine V2 selector and builder widths must match"); + +TILEXR_MOONEP_COMBINE_V2_WQE_BATCH_INLINE uint64_t +MoonEpCombineV2SelfAlignedRowBytes(uint64_t rowBytes) +{ + if (rowBytes == 0U || rowBytes > + UINT64_MAX - (kMoonEpCombineV2SelfUbAlignment - 1U)) { + return 0U; + } + return (rowBytes + kMoonEpCombineV2SelfUbAlignment - 1U) / + kMoonEpCombineV2SelfUbAlignment * + kMoonEpCombineV2SelfUbAlignment; +} + +TILEXR_MOONEP_COMBINE_V2_WQE_BATCH_INLINE uint32_t +MoonEpCombineV2SelfRowsPerBatch(uint64_t rowBytes) +{ + const uint64_t alignedRowBytes = + MoonEpCombineV2SelfAlignedRowBytes(rowBytes); + if (alignedRowBytes == 0U || + alignedRowBytes > kMoonEpCombineV2SelfRelayHalfBytes) { + return 0U; + } + const uint32_t rows = static_cast( + kMoonEpCombineV2SelfRelayHalfBytes / alignedRowBytes); + return rows < kMoonEpCombineV2SelfMaxBatchRows ? + rows : kMoonEpCombineV2SelfMaxBatchRows; +} + +TILEXR_MOONEP_COMBINE_V2_WQE_BATCH_INLINE uint64_t +MoonEpCombineV2SelfTileCount(uint64_t rowBytes) +{ + if (rowBytes == 0U) { + return 0U; + } + return rowBytes / kMoonEpCombineV2SelfRelayHalfBytes + + (rowBytes % kMoonEpCombineV2SelfRelayHalfBytes == 0U ? 0U : 1U); +} + +TILEXR_MOONEP_COMBINE_V2_WQE_BATCH_INLINE uint32_t +MoonEpCombineV2SelfTileBytes(uint64_t rowBytes, uint64_t tileIndex) +{ + if (tileIndex >= MoonEpCombineV2SelfTileCount(rowBytes)) { + return 0U; + } + const uint64_t tileOffset = + tileIndex * kMoonEpCombineV2SelfRelayHalfBytes; + const uint64_t remaining = rowBytes - tileOffset; + return static_cast( + remaining < kMoonEpCombineV2SelfRelayHalfBytes ? remaining : + kMoonEpCombineV2SelfRelayHalfBytes); +} enum MoonEpCombineV2SingleCqeResult : uint32_t { MOONEP_COMBINE_V2_SINGLE_CQE_NO_COMPLETION = 0U, @@ -41,6 +112,27 @@ TILEXR_MOONEP_COMBINE_V2_WQE_BATCH_INLINE uint32_t MoonEpCombineV2WqeBatchCount( return static_cast(batchCount); } +TILEXR_MOONEP_COMBINE_V2_WQE_BATCH_INLINE uint32_t +MoonEpCombineV2SelectorFirstIndex(uint32_t chunkStart, uint32_t threadId) +{ + return chunkStart + threadId; +} + +TILEXR_MOONEP_COMBINE_V2_WQE_BATCH_INLINE uint32_t +MoonEpCombineV2SelectorResumeIndex(uint32_t lastScannedIndex) +{ + return lastScannedIndex + kMoonEpCombineV2SelectorThreads; +} + +TILEXR_MOONEP_COMBINE_V2_WQE_BATCH_INLINE bool +MoonEpCombineV2SelectorIndexInChunk(uint32_t absoluteIndex, + uint32_t chunkStart, uint32_t chunkElements) +{ + return absoluteIndex >= chunkStart && + static_cast(absoluteIndex) < + static_cast(chunkStart) + chunkElements; +} + TILEXR_MOONEP_COMBINE_V2_WQE_BATCH_INLINE uint32_t MoonEpCombineV2CqePollBatchCount( uint32_t cqTail, uint32_t cqEntryCount, uint32_t batchCapacity) { diff --git a/src/moonep/combine_v2/host/tilexr_moonep_combine_v2.cpp b/src/moonep/combine_v2/host/tilexr_moonep_combine_v2.cpp index 008511b..0baeaed 100644 --- a/src/moonep/combine_v2/host/tilexr_moonep_combine_v2.cpp +++ b/src/moonep/combine_v2/host/tilexr_moonep_combine_v2.cpp @@ -49,7 +49,7 @@ extern "C" int TileXRMoonEpCombineV2(void *registeredWorkspace, params.aivCoreNum = aivCoreNum; params.activeOutputOffset = activeOutputOffset; params.dtype = dtype; - params.reduceHidden = dtype == TILEXR_MOONEP_DTYPE_BFLOAT16; + params.reduceHidden = false; params.stream = stream; return TileXRMoonEp::TileXRMoonEpRunCombineV2(params); } diff --git a/src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.h b/src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.h index 6137c39..a48c747 100644 --- a/src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.h +++ b/src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.h @@ -11,6 +11,7 @@ #include "comm_args.h" #include "combine_v2_profile.h" #include "kernel_operator.h" +#include "simt_api/device_atomic_functions.h" #include "combine_v2_schedule.h" #include "combine_v2_wqe_batch.h" #include "tilexr_udma.h" @@ -21,18 +22,33 @@ using namespace AscendC; constexpr uint32_t kUbAlignBytes = 32U; constexpr uint32_t kFullUbBytes = 216U * 1024U; -constexpr uint32_t kRelayBytes = 4U * 1024U; constexpr uint32_t kWqeBytes = 64U; -constexpr uint32_t kSixPortFinalCapacity = - TileXRMoonEp::kMoonEpCombineV2SixPortRows + 2U; -constexpr uint32_t kTwoPortFinalCapacity = - TileXRMoonEp::kMoonEpCombineV2TwoPortRows + 2U; -constexpr uint32_t kSixPortIssueBytes = kSixPortFinalCapacity * kWqeBytes; -constexpr uint32_t kTwoPortIssueBytes = kTwoPortFinalCapacity * kWqeBytes; -constexpr uint32_t kDescriptorBytes = - TileXRMoonEp::kMoonEpCombineV2LogicalBatchRows * 2U * sizeof(uint32_t); -constexpr uint32_t kBuildContextBytes = 256U; -constexpr uint32_t kWqeBuildThreads = 64U; +constexpr uint32_t kControlWqesPerLane = 2U; +constexpr uint32_t kSixPortPayloadCapacity = 192U; +constexpr uint32_t kTwoPortPayloadCapacity = 64U; +constexpr uint32_t kSixPortIssueCapacity = 194U; +constexpr uint32_t kTwoPortIssueCapacity = 66U; +constexpr uint32_t kTotalIssueCapacity = + kSixPortIssueCapacity + kTwoPortIssueCapacity; +constexpr uint32_t kSixPortIssueBytes = kSixPortIssueCapacity * kWqeBytes; +constexpr uint32_t kTwoPortIssueBytes = kTwoPortIssueCapacity * kWqeBytes; +constexpr uint32_t kTotalIssueBytes = kTotalIssueCapacity * kWqeBytes; +constexpr uint32_t kRouteEntryCapacity = + 2U * TileXRMoonEp::kMoonEpCombineV2MaxSelectorThreads; +constexpr uint32_t kThreadCursorCapacity = + TileXRMoonEp::kMoonEpCombineV2MaxSelectorThreads; +constexpr uint32_t kDstSlotBytes = + TileXRMoonEp::kMoonEpCombineV2SelectionChunkRows * sizeof(int32_t); +constexpr uint32_t kRouteEntryBytes = + kRouteEntryCapacity * 2U * sizeof(uint32_t); +constexpr uint32_t kThreadCursorBytes = + kThreadCursorCapacity * sizeof(uint32_t); +constexpr uint32_t kSelectionStateBytes = 32U; +constexpr uint32_t kWqeContextBytes = 256U; +constexpr uint32_t kSendBufferBytes = kDstSlotBytes + kTotalIssueBytes + + kRouteEntryBytes + kThreadCursorBytes + kSelectionStateBytes + + kWqeContextBytes + + 2U * TileXRMoonEp::kMoonEpCombineV2SelfRelayHalfBytes; constexpr uint32_t kReduceRouteBatch = 4U; constexpr uint32_t kReduceTileElements = 4096U; constexpr uint64_t kOperationTimeoutCycles = 10000000000ULL; @@ -40,7 +56,10 @@ constexpr uint32_t kPollNoCompletion = UINT32_MAX; constexpr uint32_t kPollInvalidState = UINT32_MAX - 1U; // Trusted-input benchmark mode compiles defensive validation and diagnostics // out of the transfer hot path. -constexpr bool kEnableSafetyChecks = true; +constexpr bool kEnableSafetyChecks = false; +constexpr TileXRMoonEp::MoonEpCombineV2ScheduleMode + kCombineV2ScheduleMode = + TileXRMoonEp::MOONEP_COMBINE_V2_SINGLE_RING; static_assert(sizeof(TileXR::UDMASqeCtx) + sizeof(TileXR::UDMASgeCtx) == kWqeBytes, "Combine V2 WRITE WQE must occupy one basic block"); @@ -49,6 +68,26 @@ static_assert(sizeof(TileXR::UDMACqeCtx) == 64U, static_assert(TileXRMoonEp::kMoonEpCombineV2ProfileStepCount == TileXRMoonEp::kMoonEpCombineV2StepCount, "Combine V2 profile must cover every schedule step"); +static_assert(TileXRMoonEp::kMoonEpCombineV2MaxSelectedPayloadWqes % 4U == 0U, + "Combine V2 maximum payload count must preserve the 3:1 QP split"); +static_assert(TileXRMoonEp::kMoonEpCombineV2MaxSelectedPayloadWqes * 3U / 4U == + kSixPortPayloadCapacity, + "Combine V2 six-port payload capacity must preserve the 3:1 QP split"); +static_assert(TileXRMoonEp::kMoonEpCombineV2MaxSelectedPayloadWqes / 4U == + kTwoPortPayloadCapacity, + "Combine V2 two-port payload capacity must preserve the 3:1 QP split"); +static_assert(kSixPortPayloadCapacity + kControlWqesPerLane == + kSixPortIssueCapacity, + "Combine V2 six-port issue buffer cannot hold payload plus controls"); +static_assert(kTwoPortPayloadCapacity + kControlWqesPerLane == + kTwoPortIssueCapacity, + "Combine V2 two-port issue buffer cannot hold payload plus controls"); +static_assert(kTotalIssueCapacity == 260U, + "Combine V2 issue buffer must contain 256 payload and four controls"); +static_assert(kSendBufferBytes == 216096U, + "Combine V2 send buffer layout must occupy 216096 bytes"); +static_assert(kFullUbBytes - kSendBufferBytes == 5088U, + "Combine V2 send buffer layout must leave 5088 bytes of UB"); template __aicore__ inline void SyncFunc() { @@ -73,11 +112,22 @@ __aicore__ inline bool TimedOut(uint64_t startCycles) kOperationTimeoutCycles; } -struct MoonEpCombineV2Descriptor { - uint32_t sourceSlot; +struct MoonEpCombineV2RouteEntry { + uint32_t sourceSlotIndex; uint32_t targetSlot; }; +struct alignas(32) MoonEpCombineV2SelectState { + uint32_t curWqeNum; + uint32_t batchBase; + uint32_t batchSelected; + uint32_t pausedThreadCount; + uint32_t reserved[4]; +}; + +static_assert(sizeof(MoonEpCombineV2SelectState) == kSelectionStateBytes, + "Combine V2 selector state must occupy 32 bytes"); + struct MoonEpCombineV2RemoteFields { uint64_t remoteRowBase; uint64_t rmtEidL; @@ -90,18 +140,32 @@ struct MoonEpCombineV2RemoteFields { uint32_t rmtTokenValue; }; +struct alignas(32) MoonEpCombineV2OperatorFields { + uint64_t rowBytes; + uint64_t reserved[3]; +}; + +struct alignas(32) MoonEpCombineV2PeerFields { + uint64_t rowBytes; + uint64_t reserved[3]; + MoonEpCombineV2RemoteFields remote[ + TileXRMoonEp::kMoonEpCombineV2LaneCount]; +}; + struct alignas(32) MoonEpCombineV2BuildContext { uint64_t localRowBase; uint64_t rowBytes; - uint32_t logicalCount; + uint64_t remoteRowBase[TileXRMoonEp::kMoonEpCombineV2LaneCount]; + uint32_t selectedCount; uint32_t sequencePhase; uint32_t head[TileXRMoonEp::kMoonEpCombineV2LaneCount]; - uint32_t laneCount[TileXRMoonEp::kMoonEpCombineV2LaneCount]; - MoonEpCombineV2RemoteFields remote[ - TileXRMoonEp::kMoonEpCombineV2LaneCount]; }; -static_assert(sizeof(MoonEpCombineV2BuildContext) <= kBuildContextBytes, +static_assert(sizeof(MoonEpCombineV2OperatorFields) <= kWqeContextBytes, + "Combine V2 operator fields exceed their UB allocation"); +static_assert(sizeof(MoonEpCombineV2PeerFields) <= kWqeContextBytes, + "Combine V2 peer fields exceed their UB allocation"); +static_assert(sizeof(MoonEpCombineV2BuildContext) <= kWqeContextBytes, "Combine V2 build context exceeds its UB allocation"); struct MoonEpCombineV2LaneState { @@ -117,14 +181,145 @@ struct MoonEpCombineV2LaneState { }; #if defined(CATLASS_ARCH) && CATLASS_ARCH == 3510 -__simt_vf__ __aicore__ LAUNCH_BOUND(kWqeBuildThreads) -inline void MoonEpCombineV2BuildPayloadVf( +__simt_vf__ __aicore__ LAUNCH_BOUND( + TileXRMoonEp::kMoonEpCombineV2BuilderThreads) +inline void MoonEpCombineV2PrefillOperatorWqesVf( __ubuf__ uint8_t *sixPortWqes, __ubuf__ uint8_t *twoPortWqes, - __ubuf__ const MoonEpCombineV2Descriptor *descriptors, + __ubuf__ const MoonEpCombineV2OperatorFields *fields) +{ + constexpr uint32_t totalCapacity = + kSixPortPayloadCapacity + kTwoPortPayloadCapacity; + for (uint32_t task = static_cast(threadIdx.x); + task < totalCapacity; + task += TileXRMoonEp::kMoonEpCombineV2BuilderThreads) { + __ubuf__ uint8_t *wqe = task < kSixPortPayloadCapacity ? + sixPortWqes + task * kWqeBytes : + twoPortWqes + (task - kSixPortPayloadCapacity) * kWqeBytes; + __ubuf__ uint32_t *words = + reinterpret_cast<__ubuf__ uint32_t *>(wqe); + for (uint32_t word = 0U; word < kWqeBytes / sizeof(uint32_t); + ++word) { + words[word] = 0U; + } + __ubuf__ TileXR::UDMASqeCtx *sqe = + reinterpret_cast<__ubuf__ TileXR::UDMASqeCtx *>(wqe); + sqe->opcode = static_cast(TileXR::UDMAOpcode::WRITE); + sqe->flag = 0U; + sqe->nf = 0U; + sqe->inlineMsgLen = 0U; + sqe->sgeNum = 1U; + __ubuf__ TileXR::UDMASgeCtx *sge = + reinterpret_cast<__ubuf__ TileXR::UDMASgeCtx *>( + wqe + sizeof(TileXR::UDMASqeCtx)); + sge->len = static_cast(fields->rowBytes); + sge->tokenId = 0U; + } +} + +__simt_vf__ __aicore__ LAUNCH_BOUND( + TileXRMoonEp::kMoonEpCombineV2BuilderThreads) +inline void MoonEpCombineV2PrefillPeerWqesVf( + __ubuf__ uint8_t *sixPortWqes, __ubuf__ uint8_t *twoPortWqes, + __ubuf__ const MoonEpCombineV2PeerFields *fields) +{ + constexpr uint32_t totalCapacity = + kSixPortPayloadCapacity + kTwoPortPayloadCapacity; + for (uint32_t task = static_cast(threadIdx.x); + task < totalCapacity; + task += TileXRMoonEp::kMoonEpCombineV2BuilderThreads) { + const uint32_t lane = task < kSixPortPayloadCapacity ? + TileXRMoonEp::MOONEP_COMBINE_V2_SIX_PORT : + TileXRMoonEp::MOONEP_COMBINE_V2_TWO_PORT; + const uint32_t laneIndex = task < kSixPortPayloadCapacity ? task : + task - kSixPortPayloadCapacity; + __ubuf__ uint8_t *wqe = + (lane == TileXRMoonEp::MOONEP_COMBINE_V2_SIX_PORT ? + sixPortWqes : twoPortWqes) + laneIndex * kWqeBytes; + __ubuf__ uint32_t *words = + reinterpret_cast<__ubuf__ uint32_t *>(wqe); + for (uint32_t word = 0U; word < kWqeBytes / sizeof(uint32_t); + ++word) { + words[word] = 0U; + } + __ubuf__ TileXR::UDMASqeCtx *sqe = + reinterpret_cast<__ubuf__ TileXR::UDMASqeCtx *>(wqe); + sqe->opcode = static_cast(TileXR::UDMAOpcode::WRITE); + sqe->flag = 0U; + sqe->nf = 0U; + sqe->tokenEn = fields->remote[lane].tokenEn; + sqe->rmtJettyType = fields->remote[lane].rmtJettyType; + sqe->targetHint = fields->remote[lane].targetHint; + sqe->inlineMsgLen = 0U; + sqe->tpId = fields->remote[lane].tpId; + sqe->sgeNum = 1U; + sqe->rmtJettyOrSegId = fields->remote[lane].rmtJettyOrSegId; + sqe->rmtTokenValue = fields->remote[lane].rmtTokenValue; + sqe->rmtEidL = fields->remote[lane].rmtEidL; + sqe->rmtEidH = fields->remote[lane].rmtEidH; + __ubuf__ TileXR::UDMASgeCtx *sge = + reinterpret_cast<__ubuf__ TileXR::UDMASgeCtx *>( + wqe + sizeof(TileXR::UDMASqeCtx)); + sge->len = static_cast(fields->rowBytes); + sge->tokenId = 0U; + } +} + +__simt_vf__ __aicore__ LAUNCH_BOUND( + TileXRMoonEp::kMoonEpCombineV2SelectorThreads) +inline void MoonEpCombineV2SelectPeerRoutesVf( + __ubuf__ const int32_t *dstSlots, uint32_t chunkStart, + uint32_t chunkElements, uint32_t peer, uint64_t slots, + bool firstPass, __ubuf__ MoonEpCombineV2SelectState *state, + __ubuf__ uint32_t *threadMaxSlotIdx, + __ubuf__ MoonEpCombineV2RouteEntry *routes) +{ + const uint32_t thread = static_cast(threadIdx.x); + uint32_t index = firstPass ? + TileXRMoonEp::MoonEpCombineV2SelectorFirstIndex(chunkStart, thread) : + TileXRMoonEp::MoonEpCombineV2SelectorResumeIndex( + threadMaxSlotIdx[thread]); + uint32_t lastScanned = index; + while (TileXRMoonEp::MoonEpCombineV2SelectorIndexInChunk( + index, chunkStart, chunkElements)) { + lastScanned = index; + const int32_t encoded = dstSlots[index - chunkStart]; + if (encoded >= 0 && + static_cast(static_cast(encoded) / slots) == + peer) { + const uint32_t old = asc_atomic_add(&state->curWqeNum, 1U); + const uint32_t routeIndex = old - state->batchBase; + if (routeIndex < kRouteEntryCapacity) { + routes[routeIndex].sourceSlotIndex = index; + routes[routeIndex].targetSlot = static_cast( + static_cast(encoded) % slots); + } + if (routeIndex >= + TileXRMoonEp::kMoonEpCombineV2PayloadBatchRows) { + threadMaxSlotIdx[thread] = lastScanned; + const uint32_t next = + TileXRMoonEp::MoonEpCombineV2SelectorResumeIndex(index); + if (TileXRMoonEp::MoonEpCombineV2SelectorIndexInChunk( + next, chunkStart, chunkElements)) { + (void)asc_atomic_add(&state->pausedThreadCount, 1U); + } + return; + } + } + index = TileXRMoonEp::MoonEpCombineV2SelectorResumeIndex(index); + } + threadMaxSlotIdx[thread] = lastScanned; +} + +__simt_vf__ __aicore__ LAUNCH_BOUND( + TileXRMoonEp::kMoonEpCombineV2BuilderThreads) +inline void MoonEpCombineV2BuildPayloadWqesVf( + __ubuf__ uint8_t *sixPortWqes, __ubuf__ uint8_t *twoPortWqes, + __ubuf__ const MoonEpCombineV2RouteEntry *routes, __ubuf__ const MoonEpCombineV2BuildContext *context) { for (uint32_t task = static_cast(threadIdx.x); - task < context->logicalCount; task += kWqeBuildThreads) { + task < context->selectedCount; + task += TileXRMoonEp::kMoonEpCombineV2BuilderThreads) { const uint32_t position = context->sequencePhase + task; const uint32_t lane = TileXRMoonEp::MoonEpCombineV2LaneForPosition(position); @@ -138,76 +333,31 @@ inline void MoonEpCombineV2BuildPayloadVf( __ubuf__ uint8_t *wqe = (lane == TileXRMoonEp::MOONEP_COMBINE_V2_SIX_PORT ? sixPortWqes : twoPortWqes) + laneIndex * kWqeBytes; - __ubuf__ uint32_t *words = - reinterpret_cast<__ubuf__ uint32_t *>(wqe); - for (uint32_t word = 0U; word < kWqeBytes / sizeof(uint32_t); - ++word) { - words[word] = 0U; - } - const uint64_t localAddr = context->localRowBase + - static_cast(descriptors[task].sourceSlot) * + static_cast(routes[task].sourceSlotIndex) * context->rowBytes; - const uint64_t remoteAddr = context->remote[lane].remoteRowBase + - static_cast(descriptors[task].targetSlot) * + const uint64_t remoteAddr = context->remoteRowBase[lane] + + static_cast(routes[task].targetSlot) * context->rowBytes; const uint32_t absoluteHead = context->head[lane] + laneIndex; __ubuf__ TileXR::UDMASqeCtx *sqe = reinterpret_cast<__ubuf__ TileXR::UDMASqeCtx *>(wqe); - sqe->opcode = static_cast(TileXR::UDMAOpcode::WRITE); - sqe->flag = 0U; - sqe->nf = 0U; - sqe->tokenEn = context->remote[lane].tokenEn; - sqe->rmtJettyType = context->remote[lane].rmtJettyType; sqe->sqeBbIdx = static_cast( absoluteHead % TileXR::TILEXR_UDMA_SQ_BB_COUNT); sqe->owner = (absoluteHead & TileXR::TILEXR_UDMA_SQ_BB_COUNT) == 0U ? 1U : 0U; - sqe->targetHint = context->remote[lane].targetHint; - sqe->inlineMsgLen = 0U; - sqe->tpId = context->remote[lane].tpId; - sqe->sgeNum = 1U; - sqe->rmtJettyOrSegId = context->remote[lane].rmtJettyOrSegId; - sqe->rmtTokenValue = context->remote[lane].rmtTokenValue; - sqe->rmtEidL = context->remote[lane].rmtEidL; - sqe->rmtEidH = context->remote[lane].rmtEidH; sqe->rmtAddrLOrTokenId = remoteAddr & 0xFFFFFFFFU; sqe->rmtAddrHOrTokenValue = (remoteAddr >> 32U) & 0xFFFFFFFFU; __ubuf__ TileXR::UDMASgeCtx *sge = reinterpret_cast<__ubuf__ TileXR::UDMASgeCtx *>( wqe + sizeof(TileXR::UDMASqeCtx)); - sge->len = static_cast(context->rowBytes); - sge->tokenId = 0U; sge->va = localAddr; } } #endif -__aicore__ inline void CopyBytesGmToGm(__gm__ uint8_t *dst, - const __gm__ uint8_t *src, uint32_t bytes, LocalTensor relay) -{ - for (uint32_t offset = 0U; offset < bytes; offset += kRelayBytes) { - const uint32_t tileBytes = bytes - offset < kRelayBytes ? - bytes - offset : kRelayBytes; - GlobalTensor srcGlobal; - srcGlobal.SetGlobalBuffer(const_cast<__gm__ uint8_t *>(src) + offset); - const DataCopyExtParams copyIn {1U, tileBytes, 0U, 0U, 0U}; - const DataCopyPadExtParams padIn {false, 0U, 0U, 0U}; - DataCopyPad(relay, srcGlobal, copyIn, padIn); - SetFlag(EVENT_ID0); - WaitFlag(EVENT_ID0); - GlobalTensor dstGlobal; - dstGlobal.SetGlobalBuffer(dst + offset); - const DataCopyExtParams copyOut {1U, tileBytes, 0U, 0U, 0U}; - DataCopyPad(dstGlobal, relay, copyOut); - SetFlag(EVENT_ID0); - WaitFlag(EVENT_ID0); - } - PipeBarrier(); -} - class MoonEpCombineV2 { public: __aicore__ inline void Init(GM_ADDR commArgsGM, @@ -223,6 +373,8 @@ class MoonEpCombineV2 { private: __aicore__ inline void InitBuffers(); + __aicore__ inline void PrefillOperatorWqes(); + __aicore__ inline bool PrefillPeerWqes(uint32_t peer); __aicore__ inline void SetFailure(uint32_t status, uint32_t step, uint32_t peer, uint32_t lane, uint32_t cqStatus = 0U, uint64_t expected = 0U, uint64_t observed = 0U); @@ -230,38 +382,52 @@ class MoonEpCombineV2 { __aicore__ inline bool ValidateDestinations(); __aicore__ inline void LoadSelectionChunk(uint64_t chunkStart, uint32_t chunkElements); - __aicore__ inline uint32_t SelectPeer(uint32_t peer, - uint32_t chunkElements); + __aicore__ inline uint32_t SelectPeerRoutes(uint32_t peer, + uint32_t chunkStart, uint32_t chunkElements, bool firstPass, + uint32_t &pausedThreadCount); __aicore__ inline bool InitLaneStates(); __aicore__ inline bool ResolveRemoteFields(uint32_t targetRank, uint32_t lane, uint64_t remoteBaseOffset, __ubuf__ MoonEpCombineV2RemoteFields *fields); __aicore__ inline uint32_t PollCqOnce( MoonEpCombineV2LaneState &state); - __aicore__ inline bool WaitAdmission(uint32_t step); - __aicore__ inline bool WaitFinalCqs(); - __aicore__ inline bool AppendDescriptor(uint32_t sourceSlot, - uint32_t targetSlot, uint32_t peer, uint32_t step); - __aicore__ inline bool BuildPayloadWqes(uint32_t peer); + __aicore__ inline bool WaitStepCqs(uint32_t step); + __aicore__ inline bool WaitStepGrant(uint32_t step); + __aicore__ inline bool BuildPayloadWqes(uint32_t selectedCount, + uint32_t sequenceBase); __aicore__ inline bool AppendControlWqe(LocalTensor issue, uint32_t outputIndex, MoonEpCombineV2LaneState &state, uint32_t targetRank, uint64_t remoteOffset, __gm__ uint64_t *localSource, uint32_t flag); + __aicore__ inline void PublishLocalGrant(uint32_t step, uint32_t lane); __aicore__ inline void CopyIssueToSq(LocalTensor issue, MoonEpCombineV2LaneState &state, uint32_t count); __aicore__ inline bool SubmitPair(uint32_t peer, uint32_t step, - bool finalBatch); + uint32_t selectedCount, uint32_t sequenceBase, bool finalBatch); __aicore__ inline bool SendRemoteStep(uint32_t peer, uint32_t step); - __aicore__ inline bool SendSelfStep(uint32_t peer); + __aicore__ inline void CopySelfRowsIn(uint32_t routeStart, + uint32_t routeCount, uint32_t localRowStride, + LocalTensor relay); + __aicore__ inline void CopySelfRowsOut(uint32_t routeStart, + uint32_t routeCount, uint32_t localRowStride, + LocalTensor relay); + __aicore__ inline void CopySelfTileIn(uint32_t routeIndex, + uint64_t rowOffset, uint32_t tileBytes, LocalTensor relay); + __aicore__ inline void CopySelfTileOut(uint32_t routeIndex, + uint64_t rowOffset, uint32_t tileBytes, LocalTensor relay); + __aicore__ inline bool CopySelfRouteBatch(uint32_t selectedCount); + __aicore__ inline bool SubmitSelfGrant(uint32_t step); + __aicore__ inline bool SendSelfStep(uint32_t peer, uint32_t step); __aicore__ inline bool WaitInboundDone(); __aicore__ inline void InitReduceBuffers(); __aicore__ inline bool ReduceHidden(); __aicore__ inline uint64_t LoadToken(__gm__ uint64_t *token); __aicore__ inline void PublishFailureAndConverge(); -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) __aicore__ inline void RecordProfilePoint(uint32_t index); + __aicore__ inline uint64_t BeginProfileMetric(); + __aicore__ inline void EndProfileMetric(uint32_t index, + uint64_t startCycles); __aicore__ inline void WriteProfile(); -#endif bool valid_{false}; TPipe *pipe_{nullptr}; @@ -294,10 +460,9 @@ class MoonEpCombineV2 { uint32_t activeCoreCount_{0U}; uint32_t stepCount_{0U}; uint32_t sourcesPerCore_{0U}; - uint32_t descriptorCount_{0U}; uint32_t issuedRows_{0U}; uint32_t selectedPeerRows_{0U}; - uint32_t currentChunkElements_{0U}; + uint64_t remoteRowBase_[TileXRMoonEp::kMoonEpCombineV2LaneCount] {}; uint32_t failureStatus_{TileXRMoonEp::MOONEP_COMBINE_V2_SUCCESS}; uint32_t failureStep_{UINT32_MAX}; uint32_t failurePeer_{UINT32_MAX}; @@ -312,16 +477,13 @@ class MoonEpCombineV2 { MoonEpCombineV2LaneState lane_[ TileXRMoonEp::kMoonEpCombineV2LaneCount] {}; - TBuf dstBuf_; - TBuf dstRankBuf_; - TBuf slotIndexBuf_; - TBuf selectedIndexBuf_; - TBuf compareMaskBuf_; - TBuf relayBuf_; - TBuf descriptorBuf_; - TBuf sixPortIssueBuf_; - TBuf twoPortIssueBuf_; - TBuf buildContextBuf_; + TBuf dstSlotBuf_; + TBuf wqeIssueBuf_; + TBuf routeEntryBuf_; + TBuf threadMaxSlotIdxBuf_; + TBuf selectStateBuf_; + TBuf wqeContextBuf_; + TQue selfCopyQueue_; TQue reduceInputQueue_; TQue reduceOutputQueue_; TBuf reduceRowBuf_; @@ -329,6 +491,8 @@ class MoonEpCombineV2 { #if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) int64_t profileTimePoint_[ TileXRMoonEp::kMoonEpCombineV2ProfileTimePointCapacity]; + uint64_t profileMetric_[ + TileXRMoonEp::kMoonEpCombineV2ProfileMetricCount]; #endif }; @@ -350,6 +514,11 @@ __aicore__ inline void MoonEpCombineV2::Init( ++index) { profileTimePoint_[index] = 0; } + for (uint32_t index = 0U; + index < TileXRMoonEp::kMoonEpCombineV2ProfileMetricCount; + ++index) { + profileMetric_[index] = 0U; + } profileTimePoint_[TileXRMoonEp::MOONEP_COMBINE_V2_TIME_INIT_BEGIN] = static_cast(profileStartCycles); #endif @@ -424,38 +593,75 @@ __aicore__ inline void MoonEpCombineV2::Init( UINT32_MAX, UINT32_MAX, UINT32_MAX); } (void)laneStatesReady; -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) RecordProfilePoint(TileXRMoonEp::MOONEP_COMBINE_V2_TIME_INIT_END); -#endif } __aicore__ inline void MoonEpCombineV2::InitBuffers() { - constexpr uint32_t chunk = - TileXRMoonEp::kMoonEpCombineV2SelectionChunkRows; - constexpr uint32_t dstBytes = chunk * sizeof(int32_t); - constexpr uint32_t indexBytes = chunk * sizeof(int16_t); - constexpr uint32_t maskBytes = (chunk / 8U + kUbAlignBytes - 1U) / - kUbAlignBytes * kUbAlignBytes; - constexpr uint32_t totalBytes = 2U * dstBytes + 2U * indexBytes + - maskBytes + kRelayBytes + kDescriptorBytes + kSixPortIssueBytes + - kTwoPortIssueBytes + kBuildContextBytes; - static_assert(totalBytes <= kFullUbBytes, + static_assert(kRouteEntryBytes == + kRouteEntryCapacity * sizeof(MoonEpCombineV2RouteEntry), + "Combine V2 route entry size changed without updating UB layout"); + static_assert(kSendBufferBytes <= kFullUbBytes, "Combine V2 send buffers exceed 216 KiB UB"); pipe_->Reset(); - pipe_->InitBuffer(dstBuf_, dstBytes); - pipe_->InitBuffer(dstRankBuf_, dstBytes); - pipe_->InitBuffer(slotIndexBuf_, indexBytes); - pipe_->InitBuffer(selectedIndexBuf_, indexBytes); - pipe_->InitBuffer(compareMaskBuf_, maskBytes); - pipe_->InitBuffer(relayBuf_, kRelayBytes); - pipe_->InitBuffer(descriptorBuf_, kDescriptorBytes); - pipe_->InitBuffer(sixPortIssueBuf_, kSixPortIssueBytes); - pipe_->InitBuffer(twoPortIssueBuf_, kTwoPortIssueBytes); - pipe_->InitBuffer(buildContextBuf_, kBuildContextBytes); - LocalTensor slotIndex = slotIndexBuf_.Get(); - CreateVecIndex(slotIndex, static_cast(0), chunk); - PipeBarrier(); + pipe_->InitBuffer(dstSlotBuf_, kDstSlotBytes); + pipe_->InitBuffer(wqeIssueBuf_, kTotalIssueBytes); + pipe_->InitBuffer(routeEntryBuf_, kRouteEntryBytes); + pipe_->InitBuffer(threadMaxSlotIdxBuf_, kThreadCursorBytes); + pipe_->InitBuffer(selectStateBuf_, kSelectionStateBytes); + pipe_->InitBuffer(wqeContextBuf_, kWqeContextBytes); + pipe_->InitBuffer(selfCopyQueue_, 2U, + TileXRMoonEp::kMoonEpCombineV2SelfRelayHalfBytes); + PrefillOperatorWqes(); +} + +__aicore__ inline void MoonEpCombineV2::PrefillOperatorWqes() +{ + __ubuf__ MoonEpCombineV2OperatorFields *fields = reinterpret_cast<__ubuf__ + MoonEpCombineV2OperatorFields *>( + wqeContextBuf_.Get().GetPhyAddr()); + fields->rowBytes = rowBytes_; + LocalTensor issue = wqeIssueBuf_.Get(); +#if defined(CATLASS_ARCH) && CATLASS_ARCH == 3510 + PipeBarrier(); + Simt::VF_CALL( + Simt::Dim3{TileXRMoonEp::kMoonEpCombineV2BuilderThreads, 1U, 1U}, + reinterpret_cast<__ubuf__ uint8_t *>( + issue.GetPhyAddr()), + reinterpret_cast<__ubuf__ uint8_t *>( + issue[kSixPortIssueBytes].GetPhyAddr()), fields); + PipeBarrier(); +#endif +} + +__aicore__ inline bool MoonEpCombineV2::PrefillPeerWqes(uint32_t peer) +{ + __ubuf__ MoonEpCombineV2PeerFields *fields = reinterpret_cast<__ubuf__ + MoonEpCombineV2PeerFields *>( + wqeContextBuf_.Get().GetPhyAddr()); + fields->rowBytes = rowBytes_; + for (uint32_t lane = 0U; + lane < TileXRMoonEp::kMoonEpCombineV2LaneCount; ++lane) { + const bool ready = ResolveRemoteFields( + peer, lane, scratchOffset_, &fields->remote[lane]); + if (kEnableSafetyChecks && !ready) { + return false; + } + (void)ready; + remoteRowBase_[lane] = fields->remote[lane].remoteRowBase; + } + LocalTensor issue = wqeIssueBuf_.Get(); +#if defined(CATLASS_ARCH) && CATLASS_ARCH == 3510 + PipeBarrier(); + Simt::VF_CALL( + Simt::Dim3{TileXRMoonEp::kMoonEpCombineV2BuilderThreads, 1U, 1U}, + reinterpret_cast<__ubuf__ uint8_t *>( + issue.GetPhyAddr()), + reinterpret_cast<__ubuf__ uint8_t *>( + issue[kSixPortIssueBytes].GetPhyAddr()), fields); + PipeBarrier(); +#endif + return true; } __aicore__ inline void MoonEpCombineV2::SetFailure( @@ -508,8 +714,11 @@ __aicore__ inline bool MoonEpCombineV2::InitLaneStates() info->memPtr == 0U)) { return false; } - const uint32_t firstPeer = TileXRMoonEp::MoonEpCombineV2Peer( - rank_, 0U, core_, rankSize_); + const uint32_t scheduledFirstPeer = TileXRMoonEp::MoonEpCombineV2Peer( + rank_, 0U, core_, rankSize_, kCombineV2ScheduleMode); + const uint32_t firstPeer = scheduledFirstPeer == + TileXRMoonEp::kMoonEpCombineV2InvalidPeer ? + rank_ : scheduledFirstPeer; for (uint32_t lane = 0U; lane < TileXRMoonEp::kMoonEpCombineV2LaneCount; ++lane) { MoonEpCombineV2LaneState &state = lane_[lane]; @@ -601,7 +810,7 @@ __aicore__ inline bool MoonEpCombineV2::ValidateDestinations() } GlobalTensor dstGlobal; dstGlobal.SetGlobalBuffer(dstGlobalAddr_, slots_); - LocalTensor dst = dstBuf_.Get(); + LocalTensor dst = dstSlotBuf_.Get(); const DataCopyExtParams params { 1U, rowCount * static_cast(sizeof(int32_t)), 0U, 0U, 0U}; @@ -634,41 +843,46 @@ __aicore__ inline void MoonEpCombineV2::LoadSelectionChunk( } GlobalTensor dstGlobal; dstGlobal.SetGlobalBuffer(dstGlobalAddr_, slots_); - LocalTensor dst = dstBuf_.Get(); + LocalTensor dst = dstSlotBuf_.Get(); const DataCopyExtParams params { 1U, chunkElements * static_cast(sizeof(int32_t)), 0U, 0U, 0U}; const DataCopyPadExtParams pad {false, 0U, 0U, 0U}; DataCopyPad(dst, dstGlobal[chunkStart], params, pad); SyncFunc(); - if (selectionBuffersUsed_) { - SyncFunc(); - } - LocalTensor dstRank = dstRankBuf_.Get(); - for (uint32_t index = 0U; index < chunkElements; ++index) { - const int32_t encoded = dst.GetValue(index); - dstRank.SetValue(index, encoded < 0 ? -1 : static_cast( - static_cast(encoded) / slots_)); - } - SyncFunc(); - PipeBarrier(); - currentChunkElements_ = chunkElements; selectionBuffersUsed_ = true; } -__aicore__ inline uint32_t MoonEpCombineV2::SelectPeer( - uint32_t peer, uint32_t chunkElements) +__aicore__ inline uint32_t MoonEpCombineV2::SelectPeerRoutes( + uint32_t peer, uint32_t chunkStart, uint32_t chunkElements, + bool firstPass, uint32_t &pausedThreadCount) { - LocalTensor mask = compareMaskBuf_.Get(); - Compares(mask, dstRankBuf_.Get(), static_cast(peer), - CMPMODE::EQ, chunkElements); - PipeBarrier(); - uint64_t selected = 0U; - GatherMask(selectedIndexBuf_.Get(), - slotIndexBuf_.Get(), mask.ReinterpretCast(), true, - chunkElements, {1U, 1U, 0U, 0U}, selected); - SyncFunc(); - return static_cast(selected); + __ubuf__ MoonEpCombineV2SelectState *state = reinterpret_cast<__ubuf__ + MoonEpCombineV2SelectState *>( + selectStateBuf_.Get().GetPhyAddr()); + state->batchBase = state->curWqeNum; + state->batchSelected = 0U; + state->pausedThreadCount = 0U; +#if defined(CATLASS_ARCH) && CATLASS_ARCH == 3510 + PipeBarrier(); + Simt::VF_CALL( + Simt::Dim3{TileXRMoonEp::kMoonEpCombineV2SelectorThreads, 1U, 1U}, + reinterpret_cast<__ubuf__ const int32_t *>( + dstSlotBuf_.Get().GetPhyAddr()), chunkStart, + chunkElements, peer, slots_, firstPass, state, + reinterpret_cast<__ubuf__ uint32_t *>( + threadMaxSlotIdxBuf_.Get().GetPhyAddr()), + reinterpret_cast<__ubuf__ MoonEpCombineV2RouteEntry *>( + routeEntryBuf_.Get().GetPhyAddr())); + PipeBarrier(); +#endif + state->batchSelected = state->curWqeNum - state->batchBase; + pausedThreadCount = state->pausedThreadCount; + if (kEnableSafetyChecks && state->batchSelected > + TileXRMoonEp::kMoonEpCombineV2MaxSelectedPayloadWqes) { + return UINT32_MAX; + } + return state->batchSelected; } __aicore__ inline uint64_t MoonEpCombineV2::LoadToken( @@ -724,56 +938,69 @@ __aicore__ inline uint32_t MoonEpCombineV2::PollCqOnce( state.cqTail, state.cqTarget) ? 0U : kPollNoCompletion; } -__aicore__ inline bool MoonEpCombineV2::WaitAdmission(uint32_t step) +__aicore__ inline bool MoonEpCombineV2::WaitStepCqs(uint32_t step) { - if (step == 0U) { - return true; - } - bool cqReady[2] = {false, false}; - bool grantReady[2] = {false, false}; - uint64_t observed[2] = {0U, 0U}; - const uint64_t expected = - TileXRMoonEp::MoonEpCombineV2Token(magic_, step); + bool ready[2] = { + TileXRMoonEp::MoonEpCombineV2CqTargetReached( + lane_[0].cqTail, lane_[0].cqTarget), + TileXRMoonEp::MoonEpCombineV2CqTargetReached( + lane_[1].cqTail, lane_[1].cqTarget)}; uint32_t cursor = 0U; - while (!(cqReady[0] && cqReady[1] && grantReady[0] && - grantReady[1])) { - for (uint32_t offset = 0U; offset < 4U; ++offset) { - const uint32_t condition = (cursor + offset) & 3U; - if (condition < 2U && !cqReady[condition]) { - const uint32_t status = PollCqOnce(lane_[condition]); - if (status == 0U) { - cqReady[condition] = true; - } else if (kEnableSafetyChecks && - status != kPollNoCompletion) { - SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_CQ_ERROR, - step, UINT32_MAX, condition, status); - return false; - } - } else if (condition >= 2U) { - const uint32_t lane = condition - 2U; - if (grantReady[lane]) { - continue; - } - const uint64_t index = - TileXRMoonEp::MoonEpCombineV2GrantIndex( - epoch_, core_, lane, step); - __gm__ uint64_t *token = reinterpret_cast<__gm__ uint64_t *>( - grantBase_ + index * - TileXRMoonEp::kMoonEpCombineV2GrantSlotBytes + - TileXRMoonEp::kMoonEpCombineV2GrantReceiveOffsetBytes); - observed[lane] = LoadToken(token); - grantReady[lane] = observed[lane] == expected; + while (!(ready[0] && ready[1])) { + for (uint32_t offset = 0U; offset < 2U; ++offset) { + const uint32_t lane = (cursor + offset) & 1U; + if (ready[lane]) { + continue; + } + const uint32_t status = PollCqOnce(lane_[lane]); + if (status == 0U) { + ready[lane] = true; + } else if (kEnableSafetyChecks && status != kPollNoCompletion) { + SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_CQ_ERROR, + step, UINT32_MAX, lane, status); + return false; } } - cursor = (cursor + 1U) & 3U; + cursor ^= 1U; if (kEnableSafetyChecks && TimedOut(operationStartCycles_)) { for (uint32_t lane = 0U; lane < 2U; ++lane) { - if (!cqReady[lane]) { + if (!ready[lane]) { SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_CQ_TIMEOUT, step, UINT32_MAX, lane); return false; } - if (!grantReady[lane]) { + } + } + } + return true; +} + +__aicore__ inline bool MoonEpCombineV2::WaitStepGrant(uint32_t step) +{ + bool ready[2] = {false, false}; + uint64_t observed[2] = {0U, 0U}; + const uint64_t expected = TileXRMoonEp::MoonEpCombineV2GrantToken( + magic_, step, stepCount_); + uint32_t cursor = 0U; + while (!(ready[0] && ready[1])) { + for (uint32_t offset = 0U; offset < 2U; ++offset) { + const uint32_t lane = (cursor + offset) & 1U; + if (ready[lane]) { + continue; + } + const uint64_t index = TileXRMoonEp::MoonEpCombineV2GrantIndex( + epoch_, core_, lane, step); + __gm__ uint64_t *token = reinterpret_cast<__gm__ uint64_t *>( + grantBase_ + index * + TileXRMoonEp::kMoonEpCombineV2GrantSlotBytes + + TileXRMoonEp::kMoonEpCombineV2GrantReceiveOffsetBytes); + observed[lane] = LoadToken(token); + ready[lane] = observed[lane] == expected; + } + cursor ^= 1U; + if (kEnableSafetyChecks && TimedOut(operationStartCycles_)) { + for (uint32_t lane = 0U; lane < 2U; ++lane) { + if (!ready[lane]) { SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_GRANT_TIMEOUT, step, UINT32_MAX, lane, 0U, expected, observed[lane]); @@ -785,71 +1012,36 @@ __aicore__ inline bool MoonEpCombineV2::WaitAdmission(uint32_t step) return true; } -__aicore__ inline bool MoonEpCombineV2::BuildPayloadWqes(uint32_t peer) +__aicore__ inline bool MoonEpCombineV2::BuildPayloadWqes( + uint32_t selectedCount, uint32_t sequenceBase) { - LocalTensor contextTensor = buildContextBuf_.Get(); + LocalTensor contextTensor = wqeContextBuf_.Get(); __ubuf__ MoonEpCombineV2BuildContext *context = reinterpret_cast<__ubuf__ MoonEpCombineV2BuildContext *>(contextTensor.GetPhyAddr()); context->localRowBase = reinterpret_cast(workspace_); context->rowBytes = rowBytes_; - context->logicalCount = descriptorCount_; - context->sequencePhase = issuedRows_ & 3U; + context->selectedCount = selectedCount; + context->sequencePhase = sequenceBase & 3U; for (uint32_t lane = 0U; lane < 2U; ++lane) { context->head[lane] = lane_[lane].head; - context->laneCount[lane] = 0U; - const bool remoteFieldsReady = ResolveRemoteFields( - peer, lane, scratchOffset_, &context->remote[lane]); - if (kEnableSafetyChecks && !remoteFieldsReady) { - return false; - } - (void)remoteFieldsReady; + context->remoteRowBase[lane] = remoteRowBase_[lane]; } - context->laneCount[1] = TileXRMoonEp::MoonEpCombineV2Qp1TokenCount( - descriptorCount_, context->sequencePhase); - context->laneCount[0] = descriptorCount_ - context->laneCount[1]; + LocalTensor issue = wqeIssueBuf_.Get(); #if defined(CATLASS_ARCH) && CATLASS_ARCH == 3510 PipeBarrier(); - Simt::VF_CALL( - Simt::Dim3{kWqeBuildThreads, 1U, 1U}, + Simt::VF_CALL( + Simt::Dim3{TileXRMoonEp::kMoonEpCombineV2BuilderThreads, 1U, 1U}, reinterpret_cast<__ubuf__ uint8_t *>( - sixPortIssueBuf_.Get().GetPhyAddr()), + issue.GetPhyAddr()), reinterpret_cast<__ubuf__ uint8_t *>( - twoPortIssueBuf_.Get().GetPhyAddr()), - reinterpret_cast<__ubuf__ const MoonEpCombineV2Descriptor *>( - descriptorBuf_.Get().GetPhyAddr()), context); + issue[kSixPortIssueBytes].GetPhyAddr()), + reinterpret_cast<__ubuf__ const MoonEpCombineV2RouteEntry *>( + routeEntryBuf_.Get().GetPhyAddr()), context); PipeBarrier(); #endif return true; } -__aicore__ inline bool MoonEpCombineV2::AppendDescriptor( - uint32_t sourceSlot, uint32_t targetSlot, uint32_t peer, uint32_t step) -{ - if (descriptorCount_ == - TileXRMoonEp::kMoonEpCombineV2LogicalBatchRows) { - const bool batchSubmitted = SubmitPair(peer, step, false); - if (kEnableSafetyChecks && !batchSubmitted) { - return false; - } - (void)batchSubmitted; - } - __ubuf__ MoonEpCombineV2Descriptor *descriptors = reinterpret_cast<__ubuf__ - MoonEpCombineV2Descriptor *>( - descriptorBuf_.Get().GetPhyAddr()); - descriptors[descriptorCount_].sourceSlot = sourceSlot; - descriptors[descriptorCount_].targetSlot = targetSlot; - ++descriptorCount_; - ++selectedPeerRows_; - if (kEnableSafetyChecks && selectedPeerRows_ > - TileXRMoonEp::kMoonEpCombineV2MaxOutstanding) { - SetFailure( - TileXRMoonEp::MOONEP_COMBINE_V2_OUTSTANDING_LIMIT, - step, peer, UINT32_MAX, selectedPeerRows_); - return false; - } - return true; -} - __aicore__ inline bool MoonEpCombineV2::AppendControlWqe( LocalTensor issue, uint32_t outputIndex, MoonEpCombineV2LaneState &state, uint32_t targetRank, @@ -911,6 +1103,21 @@ __aicore__ inline bool MoonEpCombineV2::AppendControlWqe( return true; } +__aicore__ inline void MoonEpCombineV2::PublishLocalGrant( + uint32_t step, uint32_t lane) +{ + const uint64_t grantIndex = TileXRMoonEp::MoonEpCombineV2GrantIndex( + epoch_, core_, lane, step); + __gm__ uint64_t *grant = reinterpret_cast<__gm__ uint64_t *>( + grantBase_ + grantIndex * + TileXRMoonEp::kMoonEpCombineV2GrantSlotBytes + + TileXRMoonEp::kMoonEpCombineV2GrantReceiveOffsetBytes); + *grant = TileXRMoonEp::MoonEpCombineV2GrantToken( + magic_, step, stepCount_); + TileXR::UDMACleanCacheLines(reinterpret_cast<__gm__ uint8_t *>(grant), + TileXRMoonEp::kMoonEpCombineV2TokenStrideBytes); +} + __aicore__ inline void MoonEpCombineV2::CopyIssueToSq( LocalTensor issue, MoonEpCombineV2LaneState &state, uint32_t count) @@ -939,41 +1146,51 @@ __aicore__ inline void MoonEpCombineV2::CopyIssueToSq( } __aicore__ inline bool MoonEpCombineV2::SubmitPair( - uint32_t peer, uint32_t step, bool finalBatch) + uint32_t peer, uint32_t step, uint32_t selectedCount, + uint32_t sequenceBase, bool finalBatch) { - const bool payloadReady = BuildPayloadWqes(peer); + const uint64_t buildStart = BeginProfileMetric(); + const bool payloadReady = BuildPayloadWqes(selectedCount, sequenceBase); + EndProfileMetric(TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_REMOTE_WQE_BUILD, + buildStart); if (kEnableSafetyChecks && !payloadReady) { SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_INVALID_CONFIG, step, peer, UINT32_MAX); return false; } (void)payloadReady; - const uint32_t phase = issuedRows_ & 3U; + const uint64_t submitStart = BeginProfileMetric(); + const uint32_t phase = sequenceBase & 3U; const TileXRMoonEp::MoonEpCombineV2LaneCounts batchCounts = TileXRMoonEp::MoonEpCombineV2BatchLaneCounts( - descriptorCount_, phase, step, stepCount_, false); + selectedCount, phase, step, stepCount_, false); uint32_t count[2] = {batchCounts.sixPort, batchCounts.twoPort}; + LocalTensor allIssue = wqeIssueBuf_.Get(); + LocalTensor laneIssue[2] = { + allIssue, + allIssue[kSixPortIssueBytes] + }; if (finalBatch) { const uint32_t successor = TileXRMoonEp::MoonEpCombineV2Successor( - rank_, core_, rankSize_); + rank_, step, core_, rankSize_, kCombineV2ScheduleMode); for (uint32_t lane = 0U; lane < 2U; ++lane) { - LocalTensor issue = lane == 0U ? - sixPortIssueBuf_.Get() : - twoPortIssueBuf_.Get(); + LocalTensor issue = laneIssue[lane]; __gm__ uint64_t *doneSource = reinterpret_cast<__gm__ uint64_t *>( controlSourceBase_ + static_cast(lane) * TileXRMoonEp::kMoonEpCombineV2TokenStrideBytes); - if (step + 1U < stepCount_) { + if (successor == rank_) { + PublishLocalGrant(step, lane); + } else { const uint64_t grantIndex = TileXRMoonEp::MoonEpCombineV2GrantIndex( - epoch_, core_, lane, step + 1U); + epoch_, core_, lane, step); __gm__ uint64_t *grantSource = reinterpret_cast<__gm__ uint64_t *>( grantBase_ + grantIndex * TileXRMoonEp::kMoonEpCombineV2GrantSlotBytes + TileXRMoonEp::kMoonEpCombineV2GrantSourceOffsetBytes); - *grantSource = TileXRMoonEp::MoonEpCombineV2Token( - magic_, step + 1U); + *grantSource = TileXRMoonEp::MoonEpCombineV2GrantToken( + magic_, step, stepCount_); TileXR::UDMACleanCacheLines( reinterpret_cast<__gm__ uint8_t *>(grantSource), TileXRMoonEp::kMoonEpCombineV2TokenStrideBytes); @@ -988,6 +1205,9 @@ __aicore__ inline bool MoonEpCombineV2::SubmitPair( SetFailure( TileXRMoonEp::MOONEP_COMBINE_V2_INVALID_CONFIG, step, successor, lane); + EndProfileMetric( + TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_REMOTE_SUBMIT, + submitStart); return false; } (void)grantAppended; @@ -1010,6 +1230,9 @@ __aicore__ inline bool MoonEpCombineV2::SubmitPair( SetFailure( TileXRMoonEp::MOONEP_COMBINE_V2_INVALID_CONFIG, step, peer, lane); + EndProfileMetric( + TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_REMOTE_SUBMIT, + submitStart); return false; } (void)doneAppended; @@ -1020,19 +1243,25 @@ __aicore__ inline bool MoonEpCombineV2::SubmitPair( for (uint32_t lane = 0U; lane < 2U; ++lane) { const uint32_t outstanding = lane_[lane].head - lane_[lane].tail; if (kEnableSafetyChecks && - (count[lane] == 0U || - static_cast(outstanding) + count[lane] >= + (static_cast(outstanding) + count[lane] >= TileXRMoonEp::kMoonEpCombineV2MaxOutstanding)) { SetFailure( TileXRMoonEp::MOONEP_COMBINE_V2_OUTSTANDING_LIMIT, step, peer, lane, outstanding + count[lane]); + EndProfileMetric( + TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_REMOTE_SUBMIT, + submitStart); return false; } } SyncFunc(); - CopyIssueToSq(sixPortIssueBuf_.Get(), lane_[0], count[0]); - CopyIssueToSq(twoPortIssueBuf_.Get(), lane_[1], count[1]); + if (count[0] != 0U) { + CopyIssueToSq(laneIssue[0], lane_[0], count[0]); + } + if (count[1] != 0U) { + CopyIssueToSq(laneIssue[1], lane_[1], count[1]); + } SyncFunc(); for (uint32_t lane = 0U; lane < 2U; ++lane) { lane_[lane].head += count[lane]; @@ -1044,10 +1273,14 @@ __aicore__ inline bool MoonEpCombineV2::SubmitPair( reinterpret_cast<__gm__ uint32_t *>( lane_[lane].sq->wqeCntAddr), 0); } - st_dev(lane_[0].head, reinterpret_cast<__gm__ uint32_t *>( - lane_[0].sq->dbAddr), 0); - st_dev(lane_[1].head, reinterpret_cast<__gm__ uint32_t *>( - lane_[1].sq->dbAddr), 0); + if (count[0] != 0U) { + st_dev(lane_[0].head, reinterpret_cast<__gm__ uint32_t *>( + lane_[0].sq->dbAddr), 0); + } + if (count[1] != 0U) { + st_dev(lane_[1].head, reinterpret_cast<__gm__ uint32_t *>( + lane_[1].sq->dbAddr), 0); + } if (finalBatch) { for (uint32_t lane = 0U; lane < 2U; ++lane) { lane_[lane].submittedHead = lane_[lane].head; @@ -1056,19 +1289,28 @@ __aicore__ inline bool MoonEpCombineV2::SubmitPair( lane_[lane].cqTail, true); } } - issuedRows_ += descriptorCount_; - descriptorCount_ = 0U; + issuedRows_ += selectedCount; + EndProfileMetric(TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_REMOTE_SUBMIT, + submitStart); return true; } __aicore__ inline bool MoonEpCombineV2::SendRemoteStep( uint32_t peer, uint32_t step) { - descriptorCount_ = 0U; issuedRows_ = 0U; selectedPeerRows_ = 0U; - LocalTensor selected = selectedIndexBuf_.Get(); - LocalTensor dst = dstBuf_.Get(); + __ubuf__ MoonEpCombineV2SelectState *state = reinterpret_cast<__ubuf__ + MoonEpCombineV2SelectState *>( + selectStateBuf_.Get().GetPhyAddr()); + state->curWqeNum = 0U; + const bool prefilled = PrefillPeerWqes(peer); + if (kEnableSafetyChecks && !prefilled) { + SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_INVALID_CONFIG, + step, peer, UINT32_MAX); + return false; + } + (void)prefilled; for (uint64_t chunkStart = 0U; chunkStart < slots_; chunkStart += TileXRMoonEp::kMoonEpCombineV2SelectionChunkRows) { const uint32_t chunkElements = static_cast( @@ -1076,35 +1318,300 @@ __aicore__ inline bool MoonEpCombineV2::SendRemoteStep( TileXRMoonEp::kMoonEpCombineV2SelectionChunkRows ? slots_ - chunkStart : TileXRMoonEp::kMoonEpCombineV2SelectionChunkRows); + uint64_t profileStart = BeginProfileMetric(); LoadSelectionChunk(chunkStart, chunkElements); - const uint32_t selectedCount = SelectPeer(peer, chunkElements); - for (uint32_t index = 0U; index < selectedCount; ++index) { - const uint32_t sourceInChunk = static_cast( - selected.GetValue(index)); - if (kEnableSafetyChecks && sourceInChunk >= chunkElements) { - SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_BAD_DESTINATION, + EndProfileMetric( + TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_SELECTION_LOAD, + profileStart); + bool firstPass = true; + uint32_t pausedThreadCount = 0U; + do { + profileStart = BeginProfileMetric(); + const uint32_t selectedCount = SelectPeerRoutes( + peer, static_cast(chunkStart), chunkElements, + firstPass, pausedThreadCount); + EndProfileMetric( + TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_SELECTION_SELECT, + profileStart); + if (kEnableSafetyChecks && selectedCount == UINT32_MAX) { + SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_OUTSTANDING_LIMIT, step, peer, UINT32_MAX); return false; } - const uint32_t targetSlot = static_cast( - static_cast(dst.GetValue(sourceInChunk)) % slots_); - const bool descriptorAppended = AppendDescriptor( - static_cast(chunkStart) + sourceInChunk, - targetSlot, peer, step); - if (kEnableSafetyChecks && !descriptorAppended) { - return false; + const bool finalBatch = pausedThreadCount == 0U && + chunkStart + chunkElements == slots_; + if (selectedCount != 0U || finalBatch) { + const bool submitted = SubmitPair(peer, step, selectedCount, + issuedRows_, finalBatch); + if (kEnableSafetyChecks && !submitted) { + return false; + } + (void)submitted; + selectedPeerRows_ += selectedCount; + if (kEnableSafetyChecks && selectedPeerRows_ > + TileXRMoonEp::kMoonEpCombineV2MaxOutstanding) { + SetFailure( + TileXRMoonEp::MOONEP_COMBINE_V2_OUTSTANDING_LIMIT, + step, peer, UINT32_MAX, selectedPeerRows_); + return false; + } } - (void)descriptorAppended; + firstPass = false; + } while (pausedThreadCount != 0U); + } + return true; +} + +__aicore__ inline void MoonEpCombineV2::CopySelfRowsIn( + uint32_t routeStart, uint32_t routeCount, uint32_t localRowStride, + LocalTensor relay) +{ + __ubuf__ const MoonEpCombineV2RouteEntry *routes = reinterpret_cast<__ubuf__ + const MoonEpCombineV2RouteEntry *>( + routeEntryBuf_.Get().GetPhyAddr()); + const uint32_t copyBytes = static_cast(rowBytes_); + const DataCopyExtParams copyIn {1U, copyBytes, 0U, 0U, 0U}; + const DataCopyPadExtParams padIn {false, 0U, 0U, 0U}; + for (uint32_t row = 0U; row < routeCount; ++row) { + const uint32_t sourceSlotIndex = + routes[routeStart + row].sourceSlotIndex; + GlobalTensor source; + source.SetGlobalBuffer(workspace_ + + static_cast(sourceSlotIndex) * rowBytes_); + DataCopyPad(relay[row * localRowStride], source, copyIn, padIn); + } +} + +__aicore__ inline void MoonEpCombineV2::CopySelfRowsOut( + uint32_t routeStart, uint32_t routeCount, uint32_t localRowStride, + LocalTensor relay) +{ + __ubuf__ const MoonEpCombineV2RouteEntry *routes = reinterpret_cast<__ubuf__ + const MoonEpCombineV2RouteEntry *>( + routeEntryBuf_.Get().GetPhyAddr()); + const uint32_t copyBytes = static_cast(rowBytes_); + const DataCopyExtParams copyOut {1U, copyBytes, 0U, 0U, 0U}; + for (uint32_t row = 0U; row < routeCount; ++row) { + const uint32_t targetSlot = routes[routeStart + row].targetSlot; + GlobalTensor target; + target.SetGlobalBuffer(scratch_ + + static_cast(targetSlot) * rowBytes_); + DataCopyPad(target, relay[row * localRowStride], copyOut); + } +} + +__aicore__ inline void MoonEpCombineV2::CopySelfTileIn( + uint32_t routeIndex, uint64_t rowOffset, uint32_t tileBytes, + LocalTensor relay) +{ + __ubuf__ const MoonEpCombineV2RouteEntry *routes = reinterpret_cast<__ubuf__ + const MoonEpCombineV2RouteEntry *>( + routeEntryBuf_.Get().GetPhyAddr()); + GlobalTensor source; + source.SetGlobalBuffer(workspace_ + + static_cast(routes[routeIndex].sourceSlotIndex) * rowBytes_ + + rowOffset); + const DataCopyExtParams copyIn {1U, tileBytes, 0U, 0U, 0U}; + const DataCopyPadExtParams padIn {false, 0U, 0U, 0U}; + DataCopyPad(relay, source, copyIn, padIn); +} + +__aicore__ inline void MoonEpCombineV2::CopySelfTileOut( + uint32_t routeIndex, uint64_t rowOffset, uint32_t tileBytes, + LocalTensor relay) +{ + __ubuf__ const MoonEpCombineV2RouteEntry *routes = reinterpret_cast<__ubuf__ + const MoonEpCombineV2RouteEntry *>( + routeEntryBuf_.Get().GetPhyAddr()); + GlobalTensor target; + target.SetGlobalBuffer(scratch_ + + static_cast(routes[routeIndex].targetSlot) * rowBytes_ + + rowOffset); + const DataCopyExtParams copyOut {1U, tileBytes, 0U, 0U, 0U}; + DataCopyPad(target, relay, copyOut); +} + +__aicore__ inline bool MoonEpCombineV2::CopySelfRouteBatch( + uint32_t selectedCount) +{ + if (selectedCount == 0U) { + return true; + } + const uint64_t localRowStride = + TileXRMoonEp::MoonEpCombineV2SelfAlignedRowBytes(rowBytes_); + if (kEnableSafetyChecks && (localRowStride == 0U || + selectedCount > kRouteEntryCapacity)) { + return false; + } + + const uint32_t rowsPerBatch = + TileXRMoonEp::MoonEpCombineV2SelfRowsPerBatch(rowBytes_); + if (rowsPerBatch != 0U) { + const uint32_t rowStride = static_cast(localRowStride); + uint32_t pendingStart = 0U; + uint32_t pendingRows = selectedCount < rowsPerBatch ? + selectedCount : rowsPerBatch; + LocalTensor first = selfCopyQueue_.AllocTensor(); + CopySelfRowsIn(pendingStart, pendingRows, rowStride, first); + selfCopyQueue_.EnQue(first); + + uint32_t nextStart = pendingRows; + while (nextStart < selectedCount) { + LocalTensor pending = + selfCopyQueue_.DeQue(); + SyncFunc(); + + const uint32_t remaining = selectedCount - nextStart; + const uint32_t currentRows = remaining < rowsPerBatch ? + remaining : rowsPerBatch; + LocalTensor current = + selfCopyQueue_.AllocTensor(); + CopySelfRowsIn(nextStart, currentRows, rowStride, current); + CopySelfRowsOut(pendingStart, pendingRows, rowStride, pending); + SyncFunc(); + selfCopyQueue_.FreeTensor(pending); + selfCopyQueue_.EnQue(current); + + pendingStart = nextStart; + pendingRows = currentRows; + nextStart += currentRows; + } + + LocalTensor pending = selfCopyQueue_.DeQue(); + SyncFunc(); + CopySelfRowsOut(pendingStart, pendingRows, rowStride, pending); + SyncFunc(); + selfCopyQueue_.FreeTensor(pending); + return true; + } + + const uint64_t tileCapacity = + TileXRMoonEp::kMoonEpCombineV2SelfRelayHalfBytes; + const uint32_t tilesPerRow = static_cast( + TileXRMoonEp::MoonEpCombineV2SelfTileCount(rowBytes_)); + const uint64_t totalTiles = + static_cast(selectedCount) * tilesPerRow; + uint32_t pendingRoute = 0U; + uint64_t pendingOffset = 0U; + uint32_t pendingBytes = + TileXRMoonEp::MoonEpCombineV2SelfTileBytes(rowBytes_, 0U); + LocalTensor first = selfCopyQueue_.AllocTensor(); + CopySelfTileIn(pendingRoute, pendingOffset, pendingBytes, first); + selfCopyQueue_.EnQue(first); + + for (uint64_t task = 1U; task < totalTiles; ++task) { + LocalTensor pending = selfCopyQueue_.DeQue(); + SyncFunc(); + + const uint32_t currentRoute = static_cast(task / tilesPerRow); + const uint32_t tileInRow = static_cast(task % tilesPerRow); + const uint64_t currentOffset = + static_cast(tileInRow) * tileCapacity; + const uint32_t currentBytes = + TileXRMoonEp::MoonEpCombineV2SelfTileBytes( + rowBytes_, tileInRow); + LocalTensor current = selfCopyQueue_.AllocTensor(); + CopySelfTileIn(currentRoute, currentOffset, currentBytes, current); + CopySelfTileOut( + pendingRoute, pendingOffset, pendingBytes, pending); + SyncFunc(); + selfCopyQueue_.FreeTensor(pending); + selfCopyQueue_.EnQue(current); + + pendingRoute = currentRoute; + pendingOffset = currentOffset; + pendingBytes = currentBytes; + } + + LocalTensor pending = selfCopyQueue_.DeQue(); + SyncFunc(); + CopySelfTileOut(pendingRoute, pendingOffset, pendingBytes, pending); + SyncFunc(); + selfCopyQueue_.FreeTensor(pending); + return true; +} + +__aicore__ inline bool MoonEpCombineV2::SubmitSelfGrant(uint32_t step) +{ + const uint32_t successor = TileXRMoonEp::MoonEpCombineV2Successor( + rank_, step, core_, rankSize_, kCombineV2ScheduleMode); + if (successor == rank_) { + for (uint32_t lane = 0U; lane < 2U; ++lane) { + PublishLocalGrant(step, lane); } + return true; + } + LocalTensor allIssue = wqeIssueBuf_.Get(); + LocalTensor laneIssue[2] = { + allIssue, + allIssue[kSixPortIssueBytes] + }; + for (uint32_t lane = 0U; lane < 2U; ++lane) { + const uint64_t grantIndex = TileXRMoonEp::MoonEpCombineV2GrantIndex( + epoch_, core_, lane, step); + __gm__ uint64_t *grantSource = reinterpret_cast<__gm__ uint64_t *>( + grantBase_ + grantIndex * + TileXRMoonEp::kMoonEpCombineV2GrantSlotBytes + + TileXRMoonEp::kMoonEpCombineV2GrantSourceOffsetBytes); + *grantSource = TileXRMoonEp::MoonEpCombineV2GrantToken( + magic_, step, stepCount_); + TileXR::UDMACleanCacheLines( + reinterpret_cast<__gm__ uint8_t *>(grantSource), + TileXRMoonEp::kMoonEpCombineV2TokenStrideBytes); + const bool appended = AppendControlWqe( + laneIssue[lane], 0U, lane_[lane], successor, + grantOffset_ + grantIndex * + TileXRMoonEp::kMoonEpCombineV2GrantSlotBytes + + TileXRMoonEp::kMoonEpCombineV2GrantReceiveOffsetBytes, + grantSource, + TileXR::TILEXR_UDMA_SQE_FLAG_ORDERED_COMPLETION); + if (kEnableSafetyChecks && !appended) { + SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_INVALID_CONFIG, + step, successor, lane); + return false; + } + (void)appended; + const uint32_t outstanding = lane_[lane].head - lane_[lane].tail; + if (kEnableSafetyChecks && + static_cast(outstanding) + 1U >= + TileXRMoonEp::kMoonEpCombineV2MaxOutstanding) { + SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_OUTSTANDING_LIMIT, + step, successor, lane, outstanding + 1U); + return false; + } + } + + SyncFunc(); + CopyIssueToSq(laneIssue[0], lane_[0], 1U); + CopyIssueToSq(laneIssue[1], lane_[1], 1U); + SyncFunc(); + for (uint32_t lane = 0U; lane < 2U; ++lane) { + ++lane_[lane].head; + ++lane_[lane].completionCount; + lane_[lane].submittedHead = lane_[lane].head; + lane_[lane].cqTarget = TileXRMoonEp::MoonEpCombineV2NextCqTarget( + lane_[lane].cqTail, true); + st_dev(lane_[lane].head, reinterpret_cast<__gm__ uint32_t *>( + lane_[lane].sq->headAddr), 0); + st_dev(lane_[lane].completionCount, + reinterpret_cast<__gm__ uint32_t *>( + lane_[lane].sq->wqeCntAddr), 0); + st_dev(lane_[lane].head, reinterpret_cast<__gm__ uint32_t *>( + lane_[lane].sq->dbAddr), 0); } - return SubmitPair(peer, step, true); + return true; } -__aicore__ inline bool MoonEpCombineV2::SendSelfStep(uint32_t peer) +__aicore__ inline bool MoonEpCombineV2::SendSelfStep( + uint32_t peer, uint32_t step) { - LocalTensor selected = selectedIndexBuf_.Get(); - LocalTensor dst = dstBuf_.Get(); - LocalTensor relay = relayBuf_.Get(); + if (kEnableSafetyChecks && peer != rank_) { + return false; + } + __ubuf__ MoonEpCombineV2SelectState *state = reinterpret_cast<__ubuf__ + MoonEpCombineV2SelectState *>( + selectStateBuf_.Get().GetPhyAddr()); + state->curWqeNum = 0U; for (uint64_t chunkStart = 0U; chunkStart < slots_; chunkStart += TileXRMoonEp::kMoonEpCombineV2SelectionChunkRows) { const uint32_t chunkElements = static_cast( @@ -1112,55 +1619,37 @@ __aicore__ inline bool MoonEpCombineV2::SendSelfStep(uint32_t peer) TileXRMoonEp::kMoonEpCombineV2SelectionChunkRows ? slots_ - chunkStart : TileXRMoonEp::kMoonEpCombineV2SelectionChunkRows); + uint64_t profileStart = BeginProfileMetric(); LoadSelectionChunk(chunkStart, chunkElements); - const uint32_t selectedCount = SelectPeer(peer, chunkElements); - for (uint32_t index = 0U; index < selectedCount; ++index) { - const uint32_t sourceInChunk = static_cast( - selected.GetValue(index)); - const uint32_t targetSlot = static_cast( - static_cast(dst.GetValue(sourceInChunk)) % slots_); - CopyBytesGmToGm(scratch_ + - static_cast(targetSlot) * rowBytes_, - workspace_ + (chunkStart + sourceInChunk) * rowBytes_, - static_cast(rowBytes_), relay); - } - } - return true; -} - -__aicore__ inline bool MoonEpCombineV2::WaitFinalCqs() -{ - bool ready[2] = {false, false}; - uint32_t cursor = 0U; - while (!(ready[0] && ready[1])) { - for (uint32_t offset = 0U; offset < 2U; ++offset) { - const uint32_t lane = (cursor + offset) & 1U; - if (ready[lane]) { - continue; - } - const uint32_t status = PollCqOnce(lane_[lane]); - if (status == 0U) { - ready[lane] = true; - } else if (kEnableSafetyChecks && - status != kPollNoCompletion) { - SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_CQ_ERROR, - stepCount_ - 1U, UINT32_MAX, lane, status); + EndProfileMetric( + TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_SELECTION_LOAD, + profileStart); + bool firstPass = true; + uint32_t pausedThreadCount = 0U; + do { + profileStart = BeginProfileMetric(); + const uint32_t selectedCount = SelectPeerRoutes( + peer, static_cast(chunkStart), chunkElements, + firstPass, pausedThreadCount); + EndProfileMetric( + TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_SELECTION_SELECT, + profileStart); + if (kEnableSafetyChecks && selectedCount == UINT32_MAX) { return false; } - } - cursor ^= 1U; - if (kEnableSafetyChecks && TimedOut(operationStartCycles_)) { - for (uint32_t lane = 0U; lane < 2U; ++lane) { - if (!ready[lane]) { - SetFailure( - TileXRMoonEp::MOONEP_COMBINE_V2_CQ_TIMEOUT, - stepCount_ - 1U, UINT32_MAX, lane); - return false; - } + profileStart = BeginProfileMetric(); + const bool copied = CopySelfRouteBatch(selectedCount); + EndProfileMetric( + TileXRMoonEp::MOONEP_COMBINE_V2_METRIC_SELF_COPY, + profileStart); + if (kEnableSafetyChecks && !copied) { + return false; } - } + (void)copied; + firstPass = false; + } while (pausedThreadCount != 0U); } - return true; + return SubmitSelfGrant(step); } __aicore__ inline bool MoonEpCombineV2::WaitInboundDone() @@ -1201,7 +1690,7 @@ __aicore__ inline bool MoonEpCombineV2::WaitInboundDone() core_, sourceIndex, rankSize_); const uint32_t step = TileXRMoonEp::MoonEpCombineV2ReceiveStep( - rank_, source, rankSize_); + rank_, source, rankSize_, kCombineV2ScheduleMode); const uint64_t expected = TileXRMoonEp::MoonEpCombineV2Token(magic_, step); const uint64_t index = TileXRMoonEp::MoonEpCombineV2DoneIndex( @@ -1231,7 +1720,7 @@ __aicore__ inline bool MoonEpCombineV2::WaitInboundDone() core_, sourceIndex, rankSize_); const uint32_t step = TileXRMoonEp::MoonEpCombineV2ReceiveStep( - rank_, source, rankSize_); + rank_, source, rankSize_, kCombineV2ScheduleMode); SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_DONE_TIMEOUT, step, source, lane, 0U, TileXRMoonEp::MoonEpCombineV2Token(magic_, step), @@ -1409,17 +1898,46 @@ __aicore__ inline void MoonEpCombineV2::PublishFailureAndConverge() ++convergenceRound_; } -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) __aicore__ inline void MoonEpCombineV2::RecordProfilePoint( uint32_t index) { +#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) if (index < TileXRMoonEp::kMoonEpCombineV2ProfileTimePointCount) { profileTimePoint_[index] = static_cast(GetSystemCycle()); } +#else + (void)index; + return; +#endif +} + +__aicore__ inline uint64_t MoonEpCombineV2::BeginProfileMetric() +{ +#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) + return static_cast(GetSystemCycle()); +#else + return 0U; +#endif +} + +__aicore__ inline void MoonEpCombineV2::EndProfileMetric( + uint32_t index, uint64_t startCycles) +{ +#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) + if (index < TileXRMoonEp::kMoonEpCombineV2ProfileMetricCount) { + profileMetric_[index] += + static_cast(GetSystemCycle()) - startCycles; + } +#else + (void)index; + (void)startCycles; + return; +#endif } __aicore__ inline void MoonEpCombineV2::WriteProfile() { +#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) if (profileTimePoint_[ TileXRMoonEp::MOONEP_COMBINE_V2_TIME_FINAL_END] == 0) { RecordProfilePoint( @@ -1452,6 +1970,8 @@ __aicore__ inline void MoonEpCombineV2::WriteProfile() record->blockDim = activeCoreCount_; record->timePointCount = TileXRMoonEp::kMoonEpCombineV2ProfileTimePointCount; + record->metricCount = TileXRMoonEp::kMoonEpCombineV2ProfileMetricCount; + record->reserved = 0U; for (uint32_t index = 0U; index < TileXRMoonEp::kMoonEpCombineV2ProfileTimePointCapacity; ++index) { @@ -1459,6 +1979,11 @@ __aicore__ inline void MoonEpCombineV2::WriteProfile() TileXRMoonEp::kMoonEpCombineV2ProfileTimePointCount ? profileTimePoint_[index] : 0; } + for (uint32_t index = 0U; + index < TileXRMoonEp::kMoonEpCombineV2ProfileMetricCount; + ++index) { + record->metric[index] = profileMetric_[index]; + } record->timePoint[ TileXRMoonEp::MOONEP_COMBINE_V2_DIAG_FAILURE_STATUS] = static_cast(failureStatus_); @@ -1486,8 +2011,10 @@ __aicore__ inline void MoonEpCombineV2::WriteProfile() static_cast(failureObserved_); TileXR::UDMACleanCacheLines(reinterpret_cast<__gm__ uint8_t *>(record), sizeof(*record)); -} +#else + return; #endif +} __aicore__ inline void MoonEpCombineV2::Process() { @@ -1506,50 +2033,41 @@ __aicore__ inline void MoonEpCombineV2::Process() PublishFailureAndConverge(); succeeded = succeeded && allCoresSucceeded_; } -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) RecordProfilePoint(TileXRMoonEp::MOONEP_COMBINE_V2_TIME_PREPARE_END); -#endif if (kEnableSafetyChecks && !succeeded) { -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) WriteProfile(); -#endif return; } for (uint32_t step = 0U; step < stepCount_ && succeeded; ++step) { - succeeded = WaitAdmission(step); -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) - RecordProfilePoint( - TileXRMoonEp::MOONEP_COMBINE_V2_TIME_STEP0_GRANT_END + - step * TileXRMoonEp::kMoonEpCombineV2ProfileStepPointStride); -#endif - if (!succeeded) { - break; - } const uint32_t peer = TileXRMoonEp::MoonEpCombineV2Peer( - rank_, step, core_, rankSize_); - succeeded = peer == rank_ ? SendSelfStep(peer) : + rank_, step, core_, rankSize_, kCombineV2ScheduleMode); + succeeded = peer == rank_ || + peer == TileXRMoonEp::kMoonEpCombineV2InvalidPeer ? + SendSelfStep(rank_, step) : SendRemoteStep(peer, step); -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) RecordProfilePoint( TileXRMoonEp::MOONEP_COMBINE_V2_TIME_STEP0_SEND_END + step * TileXRMoonEp::kMoonEpCombineV2ProfileStepPointStride); -#endif + if (succeeded) { + succeeded = WaitStepCqs(step); + } + if (succeeded) { + succeeded = WaitStepGrant(step); + } + RecordProfilePoint( + TileXRMoonEp::MOONEP_COMBINE_V2_TIME_STEP0_READY_END + + step * TileXRMoonEp::kMoonEpCombineV2ProfileStepPointStride); } if (succeeded) { - succeeded = WaitFinalCqs(); -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) RecordProfilePoint( - TileXRMoonEp::MOONEP_COMBINE_V2_TIME_FINAL_CQ_END); -#endif + TileXRMoonEp::MOONEP_COMBINE_V2_TIME_STEP_LOOP_END); } if (succeeded) { succeeded = WaitInboundDone(); -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) RecordProfilePoint( TileXRMoonEp::MOONEP_COMBINE_V2_TIME_INBOUND_DONE_END); -#endif } if (kEnableSafetyChecks) { PublishFailureAndConverge(); @@ -1559,10 +2077,8 @@ __aicore__ inline void MoonEpCombineV2::Process() SetFailure(TileXRMoonEp::MOONEP_COMBINE_V2_INVALID_CONFIG, UINT32_MAX, UINT32_MAX, UINT32_MAX); } -#if defined(TILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING) RecordProfilePoint(TileXRMoonEp::MOONEP_COMBINE_V2_TIME_FINAL_END); WriteProfile(); -#endif } } // namespace diff --git a/src/moonep/common/moonep_combine_schedule.h b/src/moonep/common/moonep_combine_schedule.h index 9f3b002..6059d58 100644 --- a/src/moonep/common/moonep_combine_schedule.h +++ b/src/moonep/common/moonep_combine_schedule.h @@ -19,7 +19,7 @@ constexpr uint32_t kMoonEpCombineV2GroupCount = 16U; constexpr uint32_t kMoonEpCombineV2GroupsPerHalf = 8U; constexpr uint32_t kMoonEpCombineV2StepCount = 8U; constexpr uint32_t kMoonEpCombineV2GrantStepCount = - kMoonEpCombineV2StepCount - 1U; + kMoonEpCombineV2StepCount; constexpr uint32_t kMoonEpCombineV2CoreCount = 16U; constexpr uint32_t kMoonEpCombineV2LaneCount = 2U; constexpr uint32_t kMoonEpCombineV2QpCount = 32U; @@ -51,6 +51,13 @@ enum MoonEpCombineV2Lane : uint32_t { MOONEP_COMBINE_V2_TWO_PORT = 1U, }; +enum MoonEpCombineV2ScheduleMode : uint32_t { + MOONEP_COMBINE_V2_SINGLE_RING = 0U, + MOONEP_COMBINE_V2_BIDIRECTIONAL_RING = 1U, +}; + +constexpr uint32_t kMoonEpCombineV2InvalidPeer = UINT32_MAX; + enum MoonEpCombineV2FailureStatus : uint32_t { MOONEP_COMBINE_V2_SUCCESS = 0U, MOONEP_COMBINE_V2_INVALID_CONFIG = 1U, @@ -180,8 +187,13 @@ TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t MoonEpCombineV2ControlWqesPerLane( uint32_t step, uint32_t stepCount, bool finalBatch) { - return !finalBatch ? 0U : - (step + 1U < stepCount ? 2U : 1U); + return finalBatch ? 2U : 0U; +} + +TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t +MoonEpCombineV2NextStep(uint32_t step, uint32_t stepCount) +{ + return step + 1U == stepCount ? 0U : step + 1U; } TILEXR_MOONEP_COMBINE_V2_INLINE MoonEpCombineV2LaneCounts @@ -261,7 +273,7 @@ MoonEpCombineV2DestinationValid( } TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t -MoonEpCombineV2Peer( +MoonEpCombineV2SingleRingPeer( uint32_t sourceRank, uint32_t step, uint32_t core, uint32_t rankSize) { if (rankSize <= kMoonEpCombineV2GroupSize) { @@ -282,7 +294,97 @@ MoonEpCombineV2Peer( } TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t -MoonEpCombineV2Successor( +MoonEpCombineV2RingOffset( + uint32_t position, int32_t offset, uint32_t ringSize) +{ + if (offset >= 0) { + return (position + static_cast(offset)) % ringSize; + } + return (position + ringSize - static_cast(-offset)) % ringSize; +} + +TILEXR_MOONEP_COMBINE_V2_INLINE int32_t +MoonEpCombineV2BidirectionalOffset( + uint32_t step, uint32_t lane, uint32_t ringSize, bool sameHalf) +{ + const uint32_t ordinal = step * kMoonEpCombineV2GroupSize + lane; + if (sameHalf) { + if (ordinal + 1U == ringSize) { + return 0; + } + const int32_t distance = static_cast(ordinal / 2U + 1U); + return (ordinal & 1U) == 0U ? distance : -distance; + } + if (ordinal == 0U) { + return 0; + } + if (ordinal == 1U) { + return static_cast(ringSize / 2U); + } + const uint32_t pairOrdinal = ordinal - 2U; + const int32_t distance = static_cast(pairOrdinal / 2U + 1U); + return (pairOrdinal & 1U) == 0U ? distance : -distance; +} + +TILEXR_MOONEP_COMBINE_V2_INLINE bool +MoonEpCombineV2BidirectionalScheduleEnabled( + uint32_t rankSize, MoonEpCombineV2ScheduleMode mode) +{ + return mode == MOONEP_COMBINE_V2_BIDIRECTIONAL_RING && + (rankSize == kMoonEpCombineV2GroupSize || rankSize >= 16U); +} + +TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t +MoonEpCombineV2BidirectionalPeer( + uint32_t sourceRank, uint32_t step, uint32_t core, uint32_t rankSize) +{ + if (rankSize == kMoonEpCombineV2GroupSize) { + if (core + 1U == rankSize) { + return kMoonEpCombineV2InvalidPeer; + } + return MoonEpCombineV2RingOffset(sourceRank, + MoonEpCombineV2BidirectionalOffset( + step, core, rankSize, true), rankSize); + } + + const uint32_t halfRankCount = rankSize / 2U; + const uint32_t sourceHalf = sourceRank / halfRankCount; + const uint32_t targetHalf = core / kMoonEpCombineV2GroupSize; + const uint32_t lane = core % kMoonEpCombineV2GroupSize; + const uint32_t ordinal = step * kMoonEpCombineV2GroupSize + lane; + const bool sameHalf = sourceHalf == targetHalf; + if (sameHalf && ordinal + 1U == halfRankCount) { + return kMoonEpCombineV2InvalidPeer; + } + const uint32_t targetLocal = MoonEpCombineV2RingOffset( + sourceRank % halfRankCount, + MoonEpCombineV2BidirectionalOffset( + step, lane, halfRankCount, sameHalf), + halfRankCount); + return targetHalf * halfRankCount + targetLocal; +} + +TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t +MoonEpCombineV2Peer( + uint32_t sourceRank, uint32_t step, uint32_t core, uint32_t rankSize, + MoonEpCombineV2ScheduleMode mode) +{ + return MoonEpCombineV2BidirectionalScheduleEnabled(rankSize, mode) ? + MoonEpCombineV2BidirectionalPeer( + sourceRank, step, core, rankSize) : + MoonEpCombineV2SingleRingPeer(sourceRank, step, core, rankSize); +} + +TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t +MoonEpCombineV2Peer( + uint32_t sourceRank, uint32_t step, uint32_t core, uint32_t rankSize) +{ + return MoonEpCombineV2Peer(sourceRank, step, core, rankSize, + MOONEP_COMBINE_V2_SINGLE_RING); +} + +TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t +MoonEpCombineV2SingleRingSuccessor( uint32_t sourceRank, uint32_t core, uint32_t rankSize) { if (rankSize <= kMoonEpCombineV2GroupSize) { @@ -301,7 +403,43 @@ MoonEpCombineV2Successor( } TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t -MoonEpCombineV2ReceiveStep( +MoonEpCombineV2Successor( + uint32_t sourceRank, uint32_t step, uint32_t core, uint32_t rankSize, + MoonEpCombineV2ScheduleMode mode) +{ + if (!MoonEpCombineV2BidirectionalScheduleEnabled(rankSize, mode) || + rankSize <= kMoonEpCombineV2GroupSize) { + return MoonEpCombineV2SingleRingSuccessor( + sourceRank, core, rankSize); + } + const uint32_t halfRankCount = rankSize / 2U; + const uint32_t sourceHalf = sourceRank / halfRankCount; + const uint32_t targetHalf = core / kMoonEpCombineV2GroupSize; + const uint32_t lane = core % kMoonEpCombineV2GroupSize; + const bool sameHalf = sourceHalf == targetHalf; + const uint32_t nextStep = MoonEpCombineV2NextStep( + step, MoonEpCombineV2StepCount(rankSize)); + const int32_t currentOffset = MoonEpCombineV2BidirectionalOffset( + step, lane, halfRankCount, sameHalf); + const int32_t nextOffset = MoonEpCombineV2BidirectionalOffset( + nextStep, lane, halfRankCount, sameHalf); + const uint32_t targetLocal = MoonEpCombineV2RingOffset( + sourceRank % halfRankCount, currentOffset, halfRankCount); + const uint32_t successorLocal = MoonEpCombineV2RingOffset( + targetLocal, -nextOffset, halfRankCount); + return sourceHalf * halfRankCount + successorLocal; +} + +TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t +MoonEpCombineV2Successor( + uint32_t sourceRank, uint32_t core, uint32_t rankSize) +{ + return MoonEpCombineV2SingleRingSuccessor( + sourceRank, core, rankSize); +} + +TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t +MoonEpCombineV2SingleRingReceiveStep( uint32_t destinationRank, uint32_t sourceRank, uint32_t rankSize) { if (rankSize <= kMoonEpCombineV2GroupSize) { @@ -322,6 +460,42 @@ MoonEpCombineV2ReceiveStep( return distance - 1U; } +TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t +MoonEpCombineV2ReceiveStep( + uint32_t destinationRank, uint32_t sourceRank, uint32_t rankSize, + MoonEpCombineV2ScheduleMode mode) +{ + if (!MoonEpCombineV2BidirectionalScheduleEnabled(rankSize, mode) || + rankSize <= kMoonEpCombineV2GroupSize) { + return MoonEpCombineV2SingleRingReceiveStep( + destinationRank, sourceRank, rankSize); + } + const uint32_t halfRankCount = rankSize / 2U; + const uint32_t sourceLocal = sourceRank % halfRankCount; + const uint32_t destinationLocal = destinationRank % halfRankCount; + const uint32_t clockwise = + (destinationLocal + halfRankCount - sourceLocal) % halfRankCount; + const uint32_t counterClockwise = + clockwise == 0U ? 0U : halfRankCount - clockwise; + const uint32_t distance = clockwise < counterClockwise ? + clockwise : counterClockwise; + const bool sameHalf = destinationRank / halfRankCount == + sourceRank / halfRankCount; + if (sameHalf) { + return distance == 0U ? MoonEpCombineV2StepCount(rankSize) - 1U : + (distance - 1U) / 4U; + } + return distance == halfRankCount / 2U ? 0U : distance / 4U; +} + +TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t +MoonEpCombineV2ReceiveStep( + uint32_t destinationRank, uint32_t sourceRank, uint32_t rankSize) +{ + return MoonEpCombineV2ReceiveStep(destinationRank, sourceRank, rankSize, + MOONEP_COMBINE_V2_SINGLE_RING); +} + TILEXR_MOONEP_COMBINE_V2_INLINE uint32_t MoonEpCombineV2SourceForCore( uint32_t core, uint32_t sourceIndex, uint32_t rankSize) @@ -336,6 +510,14 @@ MoonEpCombineV2Token( return (magic << 3U) | static_cast(step); } +TILEXR_MOONEP_COMBINE_V2_INLINE uint64_t +MoonEpCombineV2GrantToken( + uint64_t magic, uint32_t step, uint32_t stepCount) +{ + return MoonEpCombineV2Token( + magic, MoonEpCombineV2NextStep(step, stepCount)); +} + TILEXR_MOONEP_COMBINE_V2_INLINE bool MoonEpCombineV2TokenMatches( uint64_t token, uint64_t magic, uint32_t step, uint32_t rankSize) @@ -355,11 +537,12 @@ MoonEpCombineV2DoneIndex( TILEXR_MOONEP_COMBINE_V2_INLINE uint64_t MoonEpCombineV2GrantIndex( - uint32_t epoch, uint32_t core, uint32_t lane, uint32_t step) + uint32_t epoch, uint32_t core, uint32_t lane, + uint32_t transitionRound) { return (((static_cast(epoch) * kMoonEpCombineV2CoreCount + core) * kMoonEpCombineV2LaneCount + lane) * - kMoonEpCombineV2GrantStepCount) + (step - 1U); + kMoonEpCombineV2GrantStepCount) + transitionRound; } TILEXR_MOONEP_COMBINE_V2_INLINE uint64_t diff --git a/tests/moonep_combine_v2/CMakeLists.txt b/tests/moonep_combine_v2/CMakeLists.txt index df5a6a2..9b73c5d 100644 --- a/tests/moonep_combine_v2/CMakeLists.txt +++ b/tests/moonep_combine_v2/CMakeLists.txt @@ -112,6 +112,32 @@ add_executable(test_tilexr_moonep_combine_v2_source_guard target_compile_definitions(test_tilexr_moonep_combine_v2_source_guard PRIVATE TILEXR_SOURCE_ROOT="${TILEXR_ROOT}") +if(TARGET tilexr-moonep-combine-v2 AND TARGET tile-comm) + add_executable(tilexr_moonep_combine_v2_perf + demo/tilexr_moonep_combine_v2_hardware_probe.cpp) + target_include_directories(tilexr_moonep_combine_v2_perf PRIVATE + ${TILEXR_ROOT}/src/include + ${TILEXR_ROOT}/src/moonep/common + ${TILEXR_ROOT}/src/moonep/combine_v2/common + ${ASCEND_HOME_PATH}/${ARCH}-linux/pkg_inc + ${ASCEND_HOME_PATH}/${ARCH}-linux/pkg_inc/runtime + ${ASCEND_HOME_PATH}/${ARCH}-linux/include) + target_link_directories(tilexr_moonep_combine_v2_perf PRIVATE + ${ASCEND_HOME_PATH}/${ARCH}-linux/lib64 + ${ASCEND_DRIVER_PATH}/lib64/driver) + target_link_libraries(tilexr_moonep_combine_v2_perf PRIVATE + tilexr-moonep-combine-v2 + tile-comm + ascendcl + runtime) + target_link_options(tilexr_moonep_combine_v2_perf PRIVATE + "LINKER:-rpath-link,${ASCEND_HOME_PATH}/${ARCH}-linux/lib64" + "LINKER:-rpath-link,${ASCEND_DRIVER_PATH}/lib64/driver") + set_target_properties(tilexr_moonep_combine_v2_perf PROPERTIES + BUILD_WITH_INSTALL_RPATH TRUE + INSTALL_RPATH "$ORIGIN/../${CMAKE_INSTALL_LIBDIR};${ASCEND_HOME_PATH}/${ARCH}-linux/lib64;${ASCEND_DRIVER_PATH}/lib64/driver") +endif() + add_test(NAME test_tilexr_moonep_combine_v2_schedule COMMAND test_tilexr_moonep_combine_v2_schedule) add_test(NAME test_tilexr_moonep_combine_v2_layout @@ -133,3 +159,8 @@ install(TARGETS test_tilexr_moonep_combine_v2_public_abi test_tilexr_moonep_combine_v2_source_guard RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}) + +if(TARGET tilexr_moonep_combine_v2_perf) + install(TARGETS tilexr_moonep_combine_v2_perf + RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}) +endif() diff --git a/tests/moonep_combine_v2/demo/tilexr_moonep_combine_v2_hardware_probe.cpp b/tests/moonep_combine_v2/demo/tilexr_moonep_combine_v2_hardware_probe.cpp index a7ea417..ea98365 100644 --- a/tests/moonep_combine_v2/demo/tilexr_moonep_combine_v2_hardware_probe.cpp +++ b/tests/moonep_combine_v2/demo/tilexr_moonep_combine_v2_hardware_probe.cpp @@ -1,73 +1,660 @@ +#include +#include +#include +#include #include #include +#include +#include #include +#include +#include +#include #include +#include #include -#include +#include +#include +#include +#include #include "acl/acl.h" +#include "combine_v2_profile.h" +#include "combine_v2_schedule.h" #include "tilexr_api.h" #include "tilexr_moonep_combine_v2.h" #include "tilexr_types.h" namespace { -constexpr int kRankCount = 8; constexpr int kDeviceCount = 8; -constexpr int kCommDomain = 41; -constexpr int kInvocationCount = 2; -constexpr int64_t kBs = 8; +constexpr int kDefaultCommDomain = 141; constexpr int64_t kH = 3584; constexpr int64_t kTopK = 16; -constexpr int64_t kSlots = 128; +constexpr int kDefaultExpertCount = 64; constexpr uint32_t kAivCoreNum = 16; constexpr uint32_t kQpCount = 32; +constexpr int kBarrierPortOffset = 97; +constexpr int kBarrierTimeoutSeconds = 60; +const char *const kProfileMetricNames[ + TileXRMoonEp::kMoonEpCombineV2ProfileMetricCount] = { + "selection_load_us", + "selection_select_us", + "self_route_decode_us", + "self_copy_us", + "remote_route_decode_us", + "remote_descriptor_us", + "remote_wqe_build_us", + "remote_submit_us", +}; + +struct HostPort { + std::string host; + int port = 0; +}; + +struct Options { + std::vector batchSizes {128}; + int warmup = 20; + int iterations = 80; + int experts = kDefaultExpertCount; + int commDomain = kDefaultCommDomain; + int rank = -1; + int worldSize = 0; + int device = -1; + bool skipIterationBarriers = false; + bool profile = false; + bool allowSelfOnlyFailure = false; +}; + +struct ProfileSample { + int iteration = 0; + uint32_t core = 0U; + std::array timePoint {}; + std::array metric {}; +}; + +enum class OutputCheckResult { + Passed, + SelfOnlyFailed, + Failed, +}; + +const char *OutputCheckName(OutputCheckResult result) +{ + switch (result) { + case OutputCheckResult::Passed: + return "passed"; + case OutputCheckResult::SelfOnlyFailed: + return "self_only_failed"; + default: + return "failed"; + } +} [[noreturn]] void Abort(int rank, const std::string &step, int status) { std::cerr << "[rank " << rank << "] " << step << " failed, status=" << status << std::endl; - MPI_Abort(MPI_COMM_WORLD, status == 0 ? 1 : status); - std::abort(); + std::exit(status == 0 ? 1 : status); +} + +void Usage(std::ostream &out, const char *program) +{ + out << "Usage: " << program << " [options]\n" + << " --bs N Run one batch size (default: 128)\n" + << " --bs-list N[,N...] Run multiple batch sizes after one initialization\n" + << " --warmup N Warmup launches per batch size (default: 20)\n" + << " --iterations N Timed launches per batch size (default: 80)\n" + << " --experts N Total expert count (default: 64)\n" + << " --comm-domain N Shared-QP communication domain (default: 141)\n" + << " --rank N Global rank (required)\n" + << " --world-size N Global rank count (required)\n" + << " --device N Local device id (default: rank modulo 8)\n" + << " --skip-iteration-barriers\n" + << " Skip host barriers between warmup/timed launches\n" + << " --profile Capture per-AIV kernel cycle timestamps\n" + << " --allow-self-only-failure\n" + << " Continue timing when only Self-copy validation fails\n" + << " --help Show this help\n"; +} + +bool ParseInteger(const std::string &text, int64_t minValue, + int64_t maxValue, int64_t *value) +{ + if (value == nullptr || text.empty()) { + return false; + } + std::size_t consumed = 0; + try { + const long long parsed = std::stoll(text, &consumed, 10); + if (consumed != text.size() || parsed < minValue || parsed > maxValue) { + return false; + } + *value = static_cast(parsed); + return true; + } catch (...) { + return false; + } +} + +bool ParseBatchSizes(const std::string &text, std::vector *values) +{ + if (values == nullptr || text.empty()) { + return false; + } + std::vector parsed; + std::istringstream input(text); + std::string item; + while (std::getline(input, item, ',')) { + int64_t value = 0; + if (!ParseInteger(item, 1, std::numeric_limits::max(), + &value)) { + return false; + } + if (std::find(parsed.begin(), parsed.end(), value) == parsed.end()) { + parsed.push_back(value); + } + } + if (parsed.empty()) { + return false; + } + *values = parsed; + return true; +} + +bool ParseOptions(int argc, char **argv, Options *options, bool *showHelp, + std::string *error) +{ + if (options == nullptr || showHelp == nullptr || error == nullptr) { + return false; + } + bool batchSizeSet = false; + *showHelp = false; + for (int index = 1; index < argc; ++index) { + const std::string argument = argv[index]; + if (argument == "--help" || argument == "-h") { + *showHelp = true; + return true; + } + if (argument == "--skip-iteration-barriers") { + options->skipIterationBarriers = true; + continue; + } + if (argument == "--profile") { + options->profile = true; + continue; + } + if (argument == "--allow-self-only-failure") { + options->allowSelfOnlyFailure = true; + continue; + } + if (index + 1 >= argc) { + *error = "missing value for " + argument; + return false; + } + const std::string value = argv[++index]; + if (argument == "--bs" || argument == "--bs-list") { + if (batchSizeSet) { + *error = "--bs and --bs-list are mutually exclusive"; + return false; + } + batchSizeSet = true; + if (!ParseBatchSizes(value, &options->batchSizes) || + (argument == "--bs" && options->batchSizes.size() != 1U)) { + *error = "invalid batch size list: " + value; + return false; + } + } else if (argument == "--warmup") { + int64_t parsed = 0; + if (!ParseInteger(value, 0, std::numeric_limits::max(), + &parsed)) { + *error = "invalid warmup count: " + value; + return false; + } + options->warmup = static_cast(parsed); + } else if (argument == "--iterations") { + int64_t parsed = 0; + if (!ParseInteger(value, 1, std::numeric_limits::max(), + &parsed)) { + *error = "invalid iteration count: " + value; + return false; + } + options->iterations = static_cast(parsed); + } else if (argument == "--experts") { + int64_t parsed = 0; + if (!ParseInteger(value, 1, std::numeric_limits::max(), + &parsed)) { + *error = "invalid expert count: " + value; + return false; + } + options->experts = static_cast(parsed); + } else if (argument == "--comm-domain") { + int64_t parsed = 0; + if (!ParseInteger(value, 1, std::numeric_limits::max(), + &parsed)) { + *error = "invalid communication domain: " + value; + return false; + } + options->commDomain = static_cast(parsed); + } else if (argument == "--rank") { + int64_t parsed = 0; + if (!ParseInteger(value, 0, std::numeric_limits::max(), + &parsed)) { + *error = "invalid rank: " + value; + return false; + } + options->rank = static_cast(parsed); + } else if (argument == "--world-size") { + int64_t parsed = 0; + if (!ParseInteger(value, 1, std::numeric_limits::max(), + &parsed)) { + *error = "invalid world size: " + value; + return false; + } + options->worldSize = static_cast(parsed); + } else if (argument == "--device") { + int64_t parsed = 0; + if (!ParseInteger(value, 0, kDeviceCount - 1, &parsed)) { + *error = "invalid device id: " + value; + return false; + } + options->device = static_cast(parsed); + } else { + *error = "unknown argument: " + argument; + return false; + } + } + if (options->rank < 0 || options->worldSize <= 0 || + options->rank >= options->worldSize) { + *error = "--rank and --world-size must identify a valid rank"; + return false; + } + return true; +} + +uint16_t FloatToBfloat16(float value) +{ + uint32_t bits = 0; + std::memcpy(&bits, &value, sizeof(bits)); + const uint32_t roundToEven = 0x7fffU + ((bits >> 16U) & 1U); + return static_cast((bits + roundToEven) >> 16U); +} + +uint16_t SourceValue(int sourceRank) +{ + const float value = 1.0F + static_cast(sourceRank % 16) * 0.25F + + static_cast(sourceRank / 16) * 0.0625F; + return FloatToBfloat16(value); +} + +void CheckAcl(int rank, const std::string &step, aclError status) +{ + if (status != ACL_SUCCESS) { + Abort(rank, step, static_cast(status)); + } +} + +bool ParseHostPort(const std::string &text, HostPort *endpoint) +{ + const std::size_t separator = text.rfind(':'); + if (endpoint == nullptr || separator == std::string::npos || + separator == 0 || separator + 1 >= text.size()) { + return false; + } + int64_t port = 0; + if (!ParseInteger(text.substr(separator + 1), 1, 65535, &port)) { + return false; + } + in_addr address {}; + const std::string host = text.substr(0, separator); + if (inet_pton(AF_INET, host.c_str(), &address) != 1) { + return false; + } + endpoint->host = host; + endpoint->port = static_cast(port); + return true; +} + +bool GetBarrierEndpoint(HostPort *endpoint) +{ + const char *configured = std::getenv("TILEXR_DEMO_BARRIER_ADDR"); + if (configured != nullptr && configured[0] != '\0') { + return ParseHostPort(configured, endpoint); + } + const char *commId = std::getenv("TILEXR_COMM_ID"); + HostPort commEndpoint; + if (commId == nullptr || !ParseHostPort(commId, &commEndpoint)) { + return false; + } + endpoint->host = commEndpoint.host; + endpoint->port = commEndpoint.port + kBarrierPortOffset; + if (endpoint->port > 65535) { + endpoint->port = commEndpoint.port - kBarrierPortOffset; + } + return endpoint->port > 0; +} + +bool SendAll(int fd, const void *data, std::size_t bytes) +{ + const uint8_t *cursor = static_cast(data); + while (bytes > 0) { + const ssize_t sent = send(fd, cursor, bytes, 0); + if (sent < 0 && errno == EINTR) { + continue; + } + if (sent <= 0) { + return false; + } + cursor += sent; + bytes -= static_cast(sent); + } + return true; +} + +bool RecvAll(int fd, void *data, std::size_t bytes) +{ + uint8_t *cursor = static_cast(data); + while (bytes > 0) { + const ssize_t received = recv(fd, cursor, bytes, 0); + if (received < 0 && errno == EINTR) { + continue; + } + if (received <= 0) { + return false; + } + cursor += received; + bytes -= static_cast(received); + } + return true; +} + +void SetSocketTimeout(int fd) +{ + timeval timeout {}; + timeout.tv_sec = kBarrierTimeoutSeconds; + (void)setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof(timeout)); + (void)setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &timeout, sizeof(timeout)); +} + +bool BarrierServer(int world, const HostPort &endpoint, bool localSuccess) +{ + const int listenFd = socket(AF_INET, SOCK_STREAM, 0); + if (listenFd < 0) { + return false; + } + int reuse = 1; + (void)setsockopt(listenFd, SOL_SOCKET, SO_REUSEADDR, &reuse, sizeof(reuse)); + SetSocketTimeout(listenFd); + + sockaddr_in address {}; + address.sin_family = AF_INET; + address.sin_addr.s_addr = htonl(INADDR_ANY); + address.sin_port = htons(static_cast(endpoint.port)); + if (bind(listenFd, reinterpret_cast(&address), + sizeof(address)) != 0 || listen(listenFd, world) != 0) { + close(listenFd); + return false; + } + + bool globalSuccess = localSuccess; + bool exchangeOk = true; + std::vector clients; + clients.reserve(static_cast(world - 1)); + for (int peer = 1; peer < world; ++peer) { + const int client = accept(listenFd, nullptr, nullptr); + if (client < 0) { + exchangeOk = false; + break; + } + SetSocketTimeout(client); + uint8_t arrived = 0; + if (!RecvAll(client, &arrived, sizeof(arrived)) || arrived > 1U) { + exchangeOk = false; + } + globalSuccess = globalSuccess && arrived == 1U; + clients.push_back(client); + if (!exchangeOk) { + break; + } + } + + const uint8_t release = exchangeOk && globalSuccess ? 1U : 0U; + for (const int client : clients) { + if (!SendAll(client, &release, sizeof(release))) { + exchangeOk = false; + } + close(client); + } + close(listenFd); + return exchangeOk && globalSuccess; } -uint16_t SourceValue(int sourceRank, int sourceSlot, int column) +bool BarrierClient(const HostPort &endpoint, bool localSuccess) { - const uint32_t payload = static_cast( - sourceRank * 131 + sourceSlot * 17 + column) & 0x03ffU; - return static_cast(0x3c00U + payload); + sockaddr_in address {}; + address.sin_family = AF_INET; + address.sin_port = htons(static_cast(endpoint.port)); + if (inet_pton(AF_INET, endpoint.host.c_str(), &address.sin_addr) != 1) { + return false; + } + + const auto deadline = std::chrono::steady_clock::now() + + std::chrono::seconds(kBarrierTimeoutSeconds); + int fd = -1; + while (std::chrono::steady_clock::now() < deadline) { + fd = socket(AF_INET, SOCK_STREAM, 0); + if (fd < 0) { + return false; + } + SetSocketTimeout(fd); + if (connect(fd, reinterpret_cast(&address), + sizeof(address)) == 0) { + break; + } + close(fd); + fd = -1; + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + if (fd < 0) { + return false; + } + + const uint8_t arrived = localSuccess ? 1U : 0U; + uint8_t release = 0; + const bool ok = SendAll(fd, &arrived, sizeof(arrived)) && + RecvAll(fd, &release, sizeof(release)) && release == 1U; + close(fd); + return ok; +} + +bool BarrierAll(int rank, int world, const std::string &step, + bool localSuccess = true) +{ + if (world <= 1) { + return localSuccess; + } + HostPort endpoint; + if (!GetBarrierEndpoint(&endpoint)) { + std::cerr << "[rank " << rank + << "] invalid TILEXR_DEMO_BARRIER_ADDR" << std::endl; + return false; + } + const bool ok = rank == 0 ? BarrierServer(world, endpoint, localSuccess) : + BarrierClient(endpoint, localSuccess); + if (!ok) { + std::cerr << "[rank " << rank << "] barrier failed after " << step + << " at " << endpoint.host << ':' << endpoint.port + << std::endl; + } + return ok; +} + +void LaunchCombine(int rank, void *workspace, const int32_t *dst, + TileXRCommPtr comm, int64_t bs, aclrtStream stream, + uint64_t *activeOutputOffset) +{ + const int64_t slots = bs * kTopK; + const int ret = TileXRMoonEpCombineV2(workspace, dst, comm, bs, kH, + kTopK, slots, kAivCoreNum, activeOutputOffset, + TILEXR_MOONEP_DTYPE_BFLOAT16, stream); + if (ret != TILEXR_MOONEP_SUCCESS) { + Abort(rank, "TileXRMoonEpCombineV2", ret); + } +} + +OutputCheckResult CheckOutput(int rank, int world, int64_t bs, + const void *workspace, + uint64_t activeOutputOffset) +{ + const int64_t slots = bs * kTopK; + const std::size_t outputElements = static_cast(slots) * + static_cast(kH); + const std::size_t outputBytes = outputElements * sizeof(uint16_t); + std::vector output(outputElements); + const void *outputDevice = static_cast(workspace) + + activeOutputOffset; + CheckAcl(rank, "output D2H copy", aclrtMemcpy(output.data(), outputBytes, + outputDevice, outputBytes, ACL_MEMCPY_DEVICE_TO_HOST)); + + const int64_t slotsPerSourceRank = slots / world; + bool selfMismatch = false; + for (int64_t slot = 0; slot < slots; ++slot) { + const int sourceRank = static_cast(slot / slotsPerSourceRank); + if (sourceRank == rank && selfMismatch) { + continue; + } + const uint16_t expected = SourceValue(sourceRank); + const std::size_t rowOffset = static_cast(slot) * + static_cast(kH); + for (int64_t column = 0; column < kH; ++column) { + const std::size_t index = rowOffset + + static_cast(column); + if (output[index] != expected) { + std::cerr << "[rank " << rank << "] output mismatch" + << " bs=" << bs + << " slot=" << slot + << " column=" << column + << " source_rank=" << sourceRank + << " got=" << output[index] + << " expected=" << expected << std::endl; + if (sourceRank != rank) { + return OutputCheckResult::Failed; + } + selfMismatch = true; + break; + } + } + } + return selfMismatch ? OutputCheckResult::SelfOnlyFailed : + OutputCheckResult::Passed; +} + +void CaptureProfileSamples(int rank, int world, int iteration, + const void *workspace, uint64_t profileOffset, + std::vector *samples) +{ + if (samples == nullptr) { + Abort(rank, "profile sample destination", 1); + } + const uint32_t coreCount = + TileXRMoonEp::MoonEpCombineV2ActiveCoreCount( + static_cast(world)); + std::vector records(coreCount); + const std::size_t profileBytes = records.size() * sizeof(records[0]); + const void *profileDevice = static_cast(workspace) + + profileOffset; + CheckAcl(rank, "profile D2H copy", aclrtMemcpy(records.data(), + profileBytes, profileDevice, profileBytes, ACL_MEMCPY_DEVICE_TO_HOST)); + + for (uint32_t core = 0U; core < coreCount; ++core) { + const TileXRMoonEp::MoonEpCombineV2ProfileRecord &record = + records[core]; + if (record.marker != TileXRMoonEp::kMoonEpCombineV2ProfileMarker || + record.version != TileXRMoonEp::kMoonEpCombineV2ProfileVersion || + record.recordBytes != sizeof(record) || + record.rank != static_cast(rank) || + record.core != core || record.blockDim != coreCount || + record.timePointCount != + TileXRMoonEp::kMoonEpCombineV2ProfileTimePointCount || + record.metricCount != + TileXRMoonEp::kMoonEpCombineV2ProfileMetricCount || + record.reserved != 0U) { + Abort(rank, "profile record validation", 1); + } + ProfileSample sample; + sample.iteration = iteration; + sample.core = core; + for (uint32_t point = 0U; + point < TileXRMoonEp::kMoonEpCombineV2ProfileTimePointCount; + ++point) { + sample.timePoint[point] = record.timePoint[point]; + if (sample.timePoint[point] <= 0 || + (point > 0U && sample.timePoint[point] < + sample.timePoint[point - 1U])) { + Abort(rank, "profile timestamp validation", 1); + } + } + for (uint32_t metric = 0U; + metric < TileXRMoonEp::kMoonEpCombineV2ProfileMetricCount; + ++metric) { + sample.metric[metric] = record.metric[metric]; + } + samples->push_back(sample); + } } } // namespace int main(int argc, char **argv) { - MPI_Init(&argc, &argv); - int rank = 0; - int world = 0; - MPI_Comm_rank(MPI_COMM_WORLD, &rank); - MPI_Comm_size(MPI_COMM_WORLD, &world); - if (world != kRankCount) { - Abort(rank, "MPI world size", world); + Options options; + bool showHelp = false; + std::string parseError; + if (!ParseOptions(argc, argv, &options, &showHelp, &parseError)) { + std::cerr << parseError << '\n'; + Usage(std::cerr, argv[0]); + return 2; } - - const int device = rank % kDeviceCount; - if (aclInit(nullptr) != ACL_SUCCESS) { - Abort(rank, "aclInit", 1); + if (showHelp) { + Usage(std::cout, argv[0]); + return 0; } - if (aclrtSetDevice(device) != ACL_SUCCESS) { - Abort(rank, "aclrtSetDevice", 1); + const int rank = options.rank; + const int world = options.worldSize; + if (!TileXRMoonEp::MoonEpCombineV2RankSizeSupported( + static_cast(world))) { + Abort(rank, "unsupported Combine V2 world size", world); + } + if (options.experts % world != 0) { + Abort(rank, "expert count must be divisible by world size", + options.experts); + } + for (const int64_t bs : options.batchSizes) { + if (bs % world != 0 || bs > + std::numeric_limits::max() / kTopK) { + Abort(rank, "batch size must be divisible by world size", 1); + } } + const int device = options.device >= 0 ? options.device : + rank % kDeviceCount; + CheckAcl(rank, "aclInit", aclInit(nullptr)); + CheckAcl(rank, "aclrtSetDevice", aclrtSetDevice(device)); + aclrtStream stream = nullptr; - if (aclrtCreateStream(&stream) != ACL_SUCCESS) { - Abort(rank, "aclrtCreateStream", 1); - } + CheckAcl(rank, "aclrtCreateStream", aclrtCreateStream(&stream)); + aclrtEvent startEvent = nullptr; + aclrtEvent stopEvent = nullptr; + CheckAcl(rank, "aclrtCreateEvent start", aclrtCreateEvent(&startEvent)); + CheckAcl(rank, "aclrtCreateEvent stop", aclrtCreateEvent(&stopEvent)); TileXRCommPtr comm = nullptr; int ret = TileXRCommInitRankWithSharedQpDomain( - kCommDomain, world, rank, &comm); + options.commDomain, world, rank, &comm); if (ret != TileXR::TILEXR_SUCCESS) { Abort(rank, "TileXRCommInitRankWithSharedQpDomain", ret); } @@ -84,148 +671,222 @@ int main(int argc, char **argv) Abort(rank, "TileXRUDMAGetQpCount", ret); } - uint64_t workspaceBytes = 0; - uint64_t profileOffset = 0; - uint64_t outputOffsets[2] = {}; - ret = TileXRMoonEpCombineGetWorkspaceSizeV2( - kBs, kH, kTopK, kSlots, TILEXR_MOONEP_DTYPE_BFLOAT16, - &workspaceBytes, &profileOffset, &outputOffsets[0], &outputOffsets[1]); - if (ret != TILEXR_MOONEP_SUCCESS || workspaceBytes == 0) { - Abort(rank, "TileXRMoonEpCombineGetWorkspaceSizeV2", ret); + uint64_t maxWorkspaceBytes = 0; + int64_t maxBs = 0; + for (const int64_t bs : options.batchSizes) { + uint64_t workspaceBytes = 0; + uint64_t profileOffset = 0; + uint64_t outputOffsets[2] = {}; + ret = TileXRMoonEpCombineGetWorkspaceSizeV2(bs, kH, kTopK, + bs * kTopK, TILEXR_MOONEP_DTYPE_BFLOAT16, &workspaceBytes, + &profileOffset, &outputOffsets[0], &outputOffsets[1]); + if (ret != TILEXR_MOONEP_SUCCESS || workspaceBytes == 0) { + Abort(rank, "TileXRMoonEpCombineGetWorkspaceSizeV2", ret); + } + maxWorkspaceBytes = std::max(maxWorkspaceBytes, workspaceBytes); + maxBs = std::max(maxBs, bs); } void *workspace = nullptr; int32_t *dst = nullptr; - const size_t rowElements = static_cast(kH); - const size_t sourceElements = static_cast(kSlots) * rowElements; - const size_t sourceBytes = sourceElements * sizeof(uint16_t); - const size_t dstBytes = static_cast(kSlots) * sizeof(int32_t); - if (aclrtMalloc(&workspace, static_cast(workspaceBytes), - ACL_MEM_MALLOC_HUGE_FIRST) != ACL_SUCCESS || - aclrtMalloc(reinterpret_cast(&dst), dstBytes, - ACL_MEM_MALLOC_HUGE_FIRST) != ACL_SUCCESS) { - Abort(rank, "aclrtMalloc", 1); - } - if (aclrtMemset(workspace, static_cast(workspaceBytes), 0, - static_cast(workspaceBytes)) != ACL_SUCCESS) { - Abort(rank, "aclrtMemset workspace", 1); - } - - const int slotsPerRank = static_cast(kSlots) / world; - std::vector source(sourceElements); - std::vector destinations(static_cast(kSlots)); - for (int slot = 0; slot < kSlots; ++slot) { - const int targetRank = slot % world; - const int targetSlot = rank * slotsPerRank + slot / world; - destinations[static_cast(slot)] = - targetRank * static_cast(kSlots) + targetSlot; - for (int column = 0; column < kH; ++column) { - source[static_cast(slot) * rowElements + - static_cast(column)] = SourceValue(rank, slot, column); - } - } - if (aclrtMemcpy(workspace, static_cast(workspaceBytes), - source.data(), sourceBytes, ACL_MEMCPY_HOST_TO_DEVICE) != ACL_SUCCESS || - aclrtMemcpy(dst, dstBytes, destinations.data(), dstBytes, - ACL_MEMCPY_HOST_TO_DEVICE) != ACL_SUCCESS) { - Abort(rank, "input H2D copy", 1); - } + const std::size_t maxDstBytes = static_cast(maxBs * kTopK) * + sizeof(int32_t); + CheckAcl(rank, "aclrtMalloc workspace", aclrtMalloc(&workspace, + static_cast(maxWorkspaceBytes), ACL_MEM_MALLOC_HUGE_FIRST)); + CheckAcl(rank, "aclrtMalloc destinations", aclrtMalloc( + reinterpret_cast(&dst), maxDstBytes, + ACL_MEM_MALLOC_HUGE_FIRST)); + CheckAcl(rank, "aclrtMemset workspace", aclrtMemset(workspace, + static_cast(maxWorkspaceBytes), 0, + static_cast(maxWorkspaceBytes))); TileXRUDMAMemHandle handle = 0; ret = TileXRUDMARegister(comm, static_cast(workspace), - static_cast(workspaceBytes), &handle); + static_cast(maxWorkspaceBytes), &handle); if (ret != TileXR::TILEXR_SUCCESS) { Abort(rank, "TileXRUDMARegister", ret); } - MPI_Barrier(MPI_COMM_WORLD); - - uint64_t activeOutputOffset = 0; - uint64_t previousOutputOffset = 0; - std::vector output(sourceElements); - int localOk = 1; - for (int invocation = 0; invocation < kInvocationCount; ++invocation) { - ret = TileXRMoonEpCombineV2(workspace, dst, comm, kBs, kH, kTopK, - kSlots, kAivCoreNum, &activeOutputOffset, - TILEXR_MOONEP_DTYPE_BFLOAT16, stream); - if (ret != TILEXR_MOONEP_SUCCESS) { - Abort(rank, "TileXRMoonEpCombineV2", ret); - } - if (aclrtSynchronizeStream(stream) != ACL_SUCCESS) { - Abort(rank, "aclrtSynchronizeStream", 1); - } - if ((activeOutputOffset != outputOffsets[0] && - activeOutputOffset != outputOffsets[1]) || - (invocation != 0 && activeOutputOffset == previousOutputOffset)) { - Abort(rank, "active output offset", 1); - } - previousOutputOffset = activeOutputOffset; - - const void *outputDevice = static_cast(workspace) + - activeOutputOffset; - if (aclrtMemcpy(output.data(), sourceBytes, outputDevice, sourceBytes, - ACL_MEMCPY_DEVICE_TO_HOST) != ACL_SUCCESS) { - Abort(rank, "output D2H copy", 1); - } - - for (int targetSlot = 0; targetSlot < kSlots && localOk != 0; - ++targetSlot) { - const int sourceRank = targetSlot / slotsPerRank; - const int sourceSlot = - (targetSlot % slotsPerRank) * world + rank; - for (int column = 0; column < kH; ++column) { - const size_t index = - static_cast(targetSlot) * rowElements + - static_cast(column); - const uint16_t expected = - SourceValue(sourceRank, sourceSlot, column); - if (output[index] != expected) { - std::cerr << "[rank " << rank - << "] output mismatch invocation=" << invocation - << " target_slot=" << targetSlot - << " source_rank=" << sourceRank - << " source_slot=" << sourceSlot - << " column=" << column - << " got=" << output[index] - << " expected=" << expected << std::endl; - localOk = 0; - break; - } + if (!BarrierAll(rank, world, "workspace registration")) { + Abort(rank, "workspace registration barrier", 1); + } + if (rank == 0) { + std::cout << "COMBINE_V2_SETUP ranks=" << world + << " devices_per_host=" << kDeviceCount + << " experts=" << options.experts + << " k=" << kTopK + << " h=" << kH + << " dtype=bf16" + << " qp_count=" << qpCount + << " max_bs=" << maxBs + << " workspace_bytes=" << maxWorkspaceBytes + << std::endl; + } + + bool allCasesOk = true; + for (const int64_t bs : options.batchSizes) { + const int64_t slots = bs * kTopK; + uint64_t caseWorkspaceBytes = 0; + uint64_t profileOffset = 0; + uint64_t outputOffsets[2] = {}; + ret = TileXRMoonEpCombineGetWorkspaceSizeV2(bs, kH, kTopK, slots, + TILEXR_MOONEP_DTYPE_BFLOAT16, &caseWorkspaceBytes, + &profileOffset, &outputOffsets[0], &outputOffsets[1]); + if (ret != TILEXR_MOONEP_SUCCESS || caseWorkspaceBytes == 0) { + Abort(rank, "TileXRMoonEpCombineGetWorkspaceSizeV2 profile", ret); + } + const std::size_t sourceElements = static_cast(slots) * + static_cast(kH); + const std::size_t sourceBytes = sourceElements * sizeof(uint16_t); + const std::size_t dstBytes = static_cast(slots) * + sizeof(int32_t); + std::vector source(sourceElements, SourceValue(rank)); + std::vector destinations(static_cast(slots)); + const int64_t slotsPerRank = slots / world; + for (int64_t slot = 0; slot < slots; ++slot) { + const int64_t targetRank = slot % world; + const int64_t targetSlot = static_cast(rank) * + slotsPerRank + slot / world; + destinations[static_cast(slot)] = + static_cast(targetRank * slots + targetSlot); + } + CheckAcl(rank, "input H2D copy", aclrtMemcpy(workspace, + static_cast(maxWorkspaceBytes), source.data(), + sourceBytes, ACL_MEMCPY_HOST_TO_DEVICE)); + CheckAcl(rank, "destinations H2D copy", aclrtMemcpy(dst, + maxDstBytes, destinations.data(), dstBytes, + ACL_MEMCPY_HOST_TO_DEVICE)); + source.clear(); + source.shrink_to_fit(); + destinations.clear(); + destinations.shrink_to_fit(); + if (!BarrierAll(rank, world, "case inputs ready")) { + Abort(rank, "case input barrier", 1); + } + + uint64_t activeOutputOffset = 0; + LaunchCombine(rank, workspace, dst, comm, bs, stream, + &activeOutputOffset); + CheckAcl(rank, "correctness stream synchronization", + aclrtSynchronizeStream(stream)); + const OutputCheckResult outputResult = CheckOutput( + rank, world, bs, workspace, + activeOutputOffset); + const bool validationAccepted = + outputResult == OutputCheckResult::Passed || + (options.allowSelfOnlyFailure && + outputResult == OutputCheckResult::SelfOnlyFailed); + if (!BarrierAll(rank, world, "correctness validation", + validationAccepted)) { + allCasesOk = false; + std::cout << "COMBINE_V2_RANK_PERF bs=" << bs + << " rank=" << rank + << " correctness=failed" << std::endl; + break; + } + + for (int iteration = 0; iteration < options.warmup; ++iteration) { + LaunchCombine(rank, workspace, dst, comm, bs, stream, + &activeOutputOffset); + CheckAcl(rank, "warmup stream synchronization", + aclrtSynchronizeStream(stream)); + if (!options.skipIterationBarriers && + !BarrierAll(rank, world, "warmup iteration")) { + Abort(rank, "warmup barrier", 1); } } + + std::vector rankSamples; + rankSamples.reserve(static_cast(options.iterations)); + std::vector profileSamples; + if (options.profile) { + profileSamples.reserve(static_cast(options.iterations) * + TileXRMoonEp::MoonEpCombineV2ActiveCoreCount( + static_cast(world))); + } + for (int iteration = 0; iteration < options.iterations; ++iteration) { + CheckAcl(rank, "aclrtRecordEvent start", + aclrtRecordEvent(startEvent, stream)); + LaunchCombine(rank, workspace, dst, comm, bs, stream, + &activeOutputOffset); + CheckAcl(rank, "aclrtRecordEvent stop", + aclrtRecordEvent(stopEvent, stream)); + CheckAcl(rank, "aclrtSynchronizeEvent stop", + aclrtSynchronizeEvent(stopEvent)); + float elapsedMs = 0.0F; + CheckAcl(rank, "aclrtEventElapsedTime", + aclrtEventElapsedTime(&elapsedMs, startEvent, stopEvent)); + rankSamples.push_back(elapsedMs); + if (options.profile) { + CaptureProfileSamples(rank, world, iteration, workspace, + profileOffset, &profileSamples); + } + if (!options.skipIterationBarriers && + !BarrierAll(rank, world, "timed iteration")) { + Abort(rank, "timed iteration barrier", 1); + } + } + + for (int iteration = 0; iteration < options.iterations; ++iteration) { + std::cout << std::fixed << std::setprecision(6) + << "COMBINE_V2_SAMPLE bs=" << bs + << " iteration=" << iteration + << " rank=" << rank + << " elapsed_ms=" << rankSamples[static_cast(iteration)] + << std::endl; + } + for (const ProfileSample &sample : profileSamples) { + std::cout << "COMBINE_V2_PROFILE bs=" << bs + << " iteration=" << sample.iteration + << " rank=" << rank + << " core=" << sample.core + << " cycles_per_us=" + << TileXRMoonEp::kMoonEpCombineV2ProfileCyclesPerUs; + for (uint32_t point = 0U; + point < TileXRMoonEp::kMoonEpCombineV2ProfileTimePointCount; + ++point) { + std::cout << " t" << point << '=' << sample.timePoint[point]; + } + std::cout << std::fixed << std::setprecision(3); + for (uint32_t metric = 0U; + metric < TileXRMoonEp::kMoonEpCombineV2ProfileMetricCount; + ++metric) { + std::cout << ' ' << kProfileMetricNames[metric] << '=' + << static_cast(sample.metric[metric]) / + TileXRMoonEp::kMoonEpCombineV2ProfileCyclesPerUs; + } + std::cout << std::endl; + } + const float total = std::accumulate(rankSamples.begin(), + rankSamples.end(), 0.0F); + std::cout << std::fixed << std::setprecision(6) + << "COMBINE_V2_RANK_PERF bs=" << bs + << " rank=" << rank + << " avg_ms=" << total / static_cast(rankSamples.size()) + << " correctness=" << OutputCheckName(outputResult) + << std::endl; + allCasesOk = allCasesOk && validationAccepted; } - int globalOk = 0; - MPI_Allreduce(&localOk, &globalOk, 1, MPI_INT, MPI_MIN, MPI_COMM_WORLD); - MPI_Barrier(MPI_COMM_WORLD); + const bool casesSynchronized = BarrierAll(rank, world, + "all benchmark cases", allCasesOk); const int unregisterRet = TileXRUDMAUnregister(comm, handle); - MPI_Barrier(MPI_COMM_WORLD); - + const bool unregisterSynchronized = BarrierAll(rank, world, + "workspace unregistration", + unregisterRet == TileXR::TILEXR_SUCCESS); const int destroyRet = TileXRCommDestroy(comm); + const aclError destroyStartRet = aclrtDestroyEvent(startEvent); + const aclError destroyStopRet = aclrtDestroyEvent(stopEvent); const aclError freeDstRet = aclrtFree(dst); const aclError freeWorkspaceRet = aclrtFree(workspace); const aclError destroyStreamRet = aclrtDestroyStream(stream); const aclError resetRet = aclrtResetDevice(device); const aclError finalizeRet = aclFinalize(); - const int cleanupOk = unregisterRet == TileXR::TILEXR_SUCCESS && - destroyRet == TileXR::TILEXR_SUCCESS && freeDstRet == ACL_SUCCESS && - freeWorkspaceRet == ACL_SUCCESS && destroyStreamRet == ACL_SUCCESS && - resetRet == ACL_SUCCESS && finalizeRet == ACL_SUCCESS; - int globalCleanupOk = 0; - MPI_Allreduce(&cleanupOk, &globalCleanupOk, 1, MPI_INT, MPI_MIN, - MPI_COMM_WORLD); - - if (rank == 0) { - std::cout << "MoonEP Combine V2 hardware probe " - << (globalOk != 0 && globalCleanupOk != 0 ? "passed" : "failed") - << " logical_ranks=" << world - << " physical_devices=" << kDeviceCount - << " invocations=" << kInvocationCount - << " qp_count=" << qpCount - << " workspace_bytes=" << workspaceBytes - << " profile_offset=" << profileOffset - << " active_output_offset=" << activeOutputOffset - << std::endl; - } - MPI_Finalize(); - return globalOk != 0 && globalCleanupOk != 0 ? 0 : 1; + const bool cleanupOk = unregisterRet == TileXR::TILEXR_SUCCESS && + destroyRet == TileXR::TILEXR_SUCCESS && + destroyStartRet == ACL_SUCCESS && destroyStopRet == ACL_SUCCESS && + freeDstRet == ACL_SUCCESS && freeWorkspaceRet == ACL_SUCCESS && + destroyStreamRet == ACL_SUCCESS && resetRet == ACL_SUCCESS && + finalizeRet == ACL_SUCCESS; + return allCasesOk && casesSynchronized && unregisterSynchronized && + cleanupOk ? 0 : 1; } diff --git a/tests/moonep_combine_v2/unit/test_combine_v2_layout.cpp b/tests/moonep_combine_v2/unit/test_combine_v2_layout.cpp index ec911ed..9f6e723 100644 --- a/tests/moonep_combine_v2/unit/test_combine_v2_layout.cpp +++ b/tests/moonep_combine_v2/unit/test_combine_v2_layout.cpp @@ -30,10 +30,12 @@ void CheckStatus(int actual, int expected, const char *message) void TestTargetLayout() { using namespace TileXRMoonEp; - Check(sizeof(MoonEpCombineV2ProfileRecord) == 320U, + Check(sizeof(MoonEpCombineV2ProfileRecord) == 384U, "profile record size mismatch"); Check(kMoonEpCombineV2ProfileTimePointCount == 22U, "profile point count mismatch"); + Check(kMoonEpCombineV2ProfileMetricCount == 8U, + "profile metric count mismatch"); CombineV2Layout layout {}; CheckStatus(TileXRMoonEpBuildCombineV2Layout( @@ -50,9 +52,15 @@ void TestTargetLayout() Check(layout.scratchOffset[1] == layout.scratchOffset[0] + layout.expertBytes, "scratch epoch 1 offset mismatch"); Check(layout.doneBytes == 32768U, "done bytes mismatch"); - Check(layout.grantBytes == 229376U, "grant bytes mismatch"); + Check(layout.grantBytes == 262144U, "grant bytes mismatch"); + Check(layout.controlSourceOffset == 2818873344ULL, + "control source offset mismatch"); + Check(layout.failureOffset == 2818875392ULL, + "failure offset mismatch"); Check(layout.controlSourceBytes == 2048U, "control source bytes mismatch"); Check(layout.failureBytes == 2048U, "failure bytes mismatch"); + Check(layout.outputOffset == 2818877440ULL, + "target output offset mismatch"); Check(layout.outputOffset == layout.failureOffset + layout.failureBytes, "output offset mismatch"); Check(layout.outputBytes == 58720256U, "output bytes mismatch"); diff --git a/tests/moonep_combine_v2/unit/test_combine_v2_schedule.cpp b/tests/moonep_combine_v2/unit/test_combine_v2_schedule.cpp index 3bc7248..9ea5acb 100644 --- a/tests/moonep_combine_v2/unit/test_combine_v2_schedule.cpp +++ b/tests/moonep_combine_v2/unit/test_combine_v2_schedule.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include "combine_v2_schedule.h" #include "combine_v2_wqe_batch.h" @@ -17,6 +18,56 @@ void Check(bool condition, const char *message) } } +std::vector SelectPeerRoutesReference( + const std::vector &dst, uint32_t chunkStart, + uint32_t peer, uint32_t slots) +{ + using namespace TileXRMoonEp; + std::array cursor {}; + std::vector selected; + bool firstPass = true; + uint32_t paused = 0U; + do { + const uint32_t batchBase = static_cast(selected.size()); + paused = 0U; + for (uint32_t thread = 0U; + thread < kMoonEpCombineV2SelectorThreads; ++thread) { + uint32_t index = firstPass ? + MoonEpCombineV2SelectorFirstIndex(chunkStart, thread) : + MoonEpCombineV2SelectorResumeIndex(cursor[thread]); + uint32_t lastScanned = index; + while (MoonEpCombineV2SelectorIndexInChunk( + index, chunkStart, static_cast(dst.size()))) { + lastScanned = index; + const int32_t encoded = dst[index - chunkStart]; + if (encoded >= 0 && + static_cast(encoded) / slots == peer) { + const uint32_t routeIndex = + static_cast(selected.size()) - batchBase; + selected.push_back(index); + if (routeIndex >= kMoonEpCombineV2PayloadBatchRows) { + cursor[thread] = lastScanned; + const uint32_t next = + MoonEpCombineV2SelectorResumeIndex(index); + if (MoonEpCombineV2SelectorIndexInChunk(next, + chunkStart, static_cast(dst.size()))) { + ++paused; + } + break; + } + } + index = MoonEpCombineV2SelectorResumeIndex(index); + } + cursor[thread] = lastScanned; + } + Check(selected.size() - batchBase <= + kMoonEpCombineV2MaxSelectedPayloadWqes, + "selector reference batch exceeded the route capacity"); + firstPass = false; + } while (paused != 0U); + return selected; +} + void TestSchedule() { using namespace TileXRMoonEp; @@ -179,7 +230,7 @@ void TestTokensAndShapes() "stale magic accepted"); Check(!MoonEpCombineV2TokenMatches(token, magic, 6U, 64U), "step outside runtime schedule accepted"); - Check(MoonEpCombineV2GrantIndex(1U, 0U, 0U, 1U) == 224U, + Check(MoonEpCombineV2GrantIndex(1U, 0U, 0U, 0U) == 256U, "grant epoch layout mismatch"); Check(MoonEpCombineV2DestinationValid(0, kMoonEpCombineV2SmallSlots, 128U), "valid destination rejected"); @@ -192,16 +243,62 @@ void TestTokensAndShapes() void TestWqeBatchHelpers() { using namespace TileXRMoonEp; + Check(kMoonEpCombineV2PayloadBatchRows == 128U, + "payload batch tuning constant mismatch"); + Check(kMoonEpCombineV2SelectorThreads == 128U && + kMoonEpCombineV2BuilderThreads == 128U, + "SIMT thread counts are not derived from payload batch rows"); + Check(kMoonEpCombineV2MaxSelectedPayloadWqes == 256U, + "selector payload bound mismatch"); Check(kMoonEpCombineV2WqeBatchCapacity == 128U, "WQE batch capacity mismatch"); Check(kMoonEpCombineV2BatchQpCount == 2U, "WQE batch QP count mismatch"); + Check(kMoonEpCombineV2SelfRelayHalfBytes == 64U * 1024U, + "Self relay half size mismatch"); + Check(MoonEpCombineV2SelfRowsPerBatch(7168U) == 8U, + "H=3584 BF16 must use eight Self rows per batch"); + Check(MoonEpCombineV2SelfRowsPerBatch(8192U) == 8U, + "8 KiB rows must use eight Self rows per batch"); + Check(MoonEpCombineV2SelfRowsPerBatch(14336U) == 4U, + "14 KiB rows must use four Self rows per batch"); + Check(MoonEpCombineV2SelfRowsPerBatch(32768U) == 2U, + "32 KiB rows must use two Self rows per batch"); + Check(MoonEpCombineV2SelfRowsPerBatch(65536U) == 1U, + "64 KiB rows must use one Self row per batch"); + Check(MoonEpCombineV2SelfRowsPerBatch(65537U) == 0U, + "oversized Self rows must select tiled copy mode"); + Check(MoonEpCombineV2SelfTileCount(65537U) == 2U, + "oversized Self row tile count mismatch"); + Check(MoonEpCombineV2SelfTileBytes(65537U, 0U) == 65536U, + "oversized Self row first tile mismatch"); + Check(MoonEpCombineV2SelfTileBytes(65537U, 1U) == 1U, + "oversized Self row final partial tile mismatch"); + Check(MoonEpCombineV2SelfTileBytes(65537U, 2U) == 0U, + "out-of-range Self row tile was not rejected"); + Check(MoonEpCombineV2SelfAlignedRowBytes(7169U) == 7200U, + "Self relay row alignment mismatch"); + Check(MoonEpCombineV2SelfAlignedRowBytes(UINT64_MAX) == 0U, + "Self relay alignment overflow was not rejected"); Check(MoonEpCombineV2WqeBatchCount(0U, 0U, 16384U) == 0U, "empty WQE batch mismatch"); Check(MoonEpCombineV2WqeBatchCount(129U, 0U, 16384U) == 128U, "WQE batch capacity not enforced"); Check(MoonEpCombineV2WqeBatchCount(128U, 16380U, 16384U) == 4U, "WQE ring wrap not split"); + Check(MoonEpCombineV2SelectorFirstIndex(16384U, 127U) == 16511U, + "selector first index mismatch"); + Check(MoonEpCombineV2SelectorResumeIndex(16511U) == 16639U, + "selector resume index mismatch"); + Check(MoonEpCombineV2SelectorIndexInChunk( + 16511U, 16384U, 128U), + "selector rejected the last full-chunk index"); + Check(!MoonEpCombineV2SelectorIndexInChunk( + 16512U, 16384U, 128U), + "selector accepted an index after the chunk"); + Check(!MoonEpCombineV2SelectorIndexInChunk( + 130U, 128U, 2U), + "selector accepted a tail-thread index after the chunk"); Check(MoonEpCombineV2CqePollBatchCount(16380U, 16384U, 128U) == 4U, "CQ poll ring wrap not split"); Check(MoonEpCombineV2CqeOwnerReady(0U, 16384U, 1U), @@ -221,8 +318,10 @@ void TestWqeBatchHelpers() "CQ error detail mismatch"); for (uint32_t phase = 0; phase < 4U; ++phase) { - for (uint32_t tokenCount = 0; tokenCount <= 128U; ++tokenCount) { - std::array seen {}; + for (uint32_t tokenCount = 0; + tokenCount <= kMoonEpCombineV2MaxSelectedPayloadWqes; + ++tokenCount) { + std::array seen {}; uint32_t selectedCount = 0; for (uint32_t qp = 0; qp < kMoonEpCombineV2BatchQpCount; ++qp) { const uint32_t qpTokens = MoonEpCombineV2QpTokenCount( @@ -240,6 +339,278 @@ void TestWqeBatchHelpers() "QP split did not cover every token"); } } + + for (uint32_t phase = 0; phase < 4U; ++phase) { + const MoonEpCombineV2LaneCounts payload = + MoonEpCombineV2BatchLaneCounts( + kMoonEpCombineV2MaxSelectedPayloadWqes, phase, + 0U, kMoonEpCombineV2StepCount, false); + Check(payload.sixPort == 192U && payload.twoPort == 64U, + "maximum payload does not fit the 192/64 QP split"); + const MoonEpCombineV2LaneCounts finalBatch = + MoonEpCombineV2BatchLaneCounts( + kMoonEpCombineV2MaxSelectedPayloadWqes, phase, + 0U, kMoonEpCombineV2StepCount, true); + Check(finalBatch.sixPort == 194U && finalBatch.twoPort == 66U, + "maximum final batch does not fit the 194/66 issue split"); + } +} + +uint32_t EffectivePeer(uint32_t peer, uint32_t source) +{ + return peer == TileXRMoonEp::kMoonEpCombineV2InvalidPeer ? + source : peer; +} + +void TestFullGrantRounds() +{ + using namespace TileXRMoonEp; + Check(kMoonEpCombineV2GrantStepCount == kMoonEpCombineV2StepCount, + "grant storage does not cover every data step"); + Check(MoonEpCombineV2GrantIndex(0U, 0U, 0U, 0U) == 0U, + "first grant transition index mismatch"); + Check(MoonEpCombineV2GrantIndex(0U, 0U, 0U, 7U) == 7U, + "terminal grant transition index mismatch"); + Check(MoonEpCombineV2GrantIndex(0U, 0U, 1U, 0U) == 8U, + "grant lane stride mismatch"); + Check(MoonEpCombineV2GrantIndex(1U, 0U, 0U, 0U) == 256U, + "grant epoch stride mismatch"); + Check(MoonEpCombineV2NextStep(6U, 8U) == 7U, + "non-terminal next step mismatch"); + Check(MoonEpCombineV2NextStep(7U, 8U) == 0U, + "terminal next step did not wrap"); + + const uint64_t magic = 17U; + Check(MoonEpCombineV2GrantToken(magic, 6U, 8U) == + MoonEpCombineV2Token(magic, 7U), + "non-terminal grant token mismatch"); + Check(MoonEpCombineV2GrantToken(magic, 7U, 8U) == + MoonEpCombineV2Token(magic, 0U), + "terminal grant token did not wrap to step zero"); + Check(MoonEpCombineV2ControlWqesPerLane(7U, 8U, true) == 2U, + "final data step does not reserve grant plus done controls"); + + const MoonEpCombineV2ScheduleMode modes[] = { + MOONEP_COMBINE_V2_SINGLE_RING, + MOONEP_COMBINE_V2_BIDIRECTIONAL_RING}; + const uint32_t supported[] = {2U, 3U, 4U, 5U, 6U, 7U, 8U, + 16U, 32U, 64U, 128U}; + for (MoonEpCombineV2ScheduleMode mode : modes) { + for (uint32_t rankSize : supported) { + const uint32_t activeCores = + MoonEpCombineV2ActiveCoreCount(rankSize); + const uint32_t stepCount = MoonEpCombineV2StepCount(rankSize); + for (uint32_t source = 0U; source < rankSize; ++source) { + for (uint32_t step = 0U; step < stepCount; ++step) { + const uint32_t nextStep = + MoonEpCombineV2NextStep(step, stepCount); + for (uint32_t core = 0U; core < activeCores; ++core) { + const uint32_t currentPeer = EffectivePeer( + MoonEpCombineV2Peer( + source, step, core, rankSize, mode), + source); + const uint32_t successor = MoonEpCombineV2Successor( + source, step, core, rankSize, mode); + Check(successor < rankSize, + "cyclic grant successor out of range"); + const uint32_t nextPeer = EffectivePeer( + MoonEpCombineV2Peer(successor, nextStep, core, + rankSize, mode), + successor); + Check(currentPeer == nextPeer, + "cyclic grant successor invariant mismatch"); + } + } + } + } + } +} + +void TestBidirectionalScheduleAnchors() +{ + using namespace TileXRMoonEp; + const MoonEpCombineV2ScheduleMode mode = + MOONEP_COMBINE_V2_BIDIRECTIONAL_RING; + + const std::array expected8 = { + 1U, 7U, 2U, 6U, 3U, 5U, 4U, + kMoonEpCombineV2InvalidPeer}; + for (uint32_t core = 0U; core < expected8.size(); ++core) { + Check(MoonEpCombineV2Peer(0U, 0U, core, 8U, mode) == + expected8[core], + "8P bidirectional anchor mismatch"); + } + + const std::array expected32Step0 = { + 1U, 15U, 2U, 14U, 3U, 13U, 4U, 12U, + 16U, 24U, 17U, 31U, 18U, 30U, 19U, 29U}; + const std::array expected32Step1 = { + 5U, 11U, 6U, 10U, 7U, 9U, 8U, + kMoonEpCombineV2InvalidPeer, + 20U, 28U, 21U, 27U, 22U, 26U, 23U, 25U}; + const std::array expected64Step0 = { + 1U, 31U, 2U, 30U, 3U, 29U, 4U, 28U, + 32U, 48U, 33U, 63U, 34U, 62U, 35U, 61U}; + const std::array expected64Step3 = { + 13U, 19U, 14U, 18U, 15U, 17U, 16U, + kMoonEpCombineV2InvalidPeer, + 44U, 52U, 45U, 51U, 46U, 50U, 47U, 49U}; + const std::array expected128Step0 = { + 1U, 63U, 2U, 62U, 3U, 61U, 4U, 60U, + 64U, 96U, 65U, 127U, 66U, 126U, 67U, 125U}; + const std::array expected128Step7 = { + 29U, 35U, 30U, 34U, 31U, 33U, 32U, + kMoonEpCombineV2InvalidPeer, + 92U, 100U, 93U, 99U, 94U, 98U, 95U, 97U}; + + for (uint32_t core = 0U; core < 16U; ++core) { + Check(MoonEpCombineV2Peer(0U, 0U, core, 32U, mode) == + expected32Step0[core], + "32P bidirectional first-step anchor mismatch"); + Check(MoonEpCombineV2Peer(0U, 1U, core, 32U, mode) == + expected32Step1[core], + "32P bidirectional final-step anchor mismatch"); + Check(MoonEpCombineV2Peer(0U, 0U, core, 64U, mode) == + expected64Step0[core], + "64P bidirectional first-step anchor mismatch"); + Check(MoonEpCombineV2Peer(0U, 3U, core, 64U, mode) == + expected64Step3[core], + "64P bidirectional final-step anchor mismatch"); + Check(MoonEpCombineV2Peer(0U, 0U, core, 128U, mode) == + expected128Step0[core], + "128P bidirectional first-step anchor mismatch"); + Check(MoonEpCombineV2Peer(0U, 7U, core, 128U, mode) == + expected128Step7[core], + "128P bidirectional final-step anchor mismatch"); + } + + const std::array expectedRank32Step0 = { + 33U, 31U, 34U, 30U, 35U, 29U, 36U, 28U, + 96U, 64U, 97U, 95U, 98U, 94U, 99U, 93U}; + for (uint32_t core = 0U; core < 16U; ++core) { + Check(MoonEpCombineV2Peer(32U, 0U, core, 128U, mode) == + expectedRank32Step0[core], + "128P nonzero-center bidirectional anchor mismatch"); + } +} + +void TestBidirectionalSchedule() +{ + using namespace TileXRMoonEp; + const MoonEpCombineV2ScheduleMode single = + MOONEP_COMBINE_V2_SINGLE_RING; + const MoonEpCombineV2ScheduleMode bidirectional = + MOONEP_COMBINE_V2_BIDIRECTIONAL_RING; + const uint32_t supported[] = {2U, 3U, 4U, 5U, 6U, 7U, 8U, + 16U, 32U, 64U, 128U}; + + for (uint32_t rankSize : supported) { + const uint32_t activeCores = MoonEpCombineV2ActiveCoreCount(rankSize); + const uint32_t stepCount = MoonEpCombineV2StepCount(rankSize); + for (uint32_t source = 0U; source < rankSize; ++source) { + std::array seen {}; + uint32_t invalidCount = 0U; + uint32_t selfCount = 0U; + for (uint32_t step = 0U; step < stepCount; ++step) { + for (uint32_t core = 0U; core < activeCores; ++core) { + Check(MoonEpCombineV2Peer( + source, step, core, rankSize, single) == + MoonEpCombineV2Peer( + source, step, core, rankSize), + "explicit single-ring mode changed the old peer order"); + const uint32_t rawPeer = MoonEpCombineV2Peer( + source, step, core, rankSize, bidirectional); + if (rawPeer == kMoonEpCombineV2InvalidPeer) { + ++invalidCount; + } else { + Check(rawPeer < rankSize, + "bidirectional peer out of range"); + } + const uint32_t peer = EffectivePeer(rawPeer, source); + Check(!seen[peer], "bidirectional peer repeated"); + seen[peer] = true; + if (peer == source) { + ++selfCount; + } + Check(MoonEpCombineV2ReceiveStep( + peer, source, rankSize, bidirectional) == step, + "bidirectional receive step does not invert peer"); + if (rankSize >= 16U) { + const uint32_t halfRankCount = rankSize / 2U; + Check(peer / halfRankCount == core / 8U, + "bidirectional core crossed target half"); + } + if (step + 1U < stepCount) { + const uint32_t successor = + MoonEpCombineV2Successor(source, step, core, + rankSize, bidirectional); + Check(successor / (rankSize / 2U) == + source / (rankSize / 2U), + "bidirectional successor crossed source half"); + const uint32_t nextRawPeer = MoonEpCombineV2Peer( + successor, step + 1U, core, rankSize, + bidirectional); + Check(peer == EffectivePeer(nextRawPeer, successor), + "bidirectional successor schedule mismatch"); + } + } + } + for (uint32_t peer = 0U; peer < rankSize; ++peer) { + Check(seen[peer], + "bidirectional schedule does not cover every peer"); + } + const bool optimized = rankSize == 8U || rankSize >= 16U; + Check(invalidCount == (optimized ? 1U : 0U), + "bidirectional invalid-peer count mismatch"); + Check(selfCount == 1U, + "bidirectional schedule must visit Self once"); + } + } + + TestBidirectionalScheduleAnchors(); +} + +void TestSelectorReferenceModel() +{ + using namespace TileXRMoonEp; + const uint32_t chunkStart = 32768U; + const uint32_t slots = 16384U; + + std::vector noMatches(23U, -1); + Check(SelectPeerRoutesReference(noMatches, chunkStart, 3U, slots).empty(), + "selector reference produced a route for an empty tail chunk"); + + std::vector concentrated(16384U, -1); + std::vector expected; + for (uint32_t index = 7U; index < concentrated.size(); + index += kMoonEpCombineV2SelectorThreads) { + concentrated[index] = static_cast(5U * slots + index); + expected.push_back(chunkStart + index); + } + const std::vector selected = SelectPeerRoutesReference( + concentrated, chunkStart, 5U, slots); + Check(selected == expected, + "selector reference skipped or duplicated concentrated routes"); + Check(selected.size() == 128U, + "concentrated selector case did not exercise one owning thread"); + + std::vector allMatches(1024U); + for (uint32_t index = 0U; index < allMatches.size(); ++index) { + allMatches[index] = static_cast(2U * slots + index); + } + const std::vector allSelected = SelectPeerRoutesReference( + allMatches, chunkStart, 2U, slots); + Check(allSelected.size() == allMatches.size(), + "selector reference did not resume through every matching route"); + std::array seen {}; + for (uint32_t absolute : allSelected) { + const uint32_t relative = absolute - chunkStart; + Check(relative < seen.size() && !seen[relative], + "selector reference duplicated an all-match route"); + if (relative < seen.size()) { + seen[relative] = true; + } + } } } // namespace @@ -247,8 +618,11 @@ void TestWqeBatchHelpers() int main() { TestSchedule(); + TestBidirectionalSchedule(); + TestFullGrantRounds(); TestQpAndBatchContract(); TestTokensAndShapes(); TestWqeBatchHelpers(); + TestSelectorReferenceModel(); return failures == 0 ? 0 : 1; } diff --git a/tests/moonep_combine_v2/unit/test_combine_v2_source_guard.cpp b/tests/moonep_combine_v2/unit/test_combine_v2_source_guard.cpp index abecb75..b34c694 100644 --- a/tests/moonep_combine_v2/unit/test_combine_v2_source_guard.cpp +++ b/tests/moonep_combine_v2/unit/test_combine_v2_source_guard.cpp @@ -60,6 +60,30 @@ bool RequireBefore(const std::string &text, const std::string &first, return true; } +std::size_t CountOccurrences(const std::string &text, + const std::string &needle) +{ + std::size_t count = 0U; + for (std::size_t position = text.find(needle); + position != std::string::npos; + position = text.find(needle, position + needle.size())) { + ++count; + } + return count; +} + +std::string Section(const std::string &text, const std::string &begin, + const std::string &end) +{ + const std::size_t beginPos = text.find(begin); + if (beginPos == std::string::npos) { + return std::string(); + } + const std::size_t endPos = text.find(end, beginPos + begin.size()); + return text.substr(beginPos, endPos == std::string::npos ? + std::string::npos : endPos - beginPos); +} + } // namespace int main() @@ -73,6 +97,14 @@ int main() "/src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.cpp"); const std::string kernelImpl = Read(root + "/src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.h"); + const std::string profileHeader = Read(root + + "/src/moonep/combine_v2/common/combine_v2_profile.h"); + const std::string hardwareProbe = Read(root + + "/tests/moonep_combine_v2/demo/tilexr_moonep_combine_v2_hardware_probe.cpp"); + const std::string clusterLauncher = Read(root + + "/tools/moonep/run_combine_v2_perf_cluster.sh"); + const std::string multihostLauncher = Read(root + + "/tools/moonep/run_combine_v2_perf_multihost.sh"); const std::string host = Read(root + "/src/moonep/combine_v2/host/combine_v2_host.cpp"); const std::string launch = Read(root + @@ -81,6 +113,24 @@ int main() "/src/include/tilexr_moonep_combine_v2.h"); const std::string registration = Read(root + "/src/moonep/common/moonep_kernel_registration.h"); + const std::string peerPrefill = Section(kernelImpl, + "inline void MoonEpCombineV2PrefillPeerWqesVf(", + "inline void MoonEpCombineV2SelectPeerRoutesVf("); + const std::string submitPair = Section(kernelImpl, + "__aicore__ inline bool MoonEpCombineV2::SubmitPair(", + "__aicore__ inline bool MoonEpCombineV2::SendRemoteStep("); + const std::string selfStep = Section(kernelImpl, + "__aicore__ inline bool MoonEpCombineV2::SendSelfStep(", + "__aicore__ inline bool MoonEpCombineV2::WaitInboundDone("); + const std::string selfGrant = Section(kernelImpl, + "__aicore__ inline bool MoonEpCombineV2::SubmitSelfGrant(", + "__aicore__ inline bool MoonEpCombineV2::SendSelfStep("); + const std::string localGrant = Section(kernelImpl, + "__aicore__ inline void MoonEpCombineV2::PublishLocalGrant(", + "__aicore__ inline void MoonEpCombineV2::CopyIssueToSq("); + const std::string process = Section(kernelImpl, + "__aicore__ inline void MoonEpCombineV2::Process()", + "} // namespace"); bool ok = true; ok &= Require(rootCmake, "add_subdirectory(tests/moonep_combine_v2)", @@ -136,12 +186,152 @@ int main() "Combine V2 implementation does not use UDMA"); ok &= Require(kernelImpl, "UDMA_SHARED_QP", "Combine V2 implementation does not require shared QPs"); - ok &= Require(kernelImpl, "Simt::VF_CALL", + ok &= Require(kernelImpl, + "Simt::VF_CALL", + "Combine V2 implementation does not prefill operator WQE fields"); + ok &= Require(kernelImpl, + "Simt::VF_CALL", + "Combine V2 implementation does not prefill peer WQE fields"); + ok &= Require(peerPrefill, "words[word] = 0U;", + "Combine V2 peer prefill does not clear overwritten control WQEs"); + ok &= Require(peerPrefill, + "sqe->opcode = static_cast(TileXR::UDMAOpcode::WRITE);", + "Combine V2 peer prefill does not restore the payload opcode"); + ok &= Require(peerPrefill, "sqe->flag = 0U;", + "Combine V2 peer prefill does not restore the payload flag"); + ok &= Require(peerPrefill, + "sge->len = static_cast(fields->rowBytes);", + "Combine V2 peer prefill does not restore the payload SGE length"); + ok &= Require(peerPrefill, "sge->tokenId = 0U;", + "Combine V2 peer prefill does not restore the payload SGE token"); + ok &= Require(kernelImpl, + "Simt::VF_CALL", + "Combine V2 implementation does not use SIMT route selection"); + ok &= Require(kernelImpl, + "Simt::VF_CALL", "Combine V2 implementation does not use its SIMT WQE builder"); + ok &= Require(kernelImpl, "dstSlots[index - chunkStart]", + "Combine V2 selector does not use a chunk-relative UB index"); + ok &= Require(kernelImpl, + "const uint32_t old = asc_atomic_add(&state->curWqeNum, 1U);", + "Combine V2 selector does not reserve routes with UB atomic add"); + ok &= Require(kernelImpl, "old - state->batchBase", + "Combine V2 selector does not derive a batch-local route index"); + ok &= Require(kernelImpl, + "constexpr uint32_t kSixPortPayloadCapacity = 192U;", + "Combine V2 six-port payload capacity is missing"); + ok &= Require(kernelImpl, + "constexpr uint32_t kTwoPortPayloadCapacity = 64U;", + "Combine V2 two-port payload capacity is missing"); + ok &= Require(kernelImpl, + "constexpr uint32_t kSixPortIssueCapacity = 194U;", + "Combine V2 six-port issue capacity is missing"); + ok &= Require(kernelImpl, + "constexpr uint32_t kTwoPortIssueCapacity = 66U;", + "Combine V2 two-port issue capacity is missing"); + ok &= Require(kernelImpl, "TBuf wqeIssueBuf_;", + "Combine V2 continuous WQE issue buffer is missing"); + ok &= Require(kernelImpl, + "issue[kSixPortIssueBytes].GetPhyAddr()", + "Combine V2 two-port WQE view does not start after entry 194"); + ok &= Require(kernelImpl, + "pipe_->InitBuffer(selfCopyQueue_, 2U,", + "Combine V2 Self copy queue is not double buffered"); + ok &= Require(selfStep, "SelectPeerRoutes(", + "Combine V2 Self step does not reuse SIMT route selection"); + ok &= Require(selfStep, "CopySelfRouteBatch(selectedCount)", + "Combine V2 Self step does not consume compacted route batches"); + ok &= Require(selfStep, "SubmitSelfGrant(step)", + "Combine V2 Self step does not publish its step grant"); + ok &= Require(selfGrant, + "TileXR::TILEXR_UDMA_SQE_FLAG_ORDERED_COMPLETION", + "Combine V2 Self grant does not request ordered completion"); + ok &= Require(selfGrant, "if (successor == rank_)", + "Combine V2 Self grant does not handle a local successor"); + ok &= Require(selfGrant, "PublishLocalGrant(step, lane)", + "Combine V2 Self grant does not publish a local successor token"); + ok &= Require(localGrant, + "kMoonEpCombineV2GrantReceiveOffsetBytes", + "Combine V2 local grant does not target the receive slot"); + ok &= RequireBefore(selfGrant, "SyncFunc();", + "CopyIssueToSq(", + "Combine V2 Self grant does not establish MTE3 ordering"); + ok &= RequireBefore(selfGrant, "SyncFunc();", + "st_dev(lane_[lane].head", + "Combine V2 Self grant rings a doorbell before MTE3 completion"); + ok &= Require(kernelImpl, + "MoonEpCombineV2SelfRowsPerBatch(rowBytes_)", + "Combine V2 Self copy does not derive its runtime row batch"); + ok &= Require(kernelImpl, + "routes[routeStart + row]", + "Combine V2 Self copy does not consume RouteEntry records"); + ok &= RequireBefore(kernelImpl, "CopySelfRowsIn(", + "CopySelfRowsOut(", + "Combine V2 Self copy helpers are not ordered as copy-in/copy-out"); + ok &= RequireBefore(kernelImpl, + "SyncFunc();", + "CopySelfRowsOut(pendingStart", + "Combine V2 Self copy does not wait for MTE2 before MTE3"); + ok &= RequireBefore(kernelImpl, + "CopySelfRowsOut(pendingStart", + "selfCopyQueue_.FreeTensor(pending);", + "Combine V2 Self copy frees its relay before copy-out"); + ok &= Reject(kernelImpl, "dstRankBuf_", + "Combine V2 retained the obsolete dst-rank UB buffer"); + ok &= Reject(kernelImpl, "selectedIndexBuf_", + "Combine V2 retained the obsolete selected-index UB buffer"); + ok &= Reject(kernelImpl, "compareMaskBuf_", + "Combine V2 retained the obsolete compare-mask UB buffer"); + ok &= Reject(kernelImpl, "GatherMask(", + "Combine V2 retained the obsolete gather selection path"); + ok &= Reject(kernelImpl, "descriptorBuf_", + "Combine V2 retained the obsolete descriptor UB buffer"); + ok &= Reject(kernelImpl, "CopyBytesGmToGm(", + "Combine V2 retained the serialized Self-copy helper"); + ok &= RequireBefore(submitPair, "SyncFunc();", + "CopyIssueToSq(", + "Combine V2 does not establish scalar/SIMT-to-MTE3 ordering"); + ok &= RequireBefore(submitPair, "CopyIssueToSq(", + "SyncFunc();", + "Combine V2 does not wait for SQ MTE3 publication"); + ok &= RequireBefore(submitPair, "SyncFunc();", + "st_dev(lane_[0].head", + "Combine V2 rings a doorbell before SQ MTE3 completion"); + ok &= Require(submitPair, + "*grantSource = TileXRMoonEp::MoonEpCombineV2GrantToken(", + "Combine V2 remote path does not publish every step grant"); + ok &= Reject(submitPair, "step + 1U < stepCount_", + "Combine V2 remote path still suppresses the terminal grant"); + ok &= Reject(kernelImpl, "WaitAdmission", + "Combine V2 retained the pre-step admission wait"); + ok &= Reject(kernelImpl, "WaitFinalCqs", + "Combine V2 retained the standalone final CQ wait"); + ok &= RequireBefore(process, "SendRemoteStep(peer, step)", + "WaitStepCqs(step)", + "Combine V2 step loop does not send before waiting for CQ"); + ok &= RequireBefore(process, "WaitStepCqs(step)", + "WaitStepGrant(step)", + "Combine V2 step loop does not wait CQ before grant"); + ok &= RequireBefore(process, "WaitStepGrant(step)", + "WaitInboundDone()", + "Combine V2 final grant wait does not precede finalization"); ok &= Require(kernelImpl, "st_dev(", "Combine V2 implementation does not ring device doorbells"); ok &= Require(kernelImpl, "rank_, 0U, core_, rankSize_", "Combine V2 implementation does not use the runtime Ring schedule"); + if (CountOccurrences(kernelImpl, "kCombineV2ScheduleMode") != 7U) { + std::cerr << "Combine V2 schedule mode is not passed to all six " + "active schedule calls\n"; + ok = false; + } + ok &= Require(kernelImpl, + "TileXRMoonEp::MOONEP_COMBINE_V2_SINGLE_RING;", + "Combine V2 default schedule is not the existing single ring"); + ok &= Require(kernelImpl, + "peer == TileXRMoonEp::kMoonEpCombineV2InvalidPeer", + "Combine V2 invalid bidirectional peer does not use Self processing"); + ok &= Require(kernelImpl, "SendSelfStep(rank_, step)", + "Combine V2 Self processing does not use the local rank"); ok &= Require(kernelImpl, "step < stepCount_", "Combine V2 implementation does not use the runtime step count"); ok &= Require(kernelImpl, "core < activeCoreCount_", @@ -150,6 +340,10 @@ int main() "Combine V2 destination validation does not use runtime rank size"); ok &= Require(kernelImpl, "kMoonEpCombineV2MaxSourcesPerCore", "Combine V2 inbound Done polling is not rank generalized"); + ok &= Require(kernelImpl, "MOONEP_COMBINE_V2_METRIC_SELF_COPY", + "Combine V2 detailed self-copy profiling is missing"); + ok &= Require(kernelImpl, "MOONEP_COMBINE_V2_METRIC_REMOTE_WQE_BUILD", + "Combine V2 detailed remote WQE profiling is missing"); ok &= Reject(kernelImpl, "SyncAll();", "Combine V2 implementation still uses whole-launch barriers"); ok &= Reject(kernelImpl, "extern \"C\" __global__", @@ -179,6 +373,36 @@ int main() "Combine V2 workspace API is missing"); ok &= Require(publicHeader, "TileXRMoonEpCombineV2", "Combine V2 launch API is missing"); + ok &= Require(profileHeader, + "kMoonEpCombineV2ProfileMetricCount = 8U", + "Combine V2 profile metric ABI is missing"); + ok &= Require(profileHeader, + "kMoonEpCombineV2ProfileVersion = 3U", + "Combine V2 full-grant profile version is missing"); + ok &= Require(profileHeader, + "MOONEP_COMBINE_V2_TIME_STEP0_READY_END", + "Combine V2 profile does not expose post-grant readiness"); + ok &= Require(hardwareProbe, "self_copy_us", + "Combine V2 probe does not print self-copy profiling"); + ok &= Require(hardwareProbe, "remote_submit_us", + "Combine V2 probe does not print remote-submit profiling"); + ok &= Require(hardwareProbe, "enum class OutputCheckResult", + "Combine V2 probe does not classify correctness failures"); + ok &= Require(hardwareProbe, "if (sourceRank != rank)", + "Combine V2 probe does not inspect remote-output source ranks"); + ok &= Require(hardwareProbe, "return OutputCheckResult::Failed;", + "Combine V2 probe does not reject remote-output mismatches"); + ok &= Require(hardwareProbe, "OutputCheckResult::SelfOnlyFailed", + "Combine V2 probe does not identify disabled self-copy mismatches"); + ok &= Require(clusterLauncher, "#!/usr/bin/env bash", + "Combine V2 cluster launcher is not a Bash script"); + ok &= Require(clusterLauncher, + "bash \"${run_script}\" \"${run_args[@]}\"", + "Combine V2 cluster launcher does not start the server-side launcher"); + ok &= Reject(Lower(clusterLauncher), "powershell", + "Combine V2 cluster launcher must not depend on PowerShell"); + ok &= Reject(Lower(multihostLauncher), "powershell", + "Combine V2 multihost launcher must not depend on PowerShell"); const std::vector v2Files { root + "/src/include/tilexr_moonep_combine_v2.h", @@ -195,6 +419,8 @@ int main() root + "/src/moonep/combine_v2/host/tilexr_moonep_combine_v2.cpp", root + "/src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.h", root + "/src/moonep/combine_v2/kernels/tilexr_moonep_combine_v2_kernel.cpp", + root + "/tools/moonep/run_combine_v2_perf_cluster.sh", + root + "/tools/moonep/run_combine_v2_perf_multihost.sh", }; std::string allV2; for (const std::string &path : v2Files) { diff --git a/tools/moonep/build_combine_v2_perf.sh b/tools/moonep/build_combine_v2_perf.sh new file mode 100644 index 0000000..156493a --- /dev/null +++ b/tools/moonep/build_combine_v2_perf.sh @@ -0,0 +1,122 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SOURCE_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" +BUILD_DIR="${SOURCE_DIR}/build-combine-v2-perf" +INSTALL_DIR="${SOURCE_DIR}/install-combine-v2-perf" +CANN_PATH="${ASCEND_HOME_PATH:-/usr/local/Ascend/ascend-toolkit/latest}" +JOBS="$(nproc)" +ENABLE_PROFILING=0 + +usage() { + cat <<'EOF' +Usage: bash tools/moonep/build_combine_v2_perf.sh [options] + +Options: + --source-dir PATH TileXR source directory + --build-dir PATH CMake build directory + --install-dir PATH Staged runtime directory + --cann-path PATH CANN root containing aarch64-linux + --jobs N Parallel build jobs (default: nproc) + --enable-profiling Enable per-AIV Combine V2 kernel profiling + --help Show this help +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --source-dir) SOURCE_DIR="$2"; shift 2 ;; + --build-dir) BUILD_DIR="$2"; shift 2 ;; + --install-dir) INSTALL_DIR="$2"; shift 2 ;; + --cann-path) CANN_PATH="$2"; shift 2 ;; + --jobs) JOBS="$2"; shift 2 ;; + --enable-profiling) ENABLE_PROFILING=1; shift ;; + --help|-h) usage; exit 0 ;; + *) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac +done + +if [[ ! "${JOBS}" =~ ^[1-9][0-9]*$ ]]; then + echo "--jobs must be a positive integer" >&2 + exit 2 +fi +if [[ ! -f "${SOURCE_DIR}/CMakeLists.txt" ]]; then + echo "TileXR source not found: ${SOURCE_DIR}" >&2 + exit 1 +fi +if [[ ! -d "${CANN_PATH}/aarch64-linux" ]]; then + echo "CANN aarch64-linux directory not found: ${CANN_PATH}" >&2 + exit 1 +fi +mkdir -p "${BUILD_DIR}" "${INSTALL_DIR}/bin" "${INSTALL_DIR}/lib64" +export ASCEND_HOME_PATH="${CANN_PATH}" +export ASCEND_DRIVER_PATH="${ASCEND_DRIVER_PATH:-/usr/local/Ascend/driver}" +profiling_cmake_value=OFF +if (( ENABLE_PROFILING )); then + profiling_cmake_value=ON +fi + +cmake -S "${SOURCE_DIR}" -B "${BUILD_DIR}" \ + -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_TESTING=ON \ + -DTILEXR_BUILD_TESTS=OFF \ + -DTILEXR_BUILD_MOONEP=ON \ + -DTILEXR_BUILD_MOONEP_PLANNER=OFF \ + -DTILEXR_MOONEP_COMBINE_V2_ENABLE_PROFILING="${profiling_cmake_value}" \ + -DCMAKE_INSTALL_PREFIX="${INSTALL_DIR}" +cmake --build "${BUILD_DIR}" --target tilexr_moonep_combine_v2_perf \ + --parallel "${JOBS}" + +benchmark="${BUILD_DIR}/tests/moonep_combine_v2/tilexr_moonep_combine_v2_perf" +comm_library="${BUILD_DIR}/src/comm/libtile-comm.so" +combine_dir="${BUILD_DIR}/src/moonep/combine_v2" +if [[ ! -x "${benchmark}" || ! -f "${comm_library}" ]]; then + echo "expected benchmark artifacts were not built" >&2 + exit 1 +fi +if ! compgen -G "${combine_dir}/libtilexr-moonep-combine-v2.so*" >/dev/null; then + echo "Combine V2 library was not built under ${combine_dir}" >&2 + exit 1 +fi + +rm -f "${INSTALL_DIR}/lib64/libtile-comm.so" \ + "${INSTALL_DIR}/lib64"/libtilexr-moonep-combine-v2.so* +install -m 0755 "${benchmark}" \ + "${INSTALL_DIR}/bin/tilexr_moonep_combine_v2_perf" +install -m 0755 "${comm_library}" "${INSTALL_DIR}/lib64/libtile-comm.so" +cp -a "${combine_dir}"/libtilexr-moonep-combine-v2.so* \ + "${INSTALL_DIR}/lib64/" + +runtime_ld_path="${INSTALL_DIR}/lib64:${CANN_PATH}/aarch64-linux/lib64:${CANN_PATH}/lib64:${ASCEND_DRIVER_PATH}/lib64:${ASCEND_DRIVER_PATH}/lib64/common:${ASCEND_DRIVER_PATH}/lib64/driver" +benchmark_ldd="$(LD_LIBRARY_PATH="${runtime_ld_path}:${LD_LIBRARY_PATH:-}" \ + ldd "${INSTALL_DIR}/bin/tilexr_moonep_combine_v2_perf")" +if grep -q 'not found' <<<"${benchmark_ldd}"; then + printf '%s\n' "${benchmark_ldd}" >&2 + exit 1 +fi +if grep -Eiq 'libmpi|libmpicxx' <<<"${benchmark_ldd}"; then + echo "benchmark unexpectedly depends on MPI" >&2 + printf '%s\n' "${benchmark_ldd}" >&2 + exit 1 +fi +if readelf -d "${INSTALL_DIR}/bin/tilexr_moonep_combine_v2_perf" | \ + grep -Fq "${BUILD_DIR}"; then + echo "benchmark RPATH still references the build directory" >&2 + exit 1 +fi +staged_library_dir="$(readlink -f "${INSTALL_DIR}/lib64")" +for library in libtile-comm.so libtilexr-moonep-combine-v2.so; do + resolved_library="$(awk -v prefix="${library}" \ + 'index($1, prefix) == 1 { print $3; exit }' <<<"${benchmark_ldd}")" + if [[ -z "${resolved_library}" || + "$(readlink -f "${resolved_library}")" != "${staged_library_dir}"/* ]]; then + echo "${library} was not resolved from the staged runtime" >&2 + printf '%s\n' "${benchmark_ldd}" >&2 + exit 1 + fi +done + +echo "Combine V2 benchmark staged at ${INSTALL_DIR}" +(cd "${INSTALL_DIR}" && find bin lib64 -type f -print0 | sort -z | \ + xargs -0 sha256sum) diff --git a/tools/moonep/configs/aggregate_combine_v2_rank_samples.awk b/tools/moonep/configs/aggregate_combine_v2_rank_samples.awk new file mode 100644 index 0000000..eb309ba --- /dev/null +++ b/tools/moonep/configs/aggregate_combine_v2_rank_samples.awk @@ -0,0 +1,35 @@ +$1 == "COMBINE_V2_SAMPLE" { + rank = "" + elapsed = "" + for (field = 2; field <= NF; ++field) { + split($field, item, "=") + if (item[1] == "rank") rank = item[2] + else if (item[1] == "elapsed_ms") elapsed = item[2] + } + count[rank]++ + total[rank] += elapsed +} + +END { + for (rank = 0; rank < expected_ranks; ++rank) { + if (count[rank] != expected_iterations) { + printf "BAD rank=%d samples=%d\n", rank, count[rank] + invalid = 1 + continue + } + average = total[rank] / expected_iterations + printf "RANK_AVG rank=%d avg_ms=%.9f\n", rank, average + sum += average + if (rank == 0 || average > maximum) { + maximum = average + maximum_rank = rank + } + } + if (invalid) exit 1 + + mean = sum / expected_ranks + data_bytes = bs * 16 * 3584 * 2 + printf "AGG avg_ms=%.9f avg_alg_bw_GBps=%.9f max_ms=%.9f max_rank=%d max_alg_bw_GBps=%.9f\n", \ + mean, data_bytes / mean / 1000000, maximum, maximum_rank, \ + data_bytes / maximum / 1000000 +} diff --git a/tools/moonep/configs/hostfile_cab9_4_128p.txt b/tools/moonep/configs/hostfile_cab9_4_128p.txt new file mode 100644 index 0000000..7510327 --- /dev/null +++ b/tools/moonep/configs/hostfile_cab9_4_128p.txt @@ -0,0 +1,16 @@ +141.61.55.118:8 +141.61.55.114:8 +141.61.55.110:8 +141.61.55.106:8 +141.61.55.78:8 +141.61.55.74:8 +141.61.55.70:8 +141.61.55.66:8 +141.61.52.116:8 +141.61.52.120:8 +141.61.52.128:8 +141.61.52.124:8 +141.61.52.156:8 +141.61.52.160:8 +141.61.52.164:8 +141.61.52.167:8 diff --git a/tools/moonep/configs/prepare_cab9_4_cann_runtime.sh b/tools/moonep/configs/prepare_cab9_4_cann_runtime.sh new file mode 100644 index 0000000..79df7f3 --- /dev/null +++ b/tools/moonep/configs/prepare_cab9_4_cann_runtime.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail + +runtime_link=/home/h00580772/tilexr_selfcopy_b150/cann_runtime +cabinet9=( + 141.61.55.118 141.61.55.114 141.61.55.110 141.61.55.106 + 141.61.55.78 141.61.55.74 141.61.55.70 141.61.55.66 +) +cabinet4=( + 141.61.52.116 141.61.52.120 141.61.52.128 141.61.52.124 + 141.61.52.156 141.61.52.160 141.61.52.164 141.61.52.167 +) + +prepare_group() { + local target=$1 + shift + local host + for host in "$@"; do + ssh -o BatchMode=yes -o ConnectTimeout=10 "root@${host}" \ + "test -d '${target}/aarch64-linux' && mkdir -p '$(dirname "${runtime_link}")' && ln -sfn '${target}' '${runtime_link}' && test \"\$(readlink '${runtime_link}')\" = '${target}'" + printf '%s: %s -> %s\n' "${host}" "${runtime_link}" "${target}" + done +} + +prepare_group /home/pkg/910_B150/cann-9.1.0 "${cabinet9[@]}" +prepare_group /home/pkg/b131/cann-9.1.0 "${cabinet4[@]}" diff --git a/tools/moonep/diagnose_combine_v2_e2e.py b/tools/moonep/diagnose_combine_v2_e2e.py deleted file mode 100644 index f11c2ed..0000000 --- a/tools/moonep/diagnose_combine_v2_e2e.py +++ /dev/null @@ -1,236 +0,0 @@ -from __future__ import annotations - -import ctypes -import os -import struct -import time -from pathlib import Path - -import torch -import torch.distributed as dist - -from tools.moonep.test_npu_e2e import Buffer, make_inputs, setup - - -def mismatch_summary(actual, expected): - mismatch = actual != expected - count = int(mismatch.sum().item()) - if count == 0: - return {"count": 0} - first = mismatch.nonzero()[0] - index = tuple(int(value) for value in first.cpu().tolist()) - delta = (actual.float() - expected.float()).abs() - return { - "count": count, - "index": index, - "actual": float(actual[index].float().item()), - "expected": float(expected[index].float().item()), - "max_abs": float(delta.max().item()), - } - - -def value_summary(tensor, limit=12): - values, counts = torch.unique(tensor.float(), return_counts=True) - pairs = list(zip(values.cpu().tolist(), counts.cpu().tolist())) - return { - "unique": len(pairs), - "values": pairs[:limit], - } - - -def hidden_scratch_views(buffer, tokens, hidden_size, topk, nv_s): - runtime = buffer._context.runtime - workspace_bytes = ctypes.c_uint64() - profile_offset = ctypes.c_uint64() - scratch0_offset = ctypes.c_uint64() - scratch1_offset = ctypes.c_uint64() - ret = runtime._combine_v2_lib.TileXRMoonEpCombineGetWorkspaceSizeV2( - ctypes.c_int64(tokens), - ctypes.c_int64(hidden_size), - ctypes.c_int64(topk), - ctypes.c_int64(nv_s), - ctypes.c_uint32(11), - ctypes.byref(workspace_bytes), - ctypes.byref(profile_offset), - ctypes.byref(scratch0_offset), - ctypes.byref(scratch1_offset), - ) - runtime._check("TileXRMoonEpCombineGetWorkspaceSizeV2", ret) - context = buffer._context - raw = context._dispatch_workspace_owner - aligned_offset = context._dispatch_workspace_ptr - int(raw.data_ptr()) - workspace = raw.narrow(0, aligned_offset, context._dispatch_workspace_bytes) - scratch_bytes = nv_s * hidden_size * 2 - return tuple( - workspace.narrow(0, int(offset.value), scratch_bytes) - .cpu() - .view(torch.bfloat16) - .reshape(nv_s, hidden_size) - for offset in (scratch0_offset, scratch1_offset) - ) - - -def combine_failure_records(buffer, hidden_size, nv_s): - profile_bytes = 16 * 320 - expert_bytes = nv_s * hidden_size * 2 - profile_offset = (expert_bytes + 63) // 64 * 64 - scratch0_offset = profile_offset + (profile_bytes + 63) // 64 * 64 - scratch1_offset = scratch0_offset + expert_bytes - done_offset = (scratch1_offset + expert_bytes + 63) // 64 * 64 - done_bytes = 2 * 128 * 2 * 64 - grant_bytes = 2 * 16 * 2 * 7 * 512 - control_source_bytes = 16 * 2 * 64 - failure_offset = done_offset + done_bytes + grant_bytes + control_source_bytes - - context = buffer._context - raw = context._dispatch_workspace_owner - aligned_offset = context._dispatch_workspace_ptr - int(raw.data_ptr()) - workspace = raw.narrow(0, aligned_offset, context._dispatch_workspace_bytes) - records = workspace.narrow(0, failure_offset, 2 * 16 * 64).cpu().tolist() - result = [] - for epoch in range(2): - for core in range(16): - offset = (epoch * 16 + core) * 64 - fields = struct.unpack_from("= deadline: - raise TimeoutError("timed out exchanging Planner dst snapshots") - time.sleep(0.01) - gathered_dst = [ - torch.load(exchange_dir / f"rank_{source_rank}.pt", - map_location="cpu", weights_only=True) - for source_rank in range(rank_size) - ] - expected_dst_local = torch.full( - (plan.NvS,), -1, dtype=torch.int32, device="cpu") - for source_rank, source_dst in enumerate(gathered_dst): - for route, encoded_tensor in enumerate(source_dst): - encoded = int(encoded_tensor.item()) - if encoded // plan.NvS == rank: - expected_dst_local[encoded % plan.NvS] = ( - source_rank * plan.NvS + route) - native_plan = plan._require_native() - dst_local_bytes = native_plan.workspace[ - native_plan.dst_local_offset: - native_plan.dst_local_offset + plan.NvS * 4 - ].cpu() - actual_dst_local = dst_local_bytes.view(torch.int32) - - dispatched_first, weights_first, _, _ = buffer.dispatch( - hidden, weights, topk_experts, tokens_per_expert) - output_first, _, _ = buffer.combine( - plan=plan, hidden_nvsh=dispatched_first) - torch.npu.synchronize() - dispatched_first = dispatched_first.clone() - output_first = output_first.clone() - scratch_epochs = hidden_scratch_views( - buffer, tokens, hidden_size, topk, plan.NvS) - expected_scratch = hidden.repeat_interleave(topk, dim=0).cpu() - scratch_summaries = [ - mismatch_summary(epoch[:tokens * topk], expected_scratch) - for epoch in scratch_epochs - ] - - dispatched_second, weights_second, _, _ = buffer.dispatch( - hidden, weights, topk_experts, tokens_per_expert) - output_second, gathered_weights, _ = buffer.combine( - plan=plan, hidden_nvsh=dispatched_second, - route_weights_nvs=weights_second) - torch.npu.synchronize() - - reference = (hidden.float() * float(topk)).to(torch.bfloat16) - print(f"[combine-v2-diagnostic rank {rank}] " - f"dst_local={mismatch_summary(actual_dst_local, expected_dst_local)} " - f"dst_local_routes={int((actual_dst_local >= 0).sum().item())} " - f"dispatch={mismatch_summary(dispatched_second, dispatched_first)} " - f"first_vs_reference={mismatch_summary(output_first, reference)} " - f"second_vs_reference={mismatch_summary(output_second, reference)} " - f"second_vs_first={mismatch_summary(output_second, output_first)} " - f"weights={mismatch_summary(gathered_weights, weights)} " - f"scratch={scratch_summaries} " - f"first_values={value_summary(output_first)}", - flush=True) - - for iteration in range(8): - hidden_loop, weights_loop, _, plan_loop = buffer.dispatch( - hidden, weights, topk_experts, tokens_per_expert, - zero_copy=True) - output_loop, gathered_loop, _ = buffer.combine( - plan=plan_loop, - hidden_nvsh=hidden_loop, - route_weights_nvs=weights_loop, - zero_copy=True, - ) - torch.npu.synchronize() - failures = combine_failure_records(buffer, hidden_size, plan_loop.NvS) - record_magics = { - epoch: max((record["magic"] for record in failures - if record["epoch"] == epoch), default=0) - for epoch in range(2) - } - nonzero_records = [ - record for record in failures if record["status"] != 0 - ] - output_summary = mismatch_summary(output_loop, reference) - weight_summary = mismatch_summary(gathered_loop, weights) - print(f"[combine-v2-loop rank {rank} iteration {iteration}] " - f"hidden={output_summary} weights={weight_summary} " - f"record_magics={record_magics} " - f"nonzero_records={nonzero_records}", flush=True) - if output_summary["count"] or weight_summary["count"] or any( - record["status"] != 0 for record in failures): - raise AssertionError("Combine V2 repeated-call diagnostic failed") - finally: - buffer.destroy() - if dist.is_initialized(): - dist.destroy_process_group() - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/tools/moonep/run_combine_v2_perf_cluster.sh b/tools/moonep/run_combine_v2_perf_cluster.sh new file mode 100644 index 0000000..866bba5 --- /dev/null +++ b/tools/moonep/run_combine_v2_perf_cluster.sh @@ -0,0 +1,210 @@ +#!/usr/bin/env bash +set -euo pipefail + +REMOTE_ROOT="" +HOSTFILE="" +SOURCE_DIR="" +BUILD_DIR="" +INSTALL_DIR="" +CANN_PATH="${ASCEND_HOME_PATH:-/usr/local/Ascend/ascend-toolkit/latest}" +SSH_USER="$(id -un)" +BS="" +BS_LIST="" +WARMUP=20 +ITERATIONS=80 +EXPERTS=64 +COMM_DOMAIN=141 +COMM_PORT=10067 +WAIT_SECONDS=120 +RETRY_SECONDS=15 +RANK_TIMEOUT=600 +BUILD_JOBS="$(nproc)" +LOG_FILE="" +SKIP_BUILD=0 +SKIP_RUNTIME_SYNC=0 +SKIP_ITERATION_BARRIERS=0 +PROFILE=0 +SKIP_NPU_PREFLIGHT=0 +ALLOW_SELF_ONLY_FAILURE=0 + +usage() { + cat <<'EOF' +Usage: bash tools/moonep/run_combine_v2_perf_cluster.sh --remote-root PATH [options] + +Options: + --remote-root PATH Remote task root containing source/build/install/logs + --source-dir PATH Source directory (default: REMOTE_ROOT/source) + --build-dir PATH Build directory (default: REMOTE_ROOT/build) + --install-dir PATH Runtime directory (default: REMOTE_ROOT/install) + --hostfile PATH Rank hostfile (default: REMOTE_ROOT/hostfile) + --cann-path PATH CANN root + --ssh-user USER SSH user for worker launch/sync (default: current user) + --bs N Run one batch size (default: 128) + --bs-list N[,N...] Run multiple batch sizes + --warmup N Warmup launches per BS (default: 20) + --iterations N Timed launches per BS (default: 80) + --experts N Total expert count (default: 64) + --comm-domain N Shared-QP domain (default: 141) + --comm-port N Bootstrap TCP port on first host (default: 10067) + --wait-seconds N Maximum NPU wait (default: 120) + --retry-seconds N NPU retry interval (default: 15) + --rank-timeout N Per-rank timeout (default: 600) + --build-jobs N Parallel build jobs (default: nproc) + --log-file PATH Controller log path + --skip-build Reuse an existing install directory + --skip-runtime-sync Do not rsync install to worker hosts + --skip-iteration-barriers Forward to benchmark launcher + --profile Capture per-AIV kernel cycle timestamps + --skip-npu-preflight Forward after manual NPU validation + --allow-self-only-failure Forward to benchmark launcher + --help Show this help +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --remote-root) REMOTE_ROOT="$2"; shift 2 ;; + --source-dir) SOURCE_DIR="$2"; shift 2 ;; + --build-dir) BUILD_DIR="$2"; shift 2 ;; + --install-dir) INSTALL_DIR="$2"; shift 2 ;; + --hostfile) HOSTFILE="$2"; shift 2 ;; + --cann-path) CANN_PATH="$2"; shift 2 ;; + --ssh-user) SSH_USER="$2"; shift 2 ;; + --bs) BS="$2"; shift 2 ;; + --bs-list) BS_LIST="$2"; shift 2 ;; + --warmup) WARMUP="$2"; shift 2 ;; + --iterations) ITERATIONS="$2"; shift 2 ;; + --experts) EXPERTS="$2"; shift 2 ;; + --comm-domain) COMM_DOMAIN="$2"; shift 2 ;; + --comm-port) COMM_PORT="$2"; shift 2 ;; + --wait-seconds) WAIT_SECONDS="$2"; shift 2 ;; + --retry-seconds) RETRY_SECONDS="$2"; shift 2 ;; + --rank-timeout) RANK_TIMEOUT="$2"; shift 2 ;; + --build-jobs) BUILD_JOBS="$2"; shift 2 ;; + --log-file) LOG_FILE="$2"; shift 2 ;; + --skip-build) SKIP_BUILD=1; shift ;; + --skip-runtime-sync) SKIP_RUNTIME_SYNC=1; shift ;; + --skip-iteration-barriers) SKIP_ITERATION_BARRIERS=1; shift ;; + --profile) PROFILE=1; shift ;; + --skip-npu-preflight) SKIP_NPU_PREFLIGHT=1; shift ;; + --allow-self-only-failure) ALLOW_SELF_ONLY_FAILURE=1; shift ;; + --help|-h) usage; exit 0 ;; + *) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac +done + +if [[ -z "${REMOTE_ROOT}" ]]; then + usage >&2 + exit 2 +fi +SOURCE_DIR="${SOURCE_DIR:-${REMOTE_ROOT}/source}" +BUILD_DIR="${BUILD_DIR:-${REMOTE_ROOT}/build}" +INSTALL_DIR="${INSTALL_DIR:-${REMOTE_ROOT}/install}" +HOSTFILE="${HOSTFILE:-${REMOTE_ROOT}/hostfile}" +if [[ -z "${LOG_FILE}" ]]; then + LOG_FILE="${REMOTE_ROOT}/logs/combine_v2_cluster_$(date +%Y%m%d_%H%M%S).log" +fi + +for path in "${REMOTE_ROOT}" "${SOURCE_DIR}" "${BUILD_DIR}" "${INSTALL_DIR}" \ + "${HOSTFILE}" "${CANN_PATH}" "${LOG_FILE}"; do + if [[ "${path}" != /* || "${path}" == *"'"* ]]; then + echo "paths must be absolute and cannot contain single quotes: ${path}" >&2 + exit 2 + fi +done +if [[ -n "${BS}" && -n "${BS_LIST}" ]]; then + echo "--bs and --bs-list are mutually exclusive" >&2 + exit 2 +fi +if [[ -z "${BS}" && -z "${BS_LIST}" ]]; then + BS=128 +fi +for value in "${WARMUP}" "${ITERATIONS}" "${EXPERTS}" "${COMM_DOMAIN}" \ + "${COMM_PORT}" "${WAIT_SECONDS}" "${RETRY_SECONDS}" "${RANK_TIMEOUT}" \ + "${BUILD_JOBS}"; do + if [[ ! "${value}" =~ ^[0-9]+$ ]]; then + echo "numeric arguments must be non-negative integers" >&2 + exit 2 + fi +done +if (( ITERATIONS == 0 || EXPERTS == 0 || COMM_DOMAIN == 0 || + COMM_PORT == 0 || COMM_PORT > 65535 || RETRY_SECONDS == 0 || + RANK_TIMEOUT == 0 || BUILD_JOBS == 0 )); then + echo "iterations, experts, domain, port, retry, timeout, and jobs must be positive" >&2 + exit 2 +fi +if [[ ! -f "${HOSTFILE}" ]]; then + echo "hostfile is missing: ${HOSTFILE}" >&2 + exit 1 +fi + +first_host="$(awk ' + /^[[:space:]]*($|#)/ { next } + { gsub(/[[:space:]]/, "", $0); split($0, item, ":"); print item[1]; exit } +' "${HOSTFILE}")" +if [[ -z "${first_host}" ]]; then + echo "hostfile has no hosts: ${HOSTFILE}" >&2 + exit 1 +fi + +build_script="${SOURCE_DIR}/tools/moonep/build_combine_v2_perf.sh" +sync_script="${SOURCE_DIR}/tools/moonep/sync_combine_v2_perf_runtime.sh" +run_script="${SOURCE_DIR}/tools/moonep/run_combine_v2_perf_multihost.sh" +for script in "${build_script}" "${sync_script}" "${run_script}"; do + if [[ ! -f "${script}" ]]; then + echo "required script is missing: ${script}" >&2 + exit 1 + fi +done + +if (( ! SKIP_BUILD )); then + bash "${build_script}" \ + --source-dir "${SOURCE_DIR}" \ + --build-dir "${BUILD_DIR}" \ + --install-dir "${INSTALL_DIR}" \ + --cann-path "${CANN_PATH}" \ + --jobs "${BUILD_JOBS}" +fi + +if (( ! SKIP_RUNTIME_SYNC )); then + bash "${sync_script}" \ + --hostfile "${HOSTFILE}" \ + --install-dir "${INSTALL_DIR}" \ + --ssh-user "${SSH_USER}" +fi + +run_args=( + --hostfile "${HOSTFILE}" + --install-dir "${INSTALL_DIR}" + --cann-path "${CANN_PATH}" + --ssh-user "${SSH_USER}" + --warmup "${WARMUP}" + --iterations "${ITERATIONS}" + --experts "${EXPERTS}" + --comm-domain "${COMM_DOMAIN}" + --comm-id "${first_host}:${COMM_PORT}" + --wait-seconds "${WAIT_SECONDS}" + --retry-seconds "${RETRY_SECONDS}" + --timeout "${RANK_TIMEOUT}" + --log-file "${LOG_FILE}" +) +if [[ -n "${BS_LIST}" ]]; then + run_args+=(--bs-list "${BS_LIST}") +else + run_args+=(--bs "${BS}") +fi +if (( SKIP_ITERATION_BARRIERS )); then + run_args+=(--skip-iteration-barriers) +fi +if (( PROFILE )); then + run_args+=(--profile) +fi +if (( SKIP_NPU_PREFLIGHT )); then + run_args+=(--skip-npu-preflight) +fi +if (( ALLOW_SELF_ONLY_FAILURE )); then + run_args+=(--allow-self-only-failure) +fi + +bash "${run_script}" "${run_args[@]}" +echo "Completed. Primary log: ${LOG_FILE}" diff --git a/tools/moonep/run_combine_v2_perf_multihost.sh b/tools/moonep/run_combine_v2_perf_multihost.sh new file mode 100644 index 0000000..1d04a17 --- /dev/null +++ b/tools/moonep/run_combine_v2_perf_multihost.sh @@ -0,0 +1,540 @@ +#!/usr/bin/env bash +set -euo pipefail + +HOSTFILE="" +INSTALL_DIR="" +CANN_PATH="${ASCEND_HOME_PATH:-/usr/local/Ascend/ascend-toolkit/latest}" +SSH_USER="$(id -un)" +BS="" +BS_LIST="" +WARMUP=20 +ITERATIONS=80 +EXPERTS=64 +COMM_DOMAIN=141 +COMM_ID="" +WAIT_SECONDS=120 +RETRY_SECONDS=15 +TIMEOUT_SECONDS=600 +LOG_FILE="" +SKIP_ITERATION_BARRIERS=0 +PROFILE=0 +SKIP_NPU_PREFLIGHT=0 +ALLOW_SELF_ONLY_FAILURE=0 + +usage() { + cat <<'EOF' +Usage: bash tools/moonep/run_combine_v2_perf_multihost.sh --hostfile PATH --install-dir PATH [options] + +Options: + --bs N Run one batch size (default: benchmark default 128) + --bs-list N[,N...] Run multiple BS points after one TileXR initialization + --warmup N Warmup launches per BS (default: 20) + --iterations N Timed launches per BS (default: 80) + --experts N Total expert count (default: 64) + --comm-domain N Shared-QP domain (default: 141) + --comm-id IP:PORT Bootstrap address (default: first host:10067) + --cann-path PATH CANN root + --ssh-user USER SSH user for rank launch (default: current user) + --wait-seconds N Maximum NPU wait (default: 120) + --retry-seconds N NPU retry interval (default: 15) + --timeout N Per-rank timeout (default: 600) + --log-file PATH Controller log path on the primary host + --skip-iteration-barriers + Skip host barriers between warmup/timed launches + --profile Capture per-AIV kernel cycle timestamps + --skip-npu-preflight Skip npu-smi process checks after manual validation + --allow-self-only-failure + Continue timing when only Self-copy validation fails + --help Show this help +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --hostfile) HOSTFILE="$2"; shift 2 ;; + --install-dir) INSTALL_DIR="$2"; shift 2 ;; + --bs) BS="$2"; shift 2 ;; + --bs-list) BS_LIST="$2"; shift 2 ;; + --warmup) WARMUP="$2"; shift 2 ;; + --iterations) ITERATIONS="$2"; shift 2 ;; + --experts) EXPERTS="$2"; shift 2 ;; + --comm-domain) COMM_DOMAIN="$2"; shift 2 ;; + --comm-id) COMM_ID="$2"; shift 2 ;; + --cann-path) CANN_PATH="$2"; shift 2 ;; + --ssh-user) SSH_USER="$2"; shift 2 ;; + --wait-seconds) WAIT_SECONDS="$2"; shift 2 ;; + --retry-seconds) RETRY_SECONDS="$2"; shift 2 ;; + --timeout) TIMEOUT_SECONDS="$2"; shift 2 ;; + --log-file) LOG_FILE="$2"; shift 2 ;; + --skip-iteration-barriers) SKIP_ITERATION_BARRIERS=1; shift ;; + --profile) PROFILE=1; shift ;; + --skip-npu-preflight) SKIP_NPU_PREFLIGHT=1; shift ;; + --allow-self-only-failure) ALLOW_SELF_ONLY_FAILURE=1; shift ;; + --help|-h) usage; exit 0 ;; + *) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac +done + +if [[ -z "${HOSTFILE}" || -z "${INSTALL_DIR}" ]]; then + usage >&2 + exit 2 +fi +if [[ -n "${BS}" && -n "${BS_LIST}" ]]; then + echo "--bs and --bs-list are mutually exclusive" >&2 + exit 2 +fi +if [[ ! "${WARMUP}" =~ ^[0-9]+$ || ! "${WAIT_SECONDS}" =~ ^[0-9]+$ ]]; then + echo "--warmup and --wait-seconds must be non-negative integers" >&2 + exit 2 +fi +for value in "${ITERATIONS}" "${EXPERTS}" "${COMM_DOMAIN}" "${RETRY_SECONDS}" \ + "${TIMEOUT_SECONDS}"; do + if [[ ! "${value}" =~ ^[1-9][0-9]*$ ]]; then + echo "iterations, domains, retry intervals, and timeouts must be positive integers" >&2 + exit 2 + fi +done +if [[ ! -f "${HOSTFILE}" || + ! -x "${INSTALL_DIR}/bin/tilexr_moonep_combine_v2_perf" ]]; then + echo "hostfile or staged benchmark is missing" >&2 + exit 1 +fi +if [[ "${INSTALL_DIR}" != /* || "${CANN_PATH}" != /* || + "${INSTALL_DIR}" == *"'"* || "${CANN_PATH}" == *"'"* ]]; then + echo "install and CANN paths must be absolute and cannot contain single quotes" >&2 + exit 2 +fi +if ! help wait 2>&1 | grep -q -- '-n'; then + echo "the launcher requires Bash 4.3 or newer for wait -n" >&2 + exit 1 +fi + +mapfile -t host_entries < <(awk ' + /^[[:space:]]*($|#)/ { next } + { gsub(/[[:space:]]/, "", $0); print $0 } +' "${HOSTFILE}") +if [[ ${#host_entries[@]} -eq 0 ]]; then + echo "hostfile has no hosts: ${HOSTFILE}" >&2 + exit 1 +fi + +ranks=0 +hosts=() +slots_by_host=() +declare -A seen_hosts=() +for entry in "${host_entries[@]}"; do + host="${entry%%:*}" + slots="${entry#*:}" + if [[ -z "${host}" || "${host}" == "${slots}" || + ! "${slots}" =~ ^[1-8]$ || -n "${seen_hosts[${host}]:-}" ]]; then + echo "invalid or duplicate hostfile entry: ${entry}" >&2 + exit 2 + fi + seen_hosts["${host}"]=1 + hosts+=("${host}") + slots_by_host+=("${slots}") + ranks=$((ranks + slots)) +done +case "${ranks}" in + 2|3|4|5|6|7|8|16|32|64|128) ;; + *) + echo "unsupported Combine V2 world size ${ranks}; expected 2-8, 16, 32, 64, or 128" >&2 + exit 2 + ;; +esac +if (( EXPERTS % ranks != 0 )); then + echo "expert count ${EXPERTS} must be divisible by world size ${ranks}" >&2 + exit 2 +fi + +requested_bs="${BS_LIST:-${BS:-128}}" +if [[ ! "${requested_bs}" =~ ^[1-9][0-9]*(,[1-9][0-9]*)*$ ]]; then + echo "batch sizes must be a comma-separated list of positive integers" >&2 + exit 2 +fi +IFS=',' read -r -a requested_batch_sizes <<<"${requested_bs}" +for batch_size in "${requested_batch_sizes[@]}"; do + if (( batch_size % ranks != 0 )); then + echo "batch size ${batch_size} must be divisible by world size ${ranks}" >&2 + exit 2 + fi +done + +if [[ -z "${COMM_ID}" ]]; then + COMM_ID="${hosts[0]}:10067" +fi +if [[ ! "${COMM_ID}" =~ ^([^:]+):([1-9][0-9]*)$ ]]; then + echo "--comm-id must use the first host and a valid TCP port" >&2 + exit 2 +fi +comm_host="${BASH_REMATCH[1]}" +comm_port="${BASH_REMATCH[2]}" +if [[ "${comm_host}" != "${hosts[0]}" || ${comm_port} -gt 65535 ]]; then + echo "--comm-id must use the first host and a valid TCP port" >&2 + exit 2 +fi +barrier_port=$((comm_port + 97)) +if (( barrier_port > 65535 )); then + barrier_port=$((comm_port - 97)) +fi +if (( barrier_port <= 0 )); then + echo "cannot derive a valid barrier port from ${comm_port}" >&2 + exit 2 +fi +BARRIER_ID="${hosts[0]}:${barrier_port}" + +if [[ -z "${LOG_FILE}" ]]; then + run_root="$(cd "${INSTALL_DIR}/.." && pwd)" + LOG_FILE="${run_root}/logs/combine_v2_${ranks}p_$(date +%Y%m%d_%H%M%S).log" +fi +if [[ "${LOG_FILE}" != /* || "${LOG_FILE}" == *"'"* ]]; then + echo "--log-file must be an absolute path without single quotes" >&2 + exit 2 +fi +mkdir -p "$(dirname "${LOG_FILE}")" +rank_log_dir="${LOG_FILE}.ranks" +preflight_log_dir="${LOG_FILE}.npu_preflight" +mkdir -p "${rank_log_dir}" "${preflight_log_dir}" + +ssh_options=(-o BatchMode=yes -o ConnectTimeout=10) +for host in "${hosts[@]}"; do + if ! ssh "${ssh_options[@]}" "${SSH_USER}@${host}" \ + "test -x '${INSTALL_DIR}/bin/tilexr_moonep_combine_v2_perf' && test -d '${CANN_PATH}/aarch64-linux' && command -v timeout >/dev/null && command -v ss >/dev/null"; then + echo "remote runtime validation failed on ${host}" >&2 + exit 1 + fi +done +if ssh "${ssh_options[@]}" "${SSH_USER}@${hosts[0]}" \ + "ss -ltnH | awk '{print \$4}' | grep -Eq ':(${comm_port}|${barrier_port})\$'"; then + echo "bootstrap or barrier port is already listening on ${hosts[0]} (${comm_port}, ${barrier_port})" >&2 + exit 1 +fi + +snapshot_processes() { + awk -F'|' ' + /Process id[[:space:]]*\|[[:space:]]*Process name/ { in_process_table = 1; next } + in_process_table && $2 ~ /^[[:space:]]*[0-9]+[[:space:]]*$/ { + pid = $3; name = $4 + gsub(/^[[:space:]]+|[[:space:]]+$/, "", pid) + gsub(/^[[:space:]]+|[[:space:]]+$/, "", name) + print pid "|" name + } + ' "$1" +} + +if (( SKIP_NPU_PREFLIGHT )); then + echo "NPU preflight skipped after manual validation on: ${hosts[*]}" | tee -a "${LOG_FILE}" +else + deadline=$((SECONDS + WAIT_SECONDS)) + attempt=0 + while true; do + attempt=$((attempt + 1)) + attempt_dir="${preflight_log_dir}/attempt_$(printf '%02d' "${attempt}")" + mkdir -p "${attempt_dir}" + blocked=() + for host in "${hosts[@]}"; do + snapshot="${attempt_dir}/${host}.log" + if ! ssh "${ssh_options[@]}" "${SSH_USER}@${host}" npu-smi info \ + >"${snapshot}" 2>&1; then + echo "npu-smi failed on ${host}; see ${snapshot}" >&2 + exit 2 + fi + while IFS='|' read -r pid name; do + [[ -z "${pid}" ]] && continue + if [[ "${name}" != tilexr_* ]]; then + blocked+=("${host}:${pid}:${name}") + fi + done < <(snapshot_processes "${snapshot}") + done + if [[ ${#blocked[@]} -eq 0 ]]; then + echo "NPU preflight passed on: ${hosts[*]}" | tee -a "${LOG_FILE}" + break + fi + echo "NPU preflight blocked by: ${blocked[*]}" | tee -a "${LOG_FILE}" >&2 + if (( SECONDS >= deadline )); then + echo "NPU preflight timed out after ${WAIT_SECONDS}s; no workload was started" | \ + tee -a "${LOG_FILE}" >&2 + exit 75 + fi + sleep "${RETRY_SECONDS}" + done +fi + +benchmark_args=( + --warmup "${WARMUP}" + --iterations "${ITERATIONS}" + --experts "${EXPERTS}" + --comm-domain "${COMM_DOMAIN}" +) +if [[ -n "${BS}" ]]; then + benchmark_args+=(--bs "${BS}") +elif [[ -n "${BS_LIST}" ]]; then + benchmark_args+=(--bs-list "${BS_LIST}") +fi +if (( SKIP_ITERATION_BARRIERS )); then + benchmark_args+=(--skip-iteration-barriers) +fi +if (( PROFILE )); then + benchmark_args+=(--profile) +fi +if (( ALLOW_SELF_ONLY_FAILURE )); then + benchmark_args+=(--allow-self-only-failure) +fi + +job_id="combine_v2_${ranks}p_$(date +%Y%m%d_%H%M%S)_$$" +remote_job_dir="$(cd "${INSTALL_DIR}/.." && pwd)/logs/.combine_v2_jobs/${job_id}" +rank_hosts=() +rank_devices=() +rank_pidfiles=() +global_rank=0 +for host_index in "${!hosts[@]}"; do + host="${hosts[${host_index}]}" + slots="${slots_by_host[${host_index}]}" + for ((local_rank = 0; local_rank < slots; ++local_rank)); do + rank_hosts[${global_rank}]="${host}" + rank_devices[${global_rank}]="${local_rank}" + rank_pidfiles[${global_rank}]="${remote_job_dir}/rank_${global_rank}.pid" + echo "RANK_MAP rank=${global_rank} host=${host} local_rank=${local_rank} device=${local_rank}" | \ + tee -a "${LOG_FILE}" + global_rank=$((global_rank + 1)) + done +done + +remote_rank_script=$(cat <<'REMOTE_SCRIPT' +set -euo pipefail +job_id=$1 +pidfile=$2 +rank_timeout=$3 +install_dir=$4 +cann_path=$5 +comm_id=$6 +barrier_id=$7 +rank=$8 +world=$9 +device=${10} +shift 10 +mkdir -p "$(dirname "${pidfile}")" +printf '%s %s\n' "$$" "${job_id}" >"${pidfile}" +child_pid="" +cleanup_rank() { + status=$? + trap - EXIT HUP INT TERM + if [[ -n "${child_pid}" ]] && kill -0 "${child_pid}" 2>/dev/null; then + kill -TERM "${child_pid}" 2>/dev/null || true + wait "${child_pid}" 2>/dev/null || true + fi + rm -f "${pidfile}" + exit "${status}" +} +trap cleanup_rank EXIT HUP INT TERM +export ASCEND_HOME_PATH="${cann_path}" +export ASCEND_DRIVER_PATH=/usr/local/Ascend/driver +export TILEXR_COMM_ID="${comm_id}" +export TILEXR_DEMO_BARRIER_ADDR="${barrier_id}" +export TILEXR_ENABLE_IPC=0 +export TILEXR_ENABLE_SDMA=0 +export LD_LIBRARY_PATH="${install_dir}/lib64:${cann_path}/aarch64-linux/lib64:${cann_path}/lib64:${ASCEND_DRIVER_PATH}/lib64:${ASCEND_DRIVER_PATH}/lib64/common:${ASCEND_DRIVER_PATH}/lib64/driver:${LD_LIBRARY_PATH:-}" +timeout --signal=TERM --kill-after=30 "${rank_timeout}" \ + "${install_dir}/bin/tilexr_moonep_combine_v2_perf" \ + --rank "${rank}" --world-size "${world}" --device "${device}" "$@" & +child_pid=$! +set +e +wait "${child_pid}" +status=$? +set -e +child_pid="" +rm -f "${pidfile}" +trap - EXIT HUP INT TERM +exit "${status}" +REMOTE_SCRIPT +) + +build_remote_command() { + local script=$1 + shift + local command quoted argument + printf -v quoted '%q' "${script}" + command="bash -c ${quoted} --" + for argument in "$@"; do + printf -v quoted '%q' "${argument}" + command+=" ${quoted}" + done + printf '%s' "${command}" +} + +launch_rank() { + local rank=$1 + local host="${rank_hosts[${rank}]}" + local device="${rank_devices[${rank}]}" + local remote_command + remote_command=$(build_remote_command "${remote_rank_script}" \ + "${job_id}" "${rank_pidfiles[${rank}]}" "${TIMEOUT_SECONDS}" \ + "${INSTALL_DIR}" "${CANN_PATH}" "${COMM_ID}" "${BARRIER_ID}" \ + "${rank}" "${ranks}" "${device}" "${benchmark_args[@]}") + exec ssh "${ssh_options[@]}" "${SSH_USER}@${host}" "${remote_command}" +} + +terminate_remote_tasks() { + local rank host pidfile cleanup_script cleanup_command + cleanup_script='pidfile=$1; job_id=$2; [[ -f "${pidfile}" ]] || exit 0; read -r pid stored_job <"${pidfile}"; [[ "${stored_job}" == "${job_id}" && -r "/proc/${pid}/cmdline" ]] || exit 0; cmdline=$(tr "\0" " " <"/proc/${pid}/cmdline"); [[ "${cmdline}" == *"${job_id}"* ]] || exit 0; kill -TERM "${pid}" 2>/dev/null || true' + for ((rank = 0; rank < ranks; ++rank)); do + host="${rank_hosts[${rank}]}" + pidfile="${rank_pidfiles[${rank}]}" + cleanup_command=$(build_remote_command "${cleanup_script}" \ + "${pidfile}" "${job_id}") + ssh "${ssh_options[@]}" "${SSH_USER}@${host}" "${cleanup_command}" \ + >/dev/null 2>&1 & + done + wait || true +} + +run_active=0 +ssh_pids=() +cleanup_controller() { + status=$? + trap - EXIT + if (( run_active )); then + for pid in "${ssh_pids[@]}"; do + kill -TERM "${pid}" 2>/dev/null || true + done + terminate_remote_tasks + for pid in "${ssh_pids[@]}"; do + wait "${pid}" 2>/dev/null || true + done + fi + exit "${status}" +} +trap cleanup_controller EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +echo "RUN job_id=${job_id} ranks=${ranks} experts=${EXPERTS} comm_id=${COMM_ID} barrier_id=${BARRIER_ID} bs=${requested_bs} warmup=${WARMUP} iterations=${ITERATIONS}" | \ + tee -a "${LOG_FILE}" +run_active=1 +for ((rank = 0; rank < ranks; ++rank)); do + rank_log="${rank_log_dir}/rank_$(printf '%04d' "${rank}").log" + launch_rank "${rank}" >"${rank_log}" 2>&1 & + ssh_pids+=("$!") +done + +completed=0 +while (( completed < ranks )); do + set +e + wait -n + rank_status=$? + set -e + completed=$((completed + 1)) + if (( rank_status != 0 )); then + echo "a rank launcher failed with status ${rank_status}; see ${rank_log_dir}" | \ + tee -a "${LOG_FILE}" >&2 + exit "${rank_status}" + fi +done +run_active=0 + +rank_logs=() +for ((rank = 0; rank < ranks; ++rank)); do + rank_log="${rank_log_dir}/rank_$(printf '%04d' "${rank}").log" + rank_logs+=("${rank_log}") + { + echo "===== RANK ${rank} host=${rank_hosts[${rank}]} device=${rank_devices[${rank}]} =====" + cat "${rank_log}" + } >>"${LOG_FILE}" +done + +rank_averages_file="${rank_log_dir}/rank_averages.tsv" +rm -f "${rank_averages_file}" +if ! awk -v ranks="${ranks}" -v iterations="${ITERATIONS}" \ + -v allow_self_only_failure="${ALLOW_SELF_ONLY_FAILURE}" \ + -v output="${rank_averages_file}" ' + $1 == "COMBINE_V2_SAMPLE" { + bs = iteration = rank = elapsed = "" + for (field = 2; field <= NF; ++field) { + split($field, item, "=") + if (item[1] == "bs") bs = item[2] + else if (item[1] == "iteration") iteration = item[2] + else if (item[1] == "rank") rank = item[2] + else if (item[1] == "elapsed_ms") elapsed = item[2] + } + sample_key = bs SUBSEP iteration SUBSEP rank + rank_key = bs SUBSEP rank + if (bs == "" || iteration == "" || rank == "" || elapsed == "" || + iteration !~ /^[0-9]+$/ || rank !~ /^[0-9]+$/ || + iteration + 0 < 0 || iteration + 0 >= iterations || + rank + 0 < 0 || rank + 0 >= ranks || sample_seen[sample_key]++) { + invalid = 1 + next + } + batches[bs] = 1 + sample_count[rank_key]++ + sample_total[rank_key] += elapsed + 0 + } + $1 == "COMBINE_V2_RANK_PERF" { + bs = rank = correctness = "" + for (field = 2; field <= NF; ++field) { + split($field, item, "=") + if (item[1] == "bs") bs = item[2] + else if (item[1] == "rank") rank = item[2] + else if (item[1] == "correctness") correctness = item[2] + } + rank_key = bs SUBSEP rank + if (correctness == "passed" || + (allow_self_only_failure && correctness == "self_only_failed")) { + rank_result[rank_key]++ + } else { + invalid = 1 + } + } + END { + batch_count = 0 + for (bs in batches) { + batch_count++ + for (rank = 0; rank < ranks; ++rank) { + rank_key = bs SUBSEP rank + if (rank_result[rank_key] != 1 || + sample_count[rank_key] != iterations) { + invalid = 1 + } else { + print bs, rank, sample_total[rank_key] / iterations >> output + } + } + } + if (batch_count == 0 || invalid) exit 1 + } +' "${rank_logs[@]}"; then + echo "rank logs do not contain one accepted result and one sample per iteration for every rank" | \ + tee -a "${LOG_FILE}" >&2 + exit 1 +fi + +sort -n -k1,1 -k2,2 "${rank_averages_file}" | awk \ + -v ranks="${ranks}" -v iterations="${ITERATIONS}" \ + -v experts="${EXPERTS}" \ + -v allow_self_only_failure="${ALLOW_SELF_ONLY_FAILURE}" ' + function emit( average, data_bytes, average_bandwidth, max_bandwidth, correctness) { + if (count == 0) return + if (count != ranks) exit 1 + average = total / count + data_bytes = current_bs * 16 * 3584 * 2 + average_bandwidth = data_bytes / average / 1000000 + max_bandwidth = data_bytes / maximum / 1000000 + correctness = allow_self_only_failure ? "self_only_failed_allowed" : "passed" + printf "COMBINE_V2_PERF bs=%s k=16 h=3584 experts=%d dtype=bf16 ranks=%d iterations=%d avg_ms=%.6f avg_alg_bw_GBps=%.6f max_ms=%.6f max_alg_bw_GBps=%.6f correctness=%s\n", current_bs, experts, ranks, iterations, average, average_bandwidth, maximum, max_bandwidth, correctness + } + current_bs != "" && $1 != current_bs { + emit() + total = 0 + count = 0 + maximum = 0 + } + { + current_bs = $1 + total += $3 + 0 + if (count == 0 || $3 + 0 > maximum) maximum = $3 + 0 + count++ + } + END { emit() } +' | tee -a "${LOG_FILE}" + +echo "Combine V2 benchmark log: ${LOG_FILE}" +echo "Per-rank logs: ${rank_log_dir}" diff --git a/tools/moonep/sync_combine_v2_perf_runtime.sh b/tools/moonep/sync_combine_v2_perf_runtime.sh new file mode 100644 index 0000000..77e05fe --- /dev/null +++ b/tools/moonep/sync_combine_v2_perf_runtime.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +set -euo pipefail + +HOSTFILE="" +INSTALL_DIR="" +SSH_USER="$(id -un)" + +usage() { + cat <<'EOF' +Usage: bash tools/moonep/sync_combine_v2_perf_runtime.sh --hostfile PATH --install-dir PATH [options] + +Options: + --hostfile PATH Rank hostfile with host:slots entries + --install-dir PATH Runtime directory to mirror at the same path on each host + --ssh-user USER SSH user (default: current user) + --help Show this help +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --hostfile) HOSTFILE="$2"; shift 2 ;; + --install-dir) INSTALL_DIR="$2"; shift 2 ;; + --ssh-user) SSH_USER="$2"; shift 2 ;; + --help|-h) usage; exit 0 ;; + *) echo "unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac +done + +if [[ -z "${HOSTFILE}" || -z "${INSTALL_DIR}" ]]; then + usage >&2 + exit 2 +fi +if [[ ! -f "${HOSTFILE}" || ! -x "${INSTALL_DIR}/bin/tilexr_moonep_combine_v2_perf" ]]; then + echo "hostfile or staged benchmark is missing" >&2 + exit 1 +fi +if [[ "${INSTALL_DIR}" != /* || "${INSTALL_DIR}" == *"'"* ]]; then + echo "--install-dir must be an absolute path without single quotes" >&2 + exit 2 +fi + +mapfile -t host_entries < <(awk ' + /^[[:space:]]*($|#)/ { next } + { gsub(/[[:space:]]/, "", $0); print $0 } +' "${HOSTFILE}") +if [[ ${#host_entries[@]} -eq 0 ]]; then + echo "hostfile has no hosts: ${HOSTFILE}" >&2 + exit 1 +fi + +manifest="$(mktemp)" +trap 'rm -f "${manifest}"' EXIT +(cd "${INSTALL_DIR}" && find bin lib64 -type f -print0 | sort -z | \ + xargs -0 sha256sum) >"${manifest}" +local_ips=" $(hostname -I 2>/dev/null || true) 127.0.0.1 localhost " + +for entry in "${host_entries[@]}"; do + host="${entry%%:*}" + if [[ "${local_ips}" == *" ${host} "* || "${host}" == "$(hostname)" ]]; then + echo "${host}: local runtime retained" + else + target="${SSH_USER}@${host}" + ssh -o BatchMode=yes "${target}" "mkdir -p '${INSTALL_DIR}'" + rsync -a --delete -e "ssh -o BatchMode=yes" \ + "${INSTALL_DIR}/" "${target}:${INSTALL_DIR}/" + echo "${host}: runtime synchronized" + fi +done + +for entry in "${host_entries[@]}"; do + host="${entry%%:*}" + if [[ "${local_ips}" == *" ${host} "* || "${host}" == "$(hostname)" ]]; then + (cd "${INSTALL_DIR}" && sha256sum -c "${manifest}") >/dev/null + else + ssh -o BatchMode=yes "${SSH_USER}@${host}" \ + "cd '${INSTALL_DIR}' && sha256sum -c -" <"${manifest}" >/dev/null + fi + echo "${host}: SHA256 verified" +done