diff --git a/AGENTS.md b/AGENTS.md index 0ab4178..94ce287 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,6 +72,7 @@ Optional CMake switches are `TILEXR_BUILD_COLLECTIVES`, `TILEXR_BUILD_EP`, `TILE ### Communication Environment Baseline -- For communication, peer-memory, MTE, `507035`, or device-dependent failures, run the installed official HCCL Test on the same devices and topology before investigating TileXR or operator code. The test mode must exercise the failing data path; use `all_reduce_test -a aiv_only` for AIV-only peer-memory cases. -- Treat the matching HCCL Test as the environment-health baseline, judging it by per-rank correctness output rather than exit status alone; operation failures may still exit with code 0. -- A pass on a different device subset, topology, or accelerator mode does not validate the failing path. If the matching HCCL Test also fails, classify the environment or platform baseline as unhealthy, preserve the bounded logs, stop operator-side debugging, and ask the user to provide a repaired or new known-good environment before continuing. Do not modify TileXR code to work around an unproven environment failure. +- Treat the installed official HCCL Test as a one-time environment qualification, not a per-task or per-change prerequisite. Before launching it, search the retained records for the same environment. Never run it again when a record already exists for that environment. +- Identify an environment by its host or container, device subset, topology, accelerator mode, CANN, driver, firmware, and HCCL Test build. Rebuilding TileXR, restarting processes, or starting another debugging task does not create a new environment. +- If no record exists, run HCCL Test exactly once on the same devices and topology before investigating communication, peer-memory, MTE, `507035`, or other device-dependent failures. Select a mode that exercises the failing data path; use `all_reduce_test -a aiv_only` for AIV-only peer-memory cases. Preserve the command, environment identity, bounded logs, and per-rank correctness output; do not judge the result from exit status alone because operation failures may still exit with code 0. +- Reuse that single result for later work. A pass on a different device subset, topology, or accelerator mode does not validate the current path; report that path as not covered instead of rerunning HCCL Test. If the one-time matching run fails, classify the environment or platform baseline as unhealthy, stop operator-side debugging, and ask for a materially repaired or new known-good environment. Do not modify TileXR code to work around an unproven environment failure. diff --git a/docs/moonep/MINDSPEED_DEBUGGING_EXPERIENCE.md b/docs/moonep/MINDSPEED_DEBUGGING_EXPERIENCE.md index 958d00c..3c2dc2a 100644 --- a/docs/moonep/MINDSPEED_DEBUGGING_EXPERIENCE.md +++ b/docs/moonep/MINDSPEED_DEBUGGING_EXPERIENCE.md @@ -98,6 +98,18 @@ root-cause oracle。 ## 推荐调试流程 +### 0. 复用一次性 HCCL 环境基线 + +官方 HCCL Test 是环境准入证据,不是每个任务或每次修改的回归测试。同一环境最多执行 +一次;开始前必须先查找已有记录,存在记录时禁止再次运行。环境身份由 host/container、 +device subset、topology、accelerator mode、CANN、driver、firmware 和 HCCL Test build +共同确定;重编译 TileXR、重启进程或开始新的调试任务都不构成新环境。 + +首次运行应选择覆盖当前失败数据路径的 mode,并保存完整命令、环境身份、逐 rank 正确性 +输出和有限日志。后续任务直接复用该结果;如果已有运行没有覆盖当前 device subset、 +topology 或 accelerator mode,应把该路径标记为基线未验证,而不是在同一环境补跑第二次。 +一次性运行失败后,停止算子侧排查,等待环境发生可证明的实质修复或更换新环境。 + ### 1. 固定运行身份 每次运行先保存以下信息: diff --git a/docs/plans/2026-08-13-moonep-dispatch-fused-epoch.md b/docs/plans/2026-08-13-moonep-dispatch-fused-epoch.md index 6a8c54d..00ec787 100644 --- a/docs/plans/2026-08-13-moonep-dispatch-fused-epoch.md +++ b/docs/plans/2026-08-13-moonep-dispatch-fused-epoch.md @@ -44,6 +44,8 @@ Update `dispatch_hot_loop.py`, reporting tests, source guards, and API tests. Run focused C++ and Python tests, target CANN 9.1 Host/Kernel build, and inspect the final diff. On `141.61.49.195`, establish NPU/CANN/source/binary provenance, -run matching HCCL Test, deploy to a task-specific directory, and execute the -approved correctness/stability/performance ladder. Record reusable fused-epoch -workspace and completion-order lessons in the maintained Dispatch design. +reuse the retained one-time matching HCCL Test result (create it only if this +environment has no prior record), deploy to a task-specific directory, and +execute the approved correctness/stability/performance ladder. Record reusable +fused-epoch workspace and completion-order lessons in the maintained Dispatch +design. diff --git a/docs/specs/2026-08-13-moonep-dispatch-fused-epoch-design.md b/docs/specs/2026-08-13-moonep-dispatch-fused-epoch-design.md index 584bad4..d18d304 100644 --- a/docs/specs/2026-08-13-moonep-dispatch-fused-epoch-design.md +++ b/docs/specs/2026-08-13-moonep-dispatch-fused-epoch-design.md @@ -87,7 +87,8 @@ UB-only WQE construction, MTE3 SQ publication, and `st_dev` doorbells. Python tests preserve the one-FFI-call contract and parse fused diagnostics. The target CANN 9.1 build must compile Host and Kernel. Hardware validation on -`141.61.49.195` starts with the matching official HCCL Test and then covers -single-rank, two-rank, and full-host Hidden/paired exactness, repeated rounds, -alternating plans, grouped/group-credit/shared-QP configurations where +`141.61.49.195` first reuses the retained one-time matching official HCCL Test +result, creating it only when that environment has no prior record, and then +covers single-rank, two-rank, and full-host Hidden/paired exactness, repeated +rounds, alternating plans, grouped/group-credit/shared-QP configurations where supported, and profiling-off/on `pair` A/B evidence.