perf(moonep): optimize combine v2 ring scheduling

docs/plans/2026-08-11-combine-v2-step0-detailed-profiling.md

@@ -0,0 +1,41 @@
+# Combine V2 Step0 Detailed Profiling Plan
+
+## Goal
+
+Expand MoonEP Combine V2 profiling so the Step0 send interval reports named,
+accumulated costs for selection, self-copy, and remote WQE submission while
+preserving the existing cumulative timestamps and failure diagnostics.
+
+## Scope
+
+- Version the internal profile record and add eight cycle counters.
+- Instrument the profiling-only kernel path with non-overlapping counters for
+  selection load/select, self route/copy, and remote route/descriptor/build/
+  publish work.
+- Copy and print the counters as named microsecond fields in the hardware probe.
+- Update layout and source-contract tests for the new record contract.
+- Build and run the existing focused tests, then build and validate the profile
+  artifact on the approved 198/226 16P environment using Mutagen source sync.
+
+## Non-Goals
+
+- Do not optimize the kernel in this change.
+- Do not change the public Combine V2 launch ABI or result calculations.
+- Do not use detailed-profile timings as the final non-profile performance result.
+
+## Implementation Order
+
+1. Extend `combine_v2_profile.h` with profile v2 metric indices and capacity.
+2. Add profiling-only metric accumulation to the kernel and serialize it.
+3. Extend the hardware probe sample validation and named output.
+4. Update focused unit expectations and add source-contract coverage.
+5. Run local/static tests, compile remotely, sync the runtime from the primary
+   server, and run the 16P BS=8192 profile case.
+
+## Risks And Verification
+
+Per-row cycle reads perturb profile builds, so counters are compiled out when
+profiling is disabled and are used only for attribution. Profile record size is
+an internal workspace ABI; host, kernel, and probe must be rebuilt and deployed
+together. Verification must cover record size/layout, profile source wiring,
+target compilation, record validation, correctness, and named metric output.

docs/plans/2026-08-11-moonep-combine-v2-128p-exp256.md

@@ -0,0 +1,45 @@
+# MoonEP Combine V2 128P ExpNum=256 Execution Plan
+
+## Goal
+
+Run the Combine V2 single-operator benchmark on cabinets 0 and 2 with 128
+ranks, `BS=8192`, `K=16`, `H=3584`, BF16, and 256 experts. Report the
+average and maximum of the 128 per-rank 80-iteration means, with equivalent
+bandwidth for both times.
+
+## Scope
+
+- Add a runtime expert-count option to the C++ benchmark and Bash launchers.
+- Accept the operator-supported world sizes through 128 ranks.
+- Compile on cabinet 2 CPU1 (`141.61.52.35`).
+- Flat-sync runtime artifacts from that server to all 16 cabinet 0/2 hosts.
+- Launch ranks through direct SSH without MPI.
+
+The expert count is benchmark metadata and a sharding contract. It is not an
+argument to the Combine V2 operator and does not change the bandwidth payload,
+which remains `BS * K * H * sizeof(BF16)` per rank.
+
+## Implementation
+
+1. Update `tests/moonep_combine_v2/demo/tilexr_moonep_combine_v2_hardware_probe.cpp`
+   to parse `--experts`, validate `experts % world_size == 0`, and report the
+   configured value.
+2. Update `tools/moonep/run_combine_v2_perf_multihost.sh` to accept and forward
+   `--experts`, validate 128P, and emit the configured value in final results.
+3. Update `tools/moonep/run_combine_v2_perf_cluster.sh` and the performance
+   guide with the same interface and semantics.
+4. Use the cabinet mapping from `D:\3_codex\512P环境信息.txt` to construct the
+   16-host, 128-rank hostfile in cabinet 0 then cabinet 2 order.
+5. Sync source with Mutagen, build and stage the non-MPI runtime on
+   `141.61.52.35`, then flat-sync artifacts to every host.
+6. Run `warmup=20`, `iterations=80`, and preserve the full per-rank logs.
+
+## Verification
+
+- Source/build checks pass and the benchmark has no MPI dependency.
+- Runtime SHA256 verification passes on all 16 hosts.
+- NPU preflight follows the 15-second retry and 120-second maximum wait rule.
+- The log contains 128 passing rank records and 10,240 timed samples.
+- Final output contains `experts=256`, `avg_ms`, `avg_alg_bw_GBps`, `max_ms`,
+  and `max_alg_bw_GBps`.
+- No benchmark processes remain after completion.

docs/plans/2026-08-12-moonep-combine-v2-self-copy.md

@@ -0,0 +1,62 @@
+# MoonEP Combine V2 Self-Copy Implementation Plan
+
+Status: Implemented. B150 compile, six focused Host tests, and bounded 8P
+correctness passed using `/home/pkg/910_B150/cann-9.1.0` on 9号柜 CPU1 for
+BS=128 and BS=8192.
+
+## Goal
+
+Implement the approved Self-copy pipeline in
+`docs/specs/2026-08-12-moonep-combine-v2-self-copy-design.md` without changing
+the Planner route ABI, Remote transport protocol, or public launch interface.
+
+## Scope
+
+The implementation touches the Combine V2 kernel, its focused source guards and
+Host reference tests, and the approved design/plan artifacts. It must preserve
+C++14 and target CANN compatibility.
+
+The experimental untracked `tilexr_moonep_combine_v2_kernel_back.h` is
+read-only reference material and is not part of the build or implementation.
+
+## Work
+
+1. Replace the separate oversized issue buffers with one continuous 260-entry
+   UB allocation. Expose logical six-port and two-port views at entries 0 and
+   194, prefill only 192/64 payload templates, append up to two controls per
+   lane, and retain the existing per-SQ MTE3 publication and ring-wrap split.
+2. Delete obsolete compare/gather/descriptor declarations, allocations, and
+   inactive implementation from the active kernel.
+3. Allocate two dedicated 64 KiB Self relay buffers with `TQue`, while retaining
+   a compile-time proof that all send buffers fit in 216 KiB UB.
+4. Reuse `LoadSelectionChunk()` and `SelectPeerRoutes()` in `SendSelfStep()`.
+   Consume each selected `RouteEntry` batch before invoking the selector again.
+5. For rows at most 64 KiB, derive rows per group from aligned `rowBytes_`, cap
+   the group at eight rows, and overlap the next MTE2 group with the pending
+   MTE3 group. For larger rows, flatten routes into at-most-64-KiB tiles and use
+   the same two-buffer pipeline.
+6. Update source guards and Host reference tests for continuous WQE offsets,
+   lane capacities, dynamic group sizing, legacy-code removal, RouteEntry-based
+   Self addressing, and synchronization ordering.
+7. Run focused local Host checks. Synchronize through Mutagen and compile with
+   `/home/pkg/b150` on an available target server. Before any hardware run,
+   apply the task's process-occupancy rules.
+
+## Constraints
+
+- Every WQE remains fully constructed in UB and is copied to SQ with MTE3.
+- Doorbells remain `st_dev` operations after SQ MTE3 completion.
+- Self does not modify SQ/CQ/control state.
+- `sourceSlotIndex` is absolute and must not receive `chunkStart` twice.
+- Copy exactly `rowBytes_`; aligned UB padding is never written to GM.
+- Do not reuse WQE, route, cursor, or old selection storage as Self relay.
+- Do not commit, push, or remove unrelated untracked files as part of this plan.
+
+## Verification
+
+- Focused Combine V2 schedule and source-guard tests pass.
+- `git diff --check` passes.
+- Target kernel compiles against `/home/pkg/b150`.
+- Bounded hardware correctness covers Self routes and representative dynamic
+  relay group sizes when a suitable target is available.
+- Profiling distinguishes selector time from the complete Self copy pipeline.