perf(moonep): replace reduce grad with owner pull

[Kur0x]

Summary

• add persistent multi-region UDMA profiles and deferred READ batching
• replace MoonEP ReduceGrad with the Ascend950 owner-pull data path and prepared-handle lifecycle
• add profile probe, benchmark tooling, compatibility updates, and supporting documentation/tests

Post-Merge Validation

Validation was rerun after conflict resolution and rebasing onto the latest main.

Validated Revisions

• PR head: c2c6b9ed4e123c7d18d612547a18a1ee87430c84
• Base main: 7c15324247c1b54190d9fb6b11a79b8096e1ea72
• GitHub merge ref: 1c3d6d7972ae3c54c04a24695961f584015b0a15
• The PR head and GitHub merge-ref trees are identical.
• The merge base is the current main.
• git diff --check origin/main...HEAD passed.
• No conflict markers were found.

Build and Host Validation

The same source snapshot was synchronized to:

• root@141.61.53.106
• root@141.61.53.110
• Remote path: /tmp/TileXR-pr121-verify-20260813-c2c6b9e
• Source archive SHA256:
  930bce692efdc2f57489769ce0182dce8f457a4d0b6207032271a3b55d07b2eb

Both Ascend950 hosts successfully rebuilt and installed the pure AICore
ReduceGrad implementation.

• Installed ReduceGrad library SHA256 on both hosts:
  f1971eeb6412e0cd65f55b12382fbf78a5fd1c017a818287f1673522b456103e
• Host .106 CTest: 33/33 passed
• Host .110 focused ReduceGrad CTest: 3/3 passed
• Full MoonEP Python suite: 239 passed
• Independent UDMA unit/source suite: 7/7 passed
• Local focused Python suite: 88 passed, 2 subtests passed
• All changed Python files passed py_compile.
• Runtime RPATH is only $ORIGIN.
• libascend_hal.so resolves from the real driver HAL directory.

Four-Rank Ascend950 Correctness

The exact correctness smoke used four physical Ascend950 devices, ranks 0-3,
with torch_npu 2.7.1.post4.dev20260417.

Confirmed behavior:

• Exact ordered FP32 accumulation passed on all ranks.
• Live local source-slot clearing passed.
• Unused and non-local source data remained protected by the runner checks.
• Default UDMA chunk size: 8388608 bytes.
• Workspace size: 203423744 bytes.
• Smoke P50/P99: 1588.75/1598.62 us.

Artifact:

/tmp/TileXR-pr121-verify-20260813-c2c6b9e/artifacts/pr121-exact-default8m/summary.json

SHA256:

ffe61438b8cb1f73980e04d1d90813cbac1f5e9e809bdd6e7ba4f5f34ae1086f

The three smoke samples are correctness evidence and are not treated as a
formal performance run.

Post-Merge Performance Regression Check

A short 5 warmup + 20 measured four-rank performance run produced:

Revision	P50	P99
PR #121 post-merge head	1539.76 us	1583.39 us
Previous formal TileXR range	1536.22-1562.40 us	1595.37-1662.54 us

The post-merge result is within or better than the previously retained TileXR
range, so no ReduceGrad performance regression was observed after conflict
resolution.

Artifact:

/tmp/TileXR-pr121-verify-20260813-c2c6b9e/artifacts/pr121-perf-5x20/summary.json

SHA256:

1452ccca346ebf21c3affef39ff1ac1617480e7936bdb8ebd1a4d17784ea6ecf

Review Disposition

Review findings outside the normal MoonEP ReduceGrad performance path were
recorded but intentionally not changed in this PR. These include malformed
single-rank device state, asymmetric fault injection, error teardown,
unreleased ABI compatibility, and overlapping direct C API calls from
different streams.

The supported Torch/MoonEP path prevents overlapping ReduceGrad launches at
the runtime level, binds each buffer to one NPU stream, and allows reuse only
after synchronization. Its focused reentrancy test passed.

Validation Boundaries

• Hardware validation covers four physical Ascend950 devices.
• No 8-rank, 16-rank, or rank-oversubscribed run was performed.
• No asymmetric device-failure or single-rank state-corruption injection was
  performed.
• No communication or device failure occurred, so the conditional HCCL
  environmental baseline was not triggered.
• The retained native MoonEP results and final TileXR results used different
  torch_npu builds. A strict same-runtime native rebuild followed by three
  interleaved 20 warmup + 50 measured A/B runs remains the next validation
  step before treating the native performance comparison as final.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 277a9e26cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2c6b9ed4e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".