perf(udma): extend grouped alltoall quiet batching

docs/plans/2026-08-02-grouped-alltoall-ingress-credit.md

@@ -0,0 +1,120 @@
+# Grouped AllToAll Ingress Credit Implementation Plan
+
+## Goal And Scope
+
+Implement the default-disabled `window=1` ingress-credit protocol approved in
+`docs/specs/2026-08-02-grouped-alltoall-ingress-credit-design.md`. The change
+must bound payload admission to one source per destination lane while
+preserving the existing path exactly when credits are disabled.
+
+This plan does not change route weights, cabinet topology interpretation,
+quiet batching, copyout assignment, or multi-pass credit semantics. Initial
+hardware acceptance is single-pass only.
+
+## Task 1: Pure Layout And Schedule Contracts
+
+**Objective and role:** Define host-testable helpers for credit enablement,
+token construction, receive ownership, and next-group peer mapping.
+
+**Background and prerequisites:** The approved spec is authoritative. Existing
+peer mapping and plan helpers live in
+`tests/udma/demo/tilexr_udma_alltoall_group_layout.h`.
+
+**Modification scope:** The grouped layout header and
+`tests/udma/unit/test_tilexr_udma_alltoall_group_layout.cpp`.
+
+**Constraints and non-goals:** Preserve existing peer mapping and default
+layout behavior. Do not add device-only dependencies to the host-testable
+header.
+
+**Acceptance and verification:** Unit tests cover 256 ranks, group zero,
+inactive final lanes, owner selection, shared-route credit identity, invocation
+separation, and default-disabled behavior. Run the grouped layout unit binary.
+
+**Artifacts and interfaces:** Stable helper contracts consumed by Host and
+Kernel work.
+
+## Task 2: Registered Credit Plane And Host Interface
+
+**Objective and role:** Reserve ping-pong credit storage and pass a validated
+ingress-window value and credit offsets into kernel launch.
+
+**Background and prerequisites:** Task 1 token and layout contracts are fixed.
+The grouped plan already owns payload, signal, and control regions.
+
+**Modification scope:** Grouped layout plan, demo environment parsing and
+allocation/launch wiring, grouped kernel declaration and launch wrapper, plus
+focused source-contract tests.
+
+**Constraints and non-goals:** `INGRESS_WINDOW=0` remains the default. Only 0
+and 1 are accepted. Existing payload and signal offsets remain stable where
+practical; registered-size overflow checks remain mandatory.
+
+**Acceptance and verification:** Unit tests prove valid/invalid environment
+and launch wiring contracts, credit planes do not overlap, and max registered
+size accounts for both planes.
+
+**Artifacts and interfaces:** Kernel receives two credit offsets and the
+ingress-window value.
+
+## Task 3: Kernel Credit Wait And Publication
+
+**Objective and role:** Gate each destination send once per group, publish a
+local request after both route-ready tokens arrive, and let the primary send
+core issue the next lane credit.
+
+**Background and prerequisites:** Tasks 1 and 2 provide mapping, storage, and
+arguments. Existing MTE token wait and UDMA NBI primitives are reused.
+
+**Modification scope:**
+`tests/udma/demo/tilexr_udma_alltoall_group_kernel.cpp`, existing UDMA helper
+interfaces only if required, and grouped source-contract tests.
+
+**Constraints and non-goals:** Do not add per-credit quiet. Only cores 32..47
+publish local requests; only primary send cores publish UDMA credits, preserving
+the shared-SQ single-producer rule. Local request slots are 64-byte isolated so
+independent receive-core cache maintenance cannot lose another lane's request.
+Primary and secondary workers wait on the same destination credit. The
+default-disabled branch must not execute a credit wait or write. If source
+lifetime cannot be made safe without per-credit quiet, stop and revise the
+design instead of weakening this constraint.
+
+**Acceptance and verification:** Static/source tests prove placement,
+single-producer ownership, request isolation, and unique shared-SQ completion
+reclamation. Compile checks validate Ascend C overloads and launch signatures.
+The exact final source passes a clean b131 Bisheng build and 4x8 hardware
+validation; 256P admission-bound validation remains pending.
+
+**Artifacts and interfaces:** Trace-visible or debug-visible credit wait and
+publish phases where trace capacity permits.
+
+## Task 4: Verification And Delivery Readiness
+
+**Objective and role:** Establish local correctness and document remaining
+hardware evidence.
+
+**Background and prerequisites:** Tasks 1 through 3 are complete.
+
+**Modification scope:** Tests and documentation required to keep validation
+commands and limitations accurate.
+
+**Constraints and non-goals:** Run only the explicitly authorized NPU matrix.
+Do not modify or stage unrelated existing worktree files.
+
+**Acceptance and verification:** Run focused unit tests, relevant Python
+parser/source tests, formatting/build checks available on Windows, inspect the
+final diff, and record the exact 2x8 then 4x8 then 256P hardware matrix. The
+implemented path has completed 4x8 functional and warmup-5/repeat-50
+performance validation; 256P admission-bound validation remains pending.
+
+**Artifacts and interfaces:** A reviewable implementation, verification
+summary, and scoped Git commit after the final checks pass.
+
+## Key Risks
+
+- Credit NBI source storage may be reused before hardware consumes it.
+- Two receive copy slices may publish duplicate credits.
+- Primary and secondary send workers may wait on different tokens by mistake.
+- A final inactive lane may wait for or publish a nonexistent peer.
+- Registered-memory growth may cross a region boundary or maximum-size check.
+- Trace storage may need expansion if new phases are recorded.

docs/specs/2026-08-02-grouped-alltoall-ingress-credit-design.md

@@ -0,0 +1,177 @@
+# Grouped AllToAll Ingress Credit Design
+
+## Status
+
+The original request-plus-UDMA-credit implementation was validated through 8x8
+hardware on 2026-08-02. It has since been replaced by direct receive-core
+publication through a dedicated credit IPC allocation and requires renewed
+hardware validation. The experimental feature remains disabled by default.
+
+## Goal
+
+Bound each destination rank to at most 16 concurrent payload source ranks in
+the grouped AllToAll data path. Preserve independent lane progress so one slow
+source does not impose a full-rank or full-group barrier.
+
+The first implementation is experimental and disabled by default. Existing
+behavior and performance remain unchanged unless ingress credits are enabled.
+
+## Non-goals
+
+- Limiting the number of physical port operations to 16. A source may still
+  use primary and secondary routes after it receives one destination credit.
+- Adding a global barrier between peer groups.
+- Waiting for receive-copy completion before releasing network capacity.
+- Changing topology-derived primary/secondary route weights.
+- Enabling credit control for multi-pass payloads in the first hardware test.
+
+## Existing Behavior
+
+At `rankSize=256` and `groupWidth=16`, every rank has 16 logical peer groups.
+Each group contains up to eight forward and eight backward peers. The 32 send
+workers are split into 16 primary-route workers and 16 secondary-route workers.
+
+Each worker advances to its next group immediately after its own quiet
+completes. Workers and ranks do not share group progress. Consequently, a
+destination may receive payloads from more than one logical group at the same
+time even though each group contains no more than 16 source ranks.
+
+## Credit Protocol
+
+There are 16 independent destination lane chains. Group zero is initially
+enabled. For group `g > 0`, a source must observe a credit from its destination
+for `(invocation, g)` before either route posts payload data.
+
+For destination rank `D`, lane `L`, and group `g`:
+
+1. The designated receive owner waits until all payload routes expected from
+   `peer(D, g, L)` have published their data-ready tokens.
+2. Before receive-copy, the owner directly writes one credit token through the
+   dedicated IPC mapping of `peer(D, g + 1, L)`, when that peer exists.
+3. The next source observes the credit in its local dedicated credit buffer and
+   may post both primary and secondary payload routes to `D`.
+
+At most one source per destination lane can therefore be admitted. With 16
+lanes, the strict payload-source bound is 16 ranks per destination.
+
+Credit dependencies are monotonic from group `g` to `g + 1`. Group zero has no
+dependency, so the protocol does not introduce a cyclic startup wait.
+
+## Credit Storage And Tokens
+
+Each rank allocates a dedicated 1 MiB IPC buffer when
+`TILEXR_ENABLE_CREDIT_IPC=1`. A credit received by source rank `S` is indexed by
+destination rank `D`; each entry occupies 512 bytes and two fixed 512 KiB
+ping-pong planes support 1024 ranks:
+
+```text
+credit[pingPongSlot][destinationRank * 512]
+```
+
+The expected value encodes at least the invocation and destination group. The
+receiver accepts a value greater than or equal to the expected token, matching
+the existing data-ready stale-token policy. Ping-pong slots prevent adjacent
+invocations from reusing a live credit location.
+
+Primary and secondary send workers for the same peer wait on the same credit.
+Credits control source-rank admission, not route admission.
+
+## Receive Ownership
+
+With 32 copyout workers, workers 0 through 15 and 16 through 31 can wait on the
+same peer signal while copying different payload slices. Only the first slice,
+kernel cores 32 through 47, owns direct credit publication. Cores 48 through 63
+never publish credits.
+
+The owner publishes its credit after both expected primary and secondary data
+tokens arrive and before MTE receive-copy begins. This avoids placing copyout
+latency on the send admission path.
+
+## Credit Submission
+
+Credit publication is one 512-byte MTE copy by the receive owner through
+`creditMems[nextSource]`. The token occupies the first 8 bytes of the slot;
+copying the complete slot gives the remote IPC write an explicit MTE3
+completion and keeps adjacent credits on separate transfer units. It does not
+consume a UDMA WQE, QP, completion, or quiet, and it does not add another
+producer to a shared UDMA SQ. Both send routes poll the same local dedicated
+credit slot through a 512-byte MTE2 copy.
+
+The dedicated allocation is independent of the existing optional communication
+IPC buffer, so grouped ingress credit can run with `TILEXR_ENABLE_IPC=0`. It
+adds one IPC mapping per peer and process, but only 1 MiB of device memory per
+rank. Host code rejects ingress-credit execution if any mapping is missing.
+
+## Configuration
+
+Add an experimental ingress window setting:
+
+```text
+TILEXR_DEMO_ALLTOALL_GROUP_INGRESS_WINDOW=0|1
+```
+
+- `0`: current behavior and default.
+- `1`: one admitted source per lane, at most 16 payload source ranks per
+  destination.
+
+`window=1` additionally requires `TILEXR_ENABLE_CREDIT_IPC=1` before
+communicator initialization.
+
+A later `window=2` extension may trade a 32-source bound for more tolerance of
+credit latency, but it is outside the first implementation.
+
+## Validation
+
+Host/unit tests must cover:
+
+- 256-rank group and lane predecessor/successor mapping.
+- No credit wait for group zero.
+- One shared credit for primary and secondary routes.
+- Single credit publisher when 32 copyout workers duplicate receive lanes.
+- Correct inactive-lane behavior at the diameter and final partial group.
+- Credit token separation across invocations and ping-pong slots.
+- Default-disabled compatibility.
+
+Hardware validation starts with 2x8 functional coverage, then 4x8, then 256P.
+The 256P comparison uses 1 GiB/rank, multi channel, single pass, warmup 5 and
+repeat 50. Evidence includes P50 and range plus per-phase `credit-wait`,
+`send-quiet`, and `receive-wait` trace durations. A 1 KiB single-channel case
+checks that the default-disabled path has no latency regression.
+
+The first trace-enabled run is diagnostic only and is not compared directly
+with trace-disabled performance.
+
+The final 4x8, warmup-5/repeat-50, trace-disabled comparison passed data
+validation on all 32 ranks:
+
+```text
+1 KiB/rank single: window0 P50 38.224 us, window1 P50 52.763 us
+1 GiB/rank multi:  window0 P50 5329.210 us, window1 P50 5317.925 us
+```
+
+This proved the previous request/UDMA-credit protocol progressed across repeated invocations
+and does not regress 1 GiB throughput in the 4x8 environment. It does not prove
+the 16-source global bound because 4x8 has only two peer groups.
+
+The 8x8 A10 run used the fixed host order `226, 223, 220, 217, 198, 195,
+192, 189`, single pass, shared QP, trace disabled, and warmup-5/repeat-50. All
+64 ranks passed data validation:
+
+```text
+1 KiB/rank single: window0 P50 73.067 us, window1 P50 105.930 us
+1 GiB/rank multi:  window0 P50 6516.335 us, window1 P50 6336.370 us
+```
+
+Under the previous request/UDMA-credit implementation, ingress credit reduced
+the 1 GiB P50 by 179.965 us (2.76%), from
+approximately 153.46 GiB/s to 157.82 GiB/s. The 1 KiB case paid 32.863 us of
+additional fixed control latency. This run exercises four peer groups and
+therefore provides stronger repeated credit-chain evidence than 4x8, but it
+still does not prove the 256P global admission bound.
+
+## Residual Risk
+
+Rank0-only trace cannot prove the global source-card bound. Hardware validation
+needs either all-rank trace or lightweight per-destination admission counters.
+The physical four-port capacity and cabinet-aware route weighting are separate
+topology problems and are not solved by this credit protocol.