perf(udma): extend grouped alltoall quiet batching - #88
Open
hechangcheng2303 wants to merge 164 commits into
Open
perf(udma): extend grouped alltoall quiet batching#88hechangcheng2303 wants to merge 164 commits into
hechangcheng2303 wants to merge 164 commits into
Conversation
added 30 commits
June 18, 2026 10:58
… comm vs local DataCopy cost
Add testType 7 for the UDMA all-to-all bigdata demo, including the host-side launch path, registered buffer layout, staged kernel flow, and profiling controls for stage timing.
Use three cores per peer for copy/send/recv roles in the bigdata all-to-all demo. Add ping-pong relay buffers for local GM-UB-GM copies, MTE flag synchronization for copy/ack control, and explicit kernel-loop token generation for UDMA buffer reuse across host-side repeated launches. Update the bigdata layout plan and source guards to cover the new control layout, profile stages, and UDMA cache clean fixes.
Use the 35-core bigdata all-to-all path for multi-node ranks and optional forced 8P testing. Allocate a 1024 MiB registered UDMA buffer with fixed 8 MiB send/recv peer slots per pass so the multi-node kernel does not reuse pingpong data slots internally. Keep the single-node bigdata path on the existing pingpong layout and update the demo launcher plus layout guards for the new offsets, worker roles, and topology checks.
- Add multi-route UDMA queue layout with per-QP route weights and explicit EID selection. - Create peer queues across selected local and remote EIDs, preserving default single-route behavior. - Expose QP-specific device UDMA helpers and weighted slicing for diagnostics. - Shrink multinode bigdata registered windows for small payloads and cover the layout changes with unit tests.
Use 35-core multinode bigdata scheduling with expanded control slots and per-core fanout flags to reduce shared flag contention. Route recv-side cross-card ack publication through UDMA signal while keeping local synchronization on MTE, and skip the slot reuse ack wait in the 35-core path after validation showed it caused repeat>1 hangs. Update UDMA layout/source guard tests for the new control layout and transport registration limit.
- Update source-guard expectations for the rebased multinode bigdata sync tuning. - Keep the followup 128-byte control slot and 32 control shard layout unchanged.
Checkpoint the current remote-put-only experiment with 64-core 32+32 send scheduling and per-segment tail flag completion checks before returning to the previously validated 16p mteack path.
Return remote-put-only scheduling to the previously validated flat 64-core send-task path with secondary tail completion checking after the 32+32 tail-flag experiment stalled in 64p testing.
Restore the validated remote-put-only performance path by removing the tail-flag polling and peer ack checks from the flat 64-core send-task flow. The 16p loop=1 8MiB/peer validation passes again without timeout debug statuses.
Publish IPC ready tokens after UDMA quiet for remote-put-only sends and make check workers wait for both remote segments before exchanging ACKs. Treat incomplete remote-put-only debug status as a fatal demo error and update layout guards.
Add a Python one-click runner that syncs local TileXR sources to remote hosts, builds the UDMA demo, runs warmup and measured alltoall launches, records host wall time, and summarizes per-rank operator latency. Cover size parsing, host selection, metric parsing, summary statistics, and remote tool environment inference.
added 19 commits
August 2, 2026 10:02
added 8 commits
August 5, 2026 20:54
ChinaLinZhen
force-pushed
the
aiv_urma_grouped_alltoall
branch
from
August 13, 2026 13:23
53f7a91 to
b4f03a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification