Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Ask the user to install missing `ascendc-development` or `superpowers-neo` skill

TileXR is a C++14 communication runtime for Huawei Ascend NPUs. It provides tile-level synchronization, optional collectives and EP libraries, IPC/MTE communication, registered-memory UDMA, and opt-in local SDMA.

- Current build target: CANN 9.1.0 and NPU driver 25.5.0 or later.
- Current build target: CANN 9.1.0 and NPU driver 25.1.rc1 or later.
- Core runtime targets Ascend 910B and 910A5. UDMA data-plane validation requires A5 / Ascend950 / 950 hardware.

## Key Paths
Expand Down
20 changes: 11 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,13 @@ Instead of stalling every rank at coarse barriers, TileXR splits a phase into ti
- **Standalone EP dispatch/combine MVP**: `libtilexr-ep.so`, `libtilexr_ep_dispatch_kernel.so`, and `libtilexr_ep_combine_kernel.so` provide TileXR-native MoE EP dispatch/combine routes under `src/ep`, independent from HCCL window helpers, `ops-transformer`, and shmem. Same-node paths use IPC peer-memory windows; cross-node dispatch/combine use TileXR-registered UDMA workspaces.
- **Tile-level synchronization**: device-side flag regions and magic values support reusable fine-grained synchronization rounds.
- **Registered-memory UDMA path**: host code registers ordinary `aclrtMalloc` device memory with `TileXRUDMARegister`; device kernels use `tilexr_udma.h` wrappers for put/get/signal.
- **On-card SDMA transport**: an opt-in (`TILEXR_ENABLE_SDMA=1`) local GM-to-GM copy path. Host code queries it with `TileXRSDMAAvailable` / `TileXRGetSDMAWorkspaceDev`; device kernels use `tilexr_sdma.h` (`SDMACopyNbi`, `SDMAWait`). Separate from UDMA: SDMA is local to one device, UDMA targets registered remote memory.
- **On-card SDMA transport**: an opt-in (`TILEXR_ENABLE_SDMA=1`) local GM-to-GM copy path. A2/A3 use the PTO backend; A5 / Ascend950 uses TileXR's direct STARS SQ backend initialized through CANN's built-in `ShmemSdmaStarsQuery`. Host code queries it with `TileXRSDMAAvailable` / `TileXRGetSDMAWorkspaceDev`; device kernels use `tilexr_sdma.h` (`SDMACopyNbi`, `SDMAWait`).
- **Operator simulator**: `op-simulator/` supports functional/performance simulation for selected AICore kernels without physical hardware.

## System Requirements

- **User**: root access or membership in the Ascend driver user group is typically required for CANN runfile installation and NPU device operations
- **NPU driver**: 25.5.0 or later, check with `npu-smi info`
- **NPU driver**: 25.1.rc1 or later, check with `npu-smi info`
- **CANN**: current build scripts and CMake are aligned to CANN 9.1.0
- **Core supported chips**: Ascend 910B, 910A5
- **UDMA runtime validation target**: A5 / Ascend950 / 950 only
Expand Down Expand Up @@ -147,7 +147,7 @@ TileXR/
|-- src/
| |-- comm/ # Core communication runtime
| | |-- udma/ # TileXR-owned HCCP/RA UDMA transport
| | `-- sdma/ # On-card PTO SDMA local copy transport
| | `-- sdma/ # On-card PTO and A5 direct SDMA backends
| |-- collectives/ # Optional TileXR collectives library
| |-- ep/ # Standalone TileXR EP dispatch MVP
| `-- include/ # Public C/C++ and device headers
Expand Down Expand Up @@ -246,12 +246,13 @@ If UDMA is unavailable, communicator initialization continues without setting `E

SDMA is a first-class local on-card GM-to-GM copy path, separate from UDMA. It is disabled by default and enabled with `TILEXR_ENABLE_SDMA=1`.

- `TileXRComm::InitSDMA()` owns a `TileXRSDMATransport` beside the UDMA transport. When enabled, it creates a PTO `pto::comm::sdma::SdmaWorkspaceManager`, stores its device workspace address in `CommArgs::sdmaWorkspacePtr`, and sets `ExtraFlag::SDMA`.
- `TileXRComm::InitSDMA()` owns a `TileXRSDMATransport` beside the UDMA transport. Runtime SoC selection keeps PTO `SdmaWorkspaceManager` on A2/A3 and selects TileXR's 48-channel direct backend on A5 / Ascend950.
- Host queries: `TileXRSDMAAvailable(comm, &available)` and `TileXRGetSDMAWorkspaceDev(comm, &workspace)`. The workspace pointer is owned by `TileXRComm` and must not be freed.
- Device API: `src/include/tilexr_sdma.h` provides `TileXR::SDMACopyNbi` and `TileXR::SDMAWait`, accepting raw same-device GM pointers. It does not register memory or validate buffer ownership.
- PTO SDMA header differences across CANN 9.0.0 / 9.1.0 are isolated in `src/include/tilexr_sdma_compat.h`.
- A5 initialization calls CANN's built-in `ShmemSdmaStarsQuery`, maps RTSQ doorbells on Host, and lets AIV write the two-SQE data/completion sequence. TileXR does not ship a custom AICPU kernel or OPP package for SDMA.
- PTO-specific compatibility remains isolated in `src/include/tilexr_sdma_compat.h`; A5 uses the installed `tilexr_sdma_a5.h` and `tilexr_sdma_a5_types.h` headers.

Enabled initialization is best-effort: if PTO SDMA headers or runtime resources are unavailable, communicator initialization continues without setting `ExtraFlag::SDMA`, and `SDMACopyNbi` returns event handle `0` while `SDMAWait` reports completion. See [docs/SDMA_TRANSPORT.md](docs/SDMA_TRANSPORT.md) for the full transport guide.
Enabled initialization is best-effort: if the selected backend or runtime resources are unavailable, communicator initialization continues without setting `ExtraFlag::SDMA`, and `SDMACopyNbi` returns event handle `0` while `SDMAWait` reports completion. See [docs/SDMA_TRANSPORT.md](docs/SDMA_TRANSPORT.md) for the full transport guide.

## Dependencies

Expand Down Expand Up @@ -296,18 +297,19 @@ See:
Build and run the SDMA unit tests against a selected CANN install, then run the data-plane demo on a device:

```bash
bash tests/sdma/build.sh /path/to/cann
bash tests/sdma/build.sh /path/to/cann # Ascend910B default
bash tests/sdma/build.sh /path/to/cann Ascend950 # A5 / Ascend950
bash tests/sdma/run_tests.sh /path/to/cann
bash tests/sdma/demo/run_tilexr_sdma_demo.sh /path/to/cann 0 64 4096 1048576
```

Expected demo success line:

```text
PASS TileXR SDMA copied <bytes> bytes correctly
PASS TileXR SDMA copied <bytes> bytes on <blocks> block(s), channels <first>..<last>, iterations <count>
```

The unit tests are hardware-free; the demo requires a usable driver HAL/device runtime and resolves `libascend_hal.so` from `/usr/local/Ascend/driver/lib64/driver`. See [docs/SDMA_TRANSPORT.md](docs/SDMA_TRANSPORT.md) for enablement, the host/device API, CANN 9.0.0 / 9.1.0 acceptance steps, and current validation status.
The unit tests are hardware-free; the demo requires a usable driver HAL/device runtime and resolves `libascend_hal.so` from `/usr/local/Ascend/driver/lib64/driver`. A5 kernels are built with `-O2`, need no custom OPP, and do not link PTO's `libnnopbase.so`; the shared Host library retains PTO support for A2/A3. See [docs/SDMA_TRANSPORT.md](docs/SDMA_TRANSPORT.md) for enablement, APIs, backend details, and the hardware acceptance matrix.

## Collectives Validation

Expand Down
31 changes: 27 additions & 4 deletions docs/BUILD_VERIFICATION.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# TileXR Build Verification

**Updated:** 2026-05-29
**Updated:** 2026-07-28

This checklist reflects the current TileXR codebase. The core runtime builds `libtile-comm.so` without a compile-time or link-time shmem dependency.

Expand All @@ -16,7 +16,7 @@ Expected:

- CANN 9.1.0 environment is visible through `ASCEND_HOME_PATH`.
- `scripts/common_env.sh` detects architecture and SOC information.
- NPU driver version is 25.5.0 or later.
- NPU driver version is 25.1.rc1 or later.

## Build Core Runtime

Expand All @@ -37,12 +37,33 @@ Check dynamic dependencies:
```bash
ldd install/lib/libtile-comm.so | grep -E "ascendcl|runtime|ascend_hal|profapi"
ldd install/lib/libtile-comm.so | grep -i shmem || true
readelf -d install/lib/libtile-comm.so | grep -E "RPATH|RUNPATH" || true
```

Expected:

- CANN runtime libraries are resolved.
- The shmem grep prints nothing for the current TileXR UDMA implementation.
- The shmem grep prints nothing; A5 SDMA dynamically invokes the CANN built-in
query and does not add a shmem or custom OPP link dependency.
- Any RPATH/RUNPATH output does not contain a CANN `devlib` directory.

## Build And Run SDMA Checks

```bash
cd /path/to/TileXR
bash tests/sdma/build.sh "$ASCEND_HOME_PATH" Ascend950
bash tests/sdma/run_tests.sh "$ASCEND_HOME_PATH"
```

Use the default `Ascend910B` target to compile-check the preserved PTO path:

```bash
bash tests/sdma/build.sh "$ASCEND_HOME_PATH"
```

On A5 / Ascend950 hardware, run the direct data-plane matrix documented in
[SDMA_TRANSPORT.md](SDMA_TRANSPORT.md). The A5 kernel must retain the repository's
`-O2` compile option, and no custom OPP environment setting is required.

## Build UDMA Tests

Expand Down Expand Up @@ -119,7 +140,7 @@ The final grep should print nothing for a clean run.
| Symptom | Likely Cause | Action |
| --- | --- | --- |
| Missing CANN headers | `common_env.sh` not sourced or CANN path mismatch | Source the environment and confirm CANN 9.1.0 layout |
| Cannot find `ascend_hal` | `devlib` path missing | Use the current top-level CMake configuration |
| Cannot find `ascend_hal` | Driver HAL path missing | Add `/usr/local/Ascend/driver/lib64/driver` to `LD_LIBRARY_PATH`; do not use CANN `devlib` |
| Demo target skipped | `bisheng` unavailable | Install/compiler configure `bisheng`, or run host-only tests |
| UDMA disabled in demo | Unsupported hardware or HCCP/RA runtime unavailable | Use A5 / Ascend950 / 950 and check CANN driver/runtime libraries |
| shmem appears in `ldd libtile-comm.so` | Unexpected dependency regression | Inspect `src/comm/CMakeLists.txt` and source includes |
Expand All @@ -139,6 +160,8 @@ ldd shmem check:
UDMA host tests:
UDMA all-gather demo:
UDMA put-signal demo:
SDMA unit/build checks:
SDMA A5 data-plane matrix:
Log directory:

Errors or warnings:
Expand Down
Loading