Ahead-of-time GPU kernels for production inference on AMD.
Netra Kernel compiles model operations into fixed-contract raw-assembly kernels and packages them as loadable Netra Engines. It brings a TensorRT-style workflow for profiles, tactics, layout planning, static memory planning, and engine building, applied to native AMD GPU kernels.
Compile for the exact serving contract. Launch only specialized kernels at runtime.
The first production backend targets gfx950/wave64, with a focus on
high-throughput FP8 inference for large language models.
Get started · Explore the architecture · Read the engine format · Integrate another engine · Contribute
- Native performance without a model-locked codebase. Kernels remain schedule-specific raw assembly, while contracts and templates make proven mechanisms reusable across compatible Qwen, Gemma, Llama, and future model operations.
- Predictable serving. Shapes, data types, layouts, quantization, launch dimensions, LDS, workspaces, and epilogues are resolved ahead of time.
- A minimal hot path. Engine initialization loads modules, resolves symbols, binds stable memory, and optionally captures HIP graphs. Serving reuses cached handles on a caller-owned stream.
- Safe specialization. Exact profile guards prevent approximate dispatch. Unsupported contracts return an explicit framework fallback.
- Evidence-based optimization. Tactics carry maturity, numerical semantics, deterministic ranking, and correctness and performance evidence.
Netra is intentionally not one dynamically generic GPU kernel. Generality lives in the compiler and kernel library; generated machine code is fully specialized.
The initial production target is Qwen3.6-35B-A3B-FP8 serving on eight AMD MI350X GPUs.
| Deployment | Result |
|---|---|
| 8× MI350X, DP8, dFlash | 78,498.66 output tokens/s mean |
| Exact request/input/output/token/cache checks | 15,360 / 15,360 passed |
Generated versus locked kernel .text |
18 / 18 identical |
The modular engine result was within 0.32% of the locked 78,748.15 tokens/s baseline and passed the no-regression statistical gate. These numbers describe this exact checkpoint, hardware topology, graph configuration, request set, and five-run measurement; they are not a universal performance claim.
See the machine-readable serving validation and validation methodology.
Model or canonical graph
│
▼
Netra Compiler
contracts · profiles · layouts · tactics · memory plan
│
▼
Specialized raw assembly + Netra Engine
fixed symbols · fixed launches · graph recipe · fallbacks
│
▼
Netra Runtime
cached HIP handles · stable bindings · caller-owned stream
The Python compiler accepts an explicit model description or canonical graph, validates its numerical and layout semantics, selects compatible tactics, and emits deterministic engine artifacts. The core compiler has no mandatory third-party Python dependencies.
The gfx950 library is organized into reusable assembler includes and focused schedule templates for routed MoE, attention, routing, GDN, and dense operations. Tactic parameters are assembler-time constants; specialization never introduces runtime shape branches.
An engine contains contracts, selected tactics, launch metadata, static memory and layout plans, generated sources, optional code objects, and a deterministic HIP graph recipe. Engine output is reproducible and contains no semantic timestamps or machine-specific build paths.
The HIP runtime exposes a stable C ABI for loading an engine, querying profiles,
binding persistent and boundary buffers, and launching on a caller-provided
stream. Direct fixed launches remain available when graph capture is disabled.
The installed SDK exports libnetra_engine.so.1, <netra/engine.h>, CMake and
pkg-config metadata, plus a framework-neutral netra_kernel.Engine binding.
The public integration surface is separated from target-specific kernels and inference-engine adapters:
netra-kernel/
├── compiler/netra_compiler/ # AOT compiler, IR, planners, frontends, backends
│ ├── backends/gfx950/ # Target-specific compiler implementation
│ └── library_data/ # Packaged kernels, manifests, and schemas
├── python/netra_kernel/ # Bundle, runtime, CLI, and adapter APIs
│ ├── runtime/ # Framework-neutral Python FFI
│ ├── sglang/ # SGLang adapter and startup plugin only
│ └── assets/ # Deployment profiles and packaged overlays
├── include/netra/engine.h # Stable framework-neutral C ABI
├── runtime/ # Native runtime implementations
├── examples/reference_engine/ # Minimal C consumer with no framework dependency
├── kernels/ # Architecture-specific kernel sources
├── manifests/ # Model, deployment, and tactic contracts
├── schemas/ # Public JSON schemas
├── docker/sglang/ # Thin-plugin and accepted-source images
├── tools/ # Build, compiler, benchmark, and CI tooling
├── tests/ # Compiler, package, and GPU validation
└── docs/ # Integration guides and acceptance evidence
netra-compiler and netra-kernel are separate installable distributions.
The compiler emits a deterministic engine directory; libnetra_engine.so.1
loads and launches it; an optional adapter translates only the surrounding
inference-engine boundary. New engines integrate through the
netra.inference_engines entry-point group without importing SGLang code.
The top-level kernels/gfx950, manifests, and schemas paths point to the
same canonical data packaged in the wheels. This preserves convenient
source-tree commands without maintaining duplicate catalogs that can drift.
Architecture-specific harnesses, profiling scripts, and historical results
remain outside the public SDK surface.
| Capability | Current support |
|---|---|
| GPU target | AMD gfx950, wave64 |
| Accepted operation families | FP8 routed MoE, BF16 routing, GQA FP8-KV attention, split-sequence verification, and GDN |
| Specialization profiles | Decode and fixed verification/prefill profiles used by the catalog |
| Quantization focus | FP8 E4M3 with explicit block-scale and layout semantics |
| Model inputs | Canonical JSON graph, Qwen adapter, and explicit model manifests |
| Reuse demonstrations | Synthetic Gemma and Llama configurations |
| Runtime modes | Direct fixed launches and initialization-time HIP graph recipes |
Gemma and Llama currently demonstrate model-independent contract reuse; they
are not advertised as checkpoint- or performance-accepted deployments. The
separate gfx1151 wave32 kernel track remains available but is not emitted by
the gfx950 compiler.
A model name is never part of a computational kernel identity. Reuse happens only when the complete contract matches:
- operation and exact dimensions;
- data types, accumulation, rounding, and reduction order;
- quantization and scale interpretation;
- tensor and weight layouts;
- ABI, launch dimensions, LDS, and workspace;
- graph-capture and determinism requirements;
- compile-time epilogue and schedule parameters.
When those fields match, different model frontends can select the same tactic and binary. When they do not, the compiler emits another specialized instance or preserves the framework fallback. This keeps the library extensible without making the runtime kernel generic.
CPU-only compiler development requires Python 3.10 or newer. ROCm is needed only to build gfx950 code objects.
git clone https://github.com/NetraRuntime/netra-kernel.git
cd netra-kernel
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e ./compiler -e '.[test]'
make checkInstall the SGLang deployment plugin and inspect the locked profiles:
python -m pip install -e '.[sglang]'
netra-sglang profilesFor gfx950, netra-sglang bundle resolves accepted artifacts by filename and
SHA-256, packages them behind one mount, and the SGLang entry-point plugin
activates their paths before graph capture. For the pinned gfx1151 base SGLang
revision, netra-sglang integrate preflights the complete source overlay in a
disposable worktree before changing the checkout. Run both the integration
command and SGLang from the same environment where netra-kernel is installed;
the generated compatibility shim deliberately imports the wheel's canonical
adapter. See the
SGLang deployment guide for fork, base-image, Docker,
compatibility, and benchmark instructions.
Users of stock SGLang can materialize the hash-checked, performance-accepted
gfx950 source snapshot with netra-sglang source, or build it directly into a
base image with docker/sglang/Dockerfile.base. This keeps their checkout
untouched while reproducing the source and tuning tables used by the serving
gate.
List the available gfx950 tactics:
netra-compile list-tactics --target gfx950 --library-root .Compile the real Qwen3.6-35B serving compatibility engine used by the validated deployment:
netra-compile compile \
--model manifests/gfx950/models/qwen36-moe-m1-golden.json \
--target gfx950 \
--profile decode_m1 \
--library-root . \
--output build/netra-engines/qwen36-35b-current-best
netra-compile explain \
--engine build/netra-engines/qwen36-35b-current-best
netra-compile validate \
--engine build/netra-engines/qwen36-35b-current-best \
--static \
--library-root .This is not a toy model. The manifest records the deployed Qwen3.6-35B-A3B-FP8 checkpoint revision, DP8 configuration, piecewise graph mode, dFlash block size, three accepted M=1 MoE kernel contracts, and every guarded external dispatch or framework fallback still used by the serving stack. CPU-only compilation records the locked artifact identities without requiring checkpoint weights or a GPU.
Cross-assemble all 18 gfx950 artifacts and 19 symbol specializations in the validated Qwen3.6-35B deployment recipe. A visible GPU is not required for this step.
ROCM_DIR=/opt/rocm \
bash tools/compiler/build_gfx950_tactic_catalog.sh \
build/qwen36-35b-current-best \
manifests/gfx950/deployments/qwen36-35b-current-best.jsonThe build checks every emitted .text section against the locked current-best
hash before reporting success. Generated assembly, code objects, disassembly,
metadata, and build-result.json are written under
build/qwen36-35b-current-best/.
Build the exact c64 M768 MoE prefill pair and its fixed-launch bridge:
ROCM_DIR=/opt/rocm \
bash tools/build/build_gfx950_moe_prefill_m768.sh \
"$PWD" "$PWD/build/gfx950-moe-prefill-m768"Compile its deterministic two-operation engine directory:
netra-compile compile \
--model manifests/gfx950/models/qwen36-35b-c64-fp8.json \
--target gfx950 \
--profile verify_m12_b64_routes768 \
--library-root . \
--output build/netra-engines/qwen36-35b-c64-m768The producer template is composed from semantic routing, gate/up, SiLU and quantization, down-partial, and metadata stages. The route reducer is a separate fixed operation because it has a different ABI, launch geometry, and reduction order. The build emits two code objects, not four legacy alias files. Unsupported row counts retain the framework MoE path.
Build the reusable HIP engine runtime:
cmake -S . -B build/gfx950-engine-sdk \
-DCMAKE_HIP_COMPILER=/opt/rocm/llvm/bin/clang++ \
-DCMAKE_HIP_ARCHITECTURES=gfx950 \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/opt/netra
cmake --build build/gfx950-engine-sdk --parallel
cmake --install build/gfx950-engine-sdkThis installs the versioned runtime, public C header, CMake target, and pkg-config file. The compatibility shell builder remains available for in-tree hardware smoke scripts.
Hardware correctness, determinism, graph replay, and performance promotion require a visible gfx950 device. Follow the hardware validation guide.
- Experiment: opt-in development candidate.
- Verified: passed its recorded correctness gates but not full deployment acceptance.
- Accepted: approved only for the exact contract and evidence scope.
- Rejected: retained as evidence when useful and never selected.
A faster microbenchmark alone does not promote a tactic. Numerical behavior, graph replay, full serving correctness, and matched end-to-end performance are part of the acceptance contract.
- Compiler architecture
- Kernel contracts
- Engine directory format
- Adding a model
- Adding a tactic
- Validation and promotion
- Installable SGLang deployment
Netra Kernel is open source under the MIT License. Contributions to the compiler, runtime, documentation, model frontends, validation infrastructure, and fixed-contract kernel library are welcome.
Read CONTRIBUTING.md, the Code of Conduct, and the security policy before opening a change.
MIT © Netra contributors