feat(rocm): AMD ROCm support (gfx1100) + Qwen3.5-MoE GGUF loader

.agents/learnings/index.yml

@@ -0,0 +1,9 @@
+qwen-moe-rocm-base-speed:
+  file: qwen-moe-rocm-base-speed.md
+  description: Read before Qwen MoE ROCm base-speed work; records negative kernel/cache/dense results and validation gates.
+llamacpp-surpass-rocm-gaps:
+  file: llamacpp-surpass-rocm-gaps.md
+  description: Read before follow-up work on native GGUF residency, fused MoE kernels, and ROCm cadence gaps versus llama.cpp.
+rocm-ollama-gap-implementation:
+  file: rocm-ollama-gap-implementation.md
+  description: Read before q8/GGUF ROCm validation or promotion; records fail-closed dispatch, replay timing semantics, and separate Gate A/B evidence.

.agents/learnings/llamacpp-surpass-rocm-gaps.md

@@ -0,0 +1,35 @@
+# llama.cpp Surpass ROCm Gaps
+
+## Context
+
+2026-08-31: Qwen3.6-35B-A3B GGUF decode work on RX 7900 XTX/gfx1100 added
+native Q4_K/Q5_K/Q6_K paths, residency planning, execution gates, and traces.
+
+## Hardest decision
+
+Keep native mixed-K support fail-closed and retain offload defaults: the full
+resident budget does not fit this 24-GiB card, while native Q5/Q6 offload only
+reached 58.98 tok/s API / 61.19 tok/s scheduler.
+
+## Alternatives rejected
+
+- Disable safety reserve to force `fused` residency — risks late OOM and invalid
+  benchmark evidence.
+- Enable grouped GGUF prefill by default — real gfx1100 execution failed during
+  launch; synthetic ABI success was insufficient.
+- Promote gfx1100 rotated-wave or Q5/Q6 path from microbench alone — candidate
+  and end-to-end evidence did not beat the incumbent or llama.cpp ROCm.
+- Treat Torch profiler CPU ranges as additive wall time — synchronization and
+  overlap inflate attribution; no scheduler edit was accepted.
+
+## Least confident
+
+Exact remaining attribution is unresolved because rocprofv3 attach was blocked
+by host ptrace policy. Native executor cadence, full resident capacity with a
+different context/budget, and a genuinely fused/grouped GGUF kernel remain open.
+
+## Reuse
+
+Read before further Qwen ROCm speed work. Preserve model/fixture hashes,
+resident execution evidence, zero fetch/remap gates, graph state, exact output
+count, and paired same-file llama.cpp measurements.

.agents/learnings/qwen-moe-rocm-base-speed.md

@@ -0,0 +1,36 @@
+# Qwen MoE ROCm Base-Speed Optimization
+
+## Context
+
+2026-08-31: Qwen3.6-35B-A3B GGUF Q4_K_M base decode on RX 7900 XTX/gfx1100.
+Plan compared FreeToken against Ollama with MTP excluded.
+
+## Hardest decision
+
+Keep legacy GGUF MMVQ as default after proving the gfx1100 rotated-wave
+candidate 9.7% slower, and close cache/dense/handoff branches when matched
+measurements showed no throughput win. Correctness and explicit target failure
+outweighed speculative promotion.
+
+## Alternatives rejected
+
+- AMD `sdot4` intrinsic rewrite — ROCm target rejects it without `dot1-insts`,
+  and candidate disassembly had no dot4 instruction.
+- Smaller or fixed MoE cache — 4,352 slots fell to 36.875 tok/s; 8,000 slots
+  did not beat auto; warmed runs had zero fetches.
+- Dense MMQ or lm_head rewrite — native MMVQ won every exact bs=1 dense case.
+- MTP/speculative path — outside requested base-speed scope.
+
+## Least confident
+
+The remaining roughly 13 tok/s gap is attributed to ROCm base execution cadence
+and GGUF runtime cost, but profiler CPU totals include overlap attribution and
+are not directly additive to API wall time. Revisit with a lower-overhead
+timeline or newer ROCm/compiler before changing dispatch again.
+
+## Reuse
+
+Read before future Qwen ROCm speed work. Preserve benchmark contract, exact
+completion validation, graph/finite-logit gates, cache hit/fetch telemetry, and
+forced-only candidate dispatch in `benchmarks/`, `python/freetoken/kernel/gguf.py`,
+and `.plans/qwen-moe-speed/`.

.agents/learnings/rocm-ollama-gap-implementation.md

@@ -0,0 +1,51 @@
+# ROCm/Ollama Gap Implementation
+
+## Context
+
+2026-09-01 implementation of `.plans/rocm-ollama-gap` across KV storage, GGUF dispatch,
+profiling, fused MoE, and promotion harnesses.
+
+## Hardest decision
+
+Keep q8 KV and gfx1100 GGUF paths explicit and fail-closed: requested CLI settings never count as
+observed execution, and candidate code never silently falls back in forced mode.
+
+## Alternatives rejected
+
+- Running Ollama/ROCm benchmarks during implementation — user reserved GPU validation for manual review.
+- Reusing model dtype for q8 accounting — allocation, packed scales, attention views, and metadata must share one descriptor.
+- Combining sampled and teacher-forced results — Gate A absolute throughput and Gate B q8 replay measure different claims.
+
+## Least confident
+
+HIP/CUDA compilation and real-model replay remain unverified. The b10434 bridge currently enforces
+512-column activation alignment and needs manual ABI, numerical, graph-capture, and performance proof.
+
+## Reuse
+
+Read before running `.plans/rocm-ollama-gap` evidence: start with `fixture-manifest.json`, enable
+dispatch tracing for per-op records, run q8 with `--kv-type q8_0`, and report Gate A/B separately.
+
+## Validation closure — 2026-09-01
+
+### Hardest decision
+
+Keep legacy GGUF MMVQ as default after exact 512-token q8 greedy output parity but no speed gain:
+candidate `69.040` versus legacy `69.194 tok/s`; fused gate/up also produced degenerate output.
+
+### Alternatives rejected
+
+- Promoting gfx1100 MMVQ or fused gate/up — candidate was slower, and fused model output repeated
+  `We.` despite direct-kernel tolerance.
+- Calling b10434 `predicted_ms` decode timing — forced-token server requests bill their forward pass
+  as one-token `prompt_eval_ms`, so replay accounting now uses that field.
+
+### Least confident
+
+Cross-runtime route parity remains unproven because b10434 server has no route instrumentation;
+short rocprof q8 graph startup also hit duplicate physical Q8 destinations before request.
+
+### Reuse
+
+Read `.plans/rocm-ollama-gap/notes-results.md` before any promotion or kernel follow-up; preserve
+legacy default, q8 opt-in, route-parity requirement, and the warm-offload measured bound.

.github/workflows/unit-nvidia.yml

@@ -0,0 +1,103 @@
+# NVIDIA unit-tests gate: compile the CUDA sources (catches USE_HIP/USE_ROCM-gated
+# refactors that break the CUDA build) and run the test suite on a hosted runner.
+#
+# The runner has no GPU: cu126 torch installs fine, GPU tests self-skip, and the unit
+# suite asserts the CUDA-path *logic* (gating, dispatch, version pairing) which is
+# exactly what a HIP port can silently drift. Compile failures surface here.
+#
+# Trigger policy follows nightly-wheels.yml: only trusted paths for expensive builds.
+# fork PRs cannot run the CUDA install (they can: hosted runner, no secrets). Keep
+# pull_request restricted to code paths; everything else is workflow_dispatch.
+
+name: Unit tests (NVIDIA)
+
+on:
+  pull_request:
+    paths:
+      - "python/**"
+      - "tests/**"
+      - "setup.py"
+      - "pyproject.toml"
+      - ".github/workflows/unit-nvidia.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: unit-nvidia-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  cuda-compile-and-unit:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    env:
+      # torch's CUDA wheel brings its own cudart headers; nvcc from the same toolkit
+      # major compiles the _pinned_tensor / gguf extensions against them.
+      TORCH_CUDA_VERSION: "126"
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install CUDA toolkit (nvcc)
+        uses: js-data/add-cuda-toolkit@v1.1.0
+        with:
+          cuda-version: "12.6"
+
+      - name: Install pytest (uv is preinstalled on hosted runners)
+        run: uv pip install --system pytest
+
+      - name: Compile CUDA extensions (nvcc gate)
+        run: |
+          uv pip install --system torch --index-url https://download.pytorch.org/whl/cu${TORCH_CUDA_VERSION}
+          uv pip install --system -e . --no-build-isolation
+
+      - name: Compile/import legacy GGUF CUDA module
+        run: |
+          PYTHONPATH=python python - <<'PY'
+          import torch
+          from freetoken.kernel import gguf
+
+          assert torch.version.hip is None, "NVIDIA job resolved a ROCm torch wheel"
+          module = gguf._module()
+          for symbol in ("ggml_dequantize", "ggml_mul_mat_vec_a8", "ggml_moe_a8_vec"):
+              assert hasattr(module, symbol), symbol
+          print("legacy GGUF CUDA module imported:", module.__name__)
+          PY
+
+      - name: Compile/import pinned single-token GGUF ABI
+        run: |
+          PYTHONPATH=python python - <<'PY'
+          from pathlib import Path
+          import torch
+          from torch.utils.cpp_extension import load
+
+          root = Path("python/freetoken/kernel/csrc/gguf")
+          module = load(
+              name="freetoken_gguf_b10434_ci",
+              sources=[str(root / "gguf_moe_gfx1100.cu"), str(root / "gguf_b10434_kernel.cu")],
+              extra_cuda_cflags=["-O3", "-DFREETOKEN_GGUF_NO_PYBIND=1"],
+              verbose=False,
+          )
+          for symbol in ("mmvq_bs1_workspace_bytes", "mmvq_bs1", "ggml_moe_mmvq_id"):
+              assert hasattr(module, symbol), symbol
+          assert module.mmvq_bs1_workspace_bytes(512, 16, 8) % 256 == 0
+          print("pinned b10434 ABI imported:", module.__name__)
+          PY
+
+      - name: Run unit suite (GPU tests self-skip)
+        run: |
+          set -o pipefail
+          python -m pytest tests/ -m "not slow" -q -x \
+            --ignore=tests/e2e --ignore=tests/dsv4 2>&1 | tail -40
+
+      - name: Summarize
+        if: always()
+        run: echo "CUDA-compile + unit gate finished; see the step log for failures."