Design discussion: first-class out-of-tree model providers

[ErikBPF]

Design discussion: first-class out-of-tree model providers

Summary

While prototyping Qwen3.6 mixed-GGUF execution on ROCm, the implementation crossed model parsing/loading, quantized kernels, expert-cache behavior, graph safety, and scheduler policy. Before making that work production-shaped, I would like to align on a narrower model-support boundary: explicitly trusted, out-of-tree model providers using FreeToken's existing lazy model registry and runtime-preparation lifecycle.

The goal is not a general plugin framework. A separately distributed model package would provide model semantics, checkpoint adapters, and model-owned execution policy while FreeToken retains scheduling, cache admission, backend mechanisms, and graph lifecycle.

Public PoCs

• Full Qwen3.6/ROCm implementation snapshot: Draft PoC: Qwen3.6 GGUF on ROCm ErikBPF/FreeToken#2
• Minimal provider-discovery seam: PoC: load model providers from entry points ErikBPF/FreeToken#1

Branch-only evidence

FreeToken source came from a fresh clone of the public runtime PoC branch. No private source or result artifact was used.

Test machine:

• 128 GB RAM
• AMD Ryzen 9 5950X CPU
• AMD Radeon RX 9070 XT 16 GB GPU (gfx1201)
• PyTorch 2.11.0+rocm7.14.0; HIP 7.14.60850

The branch built its native and GGUF HIP extensions after an explicit pip-vendored ROCm SDK root was supplied. Its shared-expert/copy microbenchmark reported:

Metric	Result
Serialized median	0.216279 ms
Overlapped median	0.182385 ms
Median-time improvement	15.67%
Graph copy verification	pass
Bit-exact output	pass

Artifact: https://github.com/ErikBPF/FreeToken/blob/draft/qwen36-rocm-full-poc/benchmarks/results/qwen36-graph-stream-overlap-gfx1201.json

Seven dependency-free tool, contract, and artifact-consistency tests pass from the branch. Python compile, shell syntax, JSON parsing, and git diff --check also pass.

Evidence boundary

The branch does not contain immutable endpoint reports or validators for historical single-stream, BS4, or FTW throughput measurements. This issue therefore makes no endpoint tok/s, batching-throughput, or GGUF-versus-FTW performance claim. The quantitative result above validates overlap mechanics only.

The clean build also exposed a reproducibility defect: setup.py needs an explicit ROCm SDK root for the pip-vendored _rocm_sdk_core layout.

Coupling exposed by the PoC

The experimental implementation spans four layers:

• The Qwen model coordinates Q8 activation reuse, shared-expert work, expert copies, and its fused routed epilogue.
• Generic layers/moe.py contains Qwen-named prepared-decode methods and environment checks.
• Generic moe/offload_cache.py contains Qwen-named copy schedules and two-bank assumptions.
• Fusion policy is read from process-global environment values.

That shape makes optimized models edit generic MoE/cache code, prevents per-model-instance policy, hides active fallback provenance, and leaves capability mismatches to hot-path attribute checks.

It also overlaps generic work already discussed in #199, #231, #252, and #300.

Existing lifecycle seam

FreeToken already has most of the lifecycle needed:

• ModelSpec names the model class, configuration parser, and weight iterator through lazy import strings.
• The engine calls prepare_for_runtime() after weights and MoE cache attachment but before KV allocation and graph capture.
• Server workers use multiprocessing spawn, so provider resolution must work independently in every process.

Proposed boundary

checkpoint architecture
  -> explicitly allowed model provider
       -> model/config/checkpoint semantics
       -> model-owned runtime policy
       -> model-specific kernels and exact fallback
            |
            v
       generic core mechanisms
       banks, views, copies, events, backend capabilities
            |
            v
       generic scheduler
       batching, KV/MoE admission, residency, telemetry

Provider-owned

• Model construction and configuration mapping.
• Checkpoint-format adapters.
• Model-specific execution choreography.
• Namespaced custom operators.
• Model-level capability checks and exact fallback.

Core-owned

• Provider discovery and explicit trust enforcement.
• Model resolution and lifecycle.
• Generic bank, view, copy, event, and graph-safe mechanisms.
• Backend capability reporting.
• KV/MoE admission, batching, cache policy, and telemetry.

Minimal provider mechanism

The provider PoC uses stdlib package entry points keyed by architecture:

• discovery reads package metadata before importing provider code
• only a distribution named in FREETOKEN_MODEL_PROVIDERS may load
• providers export the existing ModelSpec
• duplicate providers and built-in replacement attempts fail explicitly
• spawned processes resolve from installed metadata and inherited configuration

The current PoC deliberately does not implement a stable CLI/config surface, provider API-level negotiation, FreeToken version compatibility, hardware capability checks, or a kernel/backend ABI.

Trust and failure behavior for a production design

Installing a provider should not be consent to execute it. Startup should reject, before GPU allocation:

• a matching provider that was not explicitly allowed
• duplicate ownership without explicit selection
• an incompatible provider API level or FreeToken version
• a provider unsupported by the selected hardware/runtime

Startup provenance should identify the distribution, version, architecture provider, checkpoint adapter, resolved runtime policy, backend capabilities, and fallback reason.

Related contributions

Correctness mechanisms remain separate from this design discussion:

• fix(rocm): avoid indirect host pointers during HIP graph capture #316: HIP graph capture fallback for expert copies
• fix(ftw): preserve auxiliary expert banks #317: auxiliary expert-bank preservation in FTW

Their narratives now avoid endpoint performance claims not reproducible from the public PoC branch.

Non-goals

• Scheduler plugins or arbitrary graph-capture hooks.
• Silent overrides of built-in architectures.
• Migrating every built-in model in the first change.
• Reporting aggregate batching gains as single-stream model gains.
• Treating uncommitted historical endpoint runs as reproducible evidence.

Questions for maintainers

1. Is an explicitly allowlisted, out-of-tree model-provider boundary directionally acceptable?
2. Should a provider add only a checkpoint format to an existing architecture, or own the complete architecture implementation?
3. What is the minimum stable provider surface beyond the current ModelSpec fields?
4. Should tokenizer, reasoning/tool parsers, and multimodal input processing be part of the first contract?
5. What compatibility promise should apply to provider-supplied CUDA, HIP, Triton, and PyTorch operators?

[ErikBPF]

Public evidence has been narrowed to what the published PoC branch can reproduce.

• Runtime PoC: Draft PoC: Qwen3.6 GGUF on ROCm ErikBPF/FreeToken#2
• Provider seam: PoC: load model providers from entry points ErikBPF/FreeToken#1
• Committed result: https://github.com/ErikBPF/FreeToken/blob/draft/qwen36-rocm-full-poc/benchmarks/results/qwen36-graph-stream-overlap-gfx1201.json

A fresh public-branch clone built and ran on a machine with 128 GB RAM, an AMD Ryzen 9 5950X CPU, and an AMD Radeon RX 9070 XT 16 GB GPU (gfx1201). The branch-contained shared-expert/copy microbenchmark verified graph copies and bit-exact output, with median time moving from 0.216279 ms serialized to 0.182385 ms overlapped (15.67%). Seven dependency-free branch checks pass.

Historical endpoint, BS4, and FTW tok/s numbers were removed because the public branch does not contain immutable reports and validators for them. The remaining quantitative claim is a microbenchmark result only; it is not an endpoint-throughput claim. The clean run also found that pip-vendored ROCm currently needs an explicit SDK-root override during setup.py extension builds.