A model-aware distributed inference OS for bandwidth-bound MoE clusters.
The first lab is DeepSeek V4 Flash on N ≥ 2 NVIDIA DGX Spark boxes, with DwarfStar as the native engine. The architecture is for N. A two-Spark pair is the minimum viable lab, not the design.
This repository is a research corpus, not a runtime. The paper is the source of truth. Schemas, the forum, and the ledger are how the design stays challengeable as the lab starts.
What it is not: a DwarfStar fork, a vLLM port, a Kubernetes operator, or a GB200-NVL72 serving stack. Kernel bodies are out of scope. There is no container that "runs the OS" in this tree.
- Read
PAPER.md. That is the design. - Skim
docs/READING.mdif you want the short path through the spine. - Frozen interfaces live in
schemas/. - Decisions, challenges, and dead ends live in
forum/. The forum is append-only.
Decode on CX7-connected Sparks is dominated by sequential activation hops if you pipeline-split a model that already fits, and by cold-expert traffic if you oversubscribe unified memory. AI-OS-1 sees phase, layer kind, and expert ids. It admits, places, and prefetches so decode does not pay DwarfStar pipeline-parallelism's tax. Memory bandwidth (273 GB/s per box) stays the remaining bound. Inter-box ConnectX-7 (200 Gbps) is first-order. On-die NVLink-C2C is not an inter-box link.
Figures are diagram-design editorial HTML, snapshotted as PNG. Original mermaid from the paper is archived in docs/diagrams/mermaid/. Open the HTML if you want the live SVG.
Operator to admit to scheduler. Forum is append-only. DwarfStar is the engine, not the OS.
Phase-home binds the session. Prefill and decode stay in DwarfStar. CX7 is a hop, not a default decode path.
The dashed lines are write-backs. The scheduler compiles a snapshot. It does not let a research loop write a live Slice.
PAPER.md canonical design (the corpus)
workitems.json exclusive section ownership for the paper
schemas/ frozen interfaces (forum, budget, affinity, adapter)
forum/ append-only knowledge graph + index
ontology/ Tier-0 core types and Tier-1 extensions
docs/diagrams/ mermaid archive + editorial HTML + README PNGs
research/
critic/ independent rubric scores
inner-loops/ pruned permutation keepers
ledger.md writing-process hypotheses (also in PAPER appendix)
docs/READING.md how to read the paper
prompts/ provenance of the writing brief
.grok/skills/ vendored diagram-design skill
There is no src/ on purpose. Implementation starts only after Phase 0 measurements in the paper's lab plan.
Four inner loops were pruned (not swept as a Cartesian product) while writing the paper. Later work rebases onto these unless a forum Challenge supersedes them.
| Question | Keeper |
|---|---|
| Placement family | Phase-split hybrid: Flash q2 decode stays on one box; PP is prefill and capacity; gather KV if cheaper than remaining hop tax; q4 needs a TP pair |
| Affinity update | Last-window coactivation counts (2048 tokens) |
| Admit | Phase-split budget vector, p50 reservation, p95 hard cap |
| Forum grain | Publish Decisions and Experiments; roll Observations up |
Confidence uses the closed C1–C5 rubric in the paper (§10.1). New mechanisms in this repo start at C2. They cannot become C4/C5 without a lab.
| Name | Role |
|---|---|
| Session | Isolation: one conversation, one KV quota, one quality class |
| Slice | Placement: layers and experts on one node |
| Assembly | Execution: a typed set of slices plus a transport that can finish one phase |
The research plane (Ralph outer loop, Karpathy-style auto-research, future loop types) shares the forum schema with the inference plane. It does not share a mutable world state, and it cannot write a live Slice.
Design and evaluation plan. Not an implementation. See the paper's §12 for what to measure first on Sparks.
Drafting name was Keel. The public project name is AI-OS-1.
See CITATION.cff.
Apache License 2.0. See LICENSE. The editorial diagram skill is MIT, Cathryn Lavery; see THIRD_PARTY_LICENSES.md.