Slipstream

Run 95.5 GiB Qwen3.8-Flash-Next on a single 64 GB Mac at 41–52 tok/s.

Slipstream is a lean, high-performance C++ and Metal inference engine built specifically for Apple Silicon. It combines SSD expert streaming with predictive read-ahead and Prompt Lookup + MTP speculative drafting to serve frontier-scale models that exceed your Mac's physical RAM.

It serves Qwen3.8-Flash-Next V3 (125.7B parameters, 512 routed experts, 7.3B active per token) and its Swift KV-sparse variant at 1.76x the speed of llama.cpp, with context scaling tested all the way out to 130,000 tokens without decode collapse.

Everything is open source under Apache-2.0.

Daily driver (the one local model to run)

Main model: Swift-Flash-Next V3 lives at ~/models/swift-qwen38-flash-next-v3 (71 GiB, ready-to-serve prepared/ package inside). Published speed: 41–52 tok/s decode. Run one model server at a time.

# Once per boot: lift the GPU memory limit (required on 64 GB Macs)
sudo sysctl iogpu.wired_limit_mb=59392

# Serve the main model (stop any other model server first)
./slipstream serve --model ~/models/swift-qwen38-flash-next-v3/prepared --port 8090

# Check it: the served model id is local/prepared
curl -s http://127.0.0.1:8090/v1/models

Point any harness at http://127.0.0.1:8090 with model id local/prepared:

pi: pi --model slipstream/local/prepared --thinking low
omp: omp --model splash-flashnext/local/prepared --thinking=low --tools=read,write,edit,bash,grep,glob,todo --approval-mode=yolo
curl / python: OpenAI-style POST /v1/chat/completions with "model": "local/prepared"

Why Qwen3.8-Flash-Next V3?

Model Variant	HF Checkpoint (GGUF)	Active / Total Weights	Reasoning (Scorecard)	Peak Speed	RAM Needed
Swift-Flash-Next V3 (New KV-Sparse)	nitinpanj/Swift-Qwen3.8-Flash-Next-Q4_0-Q8out-v3-GGUF	7.3B / 125.7B	70.3% (GPQA 54.3%, MATH 62.9%)	41–52 tok/s	64 GB Mac
Qwen3.8-Flash-Next V3 (34k+ Downloads)	nitinpanj/qwen38-flash-next-v3	7.3B / 125.7B	67.6% (GPQA 45.7%, MATH 60.0%)	40–48 tok/s	64 GB Mac

Frontier Reasoning on a Laptop: Outperforms standard 27B dense models on GPQA Diamond (54.3% vs 45.7%) and MATH-500 (62.9% vs 60.0%), with 92% on HumanEval and 96% on GSM8K.
Fast Generation: 41–52 tok/s sustained decode on an M5 Pro (64 GB).
No Context Collapse: Maintains 32–43 tok/s out to 130,000 tokens thanks to 48 recurrent linear DeltaNet layers ($O(1)$ state growth) and only 16 full-attention layers.
KV-Sparsity (Swift): The Swift variant incorporates KV-sparse attention distilled from Swift-1.5 with spliced Q8 donor backbones, minimizing RAM growth in long agent sessions.

Quickstart (Step-by-Step)

Prerequisites

Hardware: Apple Silicon Mac with 64 GB Unified Memory (M2/M3/M4/M5 Pro/Max).
Disk: ~100 GB for a model served by its Hugging Face id: its GGUF files are converted in place on the first start, so preparing needs only a few GB more. A model in a folder of your own (or with --keep-gguf) keeps its GGUF files, and preparing it needs ~100 GB more.
macOS: macOS 15.0+ (macOS 26.4+ SDK).

Step 1: Install Slipstream

You can install Slipstream directly without compiling, or build from source:

Option A: One-line install (Recommended — No compilation required)

curl -fsSL https://raw.githubusercontent.com/npanj/slipstream/main/install.sh | sh

This automatically detects your Apple Silicon Mac, downloads the latest prebuilt release binary, verifies its SHA256 checksum, unpacks into ~/.local/share/slipstream/, and links slipstream into ~/.local/bin.

You can also download slipstream-<version>-macos26-arm-64bit.zip directly from GitHub Releases.

Option B: Build from source

git clone https://github.com/npanj/slipstream.git
cd slipstream
make -j4

Where Slipstream keeps things

What	Where	Override
Installed versions (the two newest are kept)	`~/.local/share/slipstream/<version>/`	`SLIPSTREAM_PREFIX`
The `slipstream` command (a link to the newest version)	`~/.local/bin/slipstream`	`SLIPSTREAM_BINDIR`
Models	`~/.slipstream/models/<owner>/<repo>/`	`SLIPSTREAM_MODELS`
Downloaded packages (models link into it)	the Hugging Face cache, `~/.cache/huggingface/hub/`	`HF_HUB_CACHE`
Runtime data, logs and caches of a release	`~/Library/Application Support/Slipstream-v2/`

Why ~/.local: the installer puts Slipstream in your home folder, so it needs no administrator rights, no sudo and no package manager, and removing it is deleting two paths. ~/.local/bin and ~/.local/share are the per-user locations of the XDG convention that command-line tools on macOS increasingly use: Claude Code's installer links claude into ~/.local/bin the same way, the installer is modelled on MariaDB Shell's, and pipx and uv install tools there too. Versions sit side by side, so going back is relinking ~/.local/bin/slipstream. If ~/.local/bin is not on your PATH, the installer says how to add it.

Why ~/.slipstream/models: a model is ~100 GB and outlives any one installation. Every installation (a release, a source checkout, or a frontend that runs slipstream pull) uses the same store, so a model is downloaded once and survives upgrades, reinstalls and git clean. It is the convention of other local model servers: Ollama keeps models in ~/.ollama/models, oMLX in ~/.omlx/models. The store is excluded from Time Machine when it is created (tmutil addexclusion), since everything in it can be downloaded again.

Before this, models lived inside each installation: a release kept them in ~/Library/Application Support/Slipstream-v2/models, a source checkout in install/models. Nothing is moved automatically. Either move a model into the store, keeping its <owner>/<repo> folder:

mkdir -p ~/.slipstream/models/nitinpanj
mv ~/Library/Application\ Support/Slipstream-v2/models/nitinpanj/<repo> ~/.slipstream/models/nitinpanj/

or point SLIPSTREAM_MODELS at the old folder. A folder you downloaded yourself (e.g. with huggingface-cli into ~/models/…) is served from where it is, by its path, and never changed.

Step 2: Download or Pull the Model

You can serve directly by Hugging Face repo ID (Slipstream will download and set up weights and MTP sidecars automatically):

# Download without serving:
slipstream pull nitinpanj/Swift-Qwen3.8-Flash-Next-Q4_0-Q8out-v3-GGUF

# Only check whether a repository can be served, and its size (downloads nothing):
slipstream pull nitinpanj/Swift-Qwen3.8-Flash-Next-Q4_0-Q8out-v3-GGUF --check

--check (add --json for scripts and frontends) accepts a ready-to-run Slipstream package, or one Qwen3.8-Flash-Next (qwen4exp) model's GGUF files: a single file or one complete set of split files at the top of the repository. It reads the first file's header from the Hub to check the architecture. pull and serve refuse anything else before downloading.

Or download manually using the Hugging Face CLI:

# Recommended: Swift-Qwen3.8-Flash-Next V3 (95.5 GiB, KV-sparse)
huggingface-cli download nitinpanj/Swift-Qwen3.8-Flash-Next-Q4_0-Q8out-v3-GGUF \
    --local-dir ~/models/swift-qwen38-flash-next-v3

# Or download the plain base model:
huggingface-cli download nitinpanj/qwen38-flash-next-v3 \
    --local-dir ~/models/qwen38-flash-next-v3

Step 3: Raise GPU Memory Limit (One-Time per Boot)

On 64 GB Macs, macOS defaults the wired GPU limit to ~48 GiB. Raise it to 58 GiB so the SSD expert cache and KV pool have ample headroom:

sudo sysctl iogpu.wired_limit_mb=59392

Step 4: Serve the Model

Run slipstream serve pointing at a local folder or directly at a Hugging Face repo ID:

# Serve local model directory:
slipstream serve --model ~/models/swift-qwen38-flash-next-v3 --port 8090

# Or serve directly by Hugging Face repo ID (downloads & sets up automatically):
slipstream serve --model nitinpanj/Swift-Qwen3.8-Flash-Next-Q4_0-Q8out-v3-GGUF --port 8090

# Or serve on your local network (e.g. for other devices or web UIs):
slipstream serve --model ~/models/swift-qwen38-flash-next-v3 --host 0.0.0.0 --port 8090 --api-key YOUR_KEY

First Run Note: On first launch, Slipstream detects multi-shard GGUF files and prepares optimized streaming package files into <model-dir>/prepared/ (~4–7 minutes). Subsequent launches load in ~10–15 seconds.

For a model served by its Hugging Face id, preparing uses the downloaded GGUF files up as it converts them, and deletes them once the package is complete: the package replaces them, and the model needs about its own size on disk instead of twice that. Add --keep-gguf to keep them (e.g. to use them with llama.cpp as well). A folder of your own is never changed. An interrupted preparation resumes where it stopped; if the GGUF data it still needs was already freed, it says which files to download again.

Step 5: Connect Your Tools & Clients

The server provides a standard OpenAI-compatible API on http://127.0.0.1:8090:

curl

curl -s http://127.0.0.1:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local/swift-qwen38-flash-next-v3",
    "messages": [
      {"role": "user", "content": "Write a clean, optimal Python function for interval merging."}
    ],
    "temperature": 0.0
  }'

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8090/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="local/swift-qwen38-flash-next-v3",
    messages=[{"role": "user", "content": "Explain multi-head self-attention with linear algebra."}],
    temperature=0.0,
)
print(response.choices[0].message.content)

Oh My Pi (omp) & Coding Agents

# Launch omp connected directly to Slipstream (served id is local/prepared)
omp --model splash-flashnext/local/prepared \
    --tools=read,write,edit,bash,grep,glob,todo \
    --thinking=low --approval-mode=yolo

Benchmarks & Performance

1. Head-to-Head: llama.cpp Fork vs. Slipstream

Evaluated on Apple MacBook M5 Pro (64 GB Unified Memory, Temperature 0.0):

Task Domain	Benchmark / Prompt	llama.cpp Fork	Slipstream	Speedup	llama.cpp TTFT	Slipstream TTFT
Math Reasoning	GSM8K (eggs derivation)	24.0 tok/s	43.6 tok/s	1.82x	4,024 ms	2,337 ms
Math Derivation	MATH-500 series ($p - q$)	24.3 tok/s	43.1 tok/s	1.77x	1,655 ms	1,587 ms
Constraint Logic	3-chair deduction	25.4 tok/s	46.0 tok/s	1.81x	1,469 ms	1,042 ms
Python Coding	`merge_intervals` ($O(N \log N)$)	19.7 tok/s	35.0 tok/s	1.77x	1,507 ms	1,070 ms
Systems Coding	Rust CSV parser	22.7 tok/s	37.5 tok/s	1.65x	1,257 ms	859 ms
Tech Communication	Multi-head attention	22.5 tok/s	39.4 tok/s	1.75x	1,267 ms	843 ms
OVERALL AVERAGE	Across all 6 domains	23.1 tok/s	40.8 tok/s	1.76x	1,863 ms	1,290 ms

2. Reasoning Accuracy: Swift V3 vs. Plain Base V3

Evaluated across 145 standardized items under memory guard (Seed 1234, T=0.0):

Benchmark	Items	Plain Flash-Next V3	Swift-Flash-Next V3	Accuracy Delta
AIME 2025	20	45.0% (9/20)	45.0% (9/20)	0.0%
MATH-500 (L4–5)	35	60.0% (21/35)	62.9% (22/35)	+2.9%
GPQA Diamond	35	45.7% (16/35)	54.3% (19/35)	+8.6%
GSM8K	25	96.0% (24/25)	96.0% (24/25)	0.0%
HumanEval	25	92.0% (23/25)	92.0% (23/25)	0.0%
Hard Logic	5	100.0% (5/5)	100.0% (5/5)	0.0%
OVERALL SCORECARD	145	67.6% (98/145)	70.3% (102/145)	+2.8%

3. Context Scaling: Live Telemetry to 130,000 Tokens

Measured across 3,086 live agent requests on Apple Silicon (M5 Pro 64 GB):

Context Range (Tokens)	Live Runs	Average Decode	Median Decode (p50)	Peak Decode	Avg TTFT
< 1,000	314	41.5 tok/s	41.9 tok/s	59.8 tok/s	2.16 s
1k – 4,000	21	41.0 tok/s	42.5 tok/s	64.5 tok/s	5.26 s
4k – 8,000	58	43.6 tok/s	43.2 tok/s	67.2 tok/s	7.36 s
8k – 16,000	117	43.6 tok/s	44.6 tok/s	58.2 tok/s	7.91 s
16k – 32,000	562	38.2 tok/s	40.9 tok/s	58.0 tok/s	13.59 s
32k – 64,000	1,029	35.0 tok/s	37.5 tok/s	55.6 tok/s	13.24 s
64k – 96,000	650	32.4 tok/s	34.7 tok/s	53.9 tok/s	12.81 s
96k – 130,000	364	32.9 tok/s	33.3 tok/s	43.8 tok/s	7.95 s

Foundation for Qwen4

The core primitives implemented in Slipstream:

512-route sparse MoE streaming with predictive read-ahead
Quasi-Sparse Attention (QSA) indexer and selection kernels
Hyper-connection mixing and per-layer embedding gathers
Metal GPU-mapped n-gram tables
Single-lane speculative verification with Prompt Lookup Decoding (PLD) + Multi-Token Prediction (MTP)

...were engineered to match the upcoming model generation. Assuming Qwen4 follows Flash-Next's architectural blueprint (hybrid linear recurrence + sparse attention + routed MoE experts), Slipstream can serve as a direct template to run Qwen4 locally on Apple Silicon on day one.

Call for Porting Partners: NVIDIA (CUDA) & AMD (ROCm)

Current Status: Tested exclusively on an Apple MacBook Pro (M5 Pro, 64 GB Unified Memory).
Porting: Because I do not have access to modern NVIDIA (CUDA) or AMD (ROCm) GPU hardware, I cannot build and test those backends myself.
Collaboration Offer: If anyone in the community has hardware available and wants to bring these expert-streaming and speculative decoding gains to CUDA or ROCm, I am happy to collaborate and help with the port. Open an issue or reach out!

Credits & Acknowledgments

Splash Team (Incoai): Full credit to the creators of Splash (github.com/incoai/splash). Their C++ Metal speculative decoding design and memory architecture provided the foundation for this work. We will prepare a clean PR/patch proposing these Flash-Next and SSD streaming extensions to the Splash upstream repo.
ds4 Team: For their valuable insights on Metal router numerical precision (Taylor polynomial softplus expansion) and streaming scheduling designs.
Qwen Team: For training Qwen3.8-Flash-Next and open-sourcing the hybrid linear MTP architecture.
ukisai: For the Swift-1.5 distillation work enabling KV-sparse reasoning.
bartowski & unsloth: For donor quants and quantization tooling.
mihailescu2m: For initial expert streaming concepts in llama.cpp.

License

Apache-2.0. See LICENSE.

Name		Name	Last commit message	Last commit date
.agents		.agents
.github/workflows		.github/workflows
dev		dev
docs		docs
install		install
models/qwen4exp		models/qwen4exp
runtime		runtime
server		server
.gitattributes		.gitattributes
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
DEVELOPMENT.md		DEVELOPMENT.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
install.sh		install.sh
slipstream		slipstream
slipstream-v2		slipstream-v2
splash		splash

Navigation Menu

Repository navigation

About

Repository files navigation

Slipstream

Daily driver (the one local model to run)

Why Qwen3.8-Flash-Next V3?

Quickstart (Step-by-Step)

Prerequisites

Step 1: Install Slipstream

Option A: One-line install (Recommended — No compilation required)

Option B: Build from source

Where Slipstream keeps things

Step 2: Download or Pull the Model

Step 3: Raise GPU Memory Limit (One-Time per Boot)

Step 4: Serve the Model

Step 5: Connect Your Tools & Clients

curl

Python (OpenAI SDK)

Oh My Pi (omp) & Coding Agents

Benchmarks & Performance

1. Head-to-Head: llama.cpp Fork vs. Slipstream

2. Reasoning Accuracy: Swift V3 vs. Plain Base V3

3. Context Scaling: Live Telemetry to 130,000 Tokens

Foundation for Qwen4

Call for Porting Partners: NVIDIA (CUDA) & AMD (ROCm)

Credits & Acknowledgments

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Slipstream

Daily driver (the one local model to run)

Why Qwen3.8-Flash-Next V3?

Quickstart (Step-by-Step)

Prerequisites

Step 1: Install Slipstream

Option A: One-line install (Recommended — No compilation required)

Option B: Build from source

Where Slipstream keeps things

Step 2: Download or Pull the Model

Step 3: Raise GPU Memory Limit (One-Time per Boot)

Step 4: Serve the Model

Step 5: Connect Your Tools & Clients

curl

Python (OpenAI SDK)

Oh My Pi (omp) & Coding Agents

Benchmarks & Performance

1. Head-to-Head: llama.cpp Fork vs. Slipstream

2. Reasoning Accuracy: Swift V3 vs. Plain Base V3

3. Context Scaling: Live Telemetry to 130,000 Tokens

Foundation for Qwen4

Call for Porting Partners: NVIDIA (CUDA) & AMD (ROCm)

Credits & Acknowledgments

License

About

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages