Skip to content
fireworm71Public
forked from antirez/ds4

About

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

 
 

Latest commit

 

History

847 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DwarfStar logo

DwarfStar aims to be the best way to run a few excellent large language models on consumer hardware (that is, hardware that people can actually own). To reach this goal, we are building a small native inference engine optimized first for DeepSeek V4 Flash (including the experimental vision model), DeepSeek V4.1 Flash (Metal, and text inference on CUDA), and additionally GLM 5.2 and 5.3, GLM 5.3 Flash and DeepSeek V4 PRO. The code is self-contained and deliberately narrow, not a general GGUF runner: you need to use the GGUF files the project produces, that are part of the project itself.

We test things in integration: model loading, prompt rendering, tool calls, KV state, the HTTP server, and the coding agent are built and tested together. The repository also includes tools and data for GGUF, imatrix, quality, and speed.

Supported hardware

  • Metal, the primary target, on Macs with 96 GB or more. Smaller machines can use SSD streaming. SSD streaming is also needed in order to run very large models such as full GLM 5.x (not Flash) on 128GB systems.
  • NVIDIA CUDA, the DGX Spark is our main gaol. DwarfStar also supports multi-GPU systems that are not supported by other backends, for instance it can run DeepSeek v4 Flash on Ada Lovelace cards.
  • ROCm on Strix Halo systems such as the Framework Desktop.

This project would not exist without llama.cpp and GGML, make sure to read the acknowledgements section, a big thank you to Georgi Gerganov and all the other contributors.

Model support is intentionally opportunistic. The project follows the best open weights for useful local machine sizes, especially 128 GB laptops and 256/512 GB workstations. A model may be removed when a better replacement arrives.

So, what can I do with this software?

  • You can run a very capable models in your consumer hardware, a MacBook, a DGX Spark, or a Strix Halo for example. Even if you have not enough RAM, with SSD streaming, you can run it at a decent speed.
  • You can use multiple CUDA cards as a multi-user LLM server. Ada Lovelace, including L40S, is supported: newer models can run here even when their other inference implementations require newer GPUs. Our eight-L40S Flash setup has reached about 126 t/s aggregate generation with 16 sessions.
  • Using two 128 GB Macs connected with RDMA, you can run 4-bit DeepSeek Flash or GLM 5.3 Flash with tensor parallelism. Larger GLM 5.2 quants need larger machines, such as Mac Studios.
  • You can also use pipeline paralellism to glue together multiple systems to sum their RAM and run larger models.

Motivations

  • Capable open-weight models now fit on high-end personal machines.
  • DeepSeek V4 Flash and PRO, GLM 5.2, tolerate aggressive routed-expert quantization.
  • Compressed KV caches and fast local SSDs make long contexts practical.
  • The idea of an inference system specialized for a few models.

AI full disclosure

  • This software is developed with strong assistance from AI coding agents and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. The acknowledgement below is equally important: this would not exist without llama.cpp and GGML, largely written by hand.

Acknowledgements to llama.cpp and GGML

ds4.c does not link against GGML, but it exists thanks to the path opened by the llama.cpp project and the kernels, quantization formats, GGUF ecosystem, and hard-won engineering knowledge developed there. We are thankful and indebted to llama.cpp and its contributors. Their implementation, kernels, tests, and design choices were an essential reference while building this DeepSeek V4 specific inference path. Some source-level pieces are retained or adapted here under the MIT license: GGUF quant layouts and tables, CPU quant/dot logic, and certain kernels. For this reason, and because we are genuinely grateful, we keep the GGML authors copyright notice in our LICENSE file.

Status

The software is currently very fast changing. Consider it beta quality. Before each release, a big QA run is executed, however instabilities and regressions are definitely possible.

How to use this project?

I (Salvatore) believe that the way projects should be shipped and used changed because of AI. The main differences today are:

  1. With AI, users can modify the software in significant ways with low efforts, costs, and even lacking deep domain knowledge about the task they want to accomplish. For instance, a DwarfStar user with a specific hardware setup can ask a coding agent to improve the inference speed of this software for the specific hardware setup, asking the model to reach the maximum prefill and generation speed without impacting correctness, and also asking to do a deep QA pass.
  2. Similiarly, because of "1", software may be shipped in a different way than before. It must be more a working template for the biggest use cases, without trying to cover every possible setup. If DwarfStar showcases a few good implementations of tensor parallel execution, the code will work as a rail for implementing the same feature in specific conditions, for a new model, and so forth.

So, while this project attempts to be usable for the featured models and the most common hardware setups, I ask you, if you have access to coding agents, to consider using coding agents as an interface to discover the project, make modifications, create personalized setups. This way you can likely do more than what we ship, and certain things that are not documented or implemented, and that you require, are potentially very easy to achieve.

Start Here

git clone https://github.com/antirez/ds4.git
cd ds4

Choose your build. The platform guides cover prerequisites, memory sizing, and hardware-specific setups:

Platform guide Build
Metal on Apple Silicon make
DGX Spark make cuda-spark
Strix Halo / Framework Desktop make strix-halo
One or more CUDA cards, including Ada/L40S make cuda-generic

For a first run on a 96 or 128 GB machine, download DeepSeek V4 Flash Q2:

./download_model.sh ds4f-q2

Downloads go in gguf/. Repeat the command to resume an interrupted download. Leave memory for the context and runtime buffers as well as the model. See other models or use SSD streaming on a smaller Mac.

Everyday Use

Once built and with a model downloaded:

./ds4
./ds4 -p "Explain Redis streams in one paragraph."
./ds4-agent
./ds4-server --ctx 32768

The default model is ds4flash.gguf, a link updated by main-model downloads. Pass -m FILE to choose explicitly. Commands normally run from the repository root; use --chdir /path/to/ds4 when launching elsewhere.

The server listens at http://127.0.0.1:8000 by default; see serving for API access and multiple sessions.

The interactive CLI keeps a multi-turn conversation. Use /help, /read FILE, /ctx N, and /quit. Ctrl+C interrupts generation and returns to the prompt. Run each binary with --help for its full options.

Native coding agent

ds4-agent runs inference directly, without a separate HTTP server. It keeps the token history and live model state together, shows prefill progress, and uses the model's native tool format. DeepSeek and GLM have their own templates.

Use /hints on for occasional, brief explanations of the programming choices behind the work, and /hints off to stop them. Changes take effect at the next conversation boundary without rebuilding the cached context. New and resumed sessions start with hints off.

Sessions are stored in ~/.ds4/kvcache:

Command Action
/save Save the current session
/list List saved sessions
/switch <sha> Resume a session
/del <sha> Delete a saved session
/strip <sha> Keep text and title, removing the large KV payload

Compatible local KV snapshots avoid rebuilding the prompt. Stripped sessions and network TP restores require prefill. Sessions containing images cannot yet be saved. Saved conversations and traces may contain private information.

For Pi, OpenCode, Codex CLI, or Claude Code, use ds4-server instead and follow the client setup guide.

Models, images, and speculation

Models and vision lists the supported downloads and memory requirements. DeepSeek Vision Experimental uses a different checkpoint from Flash 0731; GLM 5.3 Flash adds vision to the same text model.

DeepSeek V4.1 Flash text and vision run on Metal; text also runs on a DGX Spark. Q2 runs with SSD streaming on one 128 GB Mac or Spark, or resident across two Macs or two Sparks using RDMA. Q4 needs SSD streaming or a 512 GB Mac. Engram tables remain on disk in every mode, so use a fast local SSD. See the model guide for downloads and setup.

With the matching encoder passed as --vision FILE, use /read image.png in the CLI or view_image in the native agent.

Speculative decoding is opt-in. GLM uses --mtp; V4 Flash DSpark needs a matching support GGUF. It can improve generation, but not every workload benefits. Read speculative decoding for setup and the difference between default opportunistic sampling and --mtp-exact-sampling.

Output and power

Thinking is enabled by default. Use --nothink or /nothink for direct answers, and --think or /think to enable it again. For V4.1, ds4 and ds4-agent also accept --think-level 25 or /think 25: 1 to 100 sets the reasoning effort, and 0 disables thinking. --think selects 75, --think-max selects 100. Changing the level in a conversation rebuilds its cached prefix. The normal sampling defaults are temperature 1, top-p 1, and min-p 0.05; --temp 0 selects greedy output.

For DeepSeek V4, --power N trades throughput for lower sustained GPU load. The default is 100. V4.1 and GLM currently require --power 100.

DeepSeek V4 Flash and GLM 5.3 Flash also support directional steering. Load a vector with --dir-steering-file FILE; /steer F adjusts its scale for subsequent tokens in a local CLI or agent session, without rebuilding the existing KV cache. See steering documentation.

--prefix-file FILE preloads complete USER: / ASSISTANT: pairs before the live conversation. A turn marker must start a line, roles must alternate, and the last turn must be ASSISTANT:.

Capability Evaluation

ds4-eval runs embedded capability regression tests against a real GGUF. These are DwarfStar integration checks, not official leaderboard scores.

./ds4-eval -m ds4flash.gguf --trace /tmp/ds4-eval.txt
./ds4-eval -m ds4flash.gguf --suite hard-smoke
./ds4-eval -m ds4flash.gguf --suite hard --retry-incomplete

The default suite is core; --suite all runs core and hard cases. --list-cases lists tests without loading a model. --plain selects non-interactive output, and --regrade-trace FILE scores an existing trace without generating again. Sources and licenses are in EVAL_DATA.md. For inference correctness and release checks, read testing.

Speed

This recorded DeepSeek V4 Flash Q2 sweep uses an M5 Max with 128 GB RAM, 2048-token continued-prefill intervals, and 128 greedy generation tokens per frontier. It is a baseline, not a fresh benchmark of every commit.

M5 Max Flash Q2 throughput

See performance and benchmarking for the full numbers, DGX Spark results, comparison conditions, and benchmark commands.

Detailed Guides

Read CONTRIBUTING.md before sending a pull request.

DeepSeek V4.1 Flash optimization campaign (this fork)

This repository is a development fork of DwarfStar. On top of upstream a04f46f it carries 127 commits from a DeepSeek V4.1 Flash text-inference optimization campaign, run mostly on a pair of DGX Spark GB10 machines in tensor parallelism over RoCE RDMA. The work is organized as numbered tasks (T1–T31) whose write-ups live in misc/ (that directory is gitignored, but the notes are force-added so they are tracked). Almost everything below is opt-in or off by default; negative results are recorded as carefully as wins.

Headline results at a glance:

Area Result
Speculative decode, fast mode Copy-heavy decode 22.45 → 29.75 t/s; one 61-draft block accepted whole
Speculative decode, correct mode Loses at every depth, −14% to −25%; target-only stays default
n-gram lookup drafting Copy-heavy 22.45 → 27.50 t/s (+22.5%); prose neutral at a strict gate
Q3 mixed-quant splice NLL −5.79% vs Q2 (69/100 cases), −7% decode
MoE SSD streaming Fetch-at-depth: shallow prefill +46.8%, deep +20.4%, byte-identical
RDMA expert tier 1.75–1.79x over one NIC, byte-exact; remote-RAM promotion measured null
Auto-promotion to resident Q2 TP2 decode 14.07 → 21.45 t/s (+52.4%)
Streaming penalty Not disk: per-layer probe sync is 41.28 ms/token (58% of decode)

Speculative decoding (DSpark) on V4.1

  • Ported the DSpark drafter to V4.1: a sidecar builder (gguf-tools/deepseek41_dspark.py), the verify transaction core, the batched ds41_graph_verify_rows path, and full session integration. Dropping a spurious per-head q norm took acceptance from 0.29 to 1.18 tokens/cycle.
  • Added a load-time finiteness guard after finding that two of the three DSpark support GGUFs on disk held non-finite weights in 37 tensors, which made the drafter silently propose nothing.
  • Wired the verify into the existing TP protocol: DS4_TP_FRAME_VERIFY / VERIFY_COMMIT, argmax-only logits exchange (DS4_TP_FRAME_VERIFY_TOPS), multi-rank coordinated early exit, and a clean decline (rather than a dead generation) when speculative decode is unsupported under network TP.
  • Raised the verify/block caps 8 → 16 → 32 → 48 → 64 and added an AIMD streak manager plus direct RDMA slab staging that bypasses host bounce buffers. In the fast batched mode this reached 22.45 → 29.75 t/s on copy-heavy work, with a full 61-draft block accepted in one verify.
  • Found the correctness wall: the batched verify is not batch-size invariant. A row computed in an N-row sweep differs from the same row computed alone; ~1.3% of verified rows flip an argmax and commit a different token than serial decode. The failure is silent and dangerous in code and identifiers (ROUTER_MAX_QUEUES became ROUTER_MAX_QUEues), and generated text is a lossy detector of it, so validation uses per-row logit probes (misc/T29, misc/T30).
  • CORE_INVARIANT (per-row attention) fixes it for ~1.8% decode cost, but it removes the batching that made the verify cheap. The depth curve then slopes down single-box (5.54 → 5.01 t/s at depth 6) and on the TP2 pair (−14% to −25% at every depth, misc/T31). Target-only decode remains the default until the batched attention core is batch-size invariant.
  • A later fix for a layer-39 HC weighted-sum bug in verify rows reported 100% bit-exact greedy output and +50% in the batched mode; the T29–T31 probe work established that mode is not generally invariant, so those numbers are not treated as safe. All speculative paths stay opt-in.

Speculative n-gram engine

  • ds4_ngram.c / ds4_ngram.h: a 4-way set-associative hash table with O(1) lookup, LFU eviction, empirical continuation frequencies, and binary serialization.
  • Three tiers, consulted in priority order: context (the current generation), static (precomputed from a corpus with --build-ngram-corpus, auto-loaded), and dynamic (persisted across runs to ~/.cache/ds4/ngram_cache.bin).
  • Longest-match chaining from 4-grams down to 1-grams, syntax-boundary token clamping, confidence-aware chain truncation, and an adaptive gated hybrid that tries the free n-gram draft first and falls back to DSpark only when that is worth its cost. A lookup miss declines the cycle instead of paying the model's propose.
  • Measured: verbatim-copy decode 22.45 → 27.50 t/s (+22.5%), later 29.75 (+32.5%); prose is neutral at a strict minimum match. Hybrid fallthrough defaults to off under SSD streaming. Enable with DS4_DSPARK_NGRAM_DRAFT=1 (off by default).

Q3 mixed-quant splice

  • A "Q3splice" GGUF is the Q2 base with the trailing layers' routed experts taken from Q4; 6L, 7L and 8L variants were built, and 8L is the largest that fits the pair.
  • Quality (100-case official-continuation NLL): 8L scores 0.343799 against Q2's 0.364937 (−5.79%, winning 69/100 cases) while full Q4 scores 0.246408 (−32.5%, streaming only).
  • Speed: 8L TP2 resident runs 20.06 t/s against Q2's 21.57 (−7%) at ctx 32K with the 4K prefill chunk; against the Q4 streaming configuration it actually has to use, that is roughly +95% decode and +129% prefill.
  • The memory ceiling was mapped exactly: the pair works at 100.07 GiB planned per rank and fails at 101.86. Maximum Q4 layers by context are 10 at 32K, 9 at 128K, and 7 at 256K.
  • Layer selection is not a lever: per-layer Q2↔Q4 divergence is flat, and choosing the "best" k layers gains +0.93%, about 13x below the scoring noise floor. Take the last k.
  • "Q3 is not broken": the streaming expert cache was a single slab strided for one expert size, torn down and rebuilt twice per token on a mixed file. Resident TP2 needs no fix; the admit check was also bounded to what the host can actually hold, and mixed-quant routed experts are admitted on resident network TP.

MoE streaming optimization (SSD streaming and expert transport)

  • Fetch at depth: parallel expert preads instead of one synchronous read at a time. Shallow prefill 46.65 → 70.51 t/s (+46.8%), deep 266 → 320 t/s (+20.4%), generation +2.7%/+8.1%, all byte-identical. DS4_CUDA_EXPERT_FETCH_THREADS selects the depth (default 1).
  • Upstream's prefill read-ahead (c2c3ce3) was measured and rejected: it competes for the same saturated NVMe as the depth-8 fetch, costing −9.6% (Q4) and −14.1% (Q2).
  • RDMA expert tier (ds4_rdma_tier.c): one-sided RDMA READ striped across two NICs on two devices, 1.75x (Q2) and 1.79x (Q4) over one leg and byte-exact, with a region server, connection pooling, and read-back verification. Off unless DS4_EXPERT_TIER_RDMA is set. Remote-RAM promotion of uncovered spans measured null: every span that reaches disk is a first touch, because the expert cache already absorbs the reuse.
  • The streaming penalty is not disk. A per-layer host probe sync costs 1.032 ms × 40 layers = 41.28 ms/token, 58% of the streaming decode budget; decode disk I/O is only 8.4 ms/token.
  • Auto-promotion (Lever 5): when a shard fits the non-movable budget, the engine clears --ssd-streaming and runs resident. Q2 TP2 decode went 14.07 → 21.45 t/s (+52.4%) and the Q3 7L splice at 256K went 10.31 → 19.84 t/s (+92.4%) against the Q4 streaming baseline. DS4_FORCE_SSD_STREAMING=1 bypasses it.
  • Two attacks on the remaining cost measured null: a device-side probe bypass cost 8.2% of Q4 decode, and per-layer expert partitioning lifted the decode hit rate from 63.5% to 95.9% and cut decode I/O 16% but bought no throughput (the fetching is already overlapped). Both stay narrow and opt-in.
  • TP ranks can stream their own half of the experts (DS4_CUDA_TP_STREAMING). A routing analysis found 55.7% of decode misses are predictable one layer ahead, but decode is disk-bandwidth-bound, so the prefetcher was a null.

Transport and TP plumbing

  • Big-gate RDMA work: the per-layer TCP rendezvous became optional (−16 ms/sweep), direct RDMA slab staging removed host bounce buffers, vocab-split logits and multi-rank early exit were added, and a resident TP pair now bulk-sweeps short prompts (192-token prefill 23.56 → 37.16 t/s).
  • Fixed test_tp_tcp, which had never passed on Linux, by enforcing the TCP receive-buffer floor before the connection carries traffic.
  • Smaller changes: CLOCK second-chance expert eviction, depth-span expert fetch, I8 support in the mixed-splice tool, and a per-token NLL dump for --perplexity-file.

Diagnostics and tooling

  • Environment-gated probes for the speculative-verify investigation: per-row logit diff against serial decode, in-situ attention 1-row-vs-N-row invariance, attention dispatch logging, a safe-margin accept gate, and DSpark stage and finiteness profiling. All inert by default.
  • gguf-tools/analysis/: routing replay, a Belady-bound policy sweep, demand periodicity, cross-chunk prefill replay, and expert placement analysis.
  • tests/rdma_t1/ (the RDMA transport prototype and region server) and tests/test_ngram_cache.c.

Bottom line

The pair's practical configurations are Q2 TP2 resident (~21.5 t/s) or Q3 7L/8L TP2 resident (~20 t/s at measurably better quality), with Q4 TP2 streaming (~10 t/s) only when memory forces it; n-gram lookup drafting is worth enabling for copy-heavy work. Speculative decode stays opt-in and off by default, because it does not pay in the numerically correct mode. The single blocker on that whole track is making the batched attention core batch-size invariant.

Logo

The DwarfStar logo was designed by hand by Salvatore Sanfilippo, made more graphical with AI, and manually reworked by Ben Gnomino, whose human touch made it rock.

About

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages