dvllm

Verifiable, delegated LLM inference in C++.

dvllm splits transformer inference between two parties:

A server (the prover) that holds the large fixed-point weights and performs all the expensive matrix multiplications — optionally on an NVIDIA GPU (CUDA).
A client (the verifier) that runs the cheap parts of the network (embeddings, RMSNorm, RoPE, attention softmax, sampling) locally and cryptographically verifies every matmul the server returns, using a Matrix Commitment Scheme (MCS) over the secp256k1 curve.

The result: you can offload the heavy compute of a model you don't fully trust, and still be confident the returned activations were computed against the exact committed weights — no blind trust in the remote box.

The lineage traces back to Karpathy's llama2.c and Dylan Harden's llama3.2.c, but this repo adds fixed-point quantization, a client/server split, GPU backends, and the commitment-based verification layer.

AI notice: README was generated by AI, but corrected and verified for accuracy. For the development of the actual source, AI was used for catching bugs and supporting additional models, starting from a baseline human-written implementation.

How it works

There are three phases:

Export — a Python script pulls a model from Hugging Face and writes a single model.bin (fp32) file into ~/.dvllm/<model>/.
Convert + commit — the quantize binary reads model.bin and produces:
- model.bin.q — int16 fixed-point weights, used by the server.
- client.bin — the small weights the client keeps (embeddings, norms).
- commits/ — the cryptographic trusted setup (setup.bin) and a commitment file per weight matrix, plus embeddings.bin.
Serve + verify — the server (serverq) loads model.bin.q and answers matmul requests over TCP; the client (llama3.2q / llama2q / gemma3q) drives generation, sends activations to the server, and verifies each returned product against the commitments before continuing.

Cryptography lives in two libraries that are built for you automatically:

mcs-cpp (Rust, in src/verification.rs) — the Matrix Commitment Scheme, using arkworks secp256k1. Built via Corrosion, which CMake fetches.
simdcrypt (C++, vendored in ext/simdcrypt) — AES / PRNG / SHA-256 with AES-NI (x86) or ARM crypto extensions.

Supported models

Family	Client binary	Server binary	Quantizer	Export script	Tokenizer file	Tokenizer script
Llama 3.2	`llama3.2q`	`serverq`	`quantize`	`models/download.py`	`tokenizer.bin`	`models/llama3.2_tok.py`
Llama 2	`llama2q`	`serverq`	`quantize`	`models/download.py`	`llama2.tokenizer`	`models/llama2_tok.py`
Gemma 3	`gemma3q`	`serverq_gemma3`	`quantize_gemma3`	`models/gemma3.py`	`gemma3.tokenizer`	`models/gemma3_tok.py`

Llama 3.2 covers the 1B / 3B variants (base and -Instruct). Gemma 3 covers the 1B / 4B / 12B / 27B -it / -pt variants.

Directory layout

dvllm/
├── src/                 # C++/CUDA/Rust sources for the shipped binaries
│   ├── llama2q.cpp      #   Llama 2  client (verifier)
│   ├── llama3.2q.cpp    #   Llama 3.2 client (verifier)
│   ├── gemma3q.cpp      #   Gemma 3  client (verifier)
│   ├── serverq.cpp      #   Llama server (prover)
│   ├── serverq_gemma3.cpp  # Gemma 3 server (prover)
│   ├── quantize.cpp     #   Llama quantizer + commitment generator
│   ├── quantize_gemma3.cpp # Gemma 3 quantizer + commitment generator
│   ├── verification.rs  #   Matrix Commitment Scheme (Rust, crate `mcs-cpp`)
│   ├── cuda_int32_matmul.cu  # CUDA int matmul kernel
├── include/             # Headers: client logic, networking, verification, backends
│   └── backend/         #   cpu / cublas matmul backends
├── ext/simdcrypt/       # Vendored AES/PRNG/SHA-256 library
├── models/              # Python export + tokenizer scripts
├── extras/              # Non-verified reference clients/servers (opt-in build)
├── cmake/               # FindPCRE.cmake and friends
├── copier.sh            # Copy client files from a server box to a client box
├── CMakeLists.txt
├── Cargo.toml           # mcs-cpp Rust crate
└── requirements.txt     # Python deps

Dependencies

System / build toolchain

You need a C++20 toolchain, CMake, Rust, and a couple of libraries. git is required because CMake fetches Corrosion at configure time.

Dependency	Why
CMake ≥ 3.20	Build system
C++20 compiler	`g++` ≥ 10 or `clang++` ≥ 12
Rust + Cargo	Builds the `mcs-cpp` verification crate
PCRE (`libpcre`)	Regex splitting in the Llama 3.2 tokenizer
OpenMP	CPU parallelism (optional but strongly advised)
Git	CMake `FetchContent` pulls Corrosion

macOS (Homebrew):

xcode-select --install                      # clang, make
brew install cmake pcre libomp rust

Ubuntu / Debian:

sudo apt update
sudo apt install -y build-essential cmake git \
                    libpcre3 libpcre3-dev \
                    libomp-dev clang
# Rust (if not already installed):
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

Fedora / RHEL:

sudo dnf install -y gcc-c++ cmake git pcre-devel libomp-devel rust cargo

simdcrypt is vendored in ext/ (not a git submodule), so a plain git clone is enough — no --recursive needed.

GPU backend (optional)

The CUDA backend is auto-detected at CMake configure time. If CUDA isn't found, the server runs on CPU.

NVIDIA / CUDA — install the CUDA Toolkit (provides cuBLAS, cuRAND). CMake enables the USE_CUBLAS path automatically when CUDAToolkit is found. Adjust the target arch in CMakeLists.txt (CMAKE_CUDA_ARCHITECTURES, default native).

Python (model export)

Model export/conversion tooling runs under Python 3.9+. A virtualenv is recommended:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

requirements.txt installs torch, numpy, tiktoken, blobfile, requests, tqdm, etc. Depending on which family you export you also need:

# Hugging Face model loading (all families):
pip install transformers huggingface_hub

# Llama 2 tokenizer export only:
pip install sentencepiece

Most of these checkpoints are gated on Hugging Face. Request access on the model page, then authenticate:

pip install huggingface_hub
huggingface-cli login

Building

git clone <this-repo-url> dvllm
cd dvllm

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

The first configure fetches Corrosion and builds the Rust crate, so it takes a minute. Binaries land in build/:

build/llama2q  build/llama3.2q  build/gemma3q
build/serverq  build/serverq_gemma3
build/quantize build/quantize_gemma3

Useful configure flags:

Flag	Effect
`-DENABLE_EXTRAS=ON`	Also build the unverified reference `extras/` binaries
`-DCMAKE_CUDA_ARCHITECTURES=90`	Pin a CUDA arch instead of `native`

Note: -march=native is set globally in CMakeLists.txt. If you build on one machine and run on another, remove or adjust it to match the oldest CPU in your fleet.

The `~/.dvllm` model layout

Every binary reads and writes under ~/.dvllm/<model_name>/, where <model_name> is the Hugging Face repo id (e.g. meta-llama/Llama-3.2-1B). After a full export + quantize you'll have:

~/.dvllm/meta-llama/Llama-3.2-1B/
├── model.bin          # fp32 export           (from download.py)
├── model.bin.q        # int16 fixed-point server weights   (server needs this)
├── client.bin         # client weights        (client needs this)
└── commits/
    ├── setup.bin      # MCS trusted setup
    ├── embeddings.bin # embedding commitments
    └── w*_*.bin       # per-weight-matrix commitments

The server needs model.bin.q.
The client needs client.bin and the commits/ directory.
model.bin can be deleted after quantizing (it's large and only the quantizer reads it — see --embed-only for regenerating embedding commitments without it).

End-to-end walkthrough

The example below runs the Llama 3.2 1B model with the server and client on the same machine (localhost). Cross-machine setup is below.

# 0) activate the python env with export deps installed
source .venv/bin/activate

# 1) Export the fp32 model from Hugging Face → ~/.dvllm/meta-llama/Llama-3.2-1B/model.bin
python models/download.py --hf meta-llama/Llama-3.2-1B

# 2) Export the tokenizer (writes tokenizer.bin in the CWD)
python models/llama3.2_tok.py

# 3) Quantize + generate commitments
#    → model.bin.q, client.bin, commits/
./build/quantize meta-llama/Llama-3.2-1B

# 4) Start the server (prover). Listens on port 5065 by default.
./build/serverq meta-llama/Llama-3.2-1B

# 5) In another terminal, run the client (verifier) against the local server
./build/llama3.2q meta-llama/Llama-3.2-1B \
    -z tokenizer.bin \
    -r 127.0.0.1 \
    -n 256 \
    -i "Once upon a time"

The client streams generated text and verifies each server matmul as it goes.

Running each model family

The shape of the workflow is identical for every family — only the export script, quantizer, server, client, and tokenizer file differ (see the support table).

Llama 3.2

# Export (base or -Instruct)
python models/download.py --hf meta-llama/Llama-3.2-1B
python models/download.py --hf meta-llama/Llama-3.2-3B-Instruct

# Tokenizer  → tokenizer.bin
python models/llama3.2_tok.py

# Quantize + commit
./build/quantize meta-llama/Llama-3.2-1B

# Serve + generate
./build/serverq  meta-llama/Llama-3.2-1B
./build/llama3.2q meta-llama/Llama-3.2-1B -z tokenizer.bin -r 127.0.0.1 -i "Hello" -n 128

Llama 2

# Export
python models/download.py --hf meta-llama/Llama-2-7b-hf

# Tokenizer (needs sentencepiece)  → llama2.tokenizer
python models/llama2_tok.py

# Quantize + commit (same quantizer as Llama 3.2)
./build/quantize meta-llama/Llama-2-7b-hf

# Serve + generate (same server as Llama 3.2)
./build/serverq  meta-llama/Llama-2-7b-hf
./build/llama2q  meta-llama/Llama-2-7b-hf -z llama2.tokenizer -r 127.0.0.1 -i "Hello" -n 128

Gemma 3

Gemma 3 uses its own export script, quantizer, and server.

# Export (default is google/gemma-3-1b-it; override with -m)
python models/gemma3.py -m google/gemma-3-1b-it
python models/gemma3.py -m google/gemma-3-4b-it

# Tokenizer — parse the HF tokenizer.json  → gemma3.tokenizer
#  (download the model's tokenizer.json from HF first, or point at your HF cache)
python models/gemma3_tok.py -t /path/to/gemma-3/tokenizer.json -o gemma3.tokenizer

# Quantize + commit
./build/quantize_gemma3 google/gemma-3-1b-it

# Serve + generate
./build/serverq_gemma3 google/gemma-3-1b-it
./build/gemma3q google/gemma-3-1b-it -z gemma3.tokenizer -r 127.0.0.1 -i "Hello" -n 128

Splitting server and client across machines

The whole point is to run the prover on a beefy (possibly untrusted) GPU box and the verifier on a small trusted box.

On the server machine (does the export + quantize, holds the big weights):

python models/download.py --hf meta-llama/Llama-3.2-1B
./build/quantize meta-llama/Llama-3.2-1B
./build/serverq  meta-llama/Llama-3.2-1B --port 5065

Copy the client artifacts to the client machine. The helper copier.sh scps client.bin and the commits/ directory from a remote server into your local ~/.dvllm/<model>/:

# usage: ./copier.sh <remote_user> <server_host> <model_name>
./copier.sh ubuntu 192.168.1.50 meta-llama/Llama-3.2-1B

On the client machine, point the client at the server's IP with -r:

./build/llama3.2q meta-llama/Llama-3.2-1B \
    -z tokenizer.bin \
    -r 192.168.1.50 \
    -i "Once upon a time" -n 256

You can pass -r multiple times to spread the delegated matmuls across several servers (the client shards work across them). All servers must serve the same model.

Command reference

Export — `models/download.py` (Llama 2 / 3.2)

python models/download.py --hf <hf_repo_id> [--version N] [--dtype fp32|fp16]

Writes ~/.dvllm/<hf_repo_id>/model.bin. --checkpoint / --meta-llama are placeholders; only --hf is currently supported.

Export — `models/gemma3.py` (Gemma 3)

python models/gemma3.py -m <hf_model_id>     # default google/gemma-3-1b-it

Quantizer — `quantize` / `quantize_gemma3`

quantize <model_name> [--no-clobber] [--no-commit] [--embed-only]

Flag	Effect
`--no-clobber`	Don't overwrite existing outputs
`--no-commit`	Produce `model.bin.q` / `client.bin` but skip the `commits/`
`--embed-only`	Only regenerate `commits/embeddings.bin`, reusing everything else (doesn't need `model.bin`)

Server — `serverq` / `serverq_gemma3`

serverq <model_name> [options]

Flag	Default	Effect
`--port <int>`	`5065`	TCP port to listen on
`--force-cpu`	off	Ignore any GPU backend and run on CPU
`--copy-to-cpu`	off	Pin weights in CPU RAM for fast async transfer (more RAM)
`--device-layers <n>`	—	Number of layers kept resident on the device
`--num-devices <n>`	`1`	Number of GPUs to shard across

Loads ~/.dvllm/<model_name>/model.bin.q.

Client — `llama2q` / `llama3.2q` / `gemma3q`

<client> <model_name> [options]

Flag	Default	Effect
`-r <ip>`	—	Server IP (repeat for multiple servers; port defaults to 5065)
`-i <string>`	empty	Input prompt
`-n <int>`	`256`	Number of steps (`0` = max sequence length)
`-t <float>`	`1.0`	Temperature (`0` = greedy/deterministic)
`-p <float>`	`0.9`	Top-p nucleus sampling (`1.0` = off)
`-s <int>`	`time(NULL)`	RNG seed
`-z <path>`	family default	Path to the tokenizer file

Loads ~/.dvllm/<model_name>/client.bin plus commits/.

Sampling tip: for best results use -t 1.0 -p 0.9. To control diversity, vary either temperature (with -p 0) or top-p (with -t 1), not both.

Performance & tuning

OpenMP threads — the CPU path parallelizes matmul/attention with OpenMP. Set the thread count to your number of physical cores (not logical, to avoid cache thrashing):
```
OMP_NUM_THREADS=6 ./build/serverq meta-llama/Llama-3.2-1B
```
GPU — build with CUDA present (see GPU backend) and the server auto-selects it. Use --num-devices to shard across GPUs and --copy-to-cpu to trade RAM for faster host↔device transfers.
Fixed-point weights — the matmul weights are stored as int16 fixed-point (scaled by FP_SCALE, see include/constants.hpp); norms and other sensitive parameters stay fp32. This keeps the server's matmuls in integer arithmetic — which is both faster and what the commitment scheme verifies over — at a small precision cost versus fp32.
-march=native — enabled by default for the best vector instructions on the build host. Remove it in CMakeLists.txt if you build and run on different CPUs.

Troubleshooting

403 / gated model on export — request access on the model's Hugging Face page and run huggingface-cli login.
transformers package is required — pip install transformers huggingface_hub.
Llama 2 tokenizer export fails — pip install sentencepiece.
CMake can't find PCRE — install libpcre3-dev (Debian) / pcre-devel (Fedora) / brew install pcre (macOS). The Llama 3.2 / Gemma clients need it.
Server can't open model.bin.q — you skipped the quantize step, or the <model_name> doesn't match the directory under ~/.dvllm/.
Client can't find commits/ or client.bin — run quantize on the client machine, or copy them over with copier.sh.
Client and server disagree / verification fails — make sure both sides were built from the same commit and the server is serving the exact model the client quantized (the commitments are model-specific).
Illegal instruction at runtime — you built with -march=native on a newer CPU than you're running on. Rebuild without it.

License

MIT. See LICENSE. The mcs-cpp crate is BSD-3-Clause (see Cargo.toml).

Navigation Menu

Repository navigation

About

Repository files navigation

dvllm

Table of contents

How it works

Supported models

Directory layout

Dependencies

System / build toolchain

GPU backend (optional)

Python (model export)

Building

The `~/.dvllm` model layout

End-to-end walkthrough

Running each model family

Llama 3.2

Llama 2

Gemma 3

Splitting server and client across machines

Command reference

Export — `models/download.py` (Llama 2 / 3.2)

Export — `models/gemma3.py` (Gemma 3)

Quantizer — `quantize` / `quantize_gemma3`

Server — `serverq` / `serverq_gemma3`

Client — `llama2q` / `llama3.2q` / `gemma3q`

Performance & tuning

Troubleshooting

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
cmake		cmake
ext/simdcrypt		ext/simdcrypt
extras		extras
include		include
models		models
src		src
.gitignore		.gitignore
CMakeLists.txt		CMakeLists.txt
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
copier.sh		copier.sh
gemma3.tokenizer		gemma3.tokenizer
llama2.tokenizer		llama2.tokenizer
matmul.cpp		matmul.cpp
requirements.txt		requirements.txt
tokenizer.model		tokenizer.model

Latest commit

History

Folders and files

Repository files navigation

dvllm

Table of contents

How it works

Supported models

Directory layout

Dependencies

System / build toolchain

GPU backend (optional)

Python (model export)

Building

The ~/.dvllm model layout

End-to-end walkthrough

Running each model family

Llama 3.2

Llama 2

Gemma 3

Splitting server and client across machines

Command reference

Export — models/download.py (Llama 2 / 3.2)

Export — models/gemma3.py (Gemma 3)

Quantizer — quantize / quantize_gemma3

Server — serverq / serverq_gemma3

Client — llama2q / llama3.2q / gemma3q

Performance & tuning

Troubleshooting

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

The `~/.dvllm` model layout

Export — `models/download.py` (Llama 2 / 3.2)

Export — `models/gemma3.py` (Gemma 3)

Quantizer — `quantize` / `quantize_gemma3`

Server — `serverq` / `serverq_gemma3`

Client — `llama2q` / `llama3.2q` / `gemma3q`