Everything on this page is a measurement, not a promise. If you run colibrì on hardware not listed here, please open an issue with your numbers — real datapoints are what move this project.
Detailed GPU experiment: GLM-5.2 on 6× RTX 5090 — full expert residency across VRAM+RAM reaches 6.84 tok/s single-request decode.
| metric | value |
|---|---|
| model on disk (int4 container) | ~370 GB |
| resident RAM (dense, int4) | 9.9 GB |
| load time | ~30 s |
| peak RSS during chat | ~20 GB (auto-capped) |
| cold decode cost | ~11 GB disk reads/token (75 layers × 8 experts) |
| disk ceiling (this dev box's drive) | ~1 GB/s → ~0.05–0.1 tok/s cold |
| MTP speculation (int8 head) | 2.2–2.8 tok/forward measured (#8) |
This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.
Cold starts are heavy on random reads (~11 GB/token), but reads don't
meaningfully wear an SSD — colibrì's streaming is read-only. The real concerns
under heavy use are (1) swap traffic if the system runs out of RAM (writes
do wear the drive — keep a sane --ram budget; colibrì's auto-budget is designed
to stay clear of swap) and (2) sustained thermals: hours at full read duty
cycle will heat cheaper drives. Monitor drive temperature and health.
cd c && ./setup.sh # build + architecture self-test (expects ~30-32/32)
# 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads):
gcc -O2 -fopenmp iobench.c -o iobench
./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0 # buffered, 8 threads
./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1 # O_DIRECT (bypass cache)
# Caveat (#86): iobench reads a bounded ~1 GB shard, so buffered reads on a big-RAM box
# report the PAGE CACHE, not the disk. Use the O_DIRECT run (arg 1) for a true number, and
# run it on a shard you haven't touched this session (a prior buffered run caches its pages).
# On macOS there is no O_DIRECT — iobench uses F_NOCACHE, which stops *new* caching but can't
# evict pages a prior buffered run already resident-mapped, so a macOS "O_DIRECT" figure right
# after a buffered run still reads cache. Reboot or use a fresh shard for a real cold read.
# 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS):
COLI_MODEL=/path/to/glm52_i4 ./coli chat
# 3) full automated datapoint — machine info + cold/warm decode + disk, one command:
python tools/datapoint.py --snap /path/to/model --shard /path/to/container/model-00000.safetensors
# (stdlib-only; auto-selects GLM, Inkling, Kimi K3, OLMoE, Qwen3.8, Qwen3.6, or DeepSeek V4
# from config.json; evicts the page cache before engine load; then keeps one
# SERVE=1 engine alive for cold, warm-identical, and rotating-prompt measurements)
# 4) record expert usage, then pin the hottest experts in your spare RAM:
STATS=stats.txt ./coli chat
PIN=stats.txt PIN_GB=20 ./coli chat # scale PIN_GB to your free RAM
# 5) quality benchmarks (MMLU/HellaSwag/ARC):
./coli benchThe default datapoint measures serving behavior, not repeated startup: the same
engine process and cache slot are used for one cold request, one repeated-prompt
warm request, and four requests drawn in fixed order from a diverse built-in
prompt suite. The rotating-prompt median is the primary result because the
expert cache must keep adapting as requests change. Warm-identical remains in
the report only as a useful cache/KV upper bound; it is not representative of a
mixed chat workload. The reported completion count, decode tok/s, expert hit
rate, and RSS come from each engine's DONE frame.
Use repeated --rotate-prompt '...' arguments to replace the built-in suite
(at least two distinct prompts), or --rotating-runs N to run more fixed-order
samples. Use --mode fresh-process only when measuring startup or OS page-cache
effects; that compatibility mode launches a new engine for every row and
therefore does not retain in-process caches.
Decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost once the cache is warm; RAM turns cold reads into free cache hits.
| machine | expected |
|---|---|
| the dev box (WSL2 VHDX, ~1 GB/s, 25 GB RAM) | ~0.05–0.1 tok/s cold — proven baseline |
| native Linux, PCIe4 NVMe (~3–5 GB/s random), 32 GB | ~0.5–1 tok/s |
| PCIe5 NVMe or 2×NVMe RAID0 (~8–12 GB/s), 64 GB (PIN ~40 GB of hot experts) | ~2–4 tok/s |
| 128–256 GB RAM, 12 cores (hot experts cached) | ~2–4 tok/s — matmul-bound: ~80 GFLOP/token vs ~250 GFLOP/s of our AVX2 kernels |
| same RAM + 24–32 cores, or AVX-512/VNNI kernels | ~5–15 tok/s — interactive; kernel work is the multiplier |
These are estimates, not measurements.
Real numbers from real machines, stock build (setup.sh, gcc 13), greedy decoding, --ngen 32, MTP active:
| machine | disk (iobench, 19 MB × 64, 8 threads) | config | measured |
|---|---|---|---|
| Intel Core Ultra 7 270K Plus (24 threads) · WSL2 · 24 GB RAM · NVMe VHDX (#2) | 1.96 GB/s buffered · 2.74 GB/s O_DIRECT | default | 0.07 tok/s · expert hit 3–4% · RSS 14.1 GB |
| 〃 | 〃 | --topp 0.7 |
0.11 tok/s · expert hit 11% · RSS 14.7 GB |
| Apple M5 Max (18 cores) · macOS · 128 GB unified · internal SSD (#4, #5) | ~4 GB/s cold (the 14.2 GB/s reading was cache-influenced — see note) | default, MTP off | 1.06 tok/s · expert hit 23% · RSS 21.8 GB |
| Apple M5 Max · macOS · 128 GB unified · 2 TB SSD · Metal backend (#72, #87) | (macOS O_DIRECT figure unreliable — see note) | Metal on · --ram 96 · 39.7 GB warm pin · MTP off |
1.83 tok/s · expert hit 66% · warmed 1.11 → 1.83 over the run |
〃 · 46.9 GB pin (2.94M-selection history) · --ram 110, 1024-token run (#103) |
〃 | Metal on (experts + attention) · MTP off | 2.06 tok/s · hit 72.5% · coherent output |
| Apple M1 Ultra (20C, 48-core GPU) · Mac Studio · macOS · 128 GB unified · internal SSD · Metal backend · fmt=2 per-row container (report) | 6.89 GB/s F_NOCACHE · 8.93 GB/s buffered | Metal on (fmt=2) · --ram 125 · --cap 33 · 46.9 GB frozen pin · NO_OMP+PIPE · MTP off · 1024-token run |
1.50 tok/s · hit 78.7% · RSS 104.8 GB · disk wait 57% of decode, SSD at ~93% of its iobench ceiling (1.31 at default flags, --ram 110) |
| Mac Mini M4 Pro · macOS · 48 GB unified · Metal backend (#107) | 6.59 GB/s F_NOCACHE (fresh shard) | Metal on · --ram 38 |
0.30 tok/s (vs 0.18 CPU-only) |
| Apple M3 (base, 4P+4E) · macOS · 16 GB unified · internal Apple SSD (#949) | 3.18 GB/s cold F_NOCACHE (post-eviction) · 7.27 GB/s buffered — no O_DIRECT on macOS, caveat #86 | OLMoE int8 · cap 16 · TEMP=0 · CPU-only | 3.69 tok/s cold → 4.18 tok/s warm · RSS 1.5–1.81 GB · load 0.7 s |
| Epyc 9654 ES · Linux · 4x16GB DDR5-4800-rdimm · Samsung PCIe Gen3 x4 NVME SSD | — | MTP=1 DIRECT=1 |
0.31 tok/s · expert hit 35% · RSS 21.52 GB |
| Ryzen AI 9 HX 370 (Framework 13) · Arch Linux · 128 GB · WD SN850X, BTRFS zstd (#12) | — | int8 MTP head · --cap 32 · 46.7 GB auto-learned PIN |
0.37 tok/s · expert hit 66% · MTP acceptance 52% (2.59 tok/fw) · RSS 105 GB |
| Ryzen 9 9950X (32 threads) · Linux · 123 GB · Crucial P3 QLC Gen3 (#31) | 1.51 GB/s buffered | default, 2 runs from cold | 0.10 tok/s · hit 53% · profile 66% disk |
| 〃 same machine, model moved to a Samsung 9100 PRO PCIe 5.0 (#31) | 8.81 GB/s O_DIRECT | 〃 (usage history retained) | 0.28 tok/s · hit 57% · profile flips: 32% disk / 57% matmul |
| Ryzen AI Max+ 395 (Framework Desktop) · Ubuntu · 128 GB LPDDR5x · Intel Optane 905p PCIe 3.0 (#39) | 3.27 GB/s buffered | int8 MTP head · fresh history (pure LRU, auto-raised cap 65) | 0.16 tok/s · hit 57% · profile 49% disk / 47% matmul |
| 〃 five runs later — learned pin 47.6 GB (#39) | 〃 | --temp 0.7 --topp 0.7 |
0.40 tok/s · hit 71% |
| Ryzen 7 9800X3D (16T) · WSL2 · 70 GB RAM · Samsung 9100 PRO PCIe 5.0 · RTX 5090 (#101) | 10.51 GB/s O_DIRECT | MTP off · learned pin 24 GB · hit 54% · OMP hot-team on | 0.41 tok/s · disk-bound (36.5 s disk vs 24.0 s matmul) · CUDA expert tier ≈ 0% (AVX-512 CPU matches the 5090) · --topp 0.7 → 0.52 tok/s |
| EPYC 7443 (24C/48T, Zen3 AVX2) · Linux · 430 GB RAM · NVMe RAID-Z1 via TrueNAS VM (#104) | ~1 GB/s (VM overhead) | 77.5 GB pin · cap auto-raised to 194/layer · MTP off | 1.00 tok/s · hit 98% · disk eliminated → RAM-bandwidth + matmul bound |
| Intel i5-12600K (10C/16T, AVX2) · native Windows 11, no WSL · 32 GB · MinGW GCC 16.1 (#113) | buffered (no O_DIRECT on MinGW) | int8 MTP head · cold, small-RAM (cap ~2/layer) | 0.08 tok/s · hit 3.7% · MTP 57% acceptance — first native-Windows datapoint |
| Ryzen 9 9950X3D2 (16C/32T, avx512-vnni) · native Linux · 121 GB · Samsung 9100 PRO PCIe Gen5 · RTX 5090 (28 GB expert tier, 1475 pinned) (#120) | 11.48 GB/s O_DIRECT | MTP=0 DIRECT=1 PIPE_WORKERS=16 PREFETCH=1 |
1.23 tok/s |
| Ryzen AI Max+ 395 (Strix Halo, 16C/32T Zen5, avx512-vnni) · Arch Linux · 128 GB unified LPDDR5x · SK hynix P41 PCIe 4.0 (#124) | — | DIRECT=1 PIPE=1 --topp 0.7 · auto-pin |
0.06 cold → 1.10 tok/s sustained · later 1.83 tok/s on current dev with DIRECT=1 PIPE=1 PILOT_REAL=1 PILOT_TWO=1 (#200) |
| Intel Core Ultra 9 185H (16C/22T, avx-vnni) · native Windows 11, no WSL · 32 GB · Crucial P3 QLC NTFS · RTX 5070 Ti (#128, #273) | — | int8 MTP head · warm cache · GPU-resident pipeline at decode | 0.03 cold → 0.5 warm CPU → 1.07 tok/s with the pipe2 decode gate (#274) |
| Dell Pro Max GB10 (DGX Spark: Grace, aarch64 i8mm/sve2) · Linux · 121 GB unified LPDDR5x · GB10 sm_121 (#136, #161) | 5.58 GB/s O_DIRECT | int8 MTP head · warm cache | 0.50 tok/s warm · 2.4 tok/s full-k8, 3.33 tok/s with CACHE_ROUTE (#199) |
| Intel i5-13600K (14C/20T, avx-vnni) · native Linux · 62 GB · Samsung 980 PRO PCIe 4.0 (NTFS/ntfs3) · RTX 5070 Ti (#605) | 5.90 GB/s O_DIRECT | MTP off · PIN=auto PIN_GB=20 DIRECT=1 PIPE=1 --ram 50 --cap 32 --topp 0.7 · 750 VRAM + 307 RAM pinned (5.8 GB) |
0.98 tok/s (peak 1.07) · hit 54.7% (pin 37.9 + lru 16.7) · 360 experts/token · RSS 42.0 GB — up from 0.56 tok/s / hit 45.4% untuned |
| 6 × RTX 5090 · dual Xeon Silver 4510 · 251 GB (author's rig, experiment log) | NVMe | CUDA_EXPERT_GB=auto PIN_GB=all full residency · COLI_CUDA_PIPE=2 TC_W4A16 · DRAFT=0 |
5.8–6.8 tok/s decode · TTFT ~13 s · hit 89–100% |
| Apple M5 Max (6P+12E) · macOS 26.5 · 128 GB unified · internal SSD · Metal (#387) | macOS figures cache-influenced (caveat #86) | MTP=0 CAP_RAISE=0 AUTOPIN=0 --ram 90 --cap 1 |
0.4 → 2.0 tok/s at identical output — two additive levers, no routing substitution, no expert pruning |
| Ryzen 9 5950X (16C/32T, Zen 3, AVX2 only) · Ubuntu 26.04 · 62.7 GB · Samsung 990 PRO Gen4 · RX 7900 XTX, ROCm 7.2.4 (#680) | 4.88 GB/s buffered · 5.87 GB/s O_DIRECT | CPU-tuned | 0.77 tok/s — the HIP expert tier underperformed the AVX2 CPU on this box |
| Ryzen AI 365 · native Windows · 32 GB DDR5-5600 · NVMe · Vulkan (#999) | 3.32 GB/s buffered · 1.53 GB/s O_DIRECT | COLI_VULKAN=1 COLI_VK_DENSE=1 COLI_VK_ATTN=1 · PIN_GB=0 |
0.19–0.24 tok/s |
| Apple M1 Max (8P+2E, 32-core GPU) · macOS 26.6 · 64 GB unified · internal 2 TB SSD · Metal (#1030) | 4.4 GB/s (engine probe) | Metal on, settings-tuned | 0.13 → 0.61 tok/s from settings alone · Metal 2.5× over CPU · a larger --ram was consistently slower |
| Ryzen 7 5800X (8C/16T, Zen 3) · Ubuntu 22.04 · 32 GB DDR4-3200 · Samsung 980 PRO (#1070) | 10.02 GB/s buffered · 2.50 GB/s O_DIRECT | DeepSeek V4 Flash · --ram 22 |
0.93 tok/s · hit 52.5% · TTFT 16.9 s |
| Ryzen 7 9850X3D (8C/16T) · native Windows 11 · 64 GB DDR5 · WD_Black SN8100 · RTX 5090 sm_120 (#1091) | up to 10.64 GB/s | COLI_CUDA=1 CUDA_DENSE=1 CUDA_EXPERT_GB=auto PIN_GB=20 DIRECT=1 |
0.89 tok/s median |
| Ryzen 9 5950X (16C/32T, Zen 3, AVX2 only) · Linux · 62.7 GB (#1119) | ~2.7 GB/s O_DIRECT | DeepSeek V4 Flash · CPU-only | 1.24 tok/s — no I/O flag and no +40% expert cache moved it, which points at the AVX2 kernels rather than storage |
| EPYC 7282 (16C, Zen 2 AVX2) · Ubuntu 24.04 · 128 GB DDR4-3200 8-channel · 2× Micron 3400 RAID 0 (#1154) | — | DeepSeek V4 Flash · COLI_TEMP=0, 128 tokens |
0.68 cold / 0.69 warm tok/s · RSS 123.8 GB · load 30 s |
| Ryzen 7 4800H · Linux · 64 GB · NVMe (#1173) | 1.65 GB/s buffered · 1.37 GB/s O_DIRECT | DeepSeek V4 Flash · rotating prompts, n=4 | 0.47 tok/s (ondemand, 2.9 GHz cap) → 0.60 tok/s (performance governor) · hit 81% · CPU 56.8 → 92.4 °C |
| i9-14900K (24C/32T) · native Windows 11 · 128 GB DDR5 · Samsung 990 Pro, SN850P partial mirror · RTX 3090 24 GB (#1183) | — | GLM-5.2 int4 · CUDA expert tier | 0.71 tok/s median — the 24 GB VRAM point on the curve |
| Apple M4 Max (16C, 40-core GPU) · macOS · 128 GB unified · XPG MARS 980 Gen5 over Thunderbolt 5 (#1210) | — (TB5 enclosure) | COLI_METAL=1 DIRECT=1 MTP=0 PIPE_WORKERS=8 · PIN_GB=90 |
0.88 tok/s at --ram 90 → 1.06–1.17 tok/s pinned · hit 68–73% · RSS 96.7 GB |
| Threadripper PRO 7965WX (24C/48T, Zen 4 avx512-vnni) · Linux · 123 GB · two NVMe on independent controllers (990 PRO 4 TB + 9100) (#1249) | 6.70 / 8.05 GB/s single-drive O_DIRECT | DIRECT=1, CPU-only (RTX 5090 present, not engaged) |
one drive 0.80 → both 1.10 tok/s, +37.5% · with DIRECT=0 the same split is worth only +16% · expert wait 98.9 → 55.5 s |
Intel i9-12900K (8P+8E, avx-vnni, no avx512) · Debian trixie · 64 GB DDR4-3200 · Samsung 990 Pro 2 TB ext4 · RTX 3090 (commit 72d3d37, 2026-07-18) |
6.44 GB/s O_DIRECT 8T · 5.78 GB/s buffered true-cold | int8 MTP head · cap 26/layer · default (MTP on) | 0.31 cold / 0.33 warm · hit 40-42% · MTP acceptance 43% (2.29 tok/fw) · RSS 47.9 GB · profile 48% disk / 34% matmul / 12% attn · ~9.9 GB read/token |
| 〃 | 〃 | MTP=0 |
0.34 tok/s · hit 54.9% · 600 experts/token vs 1,062 with MTP on |
| 〃 | 〃 | MTP=0 + CUDA expert tier (3090) |
0.35-0.36 · hit 52.9-55.1% · +13-16% over default, of which the GPU itself is +0-6% |
| 〃 second drive: 200 GB LV on a Crucial P310, 57 interleaved expert shards (139 GB, ~40%) symlinked — file-level split, not block striping | 990 Pro 6.44 + P310-LV 4.40 GB/s O_DIRECT, independent controllers | MTP=0, coli serve mux |
N=1 0.324 (+5.5%) · N=2 0.426 (+3.1%) — see the note below on why this is so much smaller than #1249 |
| 〃 | 〃 | MTP=0 REPIN=1, six consecutive N=2 rounds |
0.461 aggregate · 0.404 → 0.405 → 0.456 → 0.458 → 0.433 → 0.461: pins converge in ~3 rounds (~300 tokens) and hold |
| Apple M4 Pro (8P+4E, 24 GB unified) · macOS 15.6 · libomp, 8 physical-core threads · OLMoE int8, fully resident (v1.9.0) | model in page cache; no disk in the loop | 226-token teacher-forced eval (PPL=1), cap 32 vs 64, paired (alternating within one session), 8 pairs |
median 19.1 tok/s at cap 32 vs 16.3 at cap 64 · cap 32 wins 7 of 8 pairs · hit rate 74.1% vs 96.7% — see the note |
16 GB is below the floor for the large models (#923): an i5-12450H with 16 GB DDR5 and a DRAM-less QLC drive builds and passes the self-test on native Windows, but there is no decode number to report, because a cold token needs ~11 GB of expert reads and the RAM cap leaves almost nothing resident. On machines this size, run OLMoE or Qwen3.6 rather than GLM-5.2: the M3 row above is 16 GB unified and reaches 3.69–4.18 tok/s on OLMoE.
More expert cache is not always faster once the model is resident, but you have to measure it in pairs. On a fully-resident OLMoE the whole 7 GB container is in the page cache, so a miss is a memcpy and not a device read, and the extra residency a larger cap buys has to pay for itself against nothing. It does not: cap 32 beat cap 64 in 7 of 8 interleaved pairs, median 19.1 against 16.3 tok/s, at half the resident footprint (3.1 GB of expert slabs against 6.1 GB, on a 24 GB machine).
The pairing is not a formality, and the first version of this note was wrong without it. Unpaired sweeps on this host disagreed with each other about the sign: one five-run sweep put cap 64 ahead on median (14.96 vs 14.32), a later ten-run block put cap 32 ahead by 35% (18.21 vs 13.45), and cap 32 alone ranged from 13.97 to 20.62 tok/s across sessions. Absolute throughput drifts far more between sessions than the effect being measured, so only same-session alternation resolves it — the discipline the decode failure ledger already asks for after the WarpDecode episode. Quoting this machine's cap curve from a single ordered sweep produces whatever the machine felt like that hour.
Related to #1050, but not the
same shape: that report's miss_rate × cap bookkeeping term is lowest at cap 64
and so predicts cap 64 fastest, which is the opposite of the paired result here.
The O(cap) scan is presumably real on that host; on this one something scaling
with residency dominates it.
A file-level shard split is not a substitute for the mirror path. The +5.5% in the 12900K split row and the +37.5% in #1249 are both two NVMe drives on independent controllers, and the gap between them is the mechanism, not the hardware. Splitting whole shards across two mounts with symlinks only parallelises reads that are already concurrent, and decode on that box sat at queue depth ~1 with the drive 24-28% utilised -- there was nothing to overlap. Block-level striping and the engine's own weighted mirror parallelise within each 19 MB read, which is where the real factor lives. Recorded because the negative is the useful half: it prices the difference between splitting files and striping blocks at roughly 5% versus 37%.
Vulkan beat ROCm/HIP on RDNA4 (#523): on an RX 9070 XT, Vulkan measured 19–24% faster than the HIP path. The first version of that comparison was confounded and the reporter corrected it themselves, which is why the number is worth carrying: it is the corrected one.
With 24 GB of RAM the engine auto-caps the expert cache to 2 slots/layer, so
decode stays cold even on a fast disk — on small-RAM machines the RAM cap, not
the disk, is the binding constraint; --topp 0.7 alone bought a clean 1.6×
end-to-end speedup. The 9950X pair is the cleanest bottleneck experiment: same
machine, same history, only the disk swapped — ×5.8 disk bandwidth bought ×2.9
tokens, and the profile flipped from 66% disk to 57% matmul. But the
crossover depends on the CPU kernel: with OMP hot-team tuning on, an AVX-512 CPU
can match an RTX 5090 on expert matmul (#101),
so the GPU tier earns its VRAM only when the CPU is the weak link. The
M1 Ultra ↔ M5 Max Metal pair is the same lesson on Apple Silicon: near-equal
GPU core counts (48 vs 40) but −33% tok/s (1.50 vs 2.24), because 57% of the
M1 Ultra decode wall is serial SSD wait at ~93% of the drive's measured
ceiling — when experts stream from disk, the drive, not the GPU, sets the
rate. On multi-socket hosts, NUMA placement is a further lever:
interleaving the resident weights across nodes measured +13% (2-socket) and +40% (4-socket CPU-only)
(#82). On a 2-socket Xeon Silver
4510 host with 6× RTX 5090, selective COLI_NUMA=1 raised effective CPU-expert
bandwidth from 42.42 to 58.26/65.89 GB/s and greedy decode from 7.66 to
9.02/9.17 tok/s (64 tokens, TEMP=0 DRAFT=0, byte-identical output). Do not
blanket-interleave a GPU host: it also spreads DMA staging pages and has measured
up to a 10× regression; generated plans enable only the selective slab policy.
Three lessons from the 2026-08 datapoints. A second drive on an independent
controller is real bandwidth: the Threadripper pair measured +37.5% from
adding one, and the gain more than doubles under DIRECT=1, because buffered
reads spend the second controller's bandwidth refilling a page cache the engine
does not need (#1249). That is
the README's multi-SSD hypothesis measured rather than assumed. On AVX2-only
CPUs the kernels bind before the disk does: the Zen 3 DeepSeek V4 Flash run
did not move for any I/O flag or for 40% more expert cache
(#1119), which is the
opposite of the small-RAM machines above and says the AVX-512 kernels are where
that box's speed is. And settings are worth more than hardware on Apple
Silicon: the M5 Max went 0.4 → 2.0 and the M1 Max 0.13 → 0.61 at identical
output, with a larger --ram measuring consistently slower on the M1
(#387,
#1030).
Measured (#108): the int4
container scored 62.5% mean acc_norm on hellaswag/arc/mmlu (0-shot
log-likelihood, n=40) — but 0-shot MC scoring underserves a reasoning model, and
the OLMoE fp16-vs-int4 A/B under the same harness measured the pure quantization
cost at -8.2pp, concentrated on the hardest task (per-row int4 scales erode
the small logit margins hard questions depend on — grouped scales recover ~63%
of that loss, see #225). The
scale-granularity/rotation/lattice ablation lives in
tools/quant_ablation.py (#81).
cd c
pip install tokenizers datasets
./coli bench # hellaswag, arc_challenge, mmlu — 40 questions each
./coli bench hellaswag --limit 200 # one task, more questions
./coli bench mmlu arc_challenge --ram 100 # pick tasks, set a RAM budget