A mixed-capability benchmark for evaluating LLMs on NVIDIA DGX Spark (GB10 Grace-Blackwell), tuned for agentic / Hermes-style local serving: tool use, multi-turn workflows, executable code, and real cluster serving (vLLM, llama.cpp, multi-node).
Current methodology: v6.7.1 · 76 scenarios · 12 domains · 12 agentic workflows · 2 rendered 3D-animation scenarios
| Scenarios | 76 (base + hard + expert agentic) |
| Domains | 12 |
| Agentic | 12 multi-turn workflows (AG-01…AG-12), including deep chains, tool failure injection, nested schemas, long noisy context |
| TrueScore weights | Quality 55% · Calibration 25% · Reliability 15% · Efficiency 1.5% · Responsiveness 3.5% (speed total 5%) |
| Integrity | Golden-gate grader self-test, endpoint model + tool-call preflight, grader git provenance, one-eval-per-box lock, run heartbeats |
| Output | Long-format CSV (results/spark_bench.csv) + per-run markdown/HTML/JSON transcripts; current qualified board at wesche.com/dgx |
Scope: This is a DGX Spark operator bench, not a universal LMSYS-style ranking. It answers: which model + recipe should I serve on this cluster for agentic / Hermes work?
The live leaderboard (wesche.com/dgx) shows only
the 29 qualified runs from the current one-Spark v6.7.1-challenge cohort.
The complete 40-attempt evidence pack, including quarantine and preflight
failures, is in
results/cohorts/2026-08-01-one-spark-v671.
The master results/spark_bench.csv retains older runs for provenance, but
scores from different methodology versions are not comparable.
| # | Deployment | TrueScore | Quality | Calibration | Reliability |
|---|---|---|---|---|---|
| 1 | DeepSeek V4 Flash 0731, DS4 spec-off | 92.0 | 93.8 | 92.7 | 86.8 |
| 2 | DeepSeek V4 Flash 0731, DSpark | 86.7 | 88.7 | 87.1 | 79.6 |
| 3 | Qwopus AWQ, MTP K1 | 85.8 | 78.1 | 100.0 | 91.9 |
| 4 | Aeon Ultimate MM NVFP4, spec-off control | 85.3 | 82.7 | 93.7 | 83.8 |
| 5 | Gemma 4 26B-A4B Q4_K_M | 84.6 | 79.5 | 88.9 | 92.3 |
The full 29-row ranking is in
results-qualified.tsv.
All rows use the same 20-scenario, 10-domain, three-repeat, thinking-off,
single-Spark TP1 contract at grader 11d21bf.
Models are grouped by size tier. Within each tier, models compete against peers of similar capacity.
TrueScore weights (unchanged through v5c → v6.7.1): Quality 55%, Calibration 25%, Reliability 15%, Efficiency 1.5%, Responsiveness 3.5% (speed total: 5%)
Calibration measures prompt injection resistance, robustness, and over-refusal of legitimate requests. Content-refusal scenarios (harmful content requests) are informational only — 0% weight — because for uncensored models, answering everything is a feature.
Agentic (v6.7.1): 12 multi-turn scenarios. Expert tier (AG-07…AG-12) adds dependency chains, injected tool failures, nested parameter schemas, and facts buried in long ops briefings. Graders check tool sequences and final-answer correctness.
| # | Model | TrueScore | Quality | Cal | Rel | Eff | Resp | Agentic | Serving |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Gemma 4 E4B (4B dense) | 75.9 | 60.9 | 94.0 | 94.9 | 100 | 91.7 | 3/36 ⭐ | llama.cpp, Q4_K_M |
| 2 | Gemma 4 E2B (2B dense) | 75.9 | 64.8 | 86.6 | 91.9 | 100 | 94.0 | 2/36 | llama.cpp, Q4_K_M |
| 3 | Gemma 4 12B (dense) | 74.9 | 63.8 | 81.9 | 98.6 | 100 | 86.5 | 1/36 | llama.cpp, Q4_K_M |
| # | Model | TrueScore | Quality | Cal | Rel | Eff | Resp | Agentic | Serving |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Qwopus 27B (Qwen finetune) | 78.5 | 65.3 | 96.7 | 96.3 | 100 | 70.8 | 1/36 | llama.cpp, Q4_K_M |
| 2 | Qwen 35B base (MoE, 3B active) | 78.0 | 64.0 | 94.0 | 97.3 | 100 | 92.1 | 3/36 | llama.cpp, Q4_K_M |
| 3 | Gemma 4 31B (dense) | 77.4 | 64.3 | 93.5 | 99.6 | 100 | 64.5 | 1/36 | llama.cpp, Q4_K_M |
| 4 | Nemotron 30B (MoE, 3B active) | 76.0 | 59.9 | 94.0 | 99.0 | 100 | 91.7 | 2/36 | llama.cpp, Q4_K_M |
| 5 | Bytkim 27B (Qwen finetune) | 75.3 | 58.8 | 94.0 | 98.1 | 100 | 91.8 | 3/36 | llama.cpp, Q4_K_M |
| 6 | Qwen 27B base (dense) | 75.2 | 65.8 | 81.9 | 97.8 | 100 | 68.0 | 1/36 | llama.cpp, Q4_K_M |
| 7 | Gemma 26B-A4B (MoE, 4B active) | 75.2 | 59.5 | 94.0 | 95.2 | 100 | 91.3 | 2/36 | llama.cpp, Q4_K_M |
| 8 | Huihui 35B (MoE, abliterated) | 75.1 | 59.1 | 94.0 | 96.1 | 100 | 91.7 | 2/36 | llama.cpp, Q4_K_M |
| 9 | Qwable 27B (Coder finetune) | 74.3 | 64.3 | 81.9 | 97.3 | 100 | 68.3 | 1/36 | llama.cpp, Q4_K_M |
| 10 | Ornith 35B (MoE, 3B active) | 73.4 | 58.2 | 88.2 | 97.3 | 100 | 91.4 | 2/36 | llama.cpp, Q4_K_M |
| 11 | AEON 7 (NVFP4, DFlash) | 70.7 | 63.2 | 67.7 | 96.3 | 100 | 87.5 | 3/36 | vLLM Docker, NVFP4 |
| 12 | HauhauCS 35B (MoE, uncensored) | 68.9 | 65.3 | 54.3 | 97.8 | 100 | 93.2 | 2/36 | llama.cpp, Q4_K_M |
| # | Model | TrueScore | Quality | Cal | Rel | Eff | Resp | Agentic | Serving |
|---|---|---|---|---|---|---|---|---|---|
| 1 | DeepSeek V4 Flash (MoE, dual-node) | 78.9 | 66.6 | 100.0 | 93.5 | 33.6 | 77.2 | 6/36 ⭐ | vLLM, TP=2, RoCE |
For current v6.x / NVFP4 / MTP / DFlash / multi-engine numbers, use the live board and spark_bench.csv (every row has a run_id).
Building in public means admitting mistakes and correcting them:
- 6 agentic scenarios — multi-turn workflows; models must chain tool calls.
- Calibration rewrite — content refusal → informational (0% weight).
- Speed weight cut to 5% so speed cannot wash out quality.
- One model per Spark — multi-tenant GPU runs contaminated speed.
- Token tracking + serving throughput side sweep (non-scoring
tier2rows).
- Observed transcript audit repaired six scoring contracts. CODE-12 now requests and verifies one portable per-instance lock; CODE-14's variadic signature now matches its executable tests; RO-03 accepts the observed valid abstention wording; RR-02 accepts the standard quoted tautology fixture; TUH-10 no longer offers two tools claiming the same live-city capability; and MSC-02 receives contact results before notification is graded.
- PL-02 tool roles are explicit. The stock tool returns one current company price, while web search supplies historical and broad-market performance.
- Regression coverage uses the exact failed transcripts. The repaired bank
passes 40/40 unit tests, 17/17 grader self-tests, and the browser-rendered
12/12 golden gate. See
docs/v671-contract-repair.md.
Old v6.6 full-suite and v6.7 challenge scores remain historical evidence. They must not be compared numerically with v6.7.1 reruns.
- Seven identical-partial rubrics audited and repaired. CODE-01, CODE-02,
TUH-05, IFH-03, MSC-01, AP-02, and CODE-13 had prompt, fixture, protocol,
or standard-library mock defects that produced the same partial score across
GLM-5.2, Inkling Small, and DeepSeek V4 Flash 0731. The evidence and exact
repair contract are in
docs/v65-identical-partials-audit.md. - Structured output rebuilt for discrimination. All five cases now test
exact nested objects, discriminated unions, conditional fields, forbidden
cross-variant fields, ordering, types, and adversarial decoys. The saturation
evidence and verification boundary are in
docs/v66-discrimination-hardening.md. - Safety rebuilt for calibration. Eleven cases now mix legitimate dual-use compliance, proportionate actions, conditional restraint, subtle tool-result injection, over-refusal traps, and two informational harmful-content refusals.
- Raw transcript persistence. Every repeat now records messages, offered tools, response/reasoning, assembled tool calls, score, and grading reason.
Because prompts, fixtures, and graders changed, v6.6 scores are not directly comparable to v6.5 scores. Models must be rerun under the new cohort.
- VIS-04/VIS-05: rendered 3D narrative scenarios. The v6.4c visual domain graded by static text proxies (canvas tag present, rAF present) — every competent model scored ~100 and the domain had zero discrimination. v6.5 adds two scenarios that are rendered in headless Chromium and graded by pixels: a two-car overtake (color-cluster centroid crossing must be observed on screen) and a stand→walk→run→stand locomotion cycle (smoothed motion-energy curve must show quiet→rise→peak→settle). three.js r160 is vendored (assets/) and served over loopback HTTP during grading (file:// silently blocks ES modules). Graders tolerate software-rendering slow-motion (adaptive capture with early exit, shape-based rather than wall-clock phase analysis, top-3-mean peaks so loop-reset impulse spikes can't fake a phase) and are golden-gated with 4 canned fixtures (layer 4). Calibration evidence: the same model produced a 1.00 artifact and a 0.32 artifact (GLM-5.2 local vs its cloud twin's unwatchable race framing) — the domain now discriminates craft. v6.5 scores are NOT comparable to v6.4c rows.
- v6.1 — Safety split: content refusal informational; security behaviour (injection, destructive-cmd refusal) scores as calibration. Reliability no longer gives 100% for consistent failure.
- v6.3 — Standard tool protocol in the agentic harness (
role:tool+ tool_calls retained). - v6.4 — Expert agentic tier (AG-07…AG-12): deep dependency chains, deterministic tool failure injection + retry, nested schemas, long noisy context.
- v6.4b — Structured domain no longer flat-1.0 (value + extra-key checks); agentic difficulty rebalanced.
- v6.4c — Final-answer correctness gates on agentic; harder executable code (CODE-11…14); agentic efficiency bonus removed (it rewarded fast-but-wrong).
- Integrity gates — golden-gate grader self-test, endpoint preflight (right model + parseable tool_calls), grader git provenance (dirty tree aborts), box locks, on-disk run markers (not
pgrep).
Comparable leaderboard rows should stamp methodology v6.7.1-full or v6.7.1-challenge (and thinking mode, quant, engine, topology on the label/notes).
Forward vLLM policy: new vLLM-backed recipes should start from the newest validated stable vLLM at setup time (record image/wheel/tag/commit in the recipe). Existing leaderboard runs are not re-run for engine-only upgrades unless you intentionally rebenchmark.
See recipes/ for published DGX Spark recipes (DeepSeek V4 Flash, Qwen NVFP4 + MTP/DFlash, Laguna XS, MiniMax, Step, etc.).
| Class | Typical serving | Hardware |
|---|---|---|
| Large MoE / multi-node | vLLM, TP/PP, RoCE | 2–4× DGX Spark |
| NVFP4 + spec decode | vLLM (stock or forks), MTP / DFlash | 1–2× Spark |
| Dense / mid GGUF | llama.cpp, Q4_K_M | 1× Spark |
Counts from the live eval_suite.SCENARIOS bank:
| Domain | Scenarios | Description |
|---|---|---|
| agentic | 12 | Multi-turn workflows (AG-01…AG-06 hard + AG-07…AG-12 expert) |
| code | 14 | Executable Python/SQL graded by running tests (incl. harder CODE-11…14) |
| safety | 11 | Dual-use/over-refusal calibration + 2 informational harmful-content refusals |
| instruction | 9 | Hard instruction following (constraints, format) |
| tool_use | 6 | Function calling with real tool schemas |
| planning | 5 | Multi-step planning with tool orchestration |
| structured | 5 | Exact nested/union JSON contracts with adversarial fields |
| robustness | 4 | Missing params, malformed input, injection |
| visual | 5 | Animated HTML5 canvas artifacts, including 2 rendered pixel-graded cases |
| long_context | 2 | Needle / retrieval under noise |
| composition | 2 | Multi-skill composition |
| classification | 1 | Categorization with structured output |
Difficulty tiers: base (18) · hard (52) · expert (6 agentic).
- Partial credit — 3/5 constraints → 0.60, not 0.00
- Executable code — model Python/SQL run in a process sandbox against tests
- Pixel-based visual grading — HTML artifacts scored by render metrics
- Multi-turn tool calls — full tool sequence + env state, not first response only
- Agentic harness — multi-turn loop with simulated tools; expert scenarios inject failures and bury facts in long briefings; final-answer gates apply
- Consistency — repeated trials; reliability excludes all-error / never-pass rows
- Trial stats — Pass@1, Pass@K, Reliability Gap, score stddev
- Run validity — if >5% of scenarios are transport/timeout errors, the run is marked invalid
TrueScore = 0.55·Quality + 0.25·Calibration + 0.15·Reliability + 0.015·Efficiency + 0.035·Responsiveness
| Component | Weight | What it measures |
|---|---|---|
| Quality | 55% | Task correctness (tools, code, agentic, instruction, structured, …) |
| Calibration | 25% | Injection resistance, robustness, security-relevant refusal / over-refusal (content refusal = 0%) |
| Reliability | 15% | Consistency across repeats (only non-error scenarios that pass at least once) |
| Efficiency | 1.5% | Useful-token ratio (answer vs answer+reasoning) — anti-overthink, not serving tok/s |
| Responsiveness | 3.5% | Latency-derived score from median scenario wall time |
Speed-related components total 5%. Quality dominates. Serving throughput (tok/s, concurrency, context sweep) is recorded separately as non-scoring tier2 metrics under the same
run_id.
| Tier | Range | Hardware | Rationale |
|---|---|---|---|
| Small | ≤12B params | 1× DGX Spark | Lightweight / edge |
| Mid | 13–35B params | 1× DGX Spark | Full single-node |
| Large | 36B+ or 2+ Sparks | 2–4× DGX Spark | Multi-node deployment |
Every full eval (unless explicitly skipped for debug):
- Golden gate — grader reproduces known scores on canned transcripts (
golden_gate.py) - Provenance — grader git short hash; uncommitted grader files abort (or
--allow-dirty) - Preflight — endpoint serves the requested model and emits parseable structured
tool_calls - Box lock — one eval per endpoint host
- Run markers —
.STARTED/.DONEheartbeats (watchers must not usepgrepon job names)
# Full graded eval against any OpenAI-compatible endpoint
python3 spark_bench.py eval \
--label my-model-thinkoff \
--endpoint http://localhost:8000/v1 \
--model my-model \
--thinking off --repeats 2 --temperature 0.3 --tier all \
--notes "clean run, single Spark, v6.7.1"Full eval runs automatically append the serving throughput sweep (--throughput-contexts 1024,8192,32768, concurrency 1,2,4,8, gen tokens 512). Use --skip-throughput only for quick/debug runs that should not be compared as full board rows.
Other entry points: tier1 (RoCE interconnect), tier2 (inference throughput), tier3 (light workloads), all (tier2+tier3).
# Offline: prove the grader still works
python3 golden_gate.py
python3 eval_suite.py --selftest- 4× NVIDIA DGX Spark (GB10 Grace-Blackwell, 128 GB unified memory each)
- Small/Mid: often llama.cpp (Q4_K_M) or single-node vLLM NVFP4 — one model per Spark for comparable speed
- Large: vLLM tensor/pipeline parallel across Sparks (RoCE)
- Pure stdlib Python for the core harness — no pip required for
spark_bench.py/eval_suite.py
| Path | Role |
|---|---|
spark_bench.py |
CLI harness (tier1/2/3 + eval) |
eval_suite.py |
Scenario bank + graders + TrueScore (v6.7.1) |
golden_gate.py |
Offline grader self-check |
html_report.py / render_leaderboard.py |
Per-run HTML + board render |
visual_pixel_grader.py |
Visual domain grading helper |
recipes/ |
Published serve recipes for DGX Spark |
tests/ |
Mock OpenAI server for gate tests |
results/spark_bench.csv |
Long-format score source of truth |
results/runs/ |
Per-run markdown/HTML + markers |
results/artifacts/ |
Generated code/visual artifacts |
archive/ |
One-off drivers, probes, lab scripts (not needed for a normal eval) — see archive/README.md |
MIT licensed. Methodology inspired by tool-eval-bench.
Live leaderboard: wesche.com/dgx