What a released KV-compression retrofit costs on agentic function calling.
A paired measurement of NVIDIA's DMS 8x retrofit of Qwen3-8B against its unconverted base on the Berkeley Function Calling Leaderboard, together with the serving harness the measurement runs on. No existing serving stack loads that checkpoint for benchmarking, which is why the harness is here.
Full writeup in paper.pdf. The compression method is
DMS, arXiv 2506.05345; the benchmark client is
BFCL.
| control | Qwen/Qwen3-8B |
| treatment | nvidia/Qwen3-8B-DMS-8x, retrofitted from that checkpoint (dms_cr = 8, dms_window_size = 512) |
| held fixed | server, chat template, tool schemas, greedy decoding, scorer, 4-bit NF4, thinking disabled |
| varied | the weights file |
| tasks | 1053 single-turn (live_multiple) and 600 multi-turn, each run by both arms and compared per task |
| category | n | base | DMS | gap | p |
|---|---|---|---|---|---|
live_multiple |
1053 | 77.02 | 76.54 | 0.5 | 0.59 |
multi_turn_base |
200 | 18.5 | 13.0 | 5.5 | 0.043 |
multi_turn_miss_func |
200 | 12.5 | 8.0 | 4.5 | 0.064 |
multi_turn_miss_param |
200 | 14.0 | 8.0 | 6.0 | 0.017 |
| multi-turn pooled, per scenario | 600 | 15.0 | 9.7 | 5.3 | 0.0011 |
- On single-turn function selection the arms are indistinguishable, with the paired difference between -0.85 and 1.80 points.
- On the multi-turn categories (600 paired tasks over 200 scenarios in three variants) DMS loses 5.3 points, scenario-level sign-flip p = 0.0011, interval 2.3 to 8.5 points.
- Effective compression on real agentic prompts is far below the nominal 8x: 2.18x, 2.97x and 3.91x at 2048, 4096 and 8192 tokens, about 3x at the multi-turn median and near 1.5x at the single-turn median.
- Failures shift toward wrong responses to executed calls (117 to 155 pooled) while wrong end states hold level (318 to 316), which the paper reports as a hypothesis rather than a finding.
Absolute scores sit below the leaderboard's because both arms run 4-bit with thinking disabled; the
same harness with thinking on scores 45.0 on multi_turn_base against a published 50.5. The
remaining figures are in analysis/figures, and the paper states what this
design does not settle, including whether the loss comes from eviction or from the distilled
weights that arrive with it.
The harness serves the checkpoint locally behind the endpoint BFCL's OSS handler targets, because
the DMS model class is loaded through trust_remote_code. It expects a CUDA GPU; the measurements
here were taken on a single 16 GB card, where the DMS arm runs about 4x the latency of base.
Set G4_ROOT to wherever you keep the project, then use serve_hf.py for the server and
run_arm4.sh through mt_job.sh for one arm per job. tick2.sh scores, analyze.py produces the
paired contingency and exact tests, and pertask.py builds the per-task CSV. occupancy_real.py
and dms_gen_check.py read retained cache lengths out of the model's own cache object.
Analysis is separate and needs no GPU: analysis/scenario_test.py runs
the scenario-level permutation test and bootstrap, and
analysis/make_figures.py regenerates every figure from the per-task
CSV.