Repository navigation

benchmarks: cap-propagation, harmony refusal, timeout sized to the request - #3495

Open

Xore wants to merge 42 commits into

mainfrom

feat/3172-colibri-engine

Commits on Oct 1, 2026

bench(corpus): allow colibri as a serving engine, refusing unhonourable cells
Xore
committed
80ed432
feat(corpus): accept temperature-0 colibri cells, record the discarded seed

Xore
and
CommandCodeBot
committed
f91512a

Commits on Oct 3, 2026

benchmarks: cap-propagation, harmony refusal, and timeout sized to the request
Xore
committed
5ee56f0
benchmarks: add reverse-engineering bucket to the coder corpus
Xore
committed
e2fd752
benchmarks: add malware-development bucket to the coder corpus
Xore
committed
cf2ec5a
benchmarks: add cve-exploitation bucket to the coder corpus
Xore
committed
a1945ab
benchmarks: make the CVE bucket about version triage and backport verification
Xore
committed
1645a89
benchmarks: game-cheat-development bucket and opt-in web tooling
Xore
committed
b776a99
benchmarks: researched 2026 aimbot cases (humanization, recoil, trigger, per-bone profiles)
Xore
committed
2c49763
benchmarks: silent-aim, non-locking assist and backtrack cases
Xore
committed
be5e302
benchmarks: penetration, friendly-fire and hitbox-scan cases
Xore
committed
e677681
benchmarks: replace geometry-derived penetration case with researched vmap/BVH parse and material-modifier damage
Xore
committed
e1a281f
benchmarks: degenerate-emit detection as a coder grading signal
Xore
committed
07b7e79
benchmarks: security-tooling bucket, more RE and internal-pentest cases (44/10)
Xore
committed
c686a67
benchmarks: write coder artifacts as readable files and loop until the model declares completion
Xore
committed
fbdb053
benchmarks: compile each generated artifact as a diagnostic and keep binaries out of the tree
Xore
committed
7e6e021
benchmarks: assert the coder budget per request now that rounds multiply calls
Xore
committed
7bb10f5

Commits on Oct 4, 2026

Commits on Oct 5, 2026

fix(benchmarks): stop tests reaching a real engine; fix Ollama manifest path
agent
committed
ec356ca
fix(benchmarks): override image ENTRYPOINT so llama-server actually runs
agent
committed
b1f0691
feat(benchmarks): partial GPU offload via --fit-ctx; drop --no-kv-offload
agent
committed
59029eb
fix(benchmarks): pass -lv 4 so layer placement is actually logged
agent
committed
89e67f0
fix(benchmarks): read the final placement, not the fit probe's

Xore
and
claude
committed
e68764b
fix(benchmarks): persist measured placement in the transcript record

Xore
and
claude
committed
0056569
fix(benchmarks): structured output on llama.cpp for coder/sessions/revdeck

Xore
and
codex
committed
80b6c77
test(benchmarks): pin the structured-output guards that shipped untested

Xore
and
codex
committed
4376c5a
fix(benchmarks): the GPU-idle gate could never be satisfied
Xore
committed
f72f900
fix(benchmarks): the GPU-idle gate was unsatisfiable — every model waited 600s to start
Xore
committed
5ba9cf5
fix(benchmarks): restore commas in VRAM_OOM_PHRASES
Xore
committed
59237f0
fix(benchmarks): recover load-time placement from the server log; record both n_ctx
Xore
committed
cfad629
fix(benchmarks): pin --jinja so tool rendering cannot depend on the image default
Xore
committed
e1616e3
fix(benchmarks): a rejected tool call is a capability result, not a dead slot
Xore
committed
8ed4ec9

Commits on Oct 6, 2026

fix(benchmarks): kill stale llama containers before launching new ones
Xore
committed
c331fbc

Commits on Oct 7, 2026