An evaluation methodology for context-augmentation experiments, demonstrated through a pre-registered SVG reference-resolution study.
When added context improves a model, was it the information or the format? A two-arm comparison cannot tell. A third arm with the same format and the information destroyed can.
That idea is not new — context-aware MT, RAG and representation learning all permute content while holding format fixed. What is here is a specification of it (invariants, conformance vectors, named failure modes), the decomposition reported as the result rather than as a sanity check, and a pre-registered worked example. See FA-014 for the overclaim this replaced and who caught it.
| I want to… | Do this |
|---|---|
| Try the method — 2 minutes, no install | python examples/rag_style_control.py — runs on bare Python, no venv, no model, no network. Shows the three arms and the validation checks in a domain that is not SVG |
| Use it in my own evaluation | Copy src/fmtcontrol/ — two files, ~200 lines, standard library only. → fmtcontrol/README |
| Reimplement it in another language | SPEC.md — 9 invariants, the algorithm, 10 conformance vectors. Written to be sufficient without reading the Python |
| Decide whether to believe it | TRUST.md — 14 objections, what answers each, what is still open. Start here if you are evaluating the work |
| Reproduce the numbers | pip install -e ".[dev]" && python -m svgbench.cli report — every published figure in ~2s, no model, no renderer |
| Cite it | 10.5281/zenodo.21682241 · CITATION.cff |
| Report an implementation or replication | CONTRIBUTING.md — the two contributions worth the most, and both currently at zero |
Stability. Within a major version of fmtcontrol, the published conformance vectors
never change. Any change to the permutation, digest, or RNG is a major bump, because it
means results from the previous version no longer reproduce.
→ SPEC.md §9
Before you invest time: the central claim is not yet exercised. Across three pre-registered studies and four models, no admissible arm has produced an effect for the control to decompose. The instrument is validated; the hypothesis it was built to test is still open.
TRUST.mdsays exactly what is settled and what is not.
- Question. Does context augmentation help because of the information it supplies, or merely the format it arrives in?
- Method. A third arm, format-identical to the augmented one, with the values
permuted between elements. Same shape, wrong contents. →
METHOD.md - Result. A constrained null: context changed what the model said (56/180 responses) but not which element it identified (0.0444 in all three arms).
qwen2.5-coder:3b · 180 cases per arm · 30 clusters · cluster bootstrap over SVGs
| arm | identification accuracy | NO_EDIT |
malformed | abstained |
|---|---|---|---|---|
baseline — no context |
0.0444 [0.0167, 0.0778] | 0.444 | 0 | 0 |
permuted — same format, values shuffled |
0.0444 [0.0167, 0.0778] | 0.483 | 0 | 0 |
enhanced — geometry supplied |
0.0444 [0.0167, 0.0778] | 0.478 | 0 | 0 |
Random-selection reference 0.1852 · every pairwise difference +0.0000 · minimum detectable effect 0.0289
Read the MDE, not the p-values. With a difference of exactly zero no permutation can be more extreme, so p = 1.000 by construction and carries no information. The claim is that no effect larger than ~3 points occurred — not that the effect is zero.
No. That was the obvious objection to the result above, so it was pre-registered and measured as a separate study. Name the target element by id, change nothing else, and the same model on the same 180 cases scores:
| condition | target specified by | identification accuracy | NO_EDIT |
|---|---|---|---|
enhanced — geometry supplied |
description | 0.0444 | 0.478 |
named_id |
element id | 0.9278 [0.8833, 0.9667] | 0.067 |
+0.8833, paired cluster permutation over 30 SVGs, p at the test's floor.
So execution capability is not the limiting factor here. On this corpus, prompt,
scorer and model, explicit ids produce high edit accuracy while descriptive references do
not — even with exact geometry supplied.
→ docs/06-study-v2-results.md
Study V3 ran all four conditions on four models — two families, three scales — over the same frozen corpus.
| model | enhanced − permuted |
MDE | named-id |
|---|---|---|---|
qwen2.5-coder:1.5b |
+0.0111 | 0.0211 | 0.6500 |
qwen2.5-coder:3b |
+0.0000 | 0.0289 | 0.9278 |
llama3.2:3b |
−0.0111 | 0.0297 | 0.7389 |
qwen2.5-coder:7b |
+0.0944 | 0.0582 | 0.6111 |
The dissociation replicates on all four. Naming the target moves accuracy from ~0.05 to 0.61–0.93 in every model tested.
The control still has not fired — on admissible evidence. The 7B is the one model whose information effect exceeds its MDE, and it is excluded by a pre-registered data-quality falsifier that fired on its malformed rate. It stays excluded, because the rule was written before the result was seen. The observation is reported in full and motivates a pre-registered follow-up; it is simply not confirmatory evidence for the primary hypothesis.
Characterising that excluded class is what made the study interesting: 237 of 237 of
the 7B's MALFORMED responses are prose refusals — the model explaining it cannot
identify the target — with zero truncations or parse failures. The scorer was not broken;
it did exactly what it was frozen to do. Its abstention patterns simply encoded an
assumption about how models decline, calibrated on the one model available at freeze
time. FA-013 — instrument drift.
A scoring rule is an empirical hypothesis about the space of permissible model behaviour. Freezing it makes that hypothesis auditable rather than timeless.
pip install -e ".[dev]"
python -m svgbench.cli reportRegenerates every published number from committed evaluation rows in ~2 seconds. No model, no renderer, no network.
You ask for "make the top-left shape blue". The model sees:
<path id="e13415408" d="{{GEOM_1b7549de}}" fill="#8c5a3c"/>
<path id="e15485c60" d="{{GEOM_33cabe17}}" fill="#c0c0c0"/>
<path id="e0d63fea4" d="{{GEOM_c2532d12}}" fill="#8c5a3c"/>
<path id="e30176ca8" d="{{GEOM_a9f3024e}}" fill="#8c5a3c"/>Three share a fill. Nothing says which is top-left. The instruction refers to the rendered picture; the edit happens in the source text; the source text does not encode what the instruction is about.
Three hypotheses, all measured:
| H1 the model ignored the context | rejected | 56/180 responses differ |
| H2 the context never reached the model | rejected | 180/180 prompts differ |
| H3 context altered generation without improving reference resolution | supported | the only hypothesis consistent with both |
permuted and enhanced identified exactly the same 8 cases. Shuffling the geometry
between elements changed nothing about which element was acted on.
On the central claim: the treatment effect was zero, so there is no quantity to decompose. C3 is not supported because its prerequisite did not occur — a scientific dependency, not a methodological failure.
| Tier | Verifies | Needs | Time |
|---|---|---|---|
| 1 | every published number | Python | ~2 s |
| 2 | the whole scoring chain, from raw responses | Python | ~10 s |
| 3 | the corpus is a deterministic function of its seed | + renderer | ~2 min |
| 4 | the model results | + local model | ~1 h |
python -m svgbench.cli report # tier 1
python -m svgbench.cli evaluate # tier 2
python -m svgbench.cli verify --determinism # tier 3
python -m svgbench.cli run main-baseline # tier 4540 raw model responses are committed, so a sceptical reader can write their own scorer and check ours against it. Tier 4 will not reproduce bit-for-bit — local backends vary with build and threading. What reproduces is the conclusion, within the reported interval.
New here? → START_HERE.md picks the right document for what you
want.
TRUST.md |
start here — the one-page thesis, and every objection with what answers it |
METHOD.md |
the transferable part — format-matched controls, domain-independent |
src/fmtcontrol/ |
the extracted package + SPEC.md and conformance vectors |
docs/essay.md |
How we almost measured the wrong thing — the essay |
docs/04-results.md |
full write-up |
docs/05-study-v2-preregistration.md |
Study V2, registered before it was run |
docs/06-study-v2-results.md |
Study V2 result — the model can edit; it cannot identify |
docs/07-study-v3-preregistration.md |
Study V3, four models, registered before it was run |
docs/08-study-v3-results.md |
Study V3 result — and the falsifier that excluded the best data point |
VALIDITY.md |
internal / construct / external / statistical validity |
LIMITATIONS.md |
17 things this does not show |
CLAIMS.md |
every module maps to one claim, and what falsifies it |
FAILED_ASSUMPTIONS.md |
fourteen times this project proved itself wrong |
RESULTS.md |
what may not change once results exist |
docs/adr/ |
eleven decision records |
The corpus, scoring rules, predicates and analysis plan were frozen at the
instrument-freeze-v1
tag, before any model output was observed.
That claim arrives in three layers, and they are not equally strong:
| Evidence | Trust required |
|---|---|
| The frozen tree contains no committed model outputs | None. Check it yourself, below |
The tag message asserts NO MODEL OUTPUTS HAVE BEEN OBSERVED. |
The author's word |
| Commit timestamps | None worth relying on — git commit --date forges them freely |
git ls-tree -r --name-only instrument-freeze-v1 | grep -c jsonl # 0experiments/ and results/ exist at the tag as empty placeholders. No response file, no
evaluation row, no metric is part of the frozen artifact.
What that establishes, and what it does not. It is independently verifiable that no model responses formed part of the frozen instrument. It cannot establish that none were observed locally beforehand — a local run leaves no trace in git — and no mechanism available after the fact can upgrade that. It remains an author assertion, recorded in the tag message. Stated here rather than left for a sceptical reader to work out.
The tag also records the dataset hash, the config hash and the commit:
dataset a2938bb031c0220abb45df12b7bc3eaa19a33484ac15592e59c62247010d2b35
data/frozen/<hash>/ carries a certificate listing six checks run against the bytes on
disk. Integrity is verified by tampering — editing a byte, deleting a file, renaming the
directory — not by re-implementing the hash.
One small model. One synthetic corpus. Opaque geometry tokens that do not occur in real SVGs. Four edit operations chosen because each is checkable by structural diff.
The method in METHOD.md is domain-independent. This measurement is not.
Archived on Zenodo: 10.5281/zenodo.21682240 — an archived research artifact, not a peer-reviewed publication. The DOI fixes the citation and guarantees the artifact at that version cannot change or disappear; it is not evidence about when the experiment was run (see Provenance above). That DOI is the concept record, which always resolves to the latest archived version; to cite the exact snapshot behind the measured result, use the version DOI: 10.5281/zenodo.21682241 (v1.0.1).
@software{svg_ambiguity_bench,
author = {R G, Nitish},
title = {svg-ambiguity-bench: an evaluation methodology for
context-augmentation experiments},
year = {2026},
url = {https://github.com/NITISH-R-G/svg-ambiguity-bench},
doi = {10.5281/zenodo.21682241},
version = {1.0.1}
}MIT licensed.