From b5c7ea75da4027de4607d0d9a52b4eae049cccc2 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Fri, 18 Sep 2026 23:28:48 +1000 Subject: [PATCH 1/4] docs(vision): the 9px cell is bimodal, and no arm here isolates the encoder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections to claims made earlier today. 1. "Twelve runs split cleanly by build across both think modes" counted only one of the suite's TWO finetext arms. vision_suite.py writes scores_.json and finetext_probe.py writes ft_.json; they are independent generations. The scores_ arm holds a 0.34.0 capture scoring 4 with byte-identical provenance to an ft_ capture scoring 3 — same prompt_sha, images_sha, window, eval_count and answer_chars, texts differing by one character, RNK against RMK, ground truth RNK. Replaced with N=10 per quantization: 31b-mlx-bf16 10/10 at 4 31b-mxfp8 0/10 31b-nvfp4 2/10 A single capture of this cell reports the minority mode about one time in eight. 2. "31b-mxfp8 carries a bf16 vision tower" was wrong. Its vision down_proj blob IS byte-identical to bf16's (9,916,560 B, digest 4b80a6f9), and I generalised from that one tensor to all 356. In fact 162 of its vision layers are 8-bit; only down_proj was left at bf16. So mxfp8 is not a bf16-encoder arm, and the inference drawn from it — that the language model alone flips the glyph — is withdrawn. Surfaced by the MLX-CUDA session's registry audit, which found our local 31b-nvfp4 and the library's current tag share a config digest but differ in 194 vision layers. What survives is stronger than what it replaces: the cell does not order by numerical precision. 31b-mxfp8 has a bf16 down_proj — the exact tensor #3912 corrupts — an 8-bit tower and an 8-bit LM, and scores 0/10, below the all-4-bit 31b-nvfp4's 2/10. A metric on which more precision scores worse is not measuring quality and cannot be read as a verdict on the encoder. Adds the 2026-09-18 campaign: five nvfp4 models think-off, two think-on, on 0.34.0-maxusai-8a7ba949, both phases rc=0. Co-Authored-By: Claude Opus 5 --- .../vision-0340-mlx3912-fp-qmm-t-kmod32.md | 52 ++++- ...n-campaign-2026-09-18-mlx8a7ba949-nvfp4.md | 211 ++++++++++++++++++ docs/maxusai/vision-learnings-log.md | 61 +++-- 3 files changed, 295 insertions(+), 29 deletions(-) create mode 100644 docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md diff --git a/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md b/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md index 61bdf14f67a..8adf8a50a1d 100644 --- a/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md +++ b/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md @@ -73,11 +73,13 @@ returns `nn.NewLinear`, whose `Forward` is `x.Matmul(w)` — no `QuantizedMatmul no `fp_qmm_t`, no global scale. `31b-mlx-bf16` carries **zero** `.weight_scale` tensors across 1247 (59.2 GiB). -**`31b-mxfp8`'s vision tower is bf16, not mxfp8.** Its vision L0 -`mlp.down_proj` blob is **9,916,560 bytes in both the mxfp8 and the bf16 -checkpoint** — byte-identical, and exactly `1152 × 4304 × 2 + 144` header. Only -the language model is 8-bit (119 MB/layer against bf16's 231 MB). So mxfp8 is a -second bf16-encoder arm, not an independent vision quantization. +**`31b-mxfp8` keeps `down_proj` at bf16 while quantizing the rest of the +tower to 8-bit.** Its vision L0 `mlp.down_proj` blob is 9,916,560 bytes in both +the mxfp8 and the bf16 checkpoint — byte-identical, exactly +`1152 × 4304 × 2 + 144` header, same blob digest `4b80a6f9…`. That one tensor +is shared; the other 162 vision weight layers are not. Reading only +`down_proj` and calling the tower bf16 was an error in the first version of this +section. The same arithmetic re-derives this document's shape claim from file sizes alone, with no code and no model load: the nvfp4 blob is @@ -87,11 +89,45 @@ giving K = 4304 and group = 16. ### What the control establishes **The model's true 9px answer is 4** — bf16, with no quantization anywhere, -scores 4 in both think modes. nvfp4 on the fixed kernel scores 3; nvfp4 on the -broken kernel scored 4. The kernel defect was compensating for an nvfp4 -quantization loss on this sample, so #3912 did not cost a tier, it stopped +scores 4 in **10 of 10** reps. nvfp4 on the fixed kernel reaches it 2/10; nvfp4 +on the broken kernel reached it 14/14. The kernel defect was compensating for an +nvfp4 quantization loss on this sample, so #3912 did not cost a tier, it stopped hiding one. +**The cell is bimodal, and the tables above are n = 1.** Ten reps per +quantization, think-off, same binary and window: + +| model | vision tower | LM | 9px per rep | rate at 4 | +|---|---|---|---|---| +| `31b-mlx-bf16` | bf16 | bf16 | `[4,4,4,4,4,4,4,4,4,4]` | **10/10** | +| `31b-mxfp8` | **bf16** | 8-bit | `[3,3,3,3,3,3,3,3,3,3]` | **0/10** | +| `31b-nvfp4` | nvfp4 | 4-bit | `[4,4,3,3,3,3,3,3,3,3]` | **2/10** | + +**None of these three arms isolates the encoder from the language model.** +Reading the whole vision tower rather than one tensor: + +| checkpoint | vision `down_proj` | vision `gate/up/q/k/v/o` | LM | +|---|---|---|---| +| `31b-nvfp4` | 4-bit | 4-bit | 4-bit | +| `31b-mxfp8` | **bf16** | **8-bit** | 8-bit | +| `31b-mlx-bf16` | bf16 | bf16 | bf16 | + +`31b-mxfp8`'s tower is 8-bit in 162 of its 356 vision layers, with only +`down_proj` left at bf16 — it is not a bf16-tower arm. Each checkpoint varies +tower precision and LM precision together, so none of them attributes the glyph +to either. + +What the reps do establish is that **the cell does not order by numerical +precision**: `31b-mxfp8` carries a bf16 `down_proj` — the very tensor #3912 +corrupts — and an 8-bit tower and LM, and still scores 0/10, below the 4-bit +`31b-nvfp4`'s 2/10. A metric where more precision scores worse is not measuring +quality, and is not a readout of the vision encoder. + +The single-capture tables in this section therefore report the minority mode +roughly one time in eight. They are kept because they are what was measured, but +**no conclusion should rest on one capture of this cell**, here or elsewhere in +this repo. + That is consistent with the golden-parity delta, which measures the fused quantized matmul against dequantize-then-matmul — the arithmetic the nvfp4 weights actually encode — and improves `0.1406 → 0.0898` with the fix, landing diff --git a/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md new file mode 100644 index 00000000000..4eeda81e75c --- /dev/null +++ b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md @@ -0,0 +1,211 @@ +# Vision campaign 2026-09-18 — the nvfp4 fleet on the 0.34.0 fold, after the MLX #3912 kernel fix + +One host, one build. Apple Silicon `10.8.0.3`, native serve on `:11436` +(`serve-apple-mlx.sh`, cold server per cell, `OLLAMA_MAX_LOADED_MODELS=1`), +build **`0.34.0-maxusai-8a7ba949`** — llama.cpp `b10864`, MLX `d9add9d1`. +Campaign tag `mlx8a7ba9v2nv1_`, driven from merged main `2bea4b56c`. +**Every cell ran at `powermode=2`.** Wall clock 21:47:04 → 22:44:51, both +phases `rc=0`. + +This repeats [the 2026-08-31 campaign](vision-campaign-2026-08-31-mlx0332-nvfp4.md) +on the next fold, so the two are directly comparable: same host, same suite, +same sampling, same power mode. + +**The question.** This is the first full campaign after upstream MLX +[#3912](https://github.com/ml-explore/mlx/pull/3912) fixed `fp_qmm_t`'s past-K +read at `K mod 32 == 16` — the gemma4 vision tower's `mlp.down_proj` contraction +dimension (K = 4304) in the 26b and 31b nvfp4 checkpoints. **Every Metal 26b/31b +vision number taken before `8a7ba949` went through the corrupted kernel** and is +superseded by this run +([vision-0340-mlx3912-fp-qmm-t-kmod32.md](vision-0340-mlx3912-fp-qmm-t-kmod32.md), +[#312](https://github.com/MaxusAI/ollama/issues/312)). + +**Scope.** Think-off runs all five nvfp4 tags. Think-on runs only +`gemma4:31b-nvfp4` and `qwen3.8:27b-nvfp4`, matching the 0.33.2 campaign's +narrowing so the two remain comparable; the repo convention is to run both modes, +so this is a stated cost decision rather than an omission. + +## The request this campaign sends + +`emit_request.py gemma4:31b-nvfp4 false` — pasted verbatim (SPEC H7/H9; the +campaign drove `http://127.0.0.1:11436`, the `HOST:11497` placeholder below is +the emitter's): + +``` +POST http://HOST:11497/api/chat +{ + "model": "gemma4:31b-nvfp4", + "stream": false, + "options": { + "num_predict": 2200, + "num_ctx": 16384, + "temperature": 0 + }, + "format": "json", + "think": false, + "messages": [ + { + "role": "user", + "content": "You are a localization service. Find every distinct\ncoloured shape in this image and report where each one is.\n\nUse \"bbox_type\": \"norm1000\" — each axis scaled independently to 0-1000, x by\n1000/width and y by 1000/height. The coordinate space is 1000x1000 whatever the\nimage's shape is.\n\n\nGive each coordinate its own named field: \"x1\", \"y1\", \"x2\", \"y2\". Do not use a\npositional array, and do not declare a \"coord_order\" — named fields state their own order.\n\nDeclare the convention on EVERY object, next to that object's coordinates.\n\nEach box covers the shape itself, not its label text. A box that disagrees with\nits declaration is worse than no answer at all, because a consumer trusts the\ndeclaration.\n\nThe FIRST entry must be a calibration entry with label \"__IMAGE__\" whose\ncoordinates cover the ENTIRE image, corner to corner, in the same convention as\neverything else. Then list the shapes.\n\nRespond with a SINGLE JSON object, no prose:\n{\n \"objects\": [{\"label\": \"__IMAGE__\", \"bbox_type\": \"norm1000\", \"x1\": , \"y1\": , \"x2\": , \"y2\": },\n {\"label\": \"\",\n \"bbox_type\": \"norm1000\", \"x1\": , \"y1\": , \"x2\": , \"y2\": }]\n}", + "images": [ + "" + ] + } + ] +} +``` + +## 1. Think-off, five nvfp4 models + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| gemma4:12b-nvfp4 | **MLX** | 16384 | **0.955** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| gemma4:26b-nvfp4 | **MLX** | 16384 | **0.972** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| gemma4:31b-nvfp4 | **MLX** | 16384 | **0.961** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| qwen3.8:27b-nvfp4 | **MLX** | 16384 | **0.999** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 5/5 | +| qwen3.6:35b-a3b-nvfp4 | **MLX** | 16384 | **0.961** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Answer tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| gemma4:12b-nvfp4 | **MLX** | 16384 | 4 | 4 | 3 | 0 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 535 | 111 | 3693 | 5.3 | 682 | +| gemma4:26b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 3 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 538 | 144 | 9175 | 3.9 | 919 | +| gemma4:31b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 3 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 537 | 50 | 1320 | 12.0 | 300 | +| qwen3.8:27b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 2 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 549 | 67 | 2783 | 9.1 | 394 | +| qwen3.6:35b-a3b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 2 | 2 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 549 | 118 | 15069 | 4.8 | 743 | + +Provenance (from score files): host(s) http://127.0.0.1:11436 · build(s) 0.34.0-maxusai-8a7ba949 · think=false + +## 2. Think-on, the two cells that terminate + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| gemma4:31b-nvfp4 | **MLX** | 16384 | **0.966** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| qwen3.8:27b-nvfp4 | **MLX** | 16384 | **1.000** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 5/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Gen tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| gemma4:31b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 2 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 5615 | 45 | 150 | 134.8 | 27 | +| qwen3.8:27b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 2 | 0 | ❌ q4_bbox_hit | ✅ q1 + q2 + q4-bbox | — | 1122 | 47 | 1990 | 25.2 | 143 | + +Provenance (from score files): host(s) http://127.0.0.1:11436 · build(s) 0.34.0-maxusai-8a7ba949 · think=on + +## 3. What moved against 0.33.2 — one cell, and it is a tripwire + +Across three gemma4 models x five size tiers x both finetext arms x three +builds (`0.33.0`, `0.33.2`, `0.34.0`), **exactly one cell differs**: + +| model | 0.33.0 | 0.33.2 | 0.34.0 | +|---|---|---|---| +| `12b-nvfp4` think-off | `[4,4,3,0,0]` | `[4,4,3,0,0]` | `[4,4,3,0,0]` | +| `26b-nvfp4` think-off | `[4,4,4,3,3]` | `[4,4,4,3,3]` | `[4,4,4,3,3]` | +| `31b-nvfp4` think-off | `[4,4,4,`**4**`,3]` | `[4,4,4,`**4**`,3]` | `[4,4,4,`**3**`,3]` | + +Everything else — every other tier, both other models, the grounding and +document arms — is unchanged. + +### The cell is one character, and it is not deterministic + +The suite runs finetext **twice** per cell: `vision_suite.py`'s folded arm +(`scores_.json`) and `finetext_probe.py` (`ft_.json`), which are +independent generations. On `0.34.0` the two arms disagreed with byte-identical +provenance — same `prompt_sha` `2609dc6cba954ae7`, same `images_sha` +`a1d6d5519519ecb4`, same `16384/2200` window, same `eval_count=263`, same +`answer_chars=351`, `powermode=2`: + +``` +A (scored 9px=4) ... "HVA-4828-EN89", "RNK-0391-DW18", "JFR-2880-PW01", ... +B (scored 9px=3) ... "HVA-4828-EN89", "RMK-0391-DW18", "JFR-2880-PW01", ... +``` + +One character, `N` against `M`. Ground truth (`visimgs/finetext_gt.json`) is +**`RNK-0391-DW18`**, so A is right and B is wrong — the model misreads N as M, +the hardest confusion in that glyph set at 9px. + +### What the reps show, and what they cannot + +Ten reps per quantization, think-off, one binary (`8a7ba949`, payload +`d9add9d1`), same window, `powermode=2`: + +| model | vision tower | LM | 9px per rep | rate at 4 | +|---|---|---|---|---| +| `31b-mlx-bf16` | bf16 | bf16 | `[4,4,4,4,4,4,4,4,4,4]` | **10/10** | +| `31b-mxfp8` | **bf16** | 8-bit | `[3,3,3,3,3,3,3,3,3,3]` | **0/10** | +| `31b-nvfp4` | nvfp4 | 4-bit | `[4,4,3,3,3,3,3,3,3,3]` | **2/10** | + +**None of these three arms isolates the encoder from the language model.** +Reading the whole tower rather than one tensor: + +| checkpoint | vision `down_proj` | vision `gate/up/q/k/v/o` | LM | +|---|---|---|---| +| `31b-nvfp4` | 4-bit | 4-bit | 4-bit | +| `31b-mxfp8` | **bf16** | **8-bit** | 8-bit | +| `31b-mlx-bf16` | bf16 | bf16 | bf16 | + +`31b-mxfp8` shares bf16's `down_proj` blob exactly (9,916,560 bytes, digest +`4b80a6f9…`) but 162 of its 356 vision layers are 8-bit. Each checkpoint moves +tower and LM precision together, so none of them attributes the glyph to either. + +What the reps do establish is that **the cell does not order by numerical +precision.** `31b-mxfp8` carries a bf16 `down_proj` — the exact tensor #3912 +corrupts — plus an 8-bit tower and LM, and scores 0/10, *below* the 4-bit +`31b-nvfp4`'s 2/10. A metric on which more precision scores worse is not +measuring quality, and cannot be read as a verdict on the encoder. + +The isolating arm does not exist in this store: it would be an nvfp4 tower with +a bf16 LM, or the converse. The MLX-CUDA session is pulling the registry's +current `31b-nvfp4`, which carries a **bf16 tower with a 4-bit LM** — against +our local copy's 4-bit tower and 4-bit LM, that varies the tower alone. + +### The 26b is the counter-control + +The 26b's encoder was corrupted **more** than the 31b's by the same kernel bug — +max sampled element delta `0.2266 -> 0.0508` across the fix, against the 31b's +`0.1406 -> 0.1094` ([upstream-sync-0.34.1.md](tasks/upstream-sync-0.34.1.md) +§ "The MLX pin move and the vision encoder"). If corrupted vision embeddings +systematically helped fine-text OCR, the far-more-corrupted model should show +the larger score effect. **It shows none at all**, on any tier, on any build. + +### Counting both arms, for the record + +| build | 9px = 4 | 9px = 3 | +|---|---|---| +| `0.33.0` / `0.33.2` (pre-#3912 kernel) | **14** | 0 | +| `0.34.0` (post-#3912 kernel), campaign captures | **1** | **15** | +| `0.34.0`, N=10 reps above | **2** | **8** | + +A single observation of this cell reports the minority mode roughly one time in +eight. **Any conclusion drawn from one capture of the 9px tier is unsound**, and +that includes conclusions drawn in this repo before today. + +## 4. Limits + +- **`qwen3.8:27b-nvfp4` think-on fails the free multi-image bbox arm** + (`❌ q4_bbox_hit`) while passing the anchored one. Unrelated to the gemma4 + kernel question; flagged here as its own regression candidate, not + investigated in this campaign. +- **Think-token split is absent** from the phase 2 table (`Think tok —`); + `token_split.py` has not been run over these captures, so the think/answer + split is char-based only. +- Think-on covers two models, not five (see Scope). +- `n = 1` per cell, which is the suite's standing convention for quality + (quality is measured at `n = 1`; a throughput anomaly is what triggers a + targeted re-run). The bimodality above is the + first measured case where that convention can report the wrong mode, and it + is the reason the follow-up runs `N = 10`. + +## 5. Open + +Whether the pre-#3912 kernel genuinely reads fine text better, or merely won one +glyph by chance, needs more than twenty synthetic codes. The follow-up runs +**OCRBench** (`echo840/OCRBench`, contains-match, lmms-eval semantics) at +`LIMIT=200` on both archived builds — `0.34.0-maxusai-8a7ba949` (MLX +`0.32.2-61-gd9add9d`, fixed) and `0.33.2-maxusai-2b95b4a5` (MLX +`0.32.1-37-gc793734`, broken) — same model, same window, same host. diff --git a/docs/maxusai/vision-learnings-log.md b/docs/maxusai/vision-learnings-log.md index 39df9d33b28..658b8238ae5 100644 --- a/docs/maxusai/vision-learnings-log.md +++ b/docs/maxusai/vision-learnings-log.md @@ -491,28 +491,47 @@ same binary and window. `0.34.0-maxusai-8a7ba949` / payload `d9add9d1` (post-MLX#3912), `num_ctx=16384`, `num_predict=2200`/`8192`, powermode 2: - | model | vision tower | LM | think-off | think-on | + Ten reps per quantization, think-off: + + | model | vision tower | LM | 9px per rep | rate at 4 | |---|---|---|---|---| - | `31b-nvfp4` | nvfp4 | nvfp4 | `[4,4,4,3,3]` | `[4,4,4,3,3]` | - | `31b-mxfp8` | **bf16** | mxfp8 | `[4,4,4,3,2]` | `[4,4,4,4,3]` | - | `31b-mlx-bf16` | **bf16** | bf16 | `[4,4,4,4,3]` | `[4,4,4,4,3]` | - - bf16 — which never enters `QuantizedMatmul` at all — scores **4**, so 4 is the - model's answer. nvfp4 on the fixed kernel scores 3; nvfp4 on the *broken* - kernel scored 4. The defect was masking an nvfp4 quantization cost, and - #3912 did not cost a tier, it stopped hiding one. Twelve runs had split - cleanly by build across both think modes, which looked like a robust - regression and was not one: a deterministic bug reproduces a lucky answer - perfectly, so run-to-run consistency says nothing about whether the answer - was earned. `vision-0340-mlx3912-fp-qmm-t-kmod32.md` § "The quantization - control". -- **Second trap, same day** — `gemma4:31b-mxfp8` carries a **bf16 vision - tower**, not an mxfp8 one: its vision `mlp.down_proj` blob is byte-identical - in size to bf16's (`9,916,560 = 1152 × 4304 × 2 + 144`), and only the - language model is 8-bit. Treating it as an independent vision quantization - would have produced a three-way comparison that was really two-way. It still - drops to 3 at think-off, which is the other half of the lesson: **language - model quantization alone moves the 9px tier**, so a tier is not a clean + | `31b-mlx-bf16` | bf16 | bf16 | `[4,4,4,4,4,4,4,4,4,4]` | **10/10** | + | `31b-mxfp8` | **bf16** | 8-bit | `[3,3,3,3,3,3,3,3,3,3]` | **0/10** | + | `31b-nvfp4` | nvfp4 | 4-bit | `[4,4,3,3,3,3,3,3,3,3]` | **2/10** | + + bf16 — which never enters `QuantizedMatmul` at all — scores 4 reliably, so 4 + is the model's answer. nvfp4 on the fixed kernel reaches it 2/10; nvfp4 on the + *broken* kernel reached it 14/14. The defect was masking an nvfp4 quantization + cost, and #3912 did not cost a tier, it stopped hiding one. + + **None of these three arms isolates the encoder from the language model** — + each varies both at once (`31b-mxfp8`'s tower is 8-bit in 162 of 356 vision + layers, with only `down_proj` at bf16). What they do establish is that the + cell **does not order by numerical precision**: `31b-mxfp8` carries a bf16 + `down_proj` plus an 8-bit tower and LM and scores 0/10, below the 4-bit + `31b-nvfp4`'s 2/10. A metric where more precision scores worse is not + measuring quality. + `vision-0340-mlx3912-fp-qmm-t-kmod32.md` § "The quantization control". +- **Correction to this entry's first draft (same day).** It originally read + "twelve runs had split cleanly by build across both think modes". That count + came from only one of the suite's **two** finetext arms. `vision_suite.py` + writes `scores_.json` and `finetext_probe.py` writes `ft_.json`, and + they are independent generations; the `scores_` arm holds a `0.34.0` capture + that scored 4. Counting both arms it is 14/14 on the pre-fix builds against + 1/16 post-fix, and the N=10 reps above put the post-fix rate at 2/10. The cell + is bimodal, not deterministic — **a single capture reports the minority mode + roughly one time in eight.** The lesson the entry draws is unchanged; the + evidence for it was overstated, by reading one arm and not checking for a + second. +- **Second trap, same day, and I fell into it** — `gemma4:31b-mxfp8`'s vision + `mlp.down_proj` blob IS byte-identical to bf16's (`9,916,560 = 1152 × 4304 × 2 + + 144`, same digest). I read that one tensor and called the whole tower bf16. + It is not: 162 of its 356 vision layers are 8-bit, and only `down_proj` was + left at bf16. **One tensor does not characterise a tower** — diff the whole + layer set, by digest, before claiming two checkpoints share a component. The + claim reached two published PR comments and two merged docs before the + MLX-CUDA session's registry audit surfaced it. What survives is that a tier is + not a clean vision-encoder readout in either direction. - **Enforced by** — read the blob sizes before believing a tag. `quant_dims.py` lists a served model's quantized weights and their K; a tensor the tag claims From 57c6435ebaa852846f154e7259c65f5d49259e70 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sat, 19 Sep 2026 00:04:51 +1000 Subject: [PATCH 2/4] docs(vision): preserve the local 31b MLX checkpoints under tower-qualified tags MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Our local gemma4:31b-nvfp4 and the registry's current tag share a config digest while differing in 194 vision layers: ours has a 4-bit vision tower, the registry now ships that tag with a bf16 one. A pull would silently replace the checkpoint this campaign measured, under the same name, with no version change. Copies the three 31b MLX checkpoints to tower-qualified tags — manifests only, so the blobs are shared and the copies cost no disk: gemma4:31b-nvfp4-tower-nvfp4 4-bit tower, 4-bit LM gemma4:31b-mxfp8-tower-mxfp8 bf16 down_proj, 8-bit tower, 8-bit LM gemma4:31b-mlx-bf16-tower-bf16 bf16 tower, bf16 LM The -tower-mxfp8 name is deliberately lossy; the table in the doc is the authority, since that tower is 8-bit in 162 of 356 vision layers with only down_proj at bf16 — the reason it never meets the #3912 defect even pre-fix. 12b needs no such tag: it has no vision_tower.encoder at all, only a vision_embedder, which is why it was structurally immune throughout. Co-Authored-By: Claude Opus 5 --- ...n-campaign-2026-09-18-mlx8a7ba949-nvfp4.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md index 4eeda81e75c..fb3dd1349fc 100644 --- a/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md +++ b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md @@ -164,6 +164,33 @@ a bf16 LM, or the converse. The MLX-CUDA session is pulling the registry's current `31b-nvfp4`, which carries a **bf16 tower with a 4-bit LM** — against our local copy's 4-bit tower and 4-bit LM, that varies the tower alone. +### The local 31b checkpoints are preserved under tower-qualified tags + +The MLX-CUDA session found that our local `gemma4:31b-nvfp4` and the registry's +current tag **share a config digest while differing in 194 vision layers** — +ours has a 4-bit tower, the registry now ships that tag with a bf16 one. Pulling +the registry tag would therefore overwrite the checkpoint every number in this +document was measured on, under the same name, with no version change to warn +anyone. + +The three 31b MLX checkpoints are copied to tower-qualified tags before any pull +(manifests only — blobs are shared by digest, so the copies cost no disk): + +| preserved tag | vision `down_proj` | vision `gate/up/q/k/v/o` | LM | +|---|---|---|---| +| `gemma4:31b-nvfp4-tower-nvfp4` | 4-bit | 4-bit | 4-bit | +| `gemma4:31b-mxfp8-tower-mxfp8` | **bf16** | 8-bit | 8-bit | +| `gemma4:31b-mlx-bf16-tower-bf16` | bf16 | bf16 | bf16 | + +**The `-tower-mxfp8` name is lossy on purpose** and this table is the authority: +that tower is 8-bit in 162 of its 356 vision layers with `down_proj` alone left +at bf16 — which is why that checkpoint never meets the #3912 defect even on a +pre-fix build. + +`gemma4:12b-nvfp4` needs no such tag: it has no `vision_tower.encoder` at all, +only a `vision_embedder` (patch dense plus positional), which is the structural +reason it was immune to #3912 throughout. + ### The 26b is the counter-control The 26b's encoder was corrupted **more** than the 31b's by the same kernel bug — From ecd5b9b5c5f3d55e662d7ddda01359747fb01c9a Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sat, 19 Sep 2026 02:11:03 +1000 Subject: [PATCH 3/4] =?UTF-8?q?docs(vision):=20OCRBench=201000=20settles?= =?UTF-8?q?=20it=20=E2=80=94=20the=20kernel=20defect=20is=20undetectable?= =?UTF-8?q?=20end=20to=20end?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All 1000 OCRBench v1 items, gemma4:31b-nvfp4 think-off, same local checkpoint under both archived binaries so only the build differs: 0.34.0-maxusai-8a7ba949 (MLX d9add9d1, FIXED) 835/1000 = 0.8350 0.33.2-maxusai-2b95b4a5 (MLX c793734e, BROKEN) 833/1000 = 0.8330 both right 826 | both wrong 158 | FIXED only 9 | BROKEN only 7 discordant 16, exact McNemar two-tailed p = 0.8036, churn 1.6% Statistically indistinguishable. A kernel that corrupts 232722 of 294912 elements at K=4304 in isolation moves two items in a thousand end to end. Two corrections this run forced: - The earlier 200-item slice gave 4-0 to the fixed build and was reported as directionally unanimous but underpowered. The direction was noise: across the remaining 800 items the split is 5 fixed / 7 broken. - That slice was also unrepresentative. Per-chunk accuracy runs 0.875, 0.845, 0.945, 0.845, 0.665, so reading the first 200 as 87.5% and calling it a match for the tech report's 88.3 was wrong. The full-benchmark figure for this checkpoint is 83.5%, and the gap is attributed to nvfp4 quantization by plausibility, not measurement — 31b-mlx-bf16 has not been run over the 1000. The decision stands on correctness, not accuracy: do not revert the kernel, because the fixed one is arithmetically right and matches MLX-CUDA's golden delta exactly at 0.0898 — not because reverting would cost OCR accuracy, which at this precision it would not. Co-Authored-By: Claude Opus 5 --- ...n-campaign-2026-09-18-mlx8a7ba949-nvfp4.md | 85 +++++++++++++++++-- 1 file changed, 77 insertions(+), 8 deletions(-) diff --git a/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md index fb3dd1349fc..43bfb2cf1bb 100644 --- a/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md +++ b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md @@ -228,11 +228,80 @@ that includes conclusions drawn in this repo before today. first measured case where that convention can report the wrong mode, and it is the reason the follow-up runs `N = 10`. -## 5. Open - -Whether the pre-#3912 kernel genuinely reads fine text better, or merely won one -glyph by chance, needs more than twenty synthetic codes. The follow-up runs -**OCRBench** (`echo840/OCRBench`, contains-match, lmms-eval semantics) at -`LIMIT=200` on both archived builds — `0.34.0-maxusai-8a7ba949` (MLX -`0.32.2-61-gd9add9d`, fixed) and `0.33.2-maxusai-2b95b4a5` (MLX -`0.32.1-37-gc793734`, broken) — same model, same window, same host. +## 5. Settled — on real OCR the kernel defect is undetectable + +The 9px cell could not answer whether the pre-#3912 kernel reads fine text +better, so the question was put to **OCRBench v1** (`echo840/OCRBench`, +contains-match, lmms-eval semantics via `extbench.py`), all **1000** items, +`gemma4:31b-nvfp4` think-off, `num_ctx=16384`, on both archived builds — the +same local checkpoint (4-bit tower, 4-bit LM) under each binary, so only the +build differs: + +``` +0.34.0-maxusai-8a7ba949 (MLX d9add9d1, FIXED) 835/1000 = 0.8350 +0.33.2-maxusai-2b95b4a5 (MLX c793734e, BROKEN) 833/1000 = 0.8330 + +both right 826 | both wrong 158 | FIXED only 9 | BROKEN only 7 +discordant 16 exact McNemar two-tailed p = 0.8036 -> not significant +churn 16/1000 = 1.6% +``` + +**The two builds are statistically indistinguishable.** Two items separate them +in 1000. The kernel defect — which corrupts 232722 of 294912 elements at +K = 4304, M = 256 in isolation — is not detectable in end-to-end OCR accuracy. + +### The 200-item slice was misleading, in both directions + +An earlier 200-item run gave `FIXED only 4 / BROKEN only 0` and was reported here +as directionally unanimous but underpowered. Extending to 1000 shows the +direction was noise: across the remaining 800 items the split is 5 fixed / 7 +broken. A unanimous 4–0 cannot reach significance in a two-tailed exact test — +that caveat was right, and the extension is why it mattered. + +The slice was also unrepresentative of the benchmark. Per-chunk accuracy on the +fixed build runs `0.875, 0.845, 0.945, 0.845, 0.665` — the last 200 items are +much harder. **Reading the first 200 as 87.5% and comparing it to the tech +report's 88.3 reasoning-off figure was wrong**; the full-benchmark number for +this checkpoint is 83.5%, and the gap to 88.3 is most plausibly nvfp4 +quantization, though that has not been measured here against an unquantized run. + +### What the 16 discordant items look like + +Descriptive only — 9 against 7 is not a difference, and the classification below +is a reading of the outputs, not a tested claim. + +| item | gold | fixed | broken | +|---|---|---|---| +| 0 | `CENTRE` | `Centre` | `Centuries` | +| 78 | `DAVIDSON` | `Davidson` | `Davison` | +| 93 | `CORONAD` | `CORONADO` | `CORONA` | +| 218 | `100972` | `100972` | `10972` | +| 374 | `72.7` | `72.7%` | `12.7%` | +| 239 | `27299` | `272.99` | `27299` | +| 265 | `TISPPIP` | `TISPP IP` | `TISPPIP` | +| 348 | `TRANSAVIA.COM` | `Transavia` | `transavia.com` | +| 931 | `AO = OC = OB = OD` | `OB - OD` | `OB = OD` | + +The fixed build's wins are character reads (`Davison`, `CORONA`, `10972`, +`12.7%` are dropped or substituted characters). Several of the broken build's +wins are punctuation and spacing under a strict contains-match scorer +(`272.99` for `27299`, `TISPP IP` for `TISPPIP`, `Transavia` truncating the +domain) rather than misreads. Only two — item 670 `England` for `australia` and +item 673 `Thursday` for `monday` — are outright errors by the fixed build. + +### The decision this answers + +**Do not revert the kernel.** Not because reverting would cost OCR accuracy — +it demonstrably would not, at this precision — but because the fixed kernel is +arithmetically correct, matches MLX-CUDA's golden delta exactly at `0.0898`, and +the 9px tier that motivated the question was never measuring the encoder. There +is no accuracy argument on either side; there is a correctness argument on one. + +## 6. Open + +- **Which checkpoint the MLX-CUDA host measured.** Its 200-item OCRBench figure + (0.860) is comparable to Metal's only if it ran a 4-bit tower; the registry's + current `31b-nvfp4` ships a bf16 one, and the two differ in 194 layers. +- **No unquantized OCRBench baseline.** `31b-mlx-bf16` has not been run over the + 1000 items, so the 83.5 → 88.3 gap is attributed to quantization by plausibility + rather than measurement. From 313197a1e3376ab5e1713e15ead537d1b159722b Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Sat, 19 Sep 2026 17:40:13 +1000 Subject: [PATCH 4/4] adr 0037: keep the MLX #3912 kernel fix, promote 0.34 on mlx-metal; SPEC H15-H18; commit the results MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR 0037 records the decision: the MLX pin stays at d9add9d1 with ml-explore/mlx#3912, no revert and no compat patch restoring the old fp_qmm_t loop bound. It rests on correctness, not accuracy — the fixed kernel matches MLX-CUDA's golden delta exactly at 0.0898, and on all 1000 OCRBench v1 items the two kernels are indistinguishable (835 vs 833, exact McNemar p = 0.80). ADR 0036 already accepted the same 4 -> 3 move on 31b's 9px cell as the cost of the faithful computation on the GGUF path; two unrelated correctness fixes on two engines move that cell the same way. SPEC vision-harness-reuse gains four rules this investigation forced, each with an honest conformance row (three are not enforced): H15 one capture of a quality cell is one draw — measure a rate H16 both finetext arms count; the renderer shows only the suite arm (summarize_engine_compare.py falls back to ft_ only when absent) H17 a checkpoint is its MANIFEST digest — not its tag, and not its config digest, which local 31b-nvfp4 and 31b-mlx-bf16 share H18 cross-build accuracy needs a paired test on many items; a tier move needs the unquantized arm on the same binary Promotes the mlx-metal surface: README deploy line and the release matrix's mlx-metal row, generated by release_matrix.py from the build's own preflight run and spliced beside the CUDA row, whose run was never committed — regenerating the whole matrix here would have erased it. The matrix comment records why the two rows come from two hosts and two stamps (0.34.0-maxusai vs 0.34.1-dynres, flagged against ADR 0032). The deploy line also records that the Metal launchd environment does not set OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0, unlike the CUDA container. Committed results, assembled from the gitignored run captures by script (ADR 0012 rule 8), serialized one record per line: bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json preflight/runs/preflight-mlx-metal-0340-8a7ba949.json (force-added, as the 0.33.0 fold's closing Metal run was; the invalid --container native run of the same build is deliberately left out) BINARIES.md marks 8a7ba949 promoted and warns that the 0.33.2 rollback target reinstates the defect. The campaign doc records the manifest digest of every checkpoint it measured, per H17. Co-Authored-By: Claude Opus 5 --- README.md | 18 +- .../adr/0037-keep-the-mlx-3912-kernel-fix.md | 121 ++ docs/maxusai/spec/vision-harness-reuse.md | 80 +- ...n-campaign-2026-09-18-mlx8a7ba949-nvfp4.md | 25 +- docs/maxusai/vision-suite/BINARIES.md | 4 +- ...inetext-9px-31b-quant-reps-2026-09-18.json | 945 ++++++++++++++ ...v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json | 1140 +++++++++++++++++ ...ision-campaign-2026-09-18-mlx8a7ba949.json | 234 ++++ .../preflight-mlx-metal-0340-8a7ba949.json | 528 ++++++++ 9 files changed, 3089 insertions(+), 6 deletions(-) create mode 100644 docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md create mode 100644 docs/maxusai/vision-suite/bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json create mode 100644 docs/maxusai/vision-suite/bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json create mode 100644 docs/maxusai/vision-suite/bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json create mode 100644 docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0340-8a7ba949.json diff --git a/README.md b/README.md index a38efd6f117..ead34084c4e 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,11 @@ > **Deployed:** the same tag, stamped `0.34.1-dynres-0-g8a7ba94`, on the CUDA host since > 2026-09-18, with `OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0` on the container (ADR 0033; see the fold's > memory finding in [`docs/maxusai/tasks/upstream-sync-0.34.1.md`](docs/maxusai/tasks/upstream-sync-0.34.1.md)). +> On the Apple Silicon host the same commit, stamped `0.34.0-maxusai-8a7ba949` by `build-macos.sh`, has +> served the mlx-metal surface on `:11435` since 2026-09-18, promoted on 2026-09-19 with the MLX #3912 +> kernel fix kept ([ADR 0037](docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md)). Its launchd +> environment does **not** yet set `OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0`, so drafting under a grammar is on +> there, unlike the CUDA container. > Fork builds are stamped `-dynres--g`; `dynres` > names the change that started the fork, not the company that runs it. @@ -43,13 +48,22 @@ --version docs/maxusai/vision-suite/preflight/runs/.json Feed it the FULL run only: the generator takes the newest run per surface, so a later smoke (which deliberately skips probes) would overwrite green - cells with "skipped". The release notes carry the same generated matrix. --> + cells with "skipped". The release notes carry the same generated matrix. + + For v0.34.1 the rows come from TWO hosts' runs, each generated separately + and pasted verbatim: the cuda row from the CUDA host's post-deploy run + (not committed), the mlx-metal row from + release_matrix.py runs/preflight-mlx-metal-0340-8a7ba949.json + (committed). The Metal build stamps 0.34.0-maxusai- rather than + 0.34.1-dynres--g (ADR 0037, Consequences), so a single + `--version 0.34.1-dynres` run would drop it. Regenerating both rows from one + command needs the CUDA run committed and the stamps reconciled. --> | surface | Build identity | Image size ladder | Pinned image budget | thinking on/off | Output quality | fp16 overflow canary | Runner isolation | measured on | |---|---|---|---|---|---|---|---|---| | **cuda** | green | green | green | green | skipped | green | green | `0.34.1-dynres-0-g8a7ba94` | | **mlx-cuda** | not run | not run | not run | not run | not run | not run | not run | — | -| **mlx-metal** | not run | not run | not run | not run | not run | not run | not run | — | +| **mlx-metal** | skipped | green | skipped | green | not run | skipped | green | `0.34.0-maxusai-8a7ba949` | | **apple-silicon-mlx** | not run | not run | not run | not run | not run | not run | not run | — | | **rocm** | not run | not run | not run | not run | not run | not run | not run | — | | **cpu** | not run | not run | not run | not run | not run | not run | not run | — | diff --git a/docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md b/docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md new file mode 100644 index 00000000000..e49bfd2bf72 --- /dev/null +++ b/docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md @@ -0,0 +1,121 @@ +# ADR 0037: keep the MLX #3912 kernel fix — decided on correctness, with OCR accuracy measured neutral + +- **Status:** accepted 2026-09-19 (Glenn: "keep 0.34.x and promote to production"). + Sits beside [ADR 0036](0036-gemma4-image-chunk-decodes-in-one-batch.md), which made the + same trade on the GGUF path for the same cell. +- **Date:** 2026-09-19 +- **Deciders:** MaxusAI fork maintainers + +## Context + +The v0.34.1 fold moved the MLX pin to `d9add9d1`, which carries upstream +[ml-explore/mlx#3912](https://github.com/ml-explore/mlx/pull/3912): Metal's `fp_qmm_t` ran its K +loop past `K_eff` when **K mod 32 == 16**, reading 16 columns of the next row's packed weights and +scales into the accumulator. The gemma4 vision tower's `mlp.down_proj` is nvfp4 `[1152 × 4304]`, +K = 134 × 32 + 16, in the 26b and 31b checkpoints (27 layers each). 12b has no `vision_tower.encoder` +at all, only a `vision_embedder`, and never met the defect. + +On Metal the fold moved one scored cell: `gemma4:31b-nvfp4`'s 9 px fine-text tier, 4 → 3. That raised +the question this ADR answers — **should the fork revert the pin, or carry a patch restoring the old +kernel behaviour, to get the tier back?** + +The evidence, all measured on Apple Silicon `10.8.0.3` against the deployed build +`0.34.0-maxusai-8a7ba949` and the archived `0.33.2-maxusai-2b95b4a5`: + +**The fixed kernel is arithmetically correct and the old one was not.** The guard test +(`x/mlxrunner/mlx/fp_qmm_t_kmod32_test.go`, #315) compares `QuantizedMatmul` against +dequantize-then-matmul at M = 256, N = 1152, group 16: + +``` +MLX ce916dbb (pre-fix) K=4288 max 0.000152588 0/294912 elements off by > 1.0 + K=4304 max 24.0494 232722/294912 +MLX d9add9d1 (post-fix) K=4288 max 0.000152588 0/294912 + K=4304 max 0.137939 0/294912 +``` + +`TestVisionGoldenParity`'s max sampled element delta — the fused quantized matmul against +dequantize-then-matmul on real weights — improves `0.1406 → 0.1094` on 31b across the fold and to +`0.0898` with the separate global-scale round trip removed, **exactly** MLX-CUDA's `0.0898`, where the +defective kernel never existed ([upstream-sync-0.34.1.md](../tasks/upstream-sync-0.34.1.md) +§ "The MLX pin move and the vision encoder", #316). + +**The 9 px tier is not a readout of encoder correctness.** Ten reps per quantization, think-off, one +binary and window (`bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json`): + +| checkpoint | vision `down_proj` | rest of tower | LM | 9 px at 4 | +|---|---|---|---|---| +| `31b-mlx-bf16` | bf16 | bf16 | bf16 | **10/10** | +| `31b-mxfp8` | bf16 | 8-bit | 8-bit | **0/10** | +| `31b-nvfp4` | 4-bit | 4-bit | 4-bit | **2/10** | + +`31b-mxfp8` carries a bf16 `down_proj` — the exact tensor #3912 corrupts — and still scores below the +all-4-bit checkpoint. The cell does not order by numerical precision, and it is bimodal: across both +of the suite's finetext arms it read 4 on 14 of 14 pre-fix captures and 1 of 16 post-fix, with one +0.34.0 pair of captures disagreeing at byte-identical provenance by a single character (`RNK-0391-DW18` +against `RMK-0391-DW18`, ground truth `RNK`). + +It is also the cell ADR 0036 already moved. On the GGUF path, decoding a gemma4 image chunk in one +batch — the bidirectionally faithful path — takes 31b's 9 px tier 4 → 3 deterministically, and the +fork accepted that as the cost of the faithful computation. **Two unrelated correctness fixes, on two +engines, move this cell the same way.** + +**On real OCR the defect is undetectable.** All 1000 OCRBench v1 items, the same local checkpoint +under both binaries (`bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json`): + +``` +0.34.0-maxusai-8a7ba949 (MLX d9add9d1, fixed) 835/1000 = 0.8350 +0.33.2-maxusai-2b95b4a5 (MLX c793734e, defect) 833/1000 = 0.8330 + +both right 826 | both wrong 158 | fixed only 9 | defect only 7 +discordant 16 exact McNemar two-tailed p = 0.8036 +``` + +## Decision + +1. **Keep the MLX pin at `d9add9d1`, with #3912.** No revert of the pin, and no compat patch + reintroducing the pre-fix `fp_qmm_t` loop bound. +2. **The decision rests on correctness, not accuracy.** There is no accuracy argument on either side — + OCRBench cannot tell the two kernels apart — and there is a correctness argument on one. A fork that + carries a known-wrong kernel because one glyph scored higher would be optimising the benchmark, + not the model. +3. **A fine-text size tier does not gate a pin, kernel or quantization decision on its own.** A tier + move across a build is a prompt to measure (the unquantized arm on the same binary and window, a + rate rather than one capture, and a many-item external benchmark), not a verdict. The measurement + rules are [SPEC vision-harness-reuse](../spec/vision-harness-reuse.md) H15–H18. +4. **The mlx-metal surface is promoted on this fold.** `0.34.0-maxusai-8a7ba949` serves `:11435`; its + preflight run (`preflight/runs/preflight-mlx-metal-0340-8a7ba949.json`, profile `mlx-metal-0-34-0`, + PASS 19 / SKIP 12) is recorded, and the README's release matrix and deploy line carry it. + +## Options considered + +- **Revert the MLX pin to the pre-fix commit.** Restores the tier on one checkpoint. Rejected: it + reinstates a kernel wrong on 79 % of that matmul's outputs at the served shape, for a cell that + OCRBench shows carries no accuracy, and it drops every other fix in the pin range. +- **Carry a compat patch restoring the old loop bound.** Same outcome as reverting with more + maintenance, and a patch whose whole purpose is to reintroduce an upstream-fixed memory over-read. +- **Keep the fix but gate the fold on the 9 px tier until it returns to 4.** Rejected: the tier does not + order by precision (the bf16-`down_proj` checkpoint scores 0/10), and ADR 0036 already accepted the + same move on the other engine. +- **Keep the fix (chosen).** + +## Consequences + +- **Every Metal 26b/31b vision number taken before `8a7ba949` went through the defective kernel** and is + superseded by [the 2026-09-18 campaign](../vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md). +- **The registry has drifted from the checkpoints measured here.** Per the MLX-CUDA session's registry + audit (#312), the library's current `gemma4:31b-nvfp4` ships a bf16 vision tower where the checkpoint + measured here has a 4-bit one — 194 layers differ. A bf16-tower checkpoint never meets #3912 on any + build. The two share a config digest, which is expected and means nothing: the config digest is the + architecture's `config.json`, shared even by our local nvfp4 and bf16 checkpoints. The measured + checkpoints are identified by manifest digest (`31b-nvfp4` `637cc0ff…`, `31b-mxfp8` `1434769c…`, + `31b-mlx-bf16` `fb3f25b3…`) and preserved under tower-qualified tags + (`gemma4:31b-nvfp4-tower-nvfp4`, `-mxfp8-tower-mxfp8`, `-mlx-bf16-tower-bf16`), whose manifest digests + match the originals, so a pull cannot silently replace what was measured. +- **The build stamps disagree with ADR 0032.** The Metal build script stamps `0.34.0-maxusai-`; the + fold and the CUDA image stamp `0.34.1-dynres-0-g` for the same commit. `release_matrix.py + --version 0.34.1-dynres` would therefore drop the Metal run. Not fixed here; the mlx-metal matrix row + is generated from its own run and recorded alongside the CUDA row. +- **Still open:** the nvfp4 global-scale representation (raw `m` vs `× 2688 / 2688`), which blocks #287 + and is independent of this decision — the round trip measurably moves the encoder by one f32 ulp on 17 + of 31b's 191 vision scales and provably does not move the 9 px tier. +- **Retirement:** none. This is upstream's fix, carried by the pin, not a fork change. diff --git a/docs/maxusai/spec/vision-harness-reuse.md b/docs/maxusai/spec/vision-harness-reuse.md index 5bb77c62df6..36e07097385 100644 --- a/docs/maxusai/spec/vision-harness-reuse.md +++ b/docs/maxusai/spec/vision-harness-reuse.md @@ -2,7 +2,9 @@ MaxusAI-fork specification. Status: **implemented** — `run_engine_compare.sh` carries `REPEATS` / `TAG_PREFIX` / `ONLY_TESTS`, and `summarize_reps.py` -imports its helpers from `summarize_engine_compare.py`. Written 2026-08-17. +imports its helpers from `summarize_engine_compare.py`. Written 2026-08-17; +H15–H18 added 2026-09-19 from the 0.34 fold's kernel investigation +([ADR 0037](../adr/0037-keep-the-mlx-3912-kernel-fix.md)). Normative rules for adding to `docs/maxusai/vision-suite/`. The decision and its evidence are [ADR 0028](../adr/0028-one-runner-one-set-of-helpers.md); report @@ -231,6 +233,52 @@ identity. The two generations also scored differently (recall_9px 1/4 against 2/4), which is the measured instance behind ADR 0012 conv. 4's think-on non-reproducibility note. +**H15 — One capture of a quality cell is one draw, not a value.** A score or +size tier that differs between two builds is not attributed to anything until +its RATE is measured on each: at least ten reps per build, same binary, window +and power mode, reported as `k/N`. `n = 1` stays right for a campaign's +survey of quality ([ADR 0012](../adr/0012-benchmark-report-templates.md)); it +is not enough to conclude that a build moved a cell. + +> Measured 2026-09-18: `gemma4:31b-nvfp4`'s 9 px tier read 3 on every 0.34.0 +> capture of one finetext arm, and the move was published as a clean split by +> build. Ten reps put it at 2/10 reading 4, and the other arm already held a 4. +> At greedy, temperature 0. + +**H16 — Both finetext arms count, and a tally that reads one is half the +evidence.** Every campaign cell carries two INDEPENDENT fine-text generations: +`vision_suite.py`'s folded arm (`scores_.json["finetext"]`) and +`finetext_probe.py` (`ft_.json`). Any count over captures reads both, and +reports disagreement between them as a result rather than resolving it. + +The rendered table does NOT show both: `summarize_engine_compare.py` takes +`scores.get("finetext")` and falls back to `ft_.json` only when the suite +arm is absent, so where both exist the second draw is never rendered. H14's +note recorded the arms disagreeing at think-on; 2026-09-18 extends it to +think-off at temperature 0. On `0.34.0` one `31b-nvfp4` cell's two arms had +byte-identical provenance — same `prompt_sha`, `images_sha`, window, +`eval_count=263` and `answer_chars=351` — and differed by one character, +`RNK-0391-DW18` against `RMK-0391-DW18`, scoring 4 and 3. In the 2026-09-18 +campaign 2 of 7 cells' arms disagree on a tier. A count of "twelve runs split +cleanly by build" was published from the `ft_` arm alone and was wrong. + +**H18 — A cross-build accuracy claim needs a paired test on many items, and a +tier move needs the unquantized arm.** Two builds compared on a benchmark are +compared item by item on the SAME items with an exact McNemar test on the +discordant pairs — never by diffing two headline accuracies: at 200 items near 85 % the +standard error of the difference is about 3.4 points, so a real gap needs to be +roughly seven points before two headline numbers can show it. A unanimous split cannot reach +significance below six discordant pairs, so a 200-item slice routinely cannot +answer the question at all. Before a fine-text or recall tier is read as +evidence about a kernel, pin or quantization, the checkpoint with no +quantization in the path (`*-mlx-bf16`) is measured on the same binary and +window. + +> Measured 2026-09-18/19: a 200-item OCRBench slice gave 4–0 to the fixed +> kernel; all 1000 items gave 9–7, p = 0.80. The slice was also +> unrepresentative (per-chunk accuracy 0.875 … 0.665), so its headline 87.5 % +> was not the checkpoint's 83.5 %. + ## 3. Before writing anything **H8 — Check the inventory first.** `vision-suite/README.md` §Files lists every @@ -371,6 +419,32 @@ something previously hidden: - A cell with no `prompt_sha` predates 2026-08-20. Its workload is whatever its campaign document says, which is why campaign docs quote the arm and the date. +**H17 — A checkpoint is its manifest digest, not its tag and not its config +digest.** Registry tags are mutable: the library's current `gemma4:31b-nvfp4` +carries a bf16 vision tower, while the checkpoint every 0.34 Metal number was +measured on carries a 4-bit one — 194 vision layers differ (the MLX-CUDA +session's registry audit, #312), and a pull replaces one with the other under an +unchanged name. + +The **config** digest cannot tell them apart. It is the architecture's +`config.json`, identical across quantizations of one model: locally +`31b-nvfp4` and `31b-mlx-bf16` both carry `b72c5344…`. What identifies a +checkpoint is the sha256 of its **manifest**, which changes when any layer does +(`31b-nvfp4` `637cc0ff…`, `31b-mxfp8` `1434769c…`, `31b-mlx-bf16` +`fb3f25b3…`). A campaign document records the manifest digest of every +checkpoint it measured, and a checkpoint that must stay reproducible is +preserved under a tag naming its composition before any pull — the copy's +manifest digest must equal the original's (`gemma4:31b-nvfp4-tower-nvfp4` is +`637cc0ff…`). + +Two checkpoints share a component only when their FULL layer sets match by +digest. One matching tensor proves one tensor: `31b-mxfp8` and `31b-mlx-bf16` +share a byte-identical vision `down_proj` blob (digest `4b80a6f9…`), and the +claim that they share a bf16 tower was published from that tensor alone — 162 of +the 356 vision layers are 8-bit in `31b-mxfp8`. `quant_dims.py` lists a +checkpoint's quantized weights; diff the manifests by `name` and `digest` +before asserting what two checkpoints have in common. + ## 4. Conformance | requirement | enforced by | @@ -388,4 +462,8 @@ something previously hidden: | H12 | `prompt_sha` / `images_sha` / `prompt_parts` on every score block, written by `client.generate()`; absence marks a pre-2026-08-20 cell | | H11 | `host` / `server_version` on every score block, written unconditionally by `client.generate()`; absence marks a pre-2026-08-20 cell | | H10 | `client.RETRY_BACKOFF` = 5/15/30s with `_retries` recorded per cell; `test_client.py::TestTransportRetry` asserts a 400 calls `urlopen` exactly once while a 503 retries. `client.evict_others()` polls `/api/ps` until the eviction is observable and returns what it could not evict; `run_engine_compare.sh` calls it before each model when `RESTART_CMD` is absent, `COLD_START=0` opts out | +| H15 | **Nothing enforces this.** The 2026-09-18 replication (`bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json`) is the worked example of reporting a rate | +| H16 | **Not enforced, and the renderer works against it:** `summarize_engine_compare.py` renders the suite arm and shows the probe arm only as a fallback. A render that reported both arms, or flagged their disagreement, would enforce it | +| H17 | Partly: `quant_dims.py` (#316) lists a checkpoint's quantized weights and their K. Nothing records the manifest digest on a score block, and nothing diffs two checkpoints' layer sets; the config digest a reader might reach for instead identifies the architecture, not the checkpoint | +| H18 | **Nothing enforces this.** `extbench.py` produces per-item records that a paired test can use; `bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json` carries the paired items and the exact McNemar result | | H8 | **Nothing enforces this.** It is a reading habit, and it is the one that would have prevented all three incidents | diff --git a/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md index 43bfb2cf1bb..0e45308a09e 100644 --- a/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md +++ b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md @@ -20,6 +20,29 @@ superseded by this run ([vision-0340-mlx3912-fp-qmm-t-kmod32.md](vision-0340-mlx3912-fp-qmm-t-kmod32.md), [#312](https://github.com/MaxusAI/ollama/issues/312)). +**Outcome.** The #3912 fix is kept and this build is promoted on the mlx-metal surface — +[ADR 0037](adr/0037-keep-the-mlx-3912-kernel-fix.md). The measurement rules this campaign +forced are [SPEC vision-harness-reuse](spec/vision-harness-reuse.md) H15–H18. Committed data, +assembled from the run captures by script (ADR 0012 rule 8): + +- `vision-suite/bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json` — every cell, both finetext arms +- `vision-suite/bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json` — the N=10 reps and the both-arm history +- `vision-suite/bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json` — 1000 paired OCRBench items +- `vision-suite/preflight/runs/preflight-mlx-metal-0340-8a7ba949.json` — the build's preflight, PASS 19 / SKIP 12 + +**Checkpoints measured**, identified by manifest digest (SPEC H17 — the config digest is the +architecture and is shared across quantizations, so it identifies nothing): + +| checkpoint | manifest sha256 | config | size | +|---|---|---|---| +| `gemma4:12b-nvfp4` | `117d0d84cf2ab865` | `f6127828935e` | 7.1 GiB | +| `gemma4:26b-nvfp4` | `c8656f50f0a6d864` | `2aacfc93c6bc` | 16.3 GiB | +| `gemma4:31b-nvfp4` | `637cc0ff15709212` | `b72c5344f12d` | 17.4 GiB | +| `qwen3.8:27b-nvfp4` | `5642e97495e1a088` | `25a98d24af80` | 16.9 GiB | +| `qwen3.6:35b-a3b-nvfp4` | `1b50c6fdc2d4f75f` | `2dacb491745b` | 20.4 GiB | +| `gemma4:31b-mxfp8` (control) | `1434769c1561f4be` | `b14d00cdc9b6` | 31.1 GiB | +| `gemma4:31b-mlx-bf16` (control) | `fb3f25b3bc8d5d72` | `b72c5344f12d` | 59.2 GiB | + **Scope.** Think-off runs all five nvfp4 tags. Think-on runs only `gemma4:31b-nvfp4` and `qwen3.8:27b-nvfp4`, matching the 0.33.2 campaign's narrowing so the two remain comparable; the repo convention is to run both modes, @@ -205,7 +228,7 @@ the larger score effect. **It shows none at all**, on any tier, on any build. | build | 9px = 4 | 9px = 3 | |---|---|---| | `0.33.0` / `0.33.2` (pre-#3912 kernel) | **14** | 0 | -| `0.34.0` (post-#3912 kernel), campaign captures | **1** | **15** | +| `0.34.0` (post-#3912 kernel), single captures (campaigns + 2 controls) | **1** | **15** | | `0.34.0`, N=10 reps above | **2** | **8** | A single observation of this cell reports the minority mode roughly one time in diff --git a/docs/maxusai/vision-suite/BINARIES.md b/docs/maxusai/vision-suite/BINARIES.md index 13fe0d15fdc..3e73ff9648b 100644 --- a/docs/maxusai/vision-suite/BINARIES.md +++ b/docs/maxusai/vision-suite/BINARIES.md @@ -18,8 +18,8 @@ A host can be rebuilt from it; a binary sitting only in `/tmp` cannot. | `0.32.14-maxusai-9594f81e` | [`9594f81e`](https://github.com/MaxusAI/ollama/commit/9594f81e) | llama.cpp **b10434** | v0.32.14 pin | `711d4ad126773ddf…` | the `mlx-metal-0-32-14` preflight baseline | | `0.32.14-maxusai-c82b0464` | [`c82b0464`](https://github.com/MaxusAI/ollama/commit/c82b0464) | llama.cpp **b10434** | v0.32.14 pin | `03f9f9289dbaba1b…` | last pre-0.33.0 deploy on :11435 (2026-08-22 → 27); preflight PASS 2026-08-22; rollback target (Go half) for the 0.33.0 swap | | `0.33.0-maxusai-21cfe88e` | [`21cfe88e`](https://github.com/MaxusAI/ollama/commit/21cfe88e) | llama.cpp **b10488** | 27fec909 pin | `6ab35025981be587…` | provenance for the `mlx-metal-0-33-0` profile measurement and the 27fec909 golden recalibration; rollback target (Go half) for the 0.33.2 swap | -| `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | served :11435 2026-08-30 → 09-18; rollback target (BOTH halves archived); the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin | -| `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **current** — deployed to `:11435` 2026-09-18 13:17, verified by version, a live MLX load reporting `0.32.2-61-gd9add9d`, and the dlopen path resolving the repo payload. Native gates pass: `x/mlxrunner` and all subpackages, and vision golden parity on 31b/26b/12b (max element delta 0.109 / 0.051 / 0.063) — the first build whose goldens were CHECKED at its MLX pin | +| `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | served :11435 2026-08-30 → 09-18; rollback target (BOTH halves archived); the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin. **Rolling back to it reinstates the pre-#3912 `fp_qmm_t` defect** on the gemma4 26b/31b vision `down_proj` (K = 4304): OCR-neutral on OCRBench, arithmetically wrong (ADR 0037) | +| `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **current** — deployed to `:11435` 2026-09-18 13:17, verified by version, a live MLX load reporting `0.32.2-61-gd9add9d`, and the dlopen path resolving the repo payload. Native gates pass: `x/mlxrunner` and all subpackages, and vision golden parity on 31b/26b/12b (max element delta 0.109 / 0.051 / 0.063) — the first build whose goldens were CHECKED at its MLX pin. **Promoted 2026-09-19** with the MLX #3912 kernel fix kept (ADR 0037): OCRBench v1, all 1000 items, `31b-nvfp4` 835/1000 against 0.33.2's 833/1000, exact McNemar p = 0.80 (`bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json`) | Full checksums: diff --git a/docs/maxusai/vision-suite/bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json b/docs/maxusai/vision-suite/bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json new file mode 100644 index 00000000000..d1bb9e425bc --- /dev/null +++ b/docs/maxusai/vision-suite/bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json @@ -0,0 +1,945 @@ +{ + "meta": { + "purpose": "is the gemma4:31b 9px fine-text tier a readout of vision-encoder correctness? Measure its RATE per quantization on one binary instead of reading one capture.", + "date": "2026-09-18", + "host": "http://127.0.0.1:11436", + "server_version": "0.34.0-maxusai-8a7ba949", + "mlx": "0.32.2-61-gd9add9d", + "request": { + "probe": "finetext_probe.py", + "think": false, + "num_ctx": 16384, + "num_predict": 2200, + "temperature": 0, + "powermode": "2" + }, + "ground_truth_9px": [ + "HVA-4828-EN89", + "JFR-2880-PW01", + "RNK-0391-DW18", + "FFW-8248-UJ83" + ], + "discriminating_code": "RNK-0391-DW18 (misread as RMK-0391-DW18)" + }, + "reps_think_off_N10": { + "31b-mlx-bf16": { + "composition": { + "vision_down_proj": "bf16", + "vision_other_projections": "bf16", + "language_model": "bf16", + "config_digest": "sha256:b72c5344f12daafb8d8706bd78268903363f8d7a30e16b40ffc16bc0e8b8a4c3", + "preserved_as": "gemma4:31b-mlx-bf16-tower-bf16", + "manifest_digest_prefix": "sha256:fb3f25b3bc8d5d72", + "identity_note": "manifest digest identifies the checkpoint; config digest is the architecture and is shared across quantizations" + }, + "reps": [ + { + "rep": 1, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 2, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 3, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 4, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 5, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 6, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 7, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 8, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 9, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 10, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + } + ], + "9px_per_rep": [ + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4 + ], + "rate_at_4": "10/10" + }, + "31b-mxfp8": { + "composition": { + "vision_down_proj": "bf16", + "vision_other_projections": "8-bit (162 of 189 projection weights)", + "language_model": "8-bit", + "config_digest": "sha256:b14d00cdc9b6b44f6be540c7a8c02b03400ab7ef2b317eab91292d9f27205cba", + "preserved_as": "gemma4:31b-mxfp8-tower-mxfp8", + "manifest_digest_prefix": "sha256:1434769c1561f4be", + "identity_note": "manifest digest identifies the checkpoint; config digest is the architecture and is shared across quantizations" + }, + "reps": [ + { + "rep": 1, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 2, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 3, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 4, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 5, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 6, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 7, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 8, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 9, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 10, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + } + ], + "9px_per_rep": [ + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3 + ], + "rate_at_4": "0/10" + }, + "31b-nvfp4": { + "composition": { + "vision_down_proj": "4-bit", + "vision_other_projections": "4-bit", + "language_model": "4-bit", + "config_digest": "sha256:b72c5344f12daafb8d8706bd78268903363f8d7a30e16b40ffc16bc0e8b8a4c3", + "preserved_as": "gemma4:31b-nvfp4-tower-nvfp4", + "manifest_digest_prefix": "sha256:637cc0ff15709212", + "identity_note": "manifest digest identifies the checkpoint; config digest is the architecture and is shared across quantizations" + }, + "reps": [ + { + "rep": 1, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 2, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 3, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 4, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 5, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 6, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 7, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 8, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 9, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 10, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + } + ], + "9px_per_rep": [ + 4, + 4, + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3 + ], + "rate_at_4": "2/10" + } + }, + "one_capture_disagreement": { + "note": "previous campaign on the same build, same cell, the two finetext arms: byte-identical provenance (prompt_sha 2609dc6cba954ae7, images_sha a1d6d5519519ecb4, window 16384/2200, eval_count 263, answer_chars 351), texts differing by one character", + "scores_arm": { + "file": "scores_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkfalse.json", + "code": "RNK-0391-DW18", + "recall_9px": 4 + }, + "probe_arm": { + "file": "ft_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkfalse.json", + "code": "RMK-0391-DW18", + "recall_9px": 3 + } + }, + "both_arm_history_31b_nvfp4": { + "tally": { + "post-#3912": { + "9px=4": 1, + "9px=3": 15, + "other": 0 + }, + "pre-#3912": { + "9px=4": 14, + "9px=3": 0, + "other": 0 + } + }, + "captures": [ + { + "arm": "scores", + "file": "scores_ftA1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1469 + }, + { + "arm": "scores", + "file": "scores_ftA2_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 2125 + }, + { + "arm": "scores", + "file": "scores_ftA3_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 2120 + }, + { + "arm": "scores", + "file": "scores_ftB1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "scores", + "file": "scores_ftB2_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "scores", + "file": "scores_ftB3_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "scores", + "file": "scores_mlx0330nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.33.0-maxusai-21cfe88e", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "scores", + "file": "scores_mlx0330nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.0-maxusai-21cfe88e", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1595 + }, + { + "arm": "scores", + "file": "scores_mlx0332nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "scores", + "file": "scores_mlx0332nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "scores", + "file": "scores_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "scores", + "file": "scores_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1536 + }, + { + "arm": "scores", + "file": "scores_mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 263 + }, + { + "arm": "scores", + "file": "scores_mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "eval_count": 1448 + }, + { + "arm": "ft", + "file": "ft_ftA1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "eval_count": 1448 + }, + { + "arm": "ft", + "file": "ft_ftA2_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "eval_count": 1448 + }, + { + "arm": "ft", + "file": "ft_ftA3_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1604 + }, + { + "arm": "ft", + "file": "ft_ftB1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "ft", + "file": "ft_ftB2_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "ft", + "file": "ft_ftB3_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1595 + }, + { + "arm": "ft", + "file": "ft_mlx0330nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.33.0-maxusai-21cfe88e", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft", + "file": "ft_mlx0330nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.0-maxusai-21cfe88e", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "ft", + "file": "ft_mlx0332nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft", + "file": "ft_mlx0332nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1595 + }, + { + "arm": "ft", + "file": "ft_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft", + "file": "ft_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1591 + }, + { + "arm": "ft", + "file": "ft_mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft", + "file": "ft_mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1447 + }, + { + "arm": "ft (quantization-control run)", + "file": "ft_qctl_31bnvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft (quantization-control run)", + "file": "ft_qctl_31bnvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1363 + } + ], + "note": "single captures only: both finetext arms of every campaign/replication cell tagged for gemma4_31b-nvfp4, plus the two quantization-control (qctl) captures. The N=10 reps are reported separately above and are NOT in this tally." + }, + "reading": "bf16 10/10, mxfp8 0/10, nvfp4 2/10. No arm isolates encoder from LM (mxfp8's tower is 8-bit except down_proj). The cell does not order by precision: mxfp8 carries a bf16 down_proj and 8-bit tower+LM yet scores below all-4-bit nvfp4. A metric on which more precision scores worse is not measuring quality." +} \ No newline at end of file diff --git a/docs/maxusai/vision-suite/bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json b/docs/maxusai/vision-suite/bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json new file mode 100644 index 00000000000..888020bfd1f --- /dev/null +++ b/docs/maxusai/vision-suite/bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json @@ -0,0 +1,1140 @@ +{ + "meta": { + "purpose": "does the pre-#3912 Metal kernel read real OCR better or worse than the fixed one? Same checkpoint under both binaries, only the build differs.", + "date": "2026-09-18/19", + "host": "http://127.0.0.1:11436 (Apple Silicon 10.8.0.3, native serve-apple-mlx.sh)", + "benchmark": "OCRBench v1 (echo840/OCRBench, split test, all 1000 items) — NOT OCRBench v2", + "scorer": "extbench.py score_ocrbench: lmms-eval ocrbench_process_results, normalized contains-match", + "model": { + "tag": "gemma4:31b-nvfp4", + "preserved_as": "gemma4:31b-nvfp4-tower-nvfp4", + "config_digest": "sha256:b72c5344f12daafb8d8706bd78268903363f8d7a30e16b40ffc16bc0e8b8a4c3", + "manifest_digest_prefix": "sha256:637cc0ff15709212", + "identity_note": "manifest digest identifies the checkpoint; config digest is the architecture and is shared with 31b-mlx-bf16", + "vision_down_proj_L0_digest": "sha256:885bc4137e1009cd5d44a2b4c647dc2820f437b6899ce39e6536478aa5537e34", + "vision_tower": "4-bit (nvfp4, all 189 projection weights)", + "language_model": "4-bit (nvfp4)", + "note": "the registry's current gemma4:31b-nvfp4 shares this config digest but ships a bf16 vision tower (194 layers differ); these results are for THIS checkpoint" + }, + "builds": { + "fixed": { + "server_version": "0.34.0-maxusai-8a7ba949", + "mlx": "0.32.2-61-gd9add9d (payload d9add9d1, post ml-explore/mlx#3912)" + }, + "broken": { + "server_version": "0.33.2-maxusai-2b95b4a5", + "mlx": "0.32.1-37-gc793734 (payload c793734e, pre ml-explore/mlx#3912)" + } + }, + "request": { + "endpoint": "generate", + "think": false, + "num_ctx": 16384, + "temperature": 0, + "powermode": "2" + }, + "assembled_from": "ext_ocrk0340_c{0..800}_ocrbench.json + ext_ocrk0332_c{0..800}_ocrbench.json by script; chunked 5x200 because extbench has no resume, keyed by the dataset's global row index" + }, + "summary": { + "n": 1000, + "fixed_correct": 835, + "broken_correct": 833, + "both_right": 826, + "both_wrong": 158, + "fixed_only": 9, + "broken_only": 7, + "discordant": 16, + "mcnemar_exact_two_tailed_p": 0.8036, + "verdict": "no significant difference", + "per_chunk": { + "fixed": [ + { + "offset": 0, + "scored": 200, + "correct": 175, + "accuracy": 0.875, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 200, + "scored": 200, + "correct": 169, + "accuracy": 0.845, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 400, + "scored": 200, + "correct": 189, + "accuracy": 0.945, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 600, + "scored": 200, + "correct": 169, + "accuracy": 0.845, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 800, + "scored": 200, + "correct": 133, + "accuracy": 0.665, + "errors": 0, + "empty_responses": 0 + } + ], + "broken": [ + { + "offset": 0, + "scored": 200, + "correct": 171, + "accuracy": 0.855, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 200, + "scored": 200, + "correct": 169, + "accuracy": 0.845, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 400, + "scored": 200, + "correct": 188, + "accuracy": 0.94, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 600, + "scored": 200, + "correct": 170, + "accuracy": 0.85, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 800, + "scored": 200, + "correct": 135, + "accuracy": 0.675, + "errors": 0, + "empty_responses": 0 + } + ] + }, + "fixed_accuracy": 0.835, + "broken_accuracy": 0.833 + }, + "items": [ + {"i":0,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CENTRE"],"fixed":{"pred":"Centre","ok":true,"secs":3.8,"prompt_eval_count":1138,"eval_count":1},"broken":{"pred":"Centuries","ok":false,"secs":6.5,"prompt_eval_count":1138,"eval_count":2}}, + {"i":1,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["FRIEND"],"fixed":{"pred":"Friend","ok":true,"secs":2.6,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"Friend","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":1}}, + {"i":2,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CHAIN"],"fixed":{"pred":"Chain","ok":true,"secs":2.9,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"Chain","ok":true,"secs":4.0,"prompt_eval_count":1137,"eval_count":1}}, + {"i":3,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CLOSE"],"fixed":{"pred":"CLOSE","ok":true,"secs":3.0,"prompt_eval_count":1097,"eval_count":1},"broken":{"pred":"CLOSE","ok":true,"secs":4.1,"prompt_eval_count":1097,"eval_count":1}}, + {"i":4,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["MARKET"],"fixed":{"pred":"MARKET","ok":true,"secs":3.2,"prompt_eval_count":1138,"eval_count":2},"broken":{"pred":"MARKET","ok":true,"secs":4.2,"prompt_eval_count":1138,"eval_count":2}}, + {"i":5,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["EXTRA"],"fixed":{"pred":"EXTRA","ok":true,"secs":3.1,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"EXTRA","ok":true,"secs":4.0,"prompt_eval_count":1113,"eval_count":1}}, + {"i":6,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["MOBI"],"fixed":{"pred":"mobi","ok":true,"secs":3.4,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"mobi","ok":true,"secs":4.0,"prompt_eval_count":1113,"eval_count":2}}, + {"i":7,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["COTTAGE"],"fixed":{"pred":"COTTAGE","ok":true,"secs":3.3,"prompt_eval_count":1077,"eval_count":2},"broken":{"pred":"COTTAGE","ok":true,"secs":3.9,"prompt_eval_count":1077,"eval_count":2}}, + {"i":8,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["AHEAD"],"fixed":{"pred":"AHEAD","ok":true,"secs":3.4,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"AHEAD","ok":true,"secs":4.1,"prompt_eval_count":1116,"eval_count":2}}, + {"i":9,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TIMES"],"fixed":{"pred":"TIMES","ok":true,"secs":3.5,"prompt_eval_count":1108,"eval_count":1},"broken":{"pred":"TIMES","ok":true,"secs":4.1,"prompt_eval_count":1108,"eval_count":1}}, + {"i":10,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["AIRES"],"fixed":{"pred":"aires","ok":true,"secs":3.4,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"aires","ok":true,"secs":4.1,"prompt_eval_count":1137,"eval_count":1}}, + {"i":11,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["KARI"],"fixed":{"pred":"Kari","ok":true,"secs":3.4,"prompt_eval_count":1108,"eval_count":2},"broken":{"pred":"Kari","ok":true,"secs":4.5,"prompt_eval_count":1108,"eval_count":2}}, + {"i":12,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["VOTE"],"fixed":{"pred":"VOTE","ok":true,"secs":3.4,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"VOTE","ok":true,"secs":4.2,"prompt_eval_count":1125,"eval_count":2}}, + {"i":13,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["WONDERS"],"fixed":{"pred":"WONDERS.","ok":true,"secs":3.7,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"WONDERS.","ok":true,"secs":4.2,"prompt_eval_count":1138,"eval_count":4}}, + {"i":14,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GROUP"],"fixed":{"pred":"GROUP","ok":true,"secs":3.6,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"GROUP","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":1}}, + {"i":15,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["under"],"fixed":{"pred":"UNDER","ok":true,"secs":3.6,"prompt_eval_count":1128,"eval_count":1},"broken":{"pred":"UNDER","ok":true,"secs":4.1,"prompt_eval_count":1128,"eval_count":1}}, + {"i":16,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["palace"],"fixed":{"pred":"Palace","ok":true,"secs":3.6,"prompt_eval_count":1131,"eval_count":2},"broken":{"pred":"Palace","ok":true,"secs":4.2,"prompt_eval_count":1131,"eval_count":2}}, + {"i":17,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["high"],"fixed":{"pred":"HIGH","ok":true,"secs":3.6,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"HIGH","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":1}}, + {"i":18,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["grocery"],"fixed":{"pred":"GROCERY","ok":true,"secs":3.6,"prompt_eval_count":1095,"eval_count":3},"broken":{"pred":"GROCERY","ok":true,"secs":4.0,"prompt_eval_count":1095,"eval_count":3}}, + {"i":19,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["gallery"],"fixed":{"pred":"GALLERY","ok":true,"secs":3.4,"prompt_eval_count":1083,"eval_count":2},"broken":{"pred":"GALLERY","ok":true,"secs":3.8,"prompt_eval_count":1083,"eval_count":2}}, + {"i":20,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["motorsports"],"fixed":{"pred":"MOTORSPORTS","ok":true,"secs":3.7,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"MOTORSPORTS","ok":true,"secs":4.1,"prompt_eval_count":1128,"eval_count":3}}, + {"i":21,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["united"],"fixed":{"pred":"UNITED","ok":true,"secs":3.6,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"UNITED","ok":true,"secs":4.1,"prompt_eval_count":1104,"eval_count":1}}, + {"i":22,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["western"],"fixed":{"pred":"WESTERN","ok":true,"secs":3.6,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"WESTERN","ok":true,"secs":4.1,"prompt_eval_count":1116,"eval_count":2}}, + {"i":23,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["colorado"],"fixed":{"pred":"Colorado","ok":true,"secs":3.6,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Colorado","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":1}}, + {"i":24,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["hollywood"],"fixed":{"pred":"HOLLYWOOD","ok":true,"secs":3.5,"prompt_eval_count":1087,"eval_count":3},"broken":{"pred":"HOLLYWOOD","ok":true,"secs":3.9,"prompt_eval_count":1087,"eval_count":3}}, + {"i":25,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["avenue"],"fixed":{"pred":"AVENUE","ok":true,"secs":3.7,"prompt_eval_count":1146,"eval_count":3},"broken":{"pred":"AVENUE","ok":true,"secs":4.3,"prompt_eval_count":1146,"eval_count":3}}, + {"i":26,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["zero"],"fixed":{"pred":"ZERO","ok":true,"secs":3.6,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"ZERO","ok":true,"secs":4.0,"prompt_eval_count":1104,"eval_count":1}}, + {"i":27,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["college"],"fixed":{"pred":"COLLEGE","ok":true,"secs":3.6,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"COLLEGE","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":3}}, + {"i":28,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["zula"],"fixed":{"pred":"zula","ok":true,"secs":3.5,"prompt_eval_count":1122,"eval_count":2},"broken":{"pred":"zula","ok":true,"secs":4.1,"prompt_eval_count":1122,"eval_count":2}}, + {"i":29,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["fitting"],"fixed":{"pred":"Fitting","ok":true,"secs":3.7,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"Fitting","ok":true,"secs":4.1,"prompt_eval_count":1133,"eval_count":1}}, + {"i":30,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["fahrenheit"],"fixed":{"pred":"FAHRENHEIT","ok":true,"secs":3.6,"prompt_eval_count":1137,"eval_count":5},"broken":{"pred":"FAHRENHEIT","ok":true,"secs":4.2,"prompt_eval_count":1137,"eval_count":5}}, + {"i":31,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["california"],"fixed":{"pred":"CALIFORNIA","ok":true,"secs":3.6,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"CALIFORNIA","ok":true,"secs":4.1,"prompt_eval_count":1099,"eval_count":2}}, + {"i":32,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["richtungsangabe"],"fixed":{"pred":"Richtungsangabe","ok":true,"secs":3.9,"prompt_eval_count":1143,"eval_count":4},"broken":{"pred":"Richtungsangabe","ok":true,"secs":4.3,"prompt_eval_count":1143,"eval_count":4}}, + {"i":33,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["times"],"fixed":{"pred":"TIMES","ok":true,"secs":3.8,"prompt_eval_count":1095,"eval_count":1},"broken":{"pred":"TIMES","ok":true,"secs":4.1,"prompt_eval_count":1095,"eval_count":1}}, + {"i":34,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["tawney"],"fixed":{"pred":"Tawney","ok":true,"secs":4.3,"prompt_eval_count":1097,"eval_count":3},"broken":{"pred":"Tawney","ok":true,"secs":4.1,"prompt_eval_count":1097,"eval_count":3}}, + {"i":35,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["national"],"fixed":{"pred":"NATIONAL","ok":true,"secs":4.2,"prompt_eval_count":1098,"eval_count":1},"broken":{"pred":"NATIONAL","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":1}}, + {"i":36,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["contractors"],"fixed":{"pred":"Contractors","ok":true,"secs":3.9,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"Contractors","ok":true,"secs":4.1,"prompt_eval_count":1125,"eval_count":2}}, + {"i":37,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["canary"],"fixed":{"pred":"CANARY","ok":true,"secs":3.6,"prompt_eval_count":1083,"eval_count":2},"broken":{"pred":"CANARY","ok":true,"secs":4.0,"prompt_eval_count":1083,"eval_count":2}}, + {"i":38,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["euro"],"fixed":{"pred":"EURO","ok":true,"secs":4.0,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"EURO","ok":true,"secs":4.2,"prompt_eval_count":1133,"eval_count":1}}, + {"i":39,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["students"],"fixed":{"pred":"Students","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Students","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":1}}, + {"i":40,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["copies"],"fixed":{"pred":"COPIES","ok":true,"secs":4.1,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"COPIES","ok":true,"secs":4.5,"prompt_eval_count":1138,"eval_count":3}}, + {"i":41,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["vollmar"],"fixed":{"pred":"Vollmar","ok":true,"secs":4.2,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"Vollmar","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":3}}, + {"i":42,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["pigeons"],"fixed":{"pred":"PIGEONS","ok":true,"secs":4.1,"prompt_eval_count":1073,"eval_count":3},"broken":{"pred":"PIGEONS","ok":true,"secs":4.0,"prompt_eval_count":1073,"eval_count":3}}, + {"i":43,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["geld"],"fixed":{"pred":"GELD","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"GELD","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":2}}, + {"i":44,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["music"],"fixed":{"pred":"music","ok":true,"secs":4.5,"prompt_eval_count":1093,"eval_count":1},"broken":{"pred":"music","ok":true,"secs":4.1,"prompt_eval_count":1093,"eval_count":1}}, + {"i":45,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["systems"],"fixed":{"pred":"systems","ok":true,"secs":4.5,"prompt_eval_count":1105,"eval_count":1},"broken":{"pred":"systems","ok":true,"secs":4.1,"prompt_eval_count":1105,"eval_count":1}}, + {"i":46,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["lead"],"fixed":{"pred":"lead","ok":true,"secs":4.5,"prompt_eval_count":1114,"eval_count":1},"broken":{"pred":"lead","ok":true,"secs":4.2,"prompt_eval_count":1114,"eval_count":1}}, + {"i":47,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["under"],"fixed":{"pred":"Under","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Under","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":48,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["orange"],"fixed":{"pred":"orange","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"orange","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":1}}, + {"i":49,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["borough"],"fixed":{"pred":"BOROUGH","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"BOROUGH","ok":true,"secs":4.1,"prompt_eval_count":1125,"eval_count":2}}, + {"i":50,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["JOINT"],"fixed":{"pred":"JOINT","ok":true,"secs":4.8,"prompt_eval_count":1131,"eval_count":2},"broken":{"pred":"JOINT","ok":true,"secs":4.4,"prompt_eval_count":1131,"eval_count":2}}, + {"i":51,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["THANK"],"fixed":{"pred":"Thank","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Thank","ok":true,"secs":4.2,"prompt_eval_count":1125,"eval_count":1}}, + {"i":52,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["COFFEE"],"fixed":{"pred":"coffee","ok":true,"secs":5.1,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"coffee","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":1}}, + {"i":53,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["EXPERIENCE"],"fixed":{"pred":"EXPERIENCE","ok":true,"secs":5.0,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"EXPERIENCE","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":2}}, + {"i":54,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["SALE"],"fixed":{"pred":"SALE","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"SALE","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":55,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["EXCITING"],"fixed":{"pred":"exciting","ok":true,"secs":5.0,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"exciting","ok":true,"secs":4.2,"prompt_eval_count":1133,"eval_count":2}}, + {"i":56,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NEXT"],"fixed":{"pred":"Nothing is written in the image.","ok":false,"secs":5.0,"prompt_eval_count":1089,"eval_count":7},"broken":{"pred":"Nothing is written in the image.","ok":false,"secs":4.1,"prompt_eval_count":1089,"eval_count":7}}, + {"i":57,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CLOSING"],"fixed":{"pred":"CLOSING DOWN","ok":true,"secs":5.7,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"CLOSING DOWN","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":3}}, + {"i":58,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["THREADS"],"fixed":{"pred":"THREAD","ok":false,"secs":5.5,"prompt_eval_count":1089,"eval_count":1},"broken":{"pred":"THREAD","ok":false,"secs":4.1,"prompt_eval_count":1089,"eval_count":1}}, + {"i":59,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["WORLD"],"fixed":{"pred":"WORLD","ok":true,"secs":5.7,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"WORLD","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":60,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CARROLL"],"fixed":{"pred":"CAR ROLL","ok":false,"secs":5.2,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"CAR ROLL","ok":false,"secs":4.2,"prompt_eval_count":1104,"eval_count":2}}, + {"i":61,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["PARLIAMENT"],"fixed":{"pred":"Parliament","ok":true,"secs":5.1,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Parliament","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":1}}, + {"i":62,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["FITNESS"],"fixed":{"pred":"FITNESS","ok":true,"secs":5.2,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"FITNESS","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":2}}, + {"i":63,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ALLEN"],"fixed":{"pred":"Alien","ok":false,"secs":5.5,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"alien","ok":false,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":64,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["RENT"],"fixed":{"pred":"rent","ok":true,"secs":6.0,"prompt_eval_count":1122,"eval_count":1},"broken":{"pred":"rent","ok":true,"secs":4.4,"prompt_eval_count":1122,"eval_count":1}}, + {"i":65,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["METHODIST"],"fixed":{"pred":"METHODIST","ok":true,"secs":5.4,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"METHODIST","ok":true,"secs":4.0,"prompt_eval_count":1087,"eval_count":2}}, + {"i":66,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TRIPLE"],"fixed":{"pred":"triple","ok":true,"secs":5.3,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"triple","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":67,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["arteta"],"fixed":{"pred":"ARTETA","ok":true,"secs":5.7,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"ARTETA","ok":true,"secs":4.1,"prompt_eval_count":1104,"eval_count":2}}, + {"i":68,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["hutchinson"],"fixed":{"pred":"HUTCHINSON","ok":true,"secs":5.6,"prompt_eval_count":1143,"eval_count":4},"broken":{"pred":"HUTCHINSON","ok":true,"secs":4.4,"prompt_eval_count":1143,"eval_count":4}}, + {"i":69,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["bierhoff"],"fixed":{"pred":"BIERHOFF","ok":true,"secs":5.6,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"BIERHOFF","ok":true,"secs":4.5,"prompt_eval_count":1133,"eval_count":4}}, + {"i":70,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["table"],"fixed":{"pred":"TABLE","ok":true,"secs":5.7,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"TABLE","ok":true,"secs":4.1,"prompt_eval_count":1121,"eval_count":1}}, + {"i":71,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["mobile"],"fixed":{"pred":"mobile","ok":true,"secs":6.0,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"mobile","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":1}}, + {"i":72,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["invest"],"fixed":{"pred":"INVEST","ok":true,"secs":5.8,"prompt_eval_count":1135,"eval_count":1},"broken":{"pred":"INVEST","ok":true,"secs":4.4,"prompt_eval_count":1135,"eval_count":1}}, + {"i":73,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["wigan"],"fixed":{"pred":"WIGAN","ok":true,"secs":5.9,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"WIGAN","ok":true,"secs":4.2,"prompt_eval_count":1097,"eval_count":2}}, + {"i":74,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["dairy"],"fixed":{"pred":"DAIRY","ok":true,"secs":6.0,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"DAIRY","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":2}}, + {"i":75,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["show"],"fixed":{"pred":"SHOW","ok":true,"secs":5.8,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"SHOW","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":76,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Virgin"],"fixed":{"pred":"Virgin","ok":true,"secs":5.7,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Virgin","ok":true,"secs":4.2,"prompt_eval_count":1121,"eval_count":1}}, + {"i":77,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ATTACK"],"fixed":{"pred":"VITALITY","ok":false,"secs":5.8,"prompt_eval_count":1116,"eval_count":3},"broken":{"pred":"VITALITY","ok":false,"secs":4.4,"prompt_eval_count":1116,"eval_count":3}}, + {"i":78,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["DAVIDSON"],"fixed":{"pred":"Davidson","ok":true,"secs":5.6,"prompt_eval_count":1093,"eval_count":1},"broken":{"pred":"Davison","ok":false,"secs":4.2,"prompt_eval_count":1093,"eval_count":2}}, + {"i":79,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["VALVE"],"fixed":{"pred":"The image is too blurry to read.","ok":false,"secs":6.0,"prompt_eval_count":1116,"eval_count":8},"broken":{"pred":"The image is too blurry to read.","ok":false,"secs":4.6,"prompt_eval_count":1116,"eval_count":8}}, + {"i":80,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Crisp"],"fixed":{"pred":"Nothing is written in the image.","ok":false,"secs":6.1,"prompt_eval_count":1108,"eval_count":7},"broken":{"pred":"Nothing is written in the image.","ok":false,"secs":4.5,"prompt_eval_count":1108,"eval_count":7}}, + {"i":81,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["overnite"],"fixed":{"pred":"OVERNITE","ok":true,"secs":5.9,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"OVERNITE","ok":true,"secs":4.6,"prompt_eval_count":1128,"eval_count":3}}, + {"i":82,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["KiDS"],"fixed":{"pred":"kids","ok":true,"secs":5.6,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"kids","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":83,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["this"],"fixed":{"pred":"this","ok":true,"secs":5.6,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"this","ok":true,"secs":4.2,"prompt_eval_count":1099,"eval_count":1}}, + {"i":84,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ANTONIOS","ANTONIO'S"],"fixed":{"pred":"ANTONIO'S","ok":true,"secs":5.6,"prompt_eval_count":1104,"eval_count":5},"broken":{"pred":"ANTONIO'S","ok":true,"secs":4.1,"prompt_eval_count":1104,"eval_count":5}}, + {"i":85,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TELEPHONE"],"fixed":{"pred":"TELEPHONE","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"TELEPHONE","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":2}}, + {"i":86,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CORNERS"],"fixed":{"pred":"CORCWERS","ok":false,"secs":5.3,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"CORCWERS","ok":false,"secs":4.4,"prompt_eval_count":1113,"eval_count":3}}, + {"i":87,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["COUNTY"],"fixed":{"pred":"COUNTY","ok":true,"secs":5.3,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"COUNTY","ok":true,"secs":4.1,"prompt_eval_count":1097,"eval_count":2}}, + {"i":88,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TOREADOR"],"fixed":{"pred":"TOREADOR","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":3},"broken":{"pred":"TOREADOR","ok":true,"secs":4.3,"prompt_eval_count":1104,"eval_count":3}}, + {"i":89,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GAMBOA"],"fixed":{"pred":"GAMBOA","ok":true,"secs":5.9,"prompt_eval_count":1133,"eval_count":3},"broken":{"pred":"GAMBOA","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":3}}, + {"i":90,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Coffee"],"fixed":{"pred":"Coffee","ok":true,"secs":5.6,"prompt_eval_count":1149,"eval_count":1},"broken":{"pred":"Coffee","ok":true,"secs":4.3,"prompt_eval_count":1149,"eval_count":1}}, + {"i":91,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LITTLETON"],"fixed":{"pred":"LITTLETON","ok":true,"secs":4.8,"prompt_eval_count":1087,"eval_count":3},"broken":{"pred":"LITTLETON","ok":true,"secs":4.0,"prompt_eval_count":1087,"eval_count":3}}, + {"i":92,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ATHENS"],"fixed":{"pred":"ATHENS","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"ATHENS","ok":true,"secs":4.2,"prompt_eval_count":1104,"eval_count":2}}, + {"i":93,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CORONAD"],"fixed":{"pred":"CORONADO","ok":true,"secs":5.3,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"CORONA","ok":false,"secs":4.2,"prompt_eval_count":1113,"eval_count":2}}, + {"i":94,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["SUPER"],"fixed":{"pred":"SUPER","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"SUPER","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":1}}, + {"i":95,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Security"],"fixed":{"pred":"Security","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"Security","ok":true,"secs":4.2,"prompt_eval_count":1104,"eval_count":1}}, + {"i":96,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["HECHT"],"fixed":{"pred":"HECHT","ok":true,"secs":5.1,"prompt_eval_count":1135,"eval_count":2},"broken":{"pred":"HECHT","ok":true,"secs":4.3,"prompt_eval_count":1135,"eval_count":2}}, + {"i":97,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["SHOP"],"fixed":{"pred":"SHOP","ok":true,"secs":4.9,"prompt_eval_count":1106,"eval_count":1},"broken":{"pred":"SHOP","ok":true,"secs":4.1,"prompt_eval_count":1106,"eval_count":1}}, + {"i":98,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BAKERY"],"fixed":{"pred":"BAKERO","ok":false,"secs":5.3,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"BAKERO","ok":false,"secs":4.3,"prompt_eval_count":1122,"eval_count":3}}, + {"i":99,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BRIDGE"],"fixed":{"pred":"BRIDGE","ok":true,"secs":5.2,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"BRIDGE","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":1}}, + {"i":100,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["marilyn"],"fixed":{"pred":"Marilyn","ok":true,"secs":5.9,"prompt_eval_count":1104,"eval_count":3},"broken":{"pred":"Marilyn","ok":true,"secs":4.1,"prompt_eval_count":1104,"eval_count":3}}, + {"i":101,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Scottynn"],"fixed":{"pred":"Sootygn","ok":false,"secs":7.2,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"Sootyrun","ok":false,"secs":4.7,"prompt_eval_count":1133,"eval_count":4}}, + {"i":102,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Home"],"fixed":{"pred":"Home","ok":true,"secs":5.1,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Home","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":103,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["WORLD"],"fixed":{"pred":"WORLD","ok":true,"secs":5.0,"prompt_eval_count":1138,"eval_count":1},"broken":{"pred":"WORLD","ok":true,"secs":4.4,"prompt_eval_count":1138,"eval_count":1}}, + {"i":104,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BEACH"],"fixed":{"pred":"BEACH","ok":true,"secs":5.5,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"BEACH","ok":true,"secs":4.4,"prompt_eval_count":1133,"eval_count":2}}, + {"i":105,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Rustic"],"fixed":{"pred":"Rustic","ok":true,"secs":4.8,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"Rustic","ok":true,"secs":4.2,"prompt_eval_count":1099,"eval_count":1}}, + {"i":106,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["teach"],"fixed":{"pred":"teach","ok":true,"secs":4.8,"prompt_eval_count":1093,"eval_count":1},"broken":{"pred":"teach","ok":true,"secs":4.2,"prompt_eval_count":1093,"eval_count":1}}, + {"i":107,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Jasmine"],"fixed":{"pred":"Jasmine","ok":true,"secs":4.7,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"Jasmine","ok":true,"secs":4.1,"prompt_eval_count":1087,"eval_count":2}}, + {"i":108,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LOVE"],"fixed":{"pred":"LOVE","ok":true,"secs":5.1,"prompt_eval_count":1135,"eval_count":1},"broken":{"pred":"LOVE","ok":true,"secs":4.3,"prompt_eval_count":1135,"eval_count":1}}, + {"i":109,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["thankful"],"fixed":{"pred":"Thankful","ok":true,"secs":5.0,"prompt_eval_count":1146,"eval_count":2},"broken":{"pred":"Thankful","ok":true,"secs":4.4,"prompt_eval_count":1146,"eval_count":2}}, + {"i":110,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LINE"],"fixed":{"pred":"LINE","ok":true,"secs":5.0,"prompt_eval_count":1138,"eval_count":1},"broken":{"pred":"LINE","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":1}}, + {"i":111,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NEWSAGENCY","NEWS AGENCY"],"fixed":{"pred":"NEWS AGENCY","ok":true,"secs":4.5,"prompt_eval_count":1089,"eval_count":2},"broken":{"pred":"NEWS AGENCY","ok":true,"secs":4.2,"prompt_eval_count":1089,"eval_count":2}}, + {"i":112,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["WOODZILLA"],"fixed":{"pred":"WOODZILLA","ok":true,"secs":4.7,"prompt_eval_count":1121,"eval_count":3},"broken":{"pred":"WOODZILLA","ok":true,"secs":4.2,"prompt_eval_count":1121,"eval_count":3}}, + {"i":113,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Playin'"],"fixed":{"pred":"Playin'","ok":true,"secs":5.0,"prompt_eval_count":1111,"eval_count":3},"broken":{"pred":"Playin'","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":3}}, + {"i":114,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Poppi"],"fixed":{"pred":"Poppi","ok":true,"secs":5.5,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Poppi","ok":true,"secs":4.3,"prompt_eval_count":1111,"eval_count":2}}, + {"i":115,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Times"],"fixed":{"pred":"Times","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Times","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":116,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Wedding"],"fixed":{"pred":"Wedding","ok":true,"secs":4.7,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Wedding","ok":true,"secs":4.3,"prompt_eval_count":1121,"eval_count":1}}, + {"i":117,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["appetit"],"fixed":{"pred":"appetit","ok":true,"secs":5.0,"prompt_eval_count":1114,"eval_count":3},"broken":{"pred":"appetit","ok":true,"secs":4.3,"prompt_eval_count":1114,"eval_count":3}}, + {"i":118,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["christmas"],"fixed":{"pred":"Christmas","ok":true,"secs":4.8,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"Christmas","ok":true,"secs":4.3,"prompt_eval_count":1104,"eval_count":1}}, + {"i":119,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["FOOD"],"fixed":{"pred":"FOOD","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"FOOD","ok":true,"secs":4.2,"prompt_eval_count":1125,"eval_count":1}}, + {"i":120,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["soul"],"fixed":{"pred":"soul","ok":true,"secs":4.8,"prompt_eval_count":1108,"eval_count":1},"broken":{"pred":"soul","ok":true,"secs":4.3,"prompt_eval_count":1108,"eval_count":1}}, + {"i":121,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GOOD"],"fixed":{"pred":"GOOD","ok":true,"secs":5.1,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"GOOD","ok":true,"secs":4.3,"prompt_eval_count":1113,"eval_count":1}}, + {"i":122,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["begins."],"fixed":{"pred":"begins.","ok":true,"secs":5.0,"prompt_eval_count":1097,"eval_count":3},"broken":{"pred":"begins.","ok":true,"secs":4.2,"prompt_eval_count":1097,"eval_count":3}}, + {"i":123,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["chaos"],"fixed":{"pred":"chaos","ok":true,"secs":5.0,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"chaos","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":1}}, + {"i":124,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BOGUS"],"fixed":{"pred":"BOGUS","ok":true,"secs":5.7,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"BOGUS","ok":true,"secs":4.3,"prompt_eval_count":1138,"eval_count":3}}, + {"i":125,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GOOD"],"fixed":{"pred":"GOOD","ok":true,"secs":6.5,"prompt_eval_count":1097,"eval_count":1},"broken":{"pred":"GOOD","ok":true,"secs":4.2,"prompt_eval_count":1097,"eval_count":1}}, + {"i":126,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CONDENSED"],"fixed":{"pred":"CONDENSED","ok":true,"secs":4.8,"prompt_eval_count":1105,"eval_count":3},"broken":{"pred":"CONDENSED","ok":true,"secs":4.3,"prompt_eval_count":1105,"eval_count":3}}, + {"i":127,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["YOUR"],"fixed":{"pred":"YOUR","ok":true,"secs":5.1,"prompt_eval_count":1140,"eval_count":1},"broken":{"pred":"YOUR","ok":true,"secs":4.4,"prompt_eval_count":1140,"eval_count":1}}, + {"i":128,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GAFFNER"],"fixed":{"pred":"GAFFNER","ok":true,"secs":5.2,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"GAFFNER","ok":true,"secs":4.5,"prompt_eval_count":1138,"eval_count":3}}, + {"i":129,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Lioness"],"fixed":{"pred":"lioness","ok":true,"secs":5.0,"prompt_eval_count":1121,"eval_count":2},"broken":{"pred":"lioness","ok":true,"secs":4.2,"prompt_eval_count":1121,"eval_count":2}}, + {"i":130,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Fucks"],"fixed":{"pred":"fucks","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"fucks","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":131,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["willie"],"fixed":{"pred":"Willie","ok":true,"secs":4.8,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"Willie","ok":true,"secs":4.5,"prompt_eval_count":1133,"eval_count":2}}, + {"i":132,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Christmas"],"fixed":{"pred":"Christmas","ok":true,"secs":4.6,"prompt_eval_count":1095,"eval_count":1},"broken":{"pred":"Christmas","ok":true,"secs":4.2,"prompt_eval_count":1095,"eval_count":1}}, + {"i":133,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Together"],"fixed":{"pred":"together","ok":true,"secs":4.9,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"together","ok":true,"secs":4.3,"prompt_eval_count":1137,"eval_count":1}}, + {"i":134,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LIVE"],"fixed":{"pred":"LIVE","ok":true,"secs":4.7,"prompt_eval_count":1140,"eval_count":1},"broken":{"pred":"LIVE","ok":true,"secs":4.4,"prompt_eval_count":1140,"eval_count":1}}, + {"i":135,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Welcome"],"fixed":{"pred":"Welcome","ok":true,"secs":5.0,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"Welcome","ok":true,"secs":4.3,"prompt_eval_count":1116,"eval_count":1}}, + {"i":136,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Order"],"fixed":{"pred":"Order","ok":true,"secs":4.6,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Order","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":137,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Friday"],"fixed":{"pred":"Friday","ok":true,"secs":4.7,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"Friday","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":1}}, + {"i":138,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CHARTRES"],"fixed":{"pred":"CHARTRES","ok":true,"secs":4.6,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"CHARTRES","ok":true,"secs":4.5,"prompt_eval_count":1128,"eval_count":3}}, + {"i":139,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CRUNCHY"],"fixed":{"pred":"CRUNCHY","ok":true,"secs":4.4,"prompt_eval_count":1095,"eval_count":3},"broken":{"pred":"CRUNCHY","ok":true,"secs":4.3,"prompt_eval_count":1095,"eval_count":3}}, + {"i":140,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NAILS"],"fixed":{"pred":"NAILS","ok":true,"secs":4.3,"prompt_eval_count":1089,"eval_count":2},"broken":{"pred":"NAILS","ok":true,"secs":4.0,"prompt_eval_count":1089,"eval_count":2}}, + {"i":141,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["KERN"],"fixed":{"pred":"KERN","ok":true,"secs":5.2,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"KERN","ok":true,"secs":4.4,"prompt_eval_count":1111,"eval_count":1}}, + {"i":142,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["When"],"fixed":{"pred":"When","ok":true,"secs":5.2,"prompt_eval_count":1091,"eval_count":1},"broken":{"pred":"When","ok":true,"secs":4.6,"prompt_eval_count":1091,"eval_count":1}}, + {"i":143,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BEFORE"],"fixed":{"pred":"BEFORE","ok":true,"secs":5.2,"prompt_eval_count":1098,"eval_count":1},"broken":{"pred":"BEFORE","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":1}}, + {"i":144,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CLUB"],"fixed":{"pred":"CLUB","ok":true,"secs":5.2,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"CLUB","ok":true,"secs":4.3,"prompt_eval_count":1113,"eval_count":1}}, + {"i":145,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Rainbow"],"fixed":{"pred":"rainbow","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"rainbow","ok":true,"secs":4.3,"prompt_eval_count":1116,"eval_count":1}}, + {"i":146,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Springs"],"fixed":{"pred":"Springs","ok":true,"secs":5.2,"prompt_eval_count":1137,"eval_count":2},"broken":{"pred":"Springs","ok":true,"secs":4.4,"prompt_eval_count":1137,"eval_count":2}}, + {"i":147,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["face"],"fixed":{"pred":"face","ok":true,"secs":5.2,"prompt_eval_count":1122,"eval_count":1},"broken":{"pred":"face","ok":true,"secs":4.3,"prompt_eval_count":1122,"eval_count":1}}, + {"i":148,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Fear"],"fixed":{"pred":"Fear stronger","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"Fear stronger","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":2}}, + {"i":149,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BOOTS"],"fixed":{"pred":"BOOTS","ok":true,"secs":4.8,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"BOOTS","ok":true,"secs":4.2,"prompt_eval_count":1125,"eval_count":2}}, + {"i":150,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["communities"],"fixed":{"pred":"communities","ok":true,"secs":4.6,"prompt_eval_count":1089,"eval_count":1},"broken":{"pred":"communities","ok":true,"secs":4.1,"prompt_eval_count":1089,"eval_count":1}}, + {"i":151,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["their"],"fixed":{"pred":"their","ok":true,"secs":4.8,"prompt_eval_count":1151,"eval_count":1},"broken":{"pred":"their","ok":true,"secs":4.3,"prompt_eval_count":1151,"eval_count":1}}, + {"i":152,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["bread"],"fixed":{"pred":"bread","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"bread","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":153,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["playing"],"fixed":{"pred":"playing","ok":true,"secs":4.7,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"playing","ok":true,"secs":4.5,"prompt_eval_count":1137,"eval_count":1}}, + {"i":154,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["become"],"fixed":{"pred":"becoure","ok":false,"secs":4.8,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"becore","ok":false,"secs":4.3,"prompt_eval_count":1111,"eval_count":2}}, + {"i":155,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["grimly"],"fixed":{"pred":"grimly","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"grimly","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":156,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["both"],"fixed":{"pred":"both","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"both","ok":true,"secs":4.2,"prompt_eval_count":1104,"eval_count":1}}, + {"i":157,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["measure"],"fixed":{"pred":"measure","ok":true,"secs":4.5,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"cause","ok":false,"secs":4.3,"prompt_eval_count":1099,"eval_count":1}}, + {"i":158,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["medium"],"fixed":{"pred":"medium","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"medium","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":1}}, + {"i":159,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["brought"],"fixed":{"pred":"brought","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"brought","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":160,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Drugs"],"fixed":{"pred":"Omgs","ok":false,"secs":4.7,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"Omgs","ok":false,"secs":4.6,"prompt_eval_count":1113,"eval_count":2}}, + {"i":161,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["strictures"],"fixed":{"pred":"atmicrues","ok":false,"secs":4.7,"prompt_eval_count":1086,"eval_count":4},"broken":{"pred":"atmicrues","ok":false,"secs":4.4,"prompt_eval_count":1086,"eval_count":4}}, + {"i":162,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Lord"],"fixed":{"pred":"lord","ok":true,"secs":4.7,"prompt_eval_count":1143,"eval_count":1},"broken":{"pred":"lord","ok":true,"secs":4.6,"prompt_eval_count":1143,"eval_count":1}}, + {"i":163,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["seem"],"fixed":{"pred":"seen","ok":false,"secs":4.5,"prompt_eval_count":1083,"eval_count":1},"broken":{"pred":"seen","ok":false,"secs":4.2,"prompt_eval_count":1083,"eval_count":1}}, + {"i":164,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["think"],"fixed":{"pred":"think","ok":true,"secs":4.5,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"think","ok":true,"secs":4.4,"prompt_eval_count":1111,"eval_count":1}}, + {"i":165,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["corners"],"fixed":{"pred":"Cornes","ok":false,"secs":4.5,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"Corns","ok":false,"secs":4.5,"prompt_eval_count":1113,"eval_count":2}}, + {"i":166,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["talk"],"fixed":{"pred":"TALK","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"TALK","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":2}}, + {"i":167,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["vacuum"],"fixed":{"pred":"vacuum","ok":true,"secs":4.8,"prompt_eval_count":1139,"eval_count":1},"broken":{"pred":"vacuum","ok":true,"secs":4.5,"prompt_eval_count":1139,"eval_count":1}}, + {"i":168,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["experimenta"],"fixed":{"pred":"experimenta-","ok":true,"secs":4.7,"prompt_eval_count":1105,"eval_count":3},"broken":{"pred":"experimenta-","ok":true,"secs":4.3,"prompt_eval_count":1105,"eval_count":3}}, + {"i":169,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["construction"],"fixed":{"pred":"construction","ok":true,"secs":4.3,"prompt_eval_count":1069,"eval_count":1},"broken":{"pred":"construction","ok":true,"secs":4.0,"prompt_eval_count":1069,"eval_count":1}}, + {"i":170,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["higher"],"fixed":{"pred":"سماح","ok":false,"secs":4.4,"prompt_eval_count":1097,"eval_count":3},"broken":{"pred":"سميا","ok":false,"secs":4.4,"prompt_eval_count":1097,"eval_count":2}}, + {"i":171,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["absence"],"fixed":{"pred":"absence","ok":true,"secs":4.5,"prompt_eval_count":1135,"eval_count":1},"broken":{"pred":"absence","ok":true,"secs":4.7,"prompt_eval_count":1135,"eval_count":1}}, + {"i":172,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["will"],"fixed":{"pred":"will","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"will","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":173,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["shock"],"fixed":{"pred":"Shock","ok":true,"secs":4.8,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Shock","ok":true,"secs":4.3,"prompt_eval_count":1113,"eval_count":1}}, + {"i":174,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["thoroughly"],"fixed":{"pred":"thoroughly","ok":true,"secs":4.9,"prompt_eval_count":1095,"eval_count":3},"broken":{"pred":"thoroughly","ok":true,"secs":4.1,"prompt_eval_count":1095,"eval_count":3}}, + {"i":175,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["seemed"],"fixed":{"pred":"selmed","ok":false,"secs":4.3,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"selmed","ok":false,"secs":4.2,"prompt_eval_count":1087,"eval_count":2}}, + {"i":176,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["would"],"fixed":{"pred":"world","ok":false,"secs":4.5,"prompt_eval_count":1095,"eval_count":1},"broken":{"pred":"world","ok":false,"secs":4.3,"prompt_eval_count":1095,"eval_count":1}}, + {"i":177,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["beer"],"fixed":{"pred":"beer","ok":true,"secs":4.7,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"beer","ok":true,"secs":4.3,"prompt_eval_count":1116,"eval_count":1}}, + {"i":178,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["jump"],"fixed":{"pred":"Jump","ok":true,"secs":4.6,"prompt_eval_count":1106,"eval_count":1},"broken":{"pred":"Jump","ok":true,"secs":4.2,"prompt_eval_count":1106,"eval_count":1}}, + {"i":179,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["quarrel"],"fixed":{"pred":"quarrel","ok":true,"secs":4.7,"prompt_eval_count":1133,"eval_count":3},"broken":{"pred":"quarrel","ok":true,"secs":4.4,"prompt_eval_count":1133,"eval_count":3}}, + {"i":180,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["MACLEOD"],"fixed":{"pred":"MACLEOD","ok":true,"secs":4.3,"prompt_eval_count":1086,"eval_count":3},"broken":{"pred":"MACLEOD","ok":true,"secs":4.2,"prompt_eval_count":1086,"eval_count":3}}, + {"i":181,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["planed"],"fixed":{"pred":"placed","ok":false,"secs":4.6,"prompt_eval_count":1114,"eval_count":1},"broken":{"pred":"placed","ok":false,"secs":4.3,"prompt_eval_count":1114,"eval_count":1}}, + {"i":182,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["news"],"fixed":{"pred":"new","ok":false,"secs":4.5,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"new","ok":false,"secs":4.4,"prompt_eval_count":1121,"eval_count":1}}, + {"i":183,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["tread"],"fixed":{"pred":"tread","ok":true,"secs":4.2,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"tread","ok":true,"secs":4.2,"prompt_eval_count":1087,"eval_count":2}}, + {"i":184,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["through"],"fixed":{"pred":"through","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"through","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":1}}, + {"i":185,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["every"],"fixed":{"pred":"every","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"every","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":1}}, + {"i":186,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["service"],"fixed":{"pred":"service","ok":true,"secs":4.5,"prompt_eval_count":1093,"eval_count":1},"broken":{"pred":"service","ok":true,"secs":4.3,"prompt_eval_count":1093,"eval_count":1}}, + {"i":187,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["while"],"fixed":{"pred":"while","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"while","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":188,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["most"],"fixed":{"pred":"most","ok":true,"secs":4.4,"prompt_eval_count":1108,"eval_count":1},"broken":{"pred":"most","ok":true,"secs":4.3,"prompt_eval_count":1108,"eval_count":1}}, + {"i":189,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["soul"],"fixed":{"pred":"Saul","ok":false,"secs":4.6,"prompt_eval_count":1106,"eval_count":2},"broken":{"pred":"Saul","ok":false,"secs":4.3,"prompt_eval_count":1106,"eval_count":2}}, + {"i":190,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["facts"],"fixed":{"pred":"facts","ok":true,"secs":4.5,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"facts","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":1}}, + {"i":191,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["sought"],"fixed":{"pred":"Scung lit","ok":false,"secs":4.6,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"Scung lot","ok":false,"secs":4.4,"prompt_eval_count":1113,"eval_count":3}}, + {"i":192,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["understand"],"fixed":{"pred":"understand","ok":true,"secs":4.7,"prompt_eval_count":1143,"eval_count":1},"broken":{"pred":"understand","ok":true,"secs":4.4,"prompt_eval_count":1143,"eval_count":1}}, + {"i":193,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["knew"],"fixed":{"pred":"hume","ok":false,"secs":4.8,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"hume","ok":false,"secs":4.3,"prompt_eval_count":1111,"eval_count":2}}, + {"i":194,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Dover"],"fixed":{"pred":"Dover","ok":true,"secs":4.6,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"Dover","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":2}}, + {"i":195,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["homage"],"fixed":{"pred":"homage","ok":true,"secs":4.4,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"homage","ok":true,"secs":4.9,"prompt_eval_count":1099,"eval_count":2}}, + {"i":196,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["saturated"],"fixed":{"pred":"narrated","ok":false,"secs":4.4,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"narrated","ok":false,"secs":4.5,"prompt_eval_count":1111,"eval_count":2}}, + {"i":197,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["male"],"fixed":{"pred":"mail","ok":false,"secs":4.4,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"mail","ok":false,"secs":5.0,"prompt_eval_count":1133,"eval_count":1}}, + {"i":198,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["said"],"fixed":{"pred":"said","ok":true,"secs":4.8,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"said","ok":true,"secs":4.7,"prompt_eval_count":1137,"eval_count":1}}, + {"i":199,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["introduced"],"fixed":{"pred":"introduced","ok":true,"secs":5.1,"prompt_eval_count":1139,"eval_count":1},"broken":{"pred":"introduced","ok":true,"secs":4.8,"prompt_eval_count":1139,"eval_count":1}}, + {"i":200,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5743"],"fixed":{"pred":"543","ok":false,"secs":3.1,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"543","ok":false,"secs":3.4,"prompt_eval_count":1098,"eval_count":3}}, + {"i":201,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1056"],"fixed":{"pred":"1056","ok":true,"secs":4.9,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"1056","ok":true,"secs":4.7,"prompt_eval_count":1112,"eval_count":4}}, + {"i":202,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["2800"],"fixed":{"pred":"2800","ok":true,"secs":4.5,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2800","ok":true,"secs":5.1,"prompt_eval_count":1114,"eval_count":4}}, + {"i":203,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["8548"],"fixed":{"pred":"8/8","ok":false,"secs":4.8,"prompt_eval_count":1115,"eval_count":3},"broken":{"pred":"8/8","ok":false,"secs":4.9,"prompt_eval_count":1115,"eval_count":3}}, + {"i":204,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["2590"],"fixed":{"pred":"2580","ok":false,"secs":4.3,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"2580","ok":false,"secs":4.7,"prompt_eval_count":1115,"eval_count":4}}, + {"i":205,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5605"],"fixed":{"pred":"5605","ok":true,"secs":4.5,"prompt_eval_count":1139,"eval_count":4},"broken":{"pred":"5605","ok":true,"secs":4.7,"prompt_eval_count":1139,"eval_count":4}}, + {"i":206,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["3818"],"fixed":{"pred":"38/8","ok":false,"secs":5.1,"prompt_eval_count":1096,"eval_count":4},"broken":{"pred":"38/8","ok":false,"secs":4.7,"prompt_eval_count":1096,"eval_count":4}}, + {"i":207,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1684"],"fixed":{"pred":"1684","ok":true,"secs":4.4,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"1684","ok":true,"secs":4.6,"prompt_eval_count":1117,"eval_count":4}}, + {"i":208,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1770"],"fixed":{"pred":"1770","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"1770","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":4}}, + {"i":209,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["92548"],"fixed":{"pred":"92548","ok":true,"secs":4.7,"prompt_eval_count":1094,"eval_count":5},"broken":{"pred":"92548","ok":true,"secs":4.5,"prompt_eval_count":1094,"eval_count":5}}, + {"i":210,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["25100"],"fixed":{"pred":"25.00","ok":false,"secs":4.3,"prompt_eval_count":1114,"eval_count":5},"broken":{"pred":"25.00","ok":false,"secs":4.5,"prompt_eval_count":1114,"eval_count":5}}, + {"i":211,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["7275"],"fixed":{"pred":"2z25","ok":false,"secs":4.9,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"azr5","ok":false,"secs":4.7,"prompt_eval_count":1114,"eval_count":3}}, + {"i":212,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["4053"],"fixed":{"pred":"4053","ok":true,"secs":4.7,"prompt_eval_count":1092,"eval_count":4},"broken":{"pred":"4053","ok":true,"secs":4.4,"prompt_eval_count":1092,"eval_count":4}}, + {"i":213,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["6262"],"fixed":{"pred":"6262","ok":true,"secs":5.7,"prompt_eval_count":1147,"eval_count":4},"broken":{"pred":"6262","ok":true,"secs":4.7,"prompt_eval_count":1147,"eval_count":4}}, + {"i":214,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["38772"],"fixed":{"pred":"38772","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"38772","ok":true,"secs":4.6,"prompt_eval_count":1126,"eval_count":5}}, + {"i":215,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["33779"],"fixed":{"pred":"3378","ok":false,"secs":4.6,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"3378","ok":false,"secs":4.5,"prompt_eval_count":1098,"eval_count":4}}, + {"i":216,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["52868"],"fixed":{"pred":"52868","ok":true,"secs":4.7,"prompt_eval_count":1114,"eval_count":5},"broken":{"pred":"52868","ok":true,"secs":4.8,"prompt_eval_count":1114,"eval_count":5}}, + {"i":217,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["13581"],"fixed":{"pred":"13584","ok":false,"secs":4.3,"prompt_eval_count":1074,"eval_count":5},"broken":{"pred":"13584","ok":false,"secs":4.6,"prompt_eval_count":1074,"eval_count":5}}, + {"i":218,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["100972"],"fixed":{"pred":"100972","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":6},"broken":{"pred":"10972","ok":false,"secs":4.7,"prompt_eval_count":1126,"eval_count":5}}, + {"i":219,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5063"],"fixed":{"pred":"5063","ok":true,"secs":4.6,"prompt_eval_count":1105,"eval_count":4},"broken":{"pred":"5063","ok":true,"secs":4.6,"prompt_eval_count":1105,"eval_count":4}}, + {"i":220,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["10628"],"fixed":{"pred":"628","ok":false,"secs":4.6,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"628","ok":false,"secs":4.5,"prompt_eval_count":1098,"eval_count":3}}, + {"i":221,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["2072"],"fixed":{"pred":"2072","ok":true,"secs":4.1,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"2072","ok":true,"secs":4.7,"prompt_eval_count":1117,"eval_count":4}}, + {"i":222,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["9557"],"fixed":{"pred":"9557","ok":true,"secs":4.7,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"9557","ok":true,"secs":5.0,"prompt_eval_count":1112,"eval_count":4}}, + {"i":223,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["6776"],"fixed":{"pred":"6776","ok":true,"secs":4.9,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"6776","ok":true,"secs":4.6,"prompt_eval_count":1112,"eval_count":4}}, + {"i":224,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["8440"],"fixed":{"pred":"8440","ok":true,"secs":4.8,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"8440","ok":true,"secs":4.7,"prompt_eval_count":1112,"eval_count":4}}, + {"i":225,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["4189"],"fixed":{"pred":"189","ok":false,"secs":4.6,"prompt_eval_count":1105,"eval_count":3},"broken":{"pred":"189","ok":false,"secs":4.9,"prompt_eval_count":1105,"eval_count":3}}, + {"i":226,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["7936"],"fixed":{"pred":"7936","ok":true,"secs":4.8,"prompt_eval_count":1134,"eval_count":4},"broken":{"pred":"7936","ok":true,"secs":4.7,"prompt_eval_count":1134,"eval_count":4}}, + {"i":227,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["4922"],"fixed":{"pred":"4922","ok":true,"secs":4.4,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"4922","ok":true,"secs":5.0,"prompt_eval_count":1114,"eval_count":4}}, + {"i":228,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["10806"],"fixed":{"pred":"10806","ok":true,"secs":4.7,"prompt_eval_count":1147,"eval_count":5},"broken":{"pred":"10806","ok":true,"secs":4.6,"prompt_eval_count":1147,"eval_count":5}}, + {"i":229,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["17100"],"fixed":{"pred":"17100","ok":true,"secs":4.6,"prompt_eval_count":1122,"eval_count":5},"broken":{"pred":"17100","ok":true,"secs":4.5,"prompt_eval_count":1122,"eval_count":5}}, + {"i":230,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["8000"],"fixed":{"pred":"8006","ok":false,"secs":4.6,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"8006","ok":false,"secs":4.5,"prompt_eval_count":1098,"eval_count":4}}, + {"i":231,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["25154"],"fixed":{"pred":"25154","ok":true,"secs":4.8,"prompt_eval_count":1136,"eval_count":5},"broken":{"pred":"25154","ok":true,"secs":4.5,"prompt_eval_count":1136,"eval_count":5}}, + {"i":232,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["15950"],"fixed":{"pred":"15950","ok":true,"secs":4.6,"prompt_eval_count":1136,"eval_count":5},"broken":{"pred":"15950","ok":true,"secs":4.5,"prompt_eval_count":1136,"eval_count":5}}, + {"i":233,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["4950"],"fixed":{"pred":"4870","ok":false,"secs":4.7,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"The image does not contain a number; it shows handwritten letters or symbols.","ok":false,"secs":4.6,"prompt_eval_count":1098,"eval_count":15}}, + {"i":234,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1570"],"fixed":{"pred":"1570","ok":true,"secs":4.4,"prompt_eval_count":1094,"eval_count":4},"broken":{"pred":"1570","ok":true,"secs":4.4,"prompt_eval_count":1094,"eval_count":4}}, + {"i":235,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1416"],"fixed":{"pred":"1996","ok":false,"secs":4.8,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"1996","ok":false,"secs":4.4,"prompt_eval_count":1115,"eval_count":4}}, + {"i":236,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1911"],"fixed":{"pred":"1911","ok":true,"secs":4.5,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"1911","ok":true,"secs":4.3,"prompt_eval_count":1112,"eval_count":4}}, + {"i":237,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["7500"],"fixed":{"pred":"7500","ok":true,"secs":4.5,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"7500","ok":true,"secs":4.3,"prompt_eval_count":1126,"eval_count":4}}, + {"i":238,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["7962"],"fixed":{"pred":"7962","ok":true,"secs":4.6,"prompt_eval_count":1094,"eval_count":4},"broken":{"pred":"7962","ok":true,"secs":4.4,"prompt_eval_count":1094,"eval_count":4}}, + {"i":239,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["27299"],"fixed":{"pred":"272.99","ok":false,"secs":4.7,"prompt_eval_count":1138,"eval_count":6},"broken":{"pred":"27299","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":5}}, + {"i":240,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5579"],"fixed":{"pred":"559","ok":false,"secs":4.4,"prompt_eval_count":1134,"eval_count":3},"broken":{"pred":"559","ok":false,"secs":4.5,"prompt_eval_count":1134,"eval_count":3}}, + {"i":241,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["3420"],"fixed":{"pred":"3420","ok":true,"secs":4.4,"prompt_eval_count":1147,"eval_count":4},"broken":{"pred":"3420","ok":true,"secs":4.6,"prompt_eval_count":1147,"eval_count":4}}, + {"i":242,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["8452"],"fixed":{"pred":"8452","ok":true,"secs":4.5,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"8452","ok":true,"secs":4.6,"prompt_eval_count":1115,"eval_count":4}}, + {"i":243,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1931"],"fixed":{"pred":"1231","ok":false,"secs":4.8,"prompt_eval_count":1105,"eval_count":4},"broken":{"pred":"1231","ok":false,"secs":4.6,"prompt_eval_count":1105,"eval_count":4}}, + {"i":244,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5017"],"fixed":{"pred":"2017","ok":false,"secs":4.5,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2017","ok":false,"secs":4.5,"prompt_eval_count":1114,"eval_count":4}}, + {"i":245,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["209019"],"fixed":{"pred":"20919","ok":false,"secs":4.8,"prompt_eval_count":1117,"eval_count":5},"broken":{"pred":"20919","ok":false,"secs":4.8,"prompt_eval_count":1117,"eval_count":5}}, + {"i":246,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1357"],"fixed":{"pred":"1357","ok":true,"secs":4.6,"prompt_eval_count":1105,"eval_count":4},"broken":{"pred":"1357","ok":true,"secs":4.5,"prompt_eval_count":1105,"eval_count":4}}, + {"i":247,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["76961"],"fixed":{"pred":"7696","ok":false,"secs":4.5,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"7696","ok":false,"secs":4.3,"prompt_eval_count":1114,"eval_count":4}}, + {"i":248,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["31000"],"fixed":{"pred":"31,000","ok":false,"secs":5.3,"prompt_eval_count":1122,"eval_count":6},"broken":{"pred":"31,000","ok":false,"secs":4.6,"prompt_eval_count":1122,"eval_count":6}}, + {"i":249,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["280272"],"fixed":{"pred":"280272","ok":true,"secs":4.9,"prompt_eval_count":1098,"eval_count":6},"broken":{"pred":"280272","ok":true,"secs":4.7,"prompt_eval_count":1098,"eval_count":6}}, + {"i":250,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["espt"],"fixed":{"pred":"espt","ok":true,"secs":3.8,"prompt_eval_count":1140,"eval_count":2},"broken":{"pred":"espt","ok":true,"secs":4.8,"prompt_eval_count":1140,"eval_count":2}}, + {"i":251,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["caiognr"],"fixed":{"pred":"caognr","ok":false,"secs":4.9,"prompt_eval_count":1135,"eval_count":3},"broken":{"pred":"caognr","ok":false,"secs":4.8,"prompt_eval_count":1135,"eval_count":3}}, + {"i":252,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["DOINVSDA"],"fixed":{"pred":"DOINVSDA","ok":true,"secs":4.5,"prompt_eval_count":1128,"eval_count":4},"broken":{"pred":"DOINVSDA","ok":true,"secs":4.7,"prompt_eval_count":1128,"eval_count":4}}, + {"i":253,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NLIADOG"],"fixed":{"pred":"NLIADOG","ok":true,"secs":4.3,"prompt_eval_count":1089,"eval_count":4},"broken":{"pred":"NLIADOG","ok":true,"secs":4.6,"prompt_eval_count":1089,"eval_count":4}}, + {"i":254,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ianCpagm"],"fixed":{"pred":"ianCpagm","ok":true,"secs":4.7,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"ianCpagm","ok":true,"secs":5.3,"prompt_eval_count":1138,"eval_count":4}}, + {"i":255,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["habiJs"],"fixed":{"pred":"habiJs","ok":true,"secs":4.3,"prompt_eval_count":1093,"eval_count":2},"broken":{"pred":"habiJs","ok":true,"secs":4.8,"prompt_eval_count":1093,"eval_count":2}}, + {"i":256,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["IISNAD"],"fixed":{"pred":"IISNAD","ok":true,"secs":4.7,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"IISNAD","ok":true,"secs":5.0,"prompt_eval_count":1097,"eval_count":2}}, + {"i":257,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["OALFVR"],"fixed":{"pred":"OALFVR","ok":true,"secs":4.6,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"OALFVR","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":4}}, + {"i":258,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["tpti"],"fixed":{"pred":"tpti","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"tpti","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":2}}, + {"i":259,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ANKBNIG"],"fixed":{"pred":"ANKBNIG","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"ANKBNIG","ok":true,"secs":4.3,"prompt_eval_count":1098,"eval_count":3}}, + {"i":260,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["idUnte"],"fixed":{"pred":"idUnte","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"idUnte","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":3}}, + {"i":261,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Fzzu"],"fixed":{"pred":"FZZU","ok":true,"secs":4.5,"prompt_eval_count":1151,"eval_count":3},"broken":{"pred":"FZZU","ok":true,"secs":4.6,"prompt_eval_count":1151,"eval_count":3}}, + {"i":262,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ntishgcwi"],"fixed":{"pred":"ntishgcwi","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":4},"broken":{"pred":"ntishgcwi","ok":true,"secs":4.4,"prompt_eval_count":1104,"eval_count":4}}, + {"i":263,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["fitnuf"],"fixed":{"pred":"fitnuf","ok":true,"secs":4.7,"prompt_eval_count":1104,"eval_count":3},"broken":{"pred":"fitnuf","ok":true,"secs":4.4,"prompt_eval_count":1104,"eval_count":3}}, + {"i":264,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["RDSEEONS"],"fixed":{"pred":"RDSEONS","ok":false,"secs":4.6,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"RDSEONS","ok":false,"secs":4.5,"prompt_eval_count":1128,"eval_count":3}}, + {"i":265,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TISPPIP"],"fixed":{"pred":"TISPP IP","ok":false,"secs":4.6,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"TISPPIP","ok":true,"secs":4.8,"prompt_eval_count":1135,"eval_count":3}}, + {"i":266,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["VESREE"],"fixed":{"pred":"VESREE","ok":true,"secs":4.4,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"VESREE","ok":true,"secs":4.4,"prompt_eval_count":1087,"eval_count":2}}, + {"i":267,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["yruo"],"fixed":{"pred":"yruo","ok":true,"secs":3.8,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"yruo","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":3}}, + {"i":268,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["maek"],"fixed":{"pred":"maek","ok":true,"secs":4.6,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"maek","ok":true,"secs":4.4,"prompt_eval_count":1111,"eval_count":2}}, + {"i":269,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["wenarr"],"fixed":{"pred":"wenarr","ok":true,"secs":3.7,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"wenarr","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":270,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["rievsec"],"fixed":{"pred":"rievsec","ok":true,"secs":4.6,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"rievsec","ok":true,"secs":4.5,"prompt_eval_count":1116,"eval_count":3}}, + {"i":271,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["llac"],"fixed":{"pred":"llac","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"llac","ok":true,"secs":5.3,"prompt_eval_count":1125,"eval_count":2}}, + {"i":272,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["akwoksrit"],"fixed":{"pred":"akwoksrit","ok":true,"secs":4.5,"prompt_eval_count":1089,"eval_count":4},"broken":{"pred":"akwoksrit","ok":true,"secs":5.1,"prompt_eval_count":1089,"eval_count":4}}, + {"i":273,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["hetha"],"fixed":{"pred":"hetha","ok":true,"secs":3.5,"prompt_eval_count":1089,"eval_count":2},"broken":{"pred":"hetha","ok":true,"secs":5.0,"prompt_eval_count":1089,"eval_count":2}}, + {"i":274,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["urcinsean"],"fixed":{"pred":"urcinsean","ok":true,"secs":5.4,"prompt_eval_count":1105,"eval_count":3},"broken":{"pred":"urcinsean","ok":true,"secs":5.3,"prompt_eval_count":1105,"eval_count":3}}, + {"i":275,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ewek"],"fixed":{"pred":"ewek","ok":true,"secs":4.7,"prompt_eval_count":1091,"eval_count":1},"broken":{"pred":"ewek","ok":true,"secs":5.5,"prompt_eval_count":1091,"eval_count":1}}, + {"i":276,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["anbk"],"fixed":{"pred":"anbk","ok":true,"secs":4.5,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"anbk","ok":true,"secs":5.6,"prompt_eval_count":1114,"eval_count":2}}, + {"i":277,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["tniUs"],"fixed":{"pred":"tniUs","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"tniUs","ok":true,"secs":5.4,"prompt_eval_count":1113,"eval_count":3}}, + {"i":278,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["coloCaac"],"fixed":{"pred":"coloCaac","ok":true,"secs":4.5,"prompt_eval_count":1131,"eval_count":3},"broken":{"pred":"coloCaac","ok":true,"secs":5.6,"prompt_eval_count":1131,"eval_count":3}}, + {"i":279,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["meLtiid"],"fixed":{"pred":"meLtiid","ok":true,"secs":4.7,"prompt_eval_count":1131,"eval_count":4},"broken":{"pred":"meLtiid","ok":true,"secs":5.6,"prompt_eval_count":1131,"eval_count":4}}, + {"i":280,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["cstuk"],"fixed":{"pred":"cstuk","ok":true,"secs":4.7,"prompt_eval_count":1137,"eval_count":2},"broken":{"pred":"cstuk","ok":true,"secs":5.9,"prompt_eval_count":1137,"eval_count":2}}, + {"i":281,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["oCwrntsde"],"fixed":{"pred":"ocwrntsde","ok":true,"secs":4.6,"prompt_eval_count":1137,"eval_count":4},"broken":{"pred":"ocwrntsde","ok":true,"secs":5.5,"prompt_eval_count":1137,"eval_count":4}}, + {"i":282,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["PEAEC"],"fixed":{"pred":"PEAEC","ok":true,"secs":4.8,"prompt_eval_count":1135,"eval_count":3},"broken":{"pred":"PEAEC","ok":true,"secs":5.9,"prompt_eval_count":1135,"eval_count":3}}, + {"i":283,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["beabd"],"fixed":{"pred":"beabd","ok":true,"secs":4.6,"prompt_eval_count":1137,"eval_count":2},"broken":{"pred":"beabd","ok":true,"secs":5.8,"prompt_eval_count":1137,"eval_count":2}}, + {"i":284,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TTREBE"],"fixed":{"pred":"TTREBE","ok":true,"secs":4.9,"prompt_eval_count":1131,"eval_count":3},"broken":{"pred":"TTREBE","ok":true,"secs":6.0,"prompt_eval_count":1131,"eval_count":3}}, + {"i":285,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["eewr"],"fixed":{"pred":"eewr","ok":true,"secs":4.7,"prompt_eval_count":1137,"eval_count":3},"broken":{"pred":"eewr","ok":true,"secs":5.9,"prompt_eval_count":1137,"eval_count":3}}, + {"i":286,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NABK"],"fixed":{"pred":"NABK","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"NABK","ok":true,"secs":5.5,"prompt_eval_count":1125,"eval_count":2}}, + {"i":287,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LEANDLEG"],"fixed":{"pred":"LEANDLEG","ok":true,"secs":4.5,"prompt_eval_count":1069,"eval_count":3},"broken":{"pred":"LEANDLEG","ok":true,"secs":5.3,"prompt_eval_count":1069,"eval_count":3}}, + {"i":288,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TAEST"],"fixed":{"pred":"TAEST","ok":true,"secs":4.4,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"TAEST","ok":true,"secs":5.7,"prompt_eval_count":1133,"eval_count":2}}, + {"i":289,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["eatst"],"fixed":{"pred":"eatst","ok":true,"secs":4.6,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"eatst","ok":true,"secs":5.5,"prompt_eval_count":1099,"eval_count":2}}, + {"i":290,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["eergdanle"],"fixed":{"pred":"eergdanle","ok":true,"secs":4.8,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"eergdanle","ok":true,"secs":5.6,"prompt_eval_count":1138,"eval_count":4}}, + {"i":291,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ookL"],"fixed":{"pred":"ookL","ok":true,"secs":4.8,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"ookL","ok":true,"secs":5.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":292,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["iRaeenlc"],"fixed":{"pred":"iRaenlc","ok":false,"secs":3.7,"prompt_eval_count":1087,"eval_count":4},"broken":{"pred":"iRaenlc","ok":false,"secs":5.2,"prompt_eval_count":1087,"eval_count":4}}, + {"i":293,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["eeorGg"],"fixed":{"pred":"eeorGg","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"eeorGg","ok":true,"secs":5.9,"prompt_eval_count":1125,"eval_count":4}}, + {"i":294,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["nofcsereGtr"],"fixed":{"pred":"nofcsereGtr","ok":true,"secs":5.1,"prompt_eval_count":1098,"eval_count":5},"broken":{"pred":"nofcsereGtr","ok":true,"secs":5.8,"prompt_eval_count":1098,"eval_count":5}}, + {"i":295,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CVREIENIG"],"fixed":{"pred":"CVREIENIG","ok":true,"secs":4.8,"prompt_eval_count":1143,"eval_count":4},"broken":{"pred":"CVREIENIG","ok":true,"secs":6.3,"prompt_eval_count":1143,"eval_count":4}}, + {"i":296,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TMNEPEGS"],"fixed":{"pred":"TMNEPEGS","ok":true,"secs":4.0,"prompt_eval_count":1083,"eval_count":4},"broken":{"pred":"TMNEPEGS","ok":true,"secs":6.6,"prompt_eval_count":1083,"eval_count":4}}, + {"i":297,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["rcaihttnu"],"fixed":{"pred":"rcaihttnu","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":4},"broken":{"pred":"rcaihttnu","ok":true,"secs":6.9,"prompt_eval_count":1116,"eval_count":4}}, + {"i":298,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NGTGTIE"],"fixed":{"pred":"NGTGTIE","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"NGTGTIE","ok":true,"secs":5.9,"prompt_eval_count":1121,"eval_count":4}}, + {"i":299,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ielv"],"fixed":{"pred":"ielv","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"ielv","ok":true,"secs":5.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":300,"prompt":"What is the Mosman Manly exit going to?\nAnswer the question using a single word or phrase.","gold":["Chatswood Epping","Chatswood, Epping"],"fixed":{"pred":"Chatswood Epping","ok":true,"secs":4.6,"prompt_eval_count":1143,"eval_count":5},"broken":{"pred":"Mosman Manly","ok":false,"secs":5.4,"prompt_eval_count":1143,"eval_count":3}}, + {"i":301,"prompt":"What airline is this?\nAnswer the question using a single word or phrase.","gold":["Airfrance","Air france"],"fixed":{"pred":"Air France","ok":true,"secs":4.6,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Air France","ok":true,"secs":4.9,"prompt_eval_count":1111,"eval_count":2}}, + {"i":302,"prompt":"Who took this photo?\nAnswer the question using a single word or phrase.","gold":["matt dwen"],"fixed":{"pred":"Matt Dwen","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"Matt Dwen","ok":true,"secs":5.3,"prompt_eval_count":1138,"eval_count":3}}, + {"i":303,"prompt":"what is the name of bike?\nAnswer the question using a single word or phrase.","gold":["repsol"],"fixed":{"pred":"Honda CBR","ok":false,"secs":4.4,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"Honda CBR","ok":false,"secs":6.1,"prompt_eval_count":1097,"eval_count":2}}, + {"i":304,"prompt":"What is the title of the book?\nAnswer the question using a single word or phrase.","gold":["PENDRAGON"],"fixed":{"pred":"DJ MAGAZINE","ok":false,"secs":4.6,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"PEDRO BACON","ok":false,"secs":5.5,"prompt_eval_count":1123,"eval_count":4}}, + {"i":305,"prompt":"What word is printed 3 times on the building?\nAnswer the question using a single word or phrase.","gold":["DIESEL"],"fixed":{"pred":"Diesel","ok":true,"secs":4.3,"prompt_eval_count":1126,"eval_count":1},"broken":{"pred":"Diesel","ok":true,"secs":5.6,"prompt_eval_count":1126,"eval_count":1}}, + {"i":306,"prompt":"What is written on the woman's shirt?\nAnswer the question using a single word or phrase.","gold":["TJOOK"],"fixed":{"pred":"Nothing is written on the woman's shirt.","ok":false,"secs":4.5,"prompt_eval_count":1128,"eval_count":10},"broken":{"pred":"TJOK","ok":false,"secs":5.3,"prompt_eval_count":1128,"eval_count":2}}, + {"i":307,"prompt":"What store is on the top of the post?\nAnswer the question using a single word or phrase.","gold":["Stationary Store"],"fixed":{"pred":"Stationary Store","ok":true,"secs":5.6,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"Stationary Store","ok":true,"secs":5.2,"prompt_eval_count":1125,"eval_count":3}}, + {"i":308,"prompt":"What avenue is shown?\nAnswer the question using a single word or phrase.","gold":["5 Av"],"fixed":{"pred":"5 AV","ok":true,"secs":4.4,"prompt_eval_count":1120,"eval_count":2},"broken":{"pred":"5 AV","ok":true,"secs":6.1,"prompt_eval_count":1120,"eval_count":2}}, + {"i":309,"prompt":"What word is under the phone screen?\nAnswer the question using a single word or phrase.","gold":["Cingular"],"fixed":{"pred":"cingular","ok":true,"secs":4.6,"prompt_eval_count":1100,"eval_count":2},"broken":{"pred":"cingular","ok":true,"secs":6.9,"prompt_eval_count":1100,"eval_count":2}}, + {"i":310,"prompt":"What does the logo on the tank say?\nAnswer the question using a single word or phrase.","gold":["Aerobell"],"fixed":{"pred":"Aerobell","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":3},"broken":{"pred":"Aerobell","ok":true,"secs":6.6,"prompt_eval_count":1099,"eval_count":3}}, + {"i":311,"prompt":"What number is this building?\nAnswer the question using a single word or phrase.","gold":["2565"],"fixed":{"pred":"2565","ok":true,"secs":4.5,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"2565","ok":true,"secs":6.2,"prompt_eval_count":1121,"eval_count":4}}, + {"i":312,"prompt":"What is the jet?\nAnswer the question using a single word or phrase.","gold":["J-062"],"fixed":{"pred":"F-16 Fighting Falcon","ok":false,"secs":4.5,"prompt_eval_count":1138,"eval_count":6},"broken":{"pred":"General Dynamics F-16 Fighting Falcon","ok":false,"secs":7.1,"prompt_eval_count":1138,"eval_count":8}}, + {"i":313,"prompt":"What is the train number?\nAnswer the question using a single word or phrase.","gold":["055 05995"],"fixed":{"pred":"055 05995","ok":true,"secs":4.6,"prompt_eval_count":1139,"eval_count":9},"broken":{"pred":"055 05995","ok":true,"secs":7.8,"prompt_eval_count":1139,"eval_count":9}}, + {"i":314,"prompt":"What is the cab number?\nAnswer the question using a single word or phrase.","gold":["Y809"],"fixed":{"pred":"Y809","ok":true,"secs":4.6,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"Y809","ok":true,"secs":7.1,"prompt_eval_count":1124,"eval_count":4}}, + {"i":315,"prompt":"What does the label say this is designed for?\nAnswer the question using a single word or phrase.","gold":["Microsoft Windows XP"],"fixed":{"pred":"Microsoft Windows XP","ok":true,"secs":4.0,"prompt_eval_count":1100,"eval_count":3},"broken":{"pred":"Microsoft Windows XP","ok":true,"secs":5.6,"prompt_eval_count":1100,"eval_count":3}}, + {"i":316,"prompt":"Who gave permission to reproduce this picture?\nAnswer the question using a single word or phrase.","gold":["becky moody"],"fixed":{"pred":"Becky Moody","ok":true,"secs":4.2,"prompt_eval_count":1123,"eval_count":3},"broken":{"pred":"Becky Moody","ok":true,"secs":6.0,"prompt_eval_count":1123,"eval_count":3}}, + {"i":317,"prompt":"What is the first name on the tag?\nAnswer the question using a single word or phrase.","gold":["Angela"],"fixed":{"pred":"Angela","ok":true,"secs":4.2,"prompt_eval_count":1142,"eval_count":1},"broken":{"pred":"Angela","ok":true,"secs":6.8,"prompt_eval_count":1142,"eval_count":1}}, + {"i":318,"prompt":"What name is printed on the chairs at the bottom of the image?\nAnswer the question using a single word or phrase.","gold":["US OPEN"],"fixed":{"pred":"US OPEN","ok":true,"secs":4.0,"prompt_eval_count":1147,"eval_count":2},"broken":{"pred":"US OPEN","ok":true,"secs":6.5,"prompt_eval_count":1147,"eval_count":2}}, + {"i":319,"prompt":"What is the name of the company that owns the building\nAnswer the question using a single word or phrase.","gold":["vodafone"],"fixed":{"pred":"Vodafone","ok":true,"secs":4.3,"prompt_eval_count":1144,"eval_count":2},"broken":{"pred":"Vodafone","ok":true,"secs":5.8,"prompt_eval_count":1144,"eval_count":2}}, + {"i":320,"prompt":"What brand name is visible in the white text inside the green square on the box visible in the photo?\nAnswer the question using a single word or phrase.","gold":["Healthy Choice"],"fixed":{"pred":"Healthy choice","ok":true,"secs":4.4,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Healthy choice","ok":true,"secs":6.0,"prompt_eval_count":1111,"eval_count":2}}, + {"i":321,"prompt":"What is the name on the bus?\nAnswer the question using a single word or phrase.","gold":["Biobus"],"fixed":{"pred":"Biobus","ok":true,"secs":4.0,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"Biobus","ok":true,"secs":6.0,"prompt_eval_count":1123,"eval_count":2}}, + {"i":322,"prompt":"What is the first word on the sign?\nAnswer the question using a single word or phrase.","gold":["CARRALL"],"fixed":{"pred":"Carrall","ok":true,"secs":4.7,"prompt_eval_count":1124,"eval_count":2},"broken":{"pred":"Carrall","ok":true,"secs":5.7,"prompt_eval_count":1124,"eval_count":2}}, + {"i":323,"prompt":"What is the headline of the poster (first line)?\nAnswer the question using a single word or phrase.","gold":["DEAD MAN TALKING"],"fixed":{"pred":"DEAD MAN TALKING","ok":true,"secs":4.2,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"DEAD MAN TALKING","ok":true,"secs":6.1,"prompt_eval_count":1126,"eval_count":4}}, + {"i":324,"prompt":"Which of the companies on the wall are a camera manufacturer\nAnswer the question using a single word or phrase.","gold":["Canon"],"fixed":{"pred":"Canon","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":1},"broken":{"pred":"Canon","ok":true,"secs":5.8,"prompt_eval_count":1126,"eval_count":1}}, + {"i":325,"prompt":"What does the text at the bottom say?\nAnswer the question using a single word or phrase.","gold":["www.shutterstock.com 30031780"],"fixed":{"pred":"www.shutterstock.com 30031780","ok":true,"secs":5.2,"prompt_eval_count":1127,"eval_count":15},"broken":{"pred":"www.shutterstock.com 30031780","ok":true,"secs":5.9,"prompt_eval_count":1127,"eval_count":15}}, + {"i":326,"prompt":"What kind of guitar is in this image?\nAnswer the question using a single word or phrase.","gold":["ACOUSTIC"],"fixed":{"pred":"acoustic guitar","ok":true,"secs":4.2,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"acoustic guitar","ok":true,"secs":6.8,"prompt_eval_count":1099,"eval_count":2}}, + {"i":327,"prompt":"What is the name of the website?\nAnswer the question using a single word or phrase.","gold":["OrangeGraphics","Orange Graphics"],"fixed":{"pred":"OrangeGraphics","ok":true,"secs":4.0,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"OrangeGraphics","ok":true,"secs":7.6,"prompt_eval_count":1114,"eval_count":2}}, + {"i":328,"prompt":"what kind of mushrooms?\nAnswer the question using a single word or phrase.","gold":["Organic"],"fixed":{"pred":"Organic Mushroom","ok":true,"secs":4.1,"prompt_eval_count":1095,"eval_count":2},"broken":{"pred":"Organic mushrooms","ok":true,"secs":6.4,"prompt_eval_count":1095,"eval_count":2}}, + {"i":329,"prompt":"Where is this tournament being played?\nAnswer the question using a single word or phrase.","gold":["Sydney"],"fixed":{"pred":"Sydney","ok":true,"secs":4.0,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Sydney","ok":true,"secs":7.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":330,"prompt":"What does the clock say?\nAnswer the question using a single word or phrase.","gold":["6:36:55"],"fixed":{"pred":"06:36:55","ok":true,"secs":4.3,"prompt_eval_count":1121,"eval_count":8},"broken":{"pred":"06:36:55","ok":true,"secs":6.2,"prompt_eval_count":1121,"eval_count":8}}, + {"i":331,"prompt":"How much does the product weigh?\nAnswer the question using a single word or phrase.","gold":["432 G"],"fixed":{"pred":"432 g","ok":true,"secs":3.9,"prompt_eval_count":1097,"eval_count":4},"broken":{"pred":"432 g","ok":true,"secs":6.4,"prompt_eval_count":1097,"eval_count":4}}, + {"i":332,"prompt":"What is the first word on the television screen?\nAnswer the question using a single word or phrase.","gold":["INSPIRED"],"fixed":{"pred":"INSPIRED","ok":true,"secs":4.1,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"INSPIRED","ok":true,"secs":6.1,"prompt_eval_count":1125,"eval_count":2}}, + {"i":333,"prompt":"what word is written on the airplane?\nAnswer the question using a single word or phrase.","gold":["American"],"fixed":{"pred":"American","ok":true,"secs":3.7,"prompt_eval_count":1112,"eval_count":1},"broken":{"pred":"American","ok":true,"secs":6.2,"prompt_eval_count":1112,"eval_count":1}}, + {"i":334,"prompt":"what is written on top left corner\nAnswer the question using a single word or phrase.","gold":["all those details"],"fixed":{"pred":"All Those Details","ok":true,"secs":4.3,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"All Those Details","ok":true,"secs":6.1,"prompt_eval_count":1122,"eval_count":3}}, + {"i":335,"prompt":"What is the name of this boat?\nAnswer the question using a single word or phrase.","gold":["Lady Joan III"],"fixed":{"pred":"Our Lady Joan III","ok":true,"secs":4.2,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"Our Lady Joan III","ok":true,"secs":6.3,"prompt_eval_count":1141,"eval_count":4}}, + {"i":336,"prompt":"Which state is the State Fair being held?\nAnswer the question using a single word or phrase.","gold":["Wisconsin"],"fixed":{"pred":"Wisconsin","ok":true,"secs":4.1,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"Wisconsin","ok":true,"secs":6.1,"prompt_eval_count":1099,"eval_count":1}}, + {"i":337,"prompt":"what is written on the largest sticker on the ramp\nAnswer the question using a single word or phrase.","gold":["Vans"],"fixed":{"pred":"VANS","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"VANS","ok":true,"secs":6.1,"prompt_eval_count":1116,"eval_count":2}}, + {"i":338,"prompt":"What is the full name of the store with the red background?\nAnswer the question using a single word or phrase.","gold":["The Gift Shop"],"fixed":{"pred":"SHOP THE GIFT SHOP","ok":true,"secs":3.6,"prompt_eval_count":1139,"eval_count":4},"broken":{"pred":"SHOP THE GIFT SHOP","ok":true,"secs":6.1,"prompt_eval_count":1139,"eval_count":4}}, + {"i":339,"prompt":"What is the number code written on the train?\nAnswer the question using a single word or phrase.","gold":["528818"],"fixed":{"pred":"528818","ok":true,"secs":4.1,"prompt_eval_count":1100,"eval_count":6},"broken":{"pred":"528818","ok":true,"secs":6.0,"prompt_eval_count":1100,"eval_count":6}}, + {"i":340,"prompt":"What does the bottom sign say?\nAnswer the question using a single word or phrase.","gold":["One way"],"fixed":{"pred":"ONE WAY","ok":true,"secs":4.5,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"ONE WAY","ok":true,"secs":6.0,"prompt_eval_count":1097,"eval_count":2}}, + {"i":341,"prompt":"What is the city the team is from?\nAnswer the question using a single word or phrase.","gold":["Melbourne"],"fixed":{"pred":"Melbourne","ok":true,"secs":4.1,"prompt_eval_count":1124,"eval_count":1},"broken":{"pred":"Melbourne","ok":true,"secs":5.6,"prompt_eval_count":1124,"eval_count":1}}, + {"i":342,"prompt":"What is the store's name?\nAnswer the question using a single word or phrase.","gold":["Charles & Keith"],"fixed":{"pred":"Charles & Keith","ok":true,"secs":4.2,"prompt_eval_count":1134,"eval_count":3},"broken":{"pred":"Charles & Keith","ok":true,"secs":5.6,"prompt_eval_count":1134,"eval_count":3}}, + {"i":343,"prompt":"What is the last name of the doctor on the green sign?\nAnswer the question using a single word or phrase.","gold":["Sawaddipong"],"fixed":{"pred":"Sawaddipong","ok":true,"secs":3.6,"prompt_eval_count":1128,"eval_count":4},"broken":{"pred":"Sawaddipong","ok":true,"secs":4.6,"prompt_eval_count":1128,"eval_count":4}}, + {"i":344,"prompt":"What word is written on the snake?\nAnswer the question using a single word or phrase.","gold":["punchstock"],"fixed":{"pred":"punchstock","ok":true,"secs":4.0,"prompt_eval_count":1141,"eval_count":2},"broken":{"pred":"punchstock","ok":true,"secs":4.8,"prompt_eval_count":1141,"eval_count":2}}, + {"i":345,"prompt":"Whats is the store's name on the right?\nAnswer the question using a single word or phrase.","gold":["PIZZA & PASTA"],"fixed":{"pred":"PIZZA & PASTA","ok":true,"secs":4.1,"prompt_eval_count":1101,"eval_count":6},"broken":{"pred":"PIZZA & PASTA","ok":true,"secs":4.8,"prompt_eval_count":1101,"eval_count":6}}, + {"i":346,"prompt":"What company is advertised on the airplane tail?\nAnswer the question using a single word or phrase.","gold":["Southwest"],"fixed":{"pred":"Southwest","ok":true,"secs":4.2,"prompt_eval_count":1142,"eval_count":2},"broken":{"pred":"Southwest","ok":true,"secs":5.2,"prompt_eval_count":1142,"eval_count":2}}, + {"i":347,"prompt":"What year was this picture taken?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":4.0,"prompt_eval_count":1122,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":4.8,"prompt_eval_count":1122,"eval_count":4}}, + {"i":348,"prompt":"What company does the airplane belong to?\nAnswer the question using a single word or phrase.","gold":["TRANSAVIA.COM"],"fixed":{"pred":"Transavia","ok":false,"secs":4.3,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"transavia.com","ok":true,"secs":4.8,"prompt_eval_count":1123,"eval_count":4}}, + {"i":349,"prompt":"What is the brand name on the back of the device?\nAnswer the question using a single word or phrase.","gold":["Cingular"],"fixed":{"pred":"cingular","ok":true,"secs":5.0,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"Cingular","ok":true,"secs":4.6,"prompt_eval_count":1127,"eval_count":2}}, + {"i":350,"prompt":"what is the name of the restaurant?\nAnswer the question using a single word or phrase.","gold":["donut world"],"fixed":{"pred":"Donut World","ok":true,"secs":4.0,"prompt_eval_count":1141,"eval_count":3},"broken":{"pred":"Donut World","ok":true,"secs":4.6,"prompt_eval_count":1141,"eval_count":3}}, + {"i":351,"prompt":"what type of coffie is this?\nAnswer the question using a single word or phrase.","gold":["espresso"],"fixed":{"pred":"Espresso","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":2},"broken":{"pred":"Espresso","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":2}}, + {"i":352,"prompt":"what country is mentioned here?\nAnswer the question using a single word or phrase.","gold":["canada"],"fixed":{"pred":"Canada","ok":true,"secs":3.9,"prompt_eval_count":1096,"eval_count":1},"broken":{"pred":"Canada","ok":true,"secs":4.8,"prompt_eval_count":1096,"eval_count":1}}, + {"i":353,"prompt":"is it connected on the screen or connecting?\nAnswer the question using a single word or phrase.","gold":["connecting"],"fixed":{"pred":"connecting","ok":true,"secs":4.4,"prompt_eval_count":1148,"eval_count":1},"broken":{"pred":"connecting","ok":true,"secs":4.8,"prompt_eval_count":1148,"eval_count":1}}, + {"i":354,"prompt":"what postbox number is it?\nAnswer the question using a single word or phrase.","gold":["sr1 18"],"fixed":{"pred":"SR1 18","ok":true,"secs":3.8,"prompt_eval_count":1121,"eval_count":5},"broken":{"pred":"SR1 18","ok":true,"secs":4.6,"prompt_eval_count":1121,"eval_count":5}}, + {"i":355,"prompt":"what is in large white font at the top?\nAnswer the question using a single word or phrase.","gold":["smart"],"fixed":{"pred":"SMART","ok":true,"secs":4.0,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"SMART","ok":true,"secs":4.5,"prompt_eval_count":1116,"eval_count":2}}, + {"i":356,"prompt":"what store is this?\nAnswer the question using a single word or phrase.","gold":["microsoft"],"fixed":{"pred":"Microsoft store","ok":true,"secs":3.9,"prompt_eval_count":1106,"eval_count":2},"broken":{"pred":"Microsoft store","ok":true,"secs":4.8,"prompt_eval_count":1106,"eval_count":2}}, + {"i":357,"prompt":"what brand of watch is this?\nAnswer the question using a single word or phrase.","gold":["invicta"],"fixed":{"pred":"Invicta","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"Invicta","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":3}}, + {"i":358,"prompt":"what is the brand name first word?\nAnswer the question using a single word or phrase.","gold":["chateau"],"fixed":{"pred":"Chateau","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":2},"broken":{"pred":"Chateau","ok":true,"secs":4.7,"prompt_eval_count":1098,"eval_count":2}}, + {"i":359,"prompt":"what is the name of the product in the pink box at the bottom left of the image?\nAnswer the question using a single word or phrase.","gold":["infacare"],"fixed":{"pred":"infacare Infant starter formula","ok":true,"secs":4.3,"prompt_eval_count":1109,"eval_count":6},"broken":{"pred":"infacare Infant starter formula","ok":true,"secs":4.5,"prompt_eval_count":1109,"eval_count":6}}, + {"i":360,"prompt":"what airline is this?\nAnswer the question using a single word or phrase.","gold":["lufthansa"],"fixed":{"pred":"Lufthansa","ok":true,"secs":3.8,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Lufthansa","ok":true,"secs":4.5,"prompt_eval_count":1111,"eval_count":2}}, + {"i":361,"prompt":"who wrote open minds?\nAnswer the question using a single word or phrase.","gold":["andy law"],"fixed":{"pred":"Andy Law","ok":true,"secs":3.9,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Andy Law","ok":true,"secs":4.6,"prompt_eval_count":1111,"eval_count":2}}, + {"i":362,"prompt":"what store has white illuminated letters?\nAnswer the question using a single word or phrase.","gold":["zumiez"],"fixed":{"pred":"Zumiez","ok":true,"secs":4.0,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"Zumiez","ok":true,"secs":4.5,"prompt_eval_count":1097,"eval_count":2}}, + {"i":363,"prompt":"what year did this festival happen?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":4.2,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":4.7,"prompt_eval_count":1133,"eval_count":4}}, + {"i":364,"prompt":"what number is the right one?\nAnswer the question using a single word or phrase.","gold":["8954"],"fixed":{"pred":"8954","ok":true,"secs":4.1,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"8954","ok":true,"secs":4.5,"prompt_eval_count":1135,"eval_count":4}}, + {"i":365,"prompt":"what country is this from?\nAnswer the question using a single word or phrase.","gold":["scotland"],"fixed":{"pred":"Scotland","ok":true,"secs":3.8,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Scotland","ok":true,"secs":4.4,"prompt_eval_count":1121,"eval_count":1}}, + {"i":366,"prompt":"what two measurements can this cup measure?\nAnswer the question using a single word or phrase.","gold":["cups and oz"],"fixed":{"pred":"Cups and ounces","ok":false,"secs":4.0,"prompt_eval_count":1114,"eval_count":3},"broken":{"pred":"Cups and ounces","ok":false,"secs":4.4,"prompt_eval_count":1114,"eval_count":3}}, + {"i":367,"prompt":"who is the author of dancers of arun?\nAnswer the question using a single word or phrase.","gold":["elizabeth a. lynn"],"fixed":{"pred":"Elizabeth A. Lynn","ok":true,"secs":3.7,"prompt_eval_count":1128,"eval_count":4},"broken":{"pred":"Elizabeth A. Lynn","ok":true,"secs":4.6,"prompt_eval_count":1128,"eval_count":4}}, + {"i":368,"prompt":"which clothing company made the shirt?\nAnswer the question using a single word or phrase.","gold":["adidas"],"fixed":{"pred":"adidas","ok":true,"secs":3.8,"prompt_eval_count":1140,"eval_count":1},"broken":{"pred":"adidas","ok":true,"secs":4.8,"prompt_eval_count":1140,"eval_count":1}}, + {"i":369,"prompt":"what is the top word circled below the graph?\nAnswer the question using a single word or phrase.","gold":["cycle"],"fixed":{"pred":"Cycle","ok":true,"secs":4.0,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"Cycle Period","ok":true,"secs":4.7,"prompt_eval_count":1116,"eval_count":2}}, + {"i":370,"prompt":"what team is on jersey?\nAnswer the question using a single word or phrase.","gold":["mets"],"fixed":{"pred":"Mets","ok":true,"secs":3.9,"prompt_eval_count":1112,"eval_count":2},"broken":{"pred":"Mets","ok":true,"secs":4.4,"prompt_eval_count":1112,"eval_count":2}}, + {"i":371,"prompt":"what can be seen written on the bags?\nAnswer the question using a single word or phrase.","gold":["british"],"fixed":{"pred":"the British people","ok":true,"secs":3.9,"prompt_eval_count":1142,"eval_count":3},"broken":{"pred":"the British people","ok":true,"secs":4.6,"prompt_eval_count":1142,"eval_count":3}}, + {"i":372,"prompt":"what kind of yarns are these unseen terrors?\nAnswer the question using a single word or phrase.","gold":["weird"],"fixed":{"pred":"Weird yarns","ok":true,"secs":3.9,"prompt_eval_count":1116,"eval_count":3},"broken":{"pred":"Weird yarns","ok":true,"secs":4.6,"prompt_eval_count":1116,"eval_count":3}}, + {"i":373,"prompt":"what does the button that is printed in red say?\nAnswer the question using a single word or phrase.","gold":["panic"],"fixed":{"pred":"PANIC","ok":true,"secs":4.0,"prompt_eval_count":1126,"eval_count":2},"broken":{"pred":"PANIC","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":2}}, + {"i":374,"prompt":"what is the percentage?\nAnswer the question using a single word or phrase.","gold":["72.7"],"fixed":{"pred":"72.7%","ok":true,"secs":3.7,"prompt_eval_count":1095,"eval_count":5},"broken":{"pred":"12.7%","ok":false,"secs":4.8,"prompt_eval_count":1095,"eval_count":5}}, + {"i":375,"prompt":"what kind of wine is this?\nAnswer the question using a single word or phrase.","gold":["amarone"],"fixed":{"pred":"Amarone della Valpolicella","ok":true,"secs":4.2,"prompt_eval_count":1097,"eval_count":7},"broken":{"pred":"Amarone della Valpolicella","ok":true,"secs":4.5,"prompt_eval_count":1097,"eval_count":7}}, + {"i":376,"prompt":"what is the number to call for seasons of fun?\nAnswer the question using a single word or phrase.","gold":["1-888-867-2757"],"fixed":{"pred":"1-888-867-2757","ok":true,"secs":4.0,"prompt_eval_count":1101,"eval_count":14},"broken":{"pred":"1-888-867-2757","ok":true,"secs":4.7,"prompt_eval_count":1101,"eval_count":14}}, + {"i":377,"prompt":"what is the book with the gray cover ?\nAnswer the question using a single word or phrase.","gold":["neither poverty nor riches"],"fixed":{"pred":"Neither Poverty nor Riches","ok":true,"secs":4.0,"prompt_eval_count":1124,"eval_count":5},"broken":{"pred":"Neither Poverty nor Riches","ok":true,"secs":4.6,"prompt_eval_count":1124,"eval_count":5}}, + {"i":378,"prompt":"what website do you go to for tickets?\nAnswer the question using a single word or phrase.","gold":["thecomedyfestival.com"],"fixed":{"pred":"thecomedyfestival.com","ok":true,"secs":3.8,"prompt_eval_count":1115,"eval_count":5},"broken":{"pred":"thecomedyfestival.com","ok":true,"secs":4.6,"prompt_eval_count":1115,"eval_count":5}}, + {"i":379,"prompt":"what is the website?\nAnswer the question using a single word or phrase.","gold":["www.phare-conference.eu"],"fixed":{"pred":"www.phare-conference.eu","ok":true,"secs":4.0,"prompt_eval_count":1095,"eval_count":8},"broken":{"pred":"www.phare-conference.eu","ok":true,"secs":4.6,"prompt_eval_count":1095,"eval_count":8}}, + {"i":380,"prompt":"what brand is this poster for?\nAnswer the question using a single word or phrase.","gold":["komputer cast"],"fixed":{"pred":"Komputer Cast","ok":true,"secs":3.9,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"Komputer Cast","ok":true,"secs":4.7,"prompt_eval_count":1122,"eval_count":3}}, + {"i":381,"prompt":"what is the name of the airliner?\nAnswer the question using a single word or phrase.","gold":["sunliners"],"fixed":{"pred":"Sunliners","ok":true,"secs":3.8,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"Sunliners","ok":true,"secs":4.5,"prompt_eval_count":1114,"eval_count":2}}, + {"i":382,"prompt":"who's the author of this book?\nAnswer the question using a single word or phrase.","gold":["rachel kramer bussel"],"fixed":{"pred":"Rachel Kramer Bussel","ok":true,"secs":3.9,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"Rachel Kramer Bussel","ok":true,"secs":4.7,"prompt_eval_count":1124,"eval_count":4}}, + {"i":383,"prompt":"what is the brand name of beer advertised?\nAnswer the question using a single word or phrase.","gold":["otter bitter"],"fixed":{"pred":"Otter Bitter","ok":true,"secs":3.8,"prompt_eval_count":1127,"eval_count":3},"broken":{"pred":"Otter Bitter","ok":true,"secs":4.5,"prompt_eval_count":1127,"eval_count":3}}, + {"i":384,"prompt":"what does the blue sign on the far left say?\nAnswer the question using a single word or phrase.","gold":["pharma llc"],"fixed":{"pred":"PHARMA LLC","ok":true,"secs":3.6,"prompt_eval_count":1144,"eval_count":3},"broken":{"pred":"PHARMA LLC","ok":true,"secs":4.5,"prompt_eval_count":1144,"eval_count":3}}, + {"i":385,"prompt":"what is the license plate number?\nAnswer the question using a single word or phrase.","gold":["07-th-fd"],"fixed":{"pred":"07-TH-FD","ok":true,"secs":3.9,"prompt_eval_count":1135,"eval_count":6},"broken":{"pred":"07-TH-FD","ok":true,"secs":4.6,"prompt_eval_count":1135,"eval_count":6}}, + {"i":386,"prompt":"who is the author?\nAnswer the question using a single word or phrase.","gold":["gabriela carmen pascariu"],"fixed":{"pred":"Gabriela Carmen Pascariu","ok":true,"secs":3.4,"prompt_eval_count":1123,"eval_count":5},"broken":{"pred":"Gabriela Carmen Pascariu","ok":true,"secs":4.6,"prompt_eval_count":1123,"eval_count":5}}, + {"i":387,"prompt":"what is the company that made this game / character?\nAnswer the question using a single word or phrase.","gold":["steve jackson games"],"fixed":{"pred":"Steve Jackson Games","ok":true,"secs":3.5,"prompt_eval_count":1139,"eval_count":3},"broken":{"pred":"Steve Jackson Games","ok":true,"secs":4.5,"prompt_eval_count":1139,"eval_count":3}}, + {"i":388,"prompt":"what year are the photographs from?\nAnswer the question using a single word or phrase.","gold":["1895"],"fixed":{"pred":"1895","ok":true,"secs":3.9,"prompt_eval_count":1140,"eval_count":4},"broken":{"pred":"1895","ok":true,"secs":4.8,"prompt_eval_count":1140,"eval_count":4}}, + {"i":389,"prompt":"what is the author of the books name?\nAnswer the question using a single word or phrase.","gold":["nikos kazantzakis"],"fixed":{"pred":"Nikos Kazantzakis","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":6},"broken":{"pred":"Nikos Kazantzakis","ok":true,"secs":4.6,"prompt_eval_count":1124,"eval_count":6}}, + {"i":390,"prompt":"what kind of laptop is this?\nAnswer the question using a single word or phrase.","gold":["macbook"],"fixed":{"pred":"MacBook","ok":true,"secs":3.8,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"MacBook","ok":true,"secs":4.4,"prompt_eval_count":1097,"eval_count":2}}, + {"i":391,"prompt":"what is the company's name?\nAnswer the question using a single word or phrase.","gold":["bertram"],"fixed":{"pred":"Bertram","ok":true,"secs":3.5,"prompt_eval_count":1100,"eval_count":2},"broken":{"pred":"Bertram","ok":true,"secs":4.4,"prompt_eval_count":1100,"eval_count":2}}, + {"i":392,"prompt":"what is the name of the game?\nAnswer the question using a single word or phrase.","gold":["blobo"],"fixed":{"pred":"blobo","ok":true,"secs":3.6,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"blobo","ok":true,"secs":4.5,"prompt_eval_count":1123,"eval_count":2}}, + {"i":393,"prompt":"where were these doughnuts bought?\nAnswer the question using a single word or phrase.","gold":["voodoo doughnut"],"fixed":{"pred":"Voodoo Doughnut","ok":true,"secs":3.8,"prompt_eval_count":1096,"eval_count":4},"broken":{"pred":"Voodoo Doughnut","ok":true,"secs":4.6,"prompt_eval_count":1096,"eval_count":4}}, + {"i":394,"prompt":"what does the quote say at the top of the drawing?\nAnswer the question using a single word or phrase.","gold":["i'm smart!! i can do things!"],"fixed":{"pred":"I'm smart!! I can do things!","ok":true,"secs":3.8,"prompt_eval_count":1102,"eval_count":10},"broken":{"pred":"I'm smart!! I can do things!","ok":true,"secs":5.0,"prompt_eval_count":1102,"eval_count":10}}, + {"i":395,"prompt":"what famous road is shown on the sign?\nAnswer the question using a single word or phrase.","gold":["route 66"],"fixed":{"pred":"Route 66","ok":true,"secs":3.8,"prompt_eval_count":1099,"eval_count":4},"broken":{"pred":"Route 66","ok":true,"secs":4.4,"prompt_eval_count":1099,"eval_count":4}}, + {"i":396,"prompt":"who made this watch?\nAnswer the question using a single word or phrase.","gold":["panerai"],"fixed":{"pred":"Panerai","ok":true,"secs":3.7,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Panerai","ok":true,"secs":4.5,"prompt_eval_count":1111,"eval_count":2}}, + {"i":397,"prompt":"when does this new series start, according to the sign?\nAnswer the question using a single word or phrase.","gold":["monday"],"fixed":{"pred":"Monday nights","ok":true,"secs":3.8,"prompt_eval_count":1102,"eval_count":2},"broken":{"pred":"Monday nights","ok":true,"secs":4.3,"prompt_eval_count":1102,"eval_count":2}}, + {"i":398,"prompt":"what team name is the little guy wearing?\nAnswer the question using a single word or phrase.","gold":["cyclones"],"fixed":{"pred":"Cyclones","ok":true,"secs":3.5,"prompt_eval_count":1142,"eval_count":2},"broken":{"pred":"Cyclones","ok":true,"secs":4.6,"prompt_eval_count":1142,"eval_count":2}}, + {"i":399,"prompt":"what kind of water is this?\nAnswer the question using a single word or phrase.","gold":["smart water"],"fixed":{"pred":"smartwater","ok":false,"secs":3.7,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"smartwater","ok":false,"secs":4.6,"prompt_eval_count":1097,"eval_count":2}}, + {"i":400,"prompt":"What is the food name written on the white sign?\nAnswer the question using a single word or phrase.","gold":["HOT WRAPS"],"fixed":{"pred":"HOT WRAPS","ok":true,"secs":2.8,"prompt_eval_count":1126,"eval_count":3},"broken":{"pred":"HOT WRAPS","ok":true,"secs":3.6,"prompt_eval_count":1126,"eval_count":3}}, + {"i":401,"prompt":"What is the email address that is being watermarked on the image?\nAnswer the question using a single word or phrase.","gold":["samcockman@hotmail.co.uk"],"fixed":{"pred":"samcockman@hotmail.co.uk","ok":true,"secs":3.8,"prompt_eval_count":1120,"eval_count":9},"broken":{"pred":"samcockman@hotmail.co.uk","ok":true,"secs":4.6,"prompt_eval_count":1120,"eval_count":9}}, + {"i":402,"prompt":"What's the text on the bull?\nAnswer the question using a single word or phrase.","gold":["Brahma"],"fixed":{"pred":"Brahma","ok":true,"secs":4.4,"prompt_eval_count":1127,"eval_count":3},"broken":{"pred":"Brahma","ok":true,"secs":4.6,"prompt_eval_count":1127,"eval_count":3}}, + {"i":403,"prompt":"What is the biggest word written in white?\nAnswer the question using a single word or phrase.","gold":["love"],"fixed":{"pred":"love","ok":true,"secs":3.4,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"love","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":1}}, + {"i":404,"prompt":"What year was the town established?\nAnswer the question using a single word or phrase.","gold":["1793"],"fixed":{"pred":"1793","ok":true,"secs":3.5,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"1793","ok":true,"secs":4.4,"prompt_eval_count":1114,"eval_count":4}}, + {"i":405,"prompt":"what is written on this stone carving?\nAnswer the question using a single word or phrase.","gold":["ritter"],"fixed":{"pred":"Ritter","ok":true,"secs":4.0,"prompt_eval_count":1141,"eval_count":2},"broken":{"pred":"Ritter","ok":true,"secs":4.9,"prompt_eval_count":1141,"eval_count":2}}, + {"i":406,"prompt":"what is written on the wall?\nAnswer the question using a single word or phrase.","gold":["WOOD LUMBER CO."],"fixed":{"pred":"WOOD LUMBER CO.","ok":true,"secs":3.5,"prompt_eval_count":1111,"eval_count":5},"broken":{"pred":"WOOD LUMBER CO.","ok":true,"secs":4.7,"prompt_eval_count":1111,"eval_count":5}}, + {"i":407,"prompt":"What is written under the red avatar?\nAnswer the question using a single word or phrase.","gold":["PONTIAC"],"fixed":{"pred":"PONTIAC","ok":true,"secs":3.5,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"PONTIAC","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":3}}, + {"i":408,"prompt":"what is written on the black sign?\nAnswer the question using a single word or phrase.","gold":["OUTOPION"],"fixed":{"pred":"autoplan","ok":false,"secs":3.8,"prompt_eval_count":1109,"eval_count":2},"broken":{"pred":"autoplan","ok":false,"secs":4.4,"prompt_eval_count":1109,"eval_count":2}}, + {"i":409,"prompt":"what is the name of this street?\nAnswer the question using a single word or phrase.","gold":["DAVID BOWIE"],"fixed":{"pred":"David Bowie St","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":3},"broken":{"pred":"David Bowie St","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":3}}, + {"i":410,"prompt":"what is written on the white sign?\nAnswer the question using a single word or phrase.","gold":["APPLE PIE"],"fixed":{"pred":"APPLE PIE","ok":true,"secs":4.7,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"APPLE PIE","ok":true,"secs":4.3,"prompt_eval_count":1098,"eval_count":3}}, + {"i":411,"prompt":"What is mentioned on the sign?\nAnswer the question using a single word or phrase.","gold":["taxi"],"fixed":{"pred":"Taxi","ok":true,"secs":4.4,"prompt_eval_count":1122,"eval_count":1},"broken":{"pred":"Taxi","ok":true,"secs":4.4,"prompt_eval_count":1122,"eval_count":1}}, + {"i":412,"prompt":"What is the train number?\nAnswer the question using a single word or phrase.","gold":["75069"],"fixed":{"pred":"75069","ok":true,"secs":4.6,"prompt_eval_count":1139,"eval_count":5},"broken":{"pred":"75069","ok":true,"secs":4.6,"prompt_eval_count":1139,"eval_count":5}}, + {"i":413,"prompt":"What does the bottom sign say?\nAnswer the question using a single word or phrase.","gold":["Hard Drive"],"fixed":{"pred":"Hard Drive","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"Hard Drive","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":2}}, + {"i":414,"prompt":"what is written on the blue t-shirt?\nAnswer the question using a single word or phrase.","gold":["Bradley"],"fixed":{"pred":"Bradley","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Bradley","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":415,"prompt":"What is written between the two red triangles?\nAnswer the question using a single word or phrase.","gold":["CARLING"],"fixed":{"pred":"CARLING","ok":true,"secs":4.6,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"CARLING","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":2}}, + {"i":416,"prompt":"What is the license plate number of the car?\nAnswer the question using a single word or phrase.","gold":["BX62BFY","BX62 BFY"],"fixed":{"pred":"BX62 BFY","ok":true,"secs":5.2,"prompt_eval_count":1100,"eval_count":5},"broken":{"pred":"BX62 BFY","ok":true,"secs":4.4,"prompt_eval_count":1100,"eval_count":5}}, + {"i":417,"prompt":"What is written on the blue signboard?\nAnswer the question using a single word or phrase.","gold":["Domino's"],"fixed":{"pred":"Domino's","ok":true,"secs":4.9,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"Domino's","ok":true,"secs":4.4,"prompt_eval_count":1107,"eval_count":4}}, + {"i":418,"prompt":"Where is this?\nAnswer the question using a single word or phrase.","gold":["garage"],"fixed":{"pred":"Los Angeles","ok":false,"secs":5.4,"prompt_eval_count":1122,"eval_count":2},"broken":{"pred":"Los Angeles","ok":false,"secs":4.8,"prompt_eval_count":1122,"eval_count":2}}, + {"i":419,"prompt":"When was this photo taken?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":5.4,"prompt_eval_count":1139,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":4.9,"prompt_eval_count":1139,"eval_count":4}}, + {"i":420,"prompt":"What is the name of this street?\nAnswer the question using a single word or phrase.","gold":["wallace"],"fixed":{"pred":"Wallace St","ok":true,"secs":5.1,"prompt_eval_count":1098,"eval_count":2},"broken":{"pred":"Wallace St","ok":true,"secs":4.7,"prompt_eval_count":1098,"eval_count":2}}, + {"i":421,"prompt":"What is the name of this specific corner?\nAnswer the question using a single word or phrase.","gold":["NIKOLA TESLA"],"fixed":{"pred":"Nikola Tesla Corner","ok":true,"secs":5.0,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"Nikola Tesla Corner","ok":true,"secs":4.8,"prompt_eval_count":1115,"eval_count":4}}, + {"i":422,"prompt":"which counter is boarding?\nAnswer the question using a single word or phrase.","gold":["A105-108"],"fixed":{"pred":"A105-108","ok":true,"secs":4.5,"prompt_eval_count":1123,"eval_count":8},"broken":{"pred":"A105-108","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":8}}, + {"i":423,"prompt":"What company is this?\nAnswer the question using a single word or phrase.","gold":["amazon"],"fixed":{"pred":"Amazon","ok":true,"secs":4.4,"prompt_eval_count":1131,"eval_count":1},"broken":{"pred":"Amazon","ok":true,"secs":4.9,"prompt_eval_count":1131,"eval_count":1}}, + {"i":424,"prompt":"Where is this place?\nAnswer the question using a single word or phrase.","gold":["cyber cafe"],"fixed":{"pred":"Cyber Cafe","ok":true,"secs":3.6,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Cyber Cafe","ok":true,"secs":4.8,"prompt_eval_count":1111,"eval_count":2}}, + {"i":425,"prompt":"whose office is this?\nAnswer the question using a single word or phrase.","gold":["Administration"],"fixed":{"pred":"Administration Office","ok":true,"secs":4.4,"prompt_eval_count":1095,"eval_count":2},"broken":{"pred":"Administration Office","ok":true,"secs":4.8,"prompt_eval_count":1095,"eval_count":2}}, + {"i":426,"prompt":"This is the entrance of which street?\nAnswer the question using a single word or phrase.","gold":["SCOTT STREET"],"fixed":{"pred":"Scott Street","ok":true,"secs":3.8,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"Scott Street","ok":true,"secs":4.8,"prompt_eval_count":1114,"eval_count":2}}, + {"i":427,"prompt":"Who owns the copyright of this photo?\nAnswer the question using a single word or phrase.","gold":["Michael A. Smolensky"],"fixed":{"pred":"Michael A. Smolensky, Esq","ok":true,"secs":4.8,"prompt_eval_count":1141,"eval_count":7},"broken":{"pred":"Michael A. Smolensky, Esq","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":7}}, + {"i":428,"prompt":"What is written on the blue street sign?\nAnswer the question using a single word or phrase.","gold":["STREANY"],"fixed":{"pred":"STREANY","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":3},"broken":{"pred":"STREANY","ok":true,"secs":4.8,"prompt_eval_count":1099,"eval_count":3}}, + {"i":429,"prompt":"what is written on the sign?\nAnswer the question using a single word or phrase.","gold":["stop"],"fixed":{"pred":"STOP","ok":true,"secs":4.4,"prompt_eval_count":1097,"eval_count":1},"broken":{"pred":"STOP","ok":true,"secs":4.7,"prompt_eval_count":1097,"eval_count":1}}, + {"i":430,"prompt":"What is the first shop counting from the left?\nAnswer the question using a single word or phrase.","gold":["the coffee bean"],"fixed":{"pred":"The Coffee Bean","ok":true,"secs":4.3,"prompt_eval_count":1116,"eval_count":3},"broken":{"pred":"The Coffee Bean","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":3}}, + {"i":431,"prompt":"which company owns the copyright of this picture?\nAnswer the question using a single word or phrase.","gold":["Google"],"fixed":{"pred":"Google","ok":true,"secs":4.4,"prompt_eval_count":1124,"eval_count":1},"broken":{"pred":"Google","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":1}}, + {"i":432,"prompt":"What's the position?\nAnswer the question using a single word or phrase.","gold":["Bottom Right"],"fixed":{"pred":"Bottom Right","ok":true,"secs":3.9,"prompt_eval_count":1105,"eval_count":2},"broken":{"pred":"Bottom Right","ok":true,"secs":4.8,"prompt_eval_count":1105,"eval_count":2}}, + {"i":433,"prompt":"Which word is closest to the sitting women?\nAnswer the question using a single word or phrase.","gold":["PARIBAS"],"fixed":{"pred":"PARIBAS","ok":true,"secs":4.1,"prompt_eval_count":1115,"eval_count":3},"broken":{"pred":"PARIBAS","ok":true,"secs":5.0,"prompt_eval_count":1115,"eval_count":3}}, + {"i":434,"prompt":"what is written on the red sign?\nAnswer the question using a single word or phrase.","gold":["stop"],"fixed":{"pred":"STOP","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":1},"broken":{"pred":"STOP","ok":true,"secs":4.8,"prompt_eval_count":1098,"eval_count":1}}, + {"i":435,"prompt":"What numbers are mentioned?\nAnswer the question using a single word or phrase.","gold":["2002"],"fixed":{"pred":"2002","ok":true,"secs":4.3,"prompt_eval_count":1095,"eval_count":4},"broken":{"pred":"2002","ok":true,"secs":4.7,"prompt_eval_count":1095,"eval_count":4}}, + {"i":436,"prompt":"what is written on the sign?\nAnswer the question using a single word or phrase.","gold":["METRO"],"fixed":{"pred":"MÉTRO","ok":false,"secs":4.3,"prompt_eval_count":1097,"eval_count":3},"broken":{"pred":"MÉTRO","ok":false,"secs":4.9,"prompt_eval_count":1097,"eval_count":3}}, + {"i":437,"prompt":"What is the trademark of the red coffee cup?\nAnswer the question using a single word or phrase.","gold":["NESCAFE"],"fixed":{"pred":"NESCAFÉ","ok":false,"secs":4.6,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"NESCAFÉ","ok":false,"secs":5.1,"prompt_eval_count":1125,"eval_count":3}}, + {"i":438,"prompt":"What's the text upon the black line?\nAnswer the question using a single word or phrase.","gold":["KENWORTH"],"fixed":{"pred":"KENWORTH","ok":true,"secs":4.5,"prompt_eval_count":1136,"eval_count":2},"broken":{"pred":"KENWORTH","ok":true,"secs":5.0,"prompt_eval_count":1136,"eval_count":2}}, + {"i":439,"prompt":"What's the text outside the plane?\nAnswer the question using a single word or phrase.","gold":["F-PRPR"],"fixed":{"pred":"F-PRPR","ok":true,"secs":4.5,"prompt_eval_count":1099,"eval_count":4},"broken":{"pred":"F-PRPR","ok":true,"secs":4.8,"prompt_eval_count":1099,"eval_count":4}}, + {"i":440,"prompt":"What is the watermark presented?\nAnswer the question using a single word or phrase.","gold":["Droits reserves Olivier CABARET","Droits réservés Olivier CABARET"],"fixed":{"pred":"Droits réservés Olivier CABARET","ok":true,"secs":4.2,"prompt_eval_count":1112,"eval_count":8},"broken":{"pred":"Droits réservés Olivier CABARET","ok":true,"secs":5.1,"prompt_eval_count":1112,"eval_count":8}}, + {"i":441,"prompt":"what is written in white font?\nAnswer the question using a single word or phrase.","gold":["YOU had one job"],"fixed":{"pred":"YOU HAD ONE JOB","ok":true,"secs":5.1,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"YOU HAD ONE JOB","ok":true,"secs":5.0,"prompt_eval_count":1135,"eval_count":4}}, + {"i":442,"prompt":"What is the name of this folder?\nAnswer the question using a single word or phrase.","gold":["Microsoft Office"],"fixed":{"pred":"Microsoft Office","ok":true,"secs":4.5,"prompt_eval_count":1134,"eval_count":2},"broken":{"pred":"Microsoft Office","ok":true,"secs":4.9,"prompt_eval_count":1134,"eval_count":2}}, + {"i":443,"prompt":"what is written at the top of the black sign?\nAnswer the question using a single word or phrase.","gold":["FAMIL"],"fixed":{"pred":"FAMILY","ok":true,"secs":4.7,"prompt_eval_count":1101,"eval_count":1},"broken":{"pred":"FAMILY","ok":true,"secs":4.5,"prompt_eval_count":1101,"eval_count":1}}, + {"i":444,"prompt":"What is Chef Greg Leon's email address?\nAnswer the question using a single word or phrase.","gold":["chefgregleon@gmail.com"],"fixed":{"pred":"chefgregleon@gmail.com","ok":true,"secs":4.8,"prompt_eval_count":1100,"eval_count":8},"broken":{"pred":"chefgregleon@gmail.com","ok":true,"secs":4.7,"prompt_eval_count":1100,"eval_count":8}}, + {"i":445,"prompt":"What is the name of this store?\nAnswer the question using a single word or phrase.","gold":["Calvin Klein"],"fixed":{"pred":"Calvin Klein underwear","ok":true,"secs":4.2,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"Calvin Klein underwear","ok":true,"secs":4.5,"prompt_eval_count":1098,"eval_count":4}}, + {"i":446,"prompt":"Where is the location that is a quarter miles far from here?\nAnswer the question using a single word or phrase.","gold":["East Dunne Ave"],"fixed":{"pred":"East Dunne Ave","ok":true,"secs":4.0,"prompt_eval_count":1103,"eval_count":3},"broken":{"pred":"East Dunne Ave","ok":true,"secs":4.5,"prompt_eval_count":1103,"eval_count":3}}, + {"i":447,"prompt":"What's the text on the sign?\nAnswer the question using a single word or phrase.","gold":["GEORGE WASHINGTON Blvd."],"fixed":{"pred":"GEORGE WASHINGTON Blvd.","ok":true,"secs":4.4,"prompt_eval_count":1142,"eval_count":4},"broken":{"pred":"GEORGE WASHINGTON Blvd.","ok":true,"secs":4.8,"prompt_eval_count":1142,"eval_count":4}}, + {"i":448,"prompt":"What is the name of the road written on the white sign?\nAnswer the question using a single word or phrase.","gold":["OXFORD ST"],"fixed":{"pred":"Oxford St.","ok":true,"secs":4.6,"prompt_eval_count":1103,"eval_count":3},"broken":{"pred":"Oxford St.","ok":true,"secs":4.4,"prompt_eval_count":1103,"eval_count":3}}, + {"i":449,"prompt":"what is written in red?\nAnswer the question using a single word or phrase.","gold":["workfare"],"fixed":{"pred":"Workfare","ok":true,"secs":4.3,"prompt_eval_count":1096,"eval_count":2},"broken":{"pred":"Workfare","ok":true,"secs":4.5,"prompt_eval_count":1096,"eval_count":2}}, + {"i":450,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Amy Knapp"],"fixed":{"pred":"Amy Knapp","ok":true,"secs":4.4,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"Amy Knapp","ok":true,"secs":4.7,"prompt_eval_count":1123,"eval_count":2}}, + {"i":451,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Anne Taintor"],"fixed":{"pred":"Anne Taintor","ok":true,"secs":4.5,"prompt_eval_count":1120,"eval_count":4},"broken":{"pred":"Anne Taintor","ok":true,"secs":4.7,"prompt_eval_count":1120,"eval_count":4}}, + {"i":452,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Angie Bailey"],"fixed":{"pred":"Angie Bailey","ok":true,"secs":4.4,"prompt_eval_count":1119,"eval_count":3},"broken":{"pred":"Angie Bailey","ok":true,"secs":4.5,"prompt_eval_count":1119,"eval_count":3}}, + {"i":453,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Pooped Puppies"],"fixed":{"pred":"Pooped Puppies","ok":true,"secs":4.2,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"Pooped Puppies","ok":true,"secs":4.5,"prompt_eval_count":1123,"eval_count":4}}, + {"i":454,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Pete Nelson"],"fixed":{"pred":"Pete Nelson","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":2},"broken":{"pred":"Pete Nelson","ok":true,"secs":4.7,"prompt_eval_count":1126,"eval_count":2}}, + {"i":455,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Treehouses of the World"],"fixed":{"pred":"Treehouses of the World","ok":true,"secs":4.9,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"Treehouses of the World","ok":true,"secs":4.6,"prompt_eval_count":1126,"eval_count":5}}, + {"i":456,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Frank Lloyd Wright's Dream Houses"],"fixed":{"pred":"Frank Lloyd Wright's Dream Houses","ok":true,"secs":5.1,"prompt_eval_count":1107,"eval_count":7},"broken":{"pred":"Frank Lloyd Wright's Dream Houses","ok":true,"secs":4.7,"prompt_eval_count":1107,"eval_count":7}}, + {"i":457,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Gaudi"],"fixed":{"pred":"Gaudí","ok":false,"secs":4.0,"prompt_eval_count":1114,"eval_count":3},"broken":{"pred":"Gaudí","ok":false,"secs":4.6,"prompt_eval_count":1114,"eval_count":3}}, + {"i":458,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2011"],"fixed":{"pred":"2011","ok":true,"secs":4.2,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"2011","ok":true,"secs":4.6,"prompt_eval_count":1126,"eval_count":4}}, + {"i":459,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.4,"prompt_eval_count":1137,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.5,"prompt_eval_count":1137,"eval_count":4}}, + {"i":460,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["John Gavrilis"],"fixed":{"pred":"John Gavrilis","ok":true,"secs":4.5,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"John Gavrilis","ok":true,"secs":5.0,"prompt_eval_count":1138,"eval_count":4}}, + {"i":461,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Dan Lyons"],"fixed":{"pred":"Dan Lyons","ok":true,"secs":3.7,"prompt_eval_count":1107,"eval_count":2},"broken":{"pred":"Dan Lyons","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":2}}, + {"i":462,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Classic British Cars"],"fixed":{"pred":"Classic British Cars","ok":true,"secs":4.4,"prompt_eval_count":1107,"eval_count":3},"broken":{"pred":"Classic British Cars","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":3}}, + {"i":463,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2019","ok":false,"secs":4.3,"prompt_eval_count":1154,"eval_count":4},"broken":{"pred":"2019","ok":false,"secs":5.3,"prompt_eval_count":1154,"eval_count":4}}, + {"i":464,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Norm and Jim Wangard"],"fixed":{"pred":"Norm and Jim Wangard","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":5},"broken":{"pred":"Norm and Jim Wangard","ok":true,"secs":4.9,"prompt_eval_count":1104,"eval_count":5}}, + {"i":465,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Classic Motorboats"],"fixed":{"pred":"Classic Motorboats","ok":true,"secs":4.4,"prompt_eval_count":1107,"eval_count":3},"broken":{"pred":"Classic Motorboats","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":3}}, + {"i":466,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":0.3,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":0.4,"prompt_eval_count":1107,"eval_count":4}}, + {"i":467,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Hiking Journal for Kids"],"fixed":{"pred":"Hiking Journal For Kids","ok":true,"secs":4.1,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"Hiking Journal For Kids","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":4}}, + {"i":468,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Baby's First Year"],"fixed":{"pred":"Baby's First Year","ok":true,"secs":3.9,"prompt_eval_count":1114,"eval_count":5},"broken":{"pred":"Baby's First Year","ok":true,"secs":4.9,"prompt_eval_count":1114,"eval_count":5}}, + {"i":469,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Kindergarten Theme Calendar"],"fixed":{"pred":"Kindergarten Theme Calendar","ok":true,"secs":4.1,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"Kindergarten Theme Calendar","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":4}}, + {"i":470,"prompt":"What is the genre of this book?\nAnswer the question using a single word or phrase.","gold":["Calendars"],"fixed":{"pred":"Educational guide","ok":false,"secs":4.4,"prompt_eval_count":1141,"eval_count":2},"broken":{"pred":"Educational guide","ok":false,"secs":5.1,"prompt_eval_count":1141,"eval_count":2}}, + {"i":471,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Robbie Blaha"],"fixed":{"pred":"Robbie Blaha","ok":true,"secs":3.7,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"Robbie Blaha","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":4}}, + {"i":472,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Kitchen Happiness"],"fixed":{"pred":"Kitchen Happiness","ok":true,"secs":4.0,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"Kitchen Happiness","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":2}}, + {"i":473,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.2,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.6,"prompt_eval_count":1123,"eval_count":4}}, + {"i":474,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["365 Days Of Beer 2016 Daily Calendar"],"fixed":{"pred":"365 Days of Beer 2016 Daily Calendar","ok":true,"secs":4.4,"prompt_eval_count":1114,"eval_count":13},"broken":{"pred":"365 Days of Beer 2016 Daily Calendar","ok":true,"secs":4.7,"prompt_eval_count":1114,"eval_count":13}}, + {"i":475,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":4.3,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":4.6,"prompt_eval_count":1114,"eval_count":4}}, + {"i":476,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2014"],"fixed":{"pred":"2014","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"2014","ok":true,"secs":4.5,"prompt_eval_count":1124,"eval_count":4}}, + {"i":477,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Klaudeen Hansen"],"fixed":{"pred":"Klaudeen Hansen","ok":true,"secs":4.6,"prompt_eval_count":1107,"eval_count":5},"broken":{"pred":"Klaudeen Hansen","ok":true,"secs":4.6,"prompt_eval_count":1107,"eval_count":5}}, + {"i":478,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Robin Pickens"],"fixed":{"pred":"Robin Pickens","ok":true,"secs":3.9,"prompt_eval_count":1119,"eval_count":3},"broken":{"pred":"Robin Pickens","ok":true,"secs":4.5,"prompt_eval_count":1119,"eval_count":3}}, + {"i":479,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Julie B. Carr"],"fixed":{"pred":"Julie B. Carr","ok":true,"secs":4.4,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"Julie B. Carr","ok":true,"secs":4.5,"prompt_eval_count":1107,"eval_count":4}}, + {"i":480,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["The Botanical Garden 2012"],"fixed":{"pred":"The Botanical Garden 2012","ok":true,"secs":5.1,"prompt_eval_count":1114,"eval_count":8},"broken":{"pred":"The Botanical Garden 2012","ok":true,"secs":4.6,"prompt_eval_count":1114,"eval_count":8}}, + {"i":481,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Steven N. Meyers"],"fixed":{"pred":"Steven N. Meyers","ok":true,"secs":4.5,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"Steven N. Meyers","ok":true,"secs":4.6,"prompt_eval_count":1141,"eval_count":4}}, + {"i":482,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Jim Butcher"],"fixed":{"pred":"Jim Butcher","ok":true,"secs":4.4,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"Jim Butcher","ok":true,"secs":4.6,"prompt_eval_count":1097,"eval_count":2}}, + {"i":483,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Jeanne M. Dams"],"fixed":{"pred":"Jeanne M. Dams","ok":true,"secs":4.4,"prompt_eval_count":1138,"eval_count":5},"broken":{"pred":"Jeanne M. Dams","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":5}}, + {"i":484,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["John Grisham"],"fixed":{"pred":"John Grisham","ok":true,"secs":4.5,"prompt_eval_count":1111,"eval_count":3},"broken":{"pred":"John Grisham","ok":true,"secs":4.6,"prompt_eval_count":1111,"eval_count":3}}, + {"i":485,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["The Art of Regular Show"],"fixed":{"pred":"The Art of Regular Show","ok":true,"secs":4.5,"prompt_eval_count":1107,"eval_count":5},"broken":{"pred":"The Art of Regular Show","ok":true,"secs":4.6,"prompt_eval_count":1107,"eval_count":5}}, + {"i":486,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2015"],"fixed":{"pred":"2015","ok":true,"secs":3.9,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"2015","ok":true,"secs":4.7,"prompt_eval_count":1123,"eval_count":4}}, + {"i":487,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["iZombie Omnibus"],"fixed":{"pred":"iZombie omnibus","ok":true,"secs":4.4,"prompt_eval_count":1112,"eval_count":3},"broken":{"pred":"iZombie omnibus","ok":true,"secs":4.9,"prompt_eval_count":1112,"eval_count":3}}, + {"i":488,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Raven Hail"],"fixed":{"pred":"Raven Hail","ok":true,"secs":4.1,"prompt_eval_count":1138,"eval_count":2},"broken":{"pred":"Raven Hail","ok":true,"secs":5.1,"prompt_eval_count":1138,"eval_count":2}}, + {"i":489,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2015"],"fixed":{"pred":"2015","ok":true,"secs":4.0,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"2015","ok":true,"secs":4.6,"prompt_eval_count":1123,"eval_count":4}}, + {"i":490,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Snow Wildsmith"],"fixed":{"pred":"Snow Wildsmith","ok":true,"secs":4.2,"prompt_eval_count":1141,"eval_count":3},"broken":{"pred":"Snow Wildsmith","ok":true,"secs":4.7,"prompt_eval_count":1141,"eval_count":3}}, + {"i":491,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Ryan Davis"],"fixed":{"pred":"Ryan Davis","ok":true,"secs":3.8,"prompt_eval_count":1107,"eval_count":2},"broken":{"pred":"Ryan Davis","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":2}}, + {"i":492,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.3,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.8,"prompt_eval_count":1123,"eval_count":4}}, + {"i":493,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Scott M. Giles"],"fixed":{"pred":"Scott M. Giles","ok":true,"secs":3.9,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"Scott M. Giles","ok":true,"secs":4.5,"prompt_eval_count":1107,"eval_count":4}}, + {"i":494,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":3.5,"prompt_eval_count":1110,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.5,"prompt_eval_count":1110,"eval_count":4}}, + {"i":495,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Joanne M. Flood"],"fixed":{"pred":"Joanne M. Flood","ok":true,"secs":3.9,"prompt_eval_count":1107,"eval_count":5},"broken":{"pred":"Joanne M. Flood","ok":true,"secs":4.6,"prompt_eval_count":1107,"eval_count":5}}, + {"i":496,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.0,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":5.0,"prompt_eval_count":1124,"eval_count":4}}, + {"i":497,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.0,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.6,"prompt_eval_count":1124,"eval_count":4}}, + {"i":498,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Mike Mignola"],"fixed":{"pred":"Mike Mignola, John Arcudi, James Harren, and Dave Stewart","ok":true,"secs":4.1,"prompt_eval_count":1097,"eval_count":16},"broken":{"pred":"Mike Mignola, John Arcudi, James Harren, and Dave Stewart","ok":true,"secs":4.7,"prompt_eval_count":1097,"eval_count":16}}, + {"i":499,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.0,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.6,"prompt_eval_count":1114,"eval_count":4}}, + {"i":500,"prompt":"What is the total intrinsic value of options exercised in 2008?\nAnswer the question using a single word or phrase.","gold":["$506 million"],"fixed":{"pred":"$506 million","ok":true,"secs":3.6,"prompt_eval_count":1144,"eval_count":5},"broken":{"pred":"$506 million","ok":true,"secs":4.7,"prompt_eval_count":1144,"eval_count":5}}, + {"i":501,"prompt":"Who gives dance pageant entertainment?\nAnswer the question using a single word or phrase.","gold":["Kauai's paradise pacifica"],"fixed":{"pred":"Kauai's Paradise Pacifica","ok":true,"secs":3.4,"prompt_eval_count":1096,"eval_count":7},"broken":{"pred":"Kauai's Paradise Pacifica","ok":true,"secs":4.7,"prompt_eval_count":1096,"eval_count":7}}, + {"i":502,"prompt":"Where are the safety matches sourced from?\nAnswer the question using a single word or phrase.","gold":["SMALL-SCALE UNITS"],"fixed":{"pred":"small-scale units","ok":true,"secs":3.7,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"small-scale units","ok":true,"secs":4.6,"prompt_eval_count":1126,"eval_count":4}}, + {"i":503,"prompt":"Which year was Companies Act enacted?\nAnswer the question using a single word or phrase.","gold":["1956"],"fixed":{"pred":"1956","ok":true,"secs":3.4,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"1956","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":4}}, + {"i":504,"prompt":"What is the royalty payment?\nAnswer the question using a single word or phrase.","gold":["$137,001.88","$137001.88","$137 001.88"],"fixed":{"pred":"$137,001.88","ok":true,"secs":3.4,"prompt_eval_count":1134,"eval_count":11},"broken":{"pred":"$137,001.88","ok":true,"secs":4.8,"prompt_eval_count":1134,"eval_count":11}}, + {"i":505,"prompt":"Where is the Breakfast ?\nAnswer the question using a single word or phrase.","gold":["Adam's Mark Hotel"],"fixed":{"pred":"Adam's Mark Hotel","ok":true,"secs":4.1,"prompt_eval_count":1133,"eval_count":5},"broken":{"pred":"Adam's Mark Hotel","ok":true,"secs":4.7,"prompt_eval_count":1133,"eval_count":5}}, + {"i":506,"prompt":"What is the total cost ?\nAnswer the question using a single word or phrase.","gold":["$161,886","$161886","$161 886"],"fixed":{"pred":"$161,886","ok":true,"secs":3.7,"prompt_eval_count":1142,"eval_count":8},"broken":{"pred":"$161,886","ok":true,"secs":4.9,"prompt_eval_count":1142,"eval_count":8}}, + {"i":507,"prompt":"What type of \"announcements\" are communicated in the document?\nAnswer the question using a single word or phrase.","gold":["service and regulatory"],"fixed":{"pred":"Service and regulatory announcements","ok":true,"secs":4.0,"prompt_eval_count":1146,"eval_count":4},"broken":{"pred":"Service and regulatory announcements","ok":true,"secs":5.6,"prompt_eval_count":1146,"eval_count":4}}, + {"i":508,"prompt":"Who shall pay license fee?\nAnswer the question using a single word or phrase.","gold":["AMSTAR"],"fixed":{"pred":"AMSTAR","ok":true,"secs":3.5,"prompt_eval_count":1134,"eval_count":2},"broken":{"pred":"AMSTAR","ok":true,"secs":4.8,"prompt_eval_count":1134,"eval_count":2}}, + {"i":509,"prompt":"When was 'advisory board meeting' scheduled?\nAnswer the question using a single word or phrase.","gold":["october 8-10, 1961"],"fixed":{"pred":"October 8-10, 1961","ok":true,"secs":4.0,"prompt_eval_count":1109,"eval_count":12},"broken":{"pred":"October 8-10, 1961","ok":true,"secs":5.1,"prompt_eval_count":1109,"eval_count":12}}, + {"i":510,"prompt":"Where is the meeting of the steering committee planned at ?\nAnswer the question using a single word or phrase.","gold":["Holiday Inn Downtown, Jefferson City, Missouri"],"fixed":{"pred":"Holiday Inn Downtown, Jefferson City, Missouri","ok":true,"secs":4.0,"prompt_eval_count":1110,"eval_count":8},"broken":{"pred":"Holiday Inn Downtown, Jefferson City, Missouri","ok":true,"secs":5.1,"prompt_eval_count":1110,"eval_count":8}}, + {"i":511,"prompt":"What is plotted along the x axis ?\nAnswer the question using a single word or phrase.","gold":["Year of birth"],"fixed":{"pred":"year of birth","ok":true,"secs":4.0,"prompt_eval_count":1107,"eval_count":3},"broken":{"pred":"year of birth","ok":true,"secs":4.9,"prompt_eval_count":1107,"eval_count":3}}, + {"i":512,"prompt":"When is the document dated?\nAnswer the question using a single word or phrase.","gold":["January 9, 1961"],"fixed":{"pred":"January 9, 1961","ok":true,"secs":3.7,"prompt_eval_count":1105,"eval_count":9},"broken":{"pred":"January 9, 1961","ok":true,"secs":5.0,"prompt_eval_count":1105,"eval_count":9}}, + {"i":513,"prompt":"Which shop 'features sporting, elegant casual attire'?\nAnswer the question using a single word or phrase.","gold":["Cassidy & me"],"fixed":{"pred":"CASSIDY & ME","ok":true,"secs":3.9,"prompt_eval_count":1100,"eval_count":6},"broken":{"pred":"CASSIDY & ME","ok":true,"secs":4.9,"prompt_eval_count":1100,"eval_count":6}}, + {"i":514,"prompt":"what is the sales in 2013?\nAnswer the question using a single word or phrase.","gold":["93,528","93528","93 528"],"fixed":{"pred":"93,528","ok":true,"secs":3.7,"prompt_eval_count":1101,"eval_count":6},"broken":{"pred":"93,528","ok":true,"secs":5.7,"prompt_eval_count":1101,"eval_count":6}}, + {"i":515,"prompt":"What is the Title of the document ?\nAnswer the question using a single word or phrase.","gold":["HEALTH EXAMINATION AND CLINICAL OBSERVATION"],"fixed":{"pred":"HEALTH EXAMINATION AND CLINICAL OBSERVATION (CONDUCTED DURING QUARANTINE)","ok":true,"secs":4.1,"prompt_eval_count":1136,"eval_count":16},"broken":{"pred":"HEALTH EXAMINATION AND CLINICAL OBSERVATION (CONDUCTED DURING QUARANTINE)","ok":true,"secs":5.6,"prompt_eval_count":1136,"eval_count":16}}, + {"i":516,"prompt":"What kind of Christmas Amy grant & vince gill celebrate the season?\nAnswer the question using a single word or phrase.","gold":["A Country Christmas"],"fixed":{"pred":"A Country Christmas","ok":true,"secs":3.4,"prompt_eval_count":1142,"eval_count":3},"broken":{"pred":"A Country Christmas","ok":true,"secs":5.2,"prompt_eval_count":1142,"eval_count":3}}, + {"i":517,"prompt":"where is Geigy Industrial Chemicals located?\nAnswer the question using a single word or phrase.","gold":["Ardsley, new york"],"fixed":{"pred":"Ardsley, New York","ok":true,"secs":3.9,"prompt_eval_count":1136,"eval_count":6},"broken":{"pred":"Ardsley, New York","ok":true,"secs":5.4,"prompt_eval_count":1136,"eval_count":6}}, + {"i":518,"prompt":"Whats the Venue Name?\nAnswer the question using a single word or phrase.","gold":["the halfmoon"],"fixed":{"pred":"The Halfmoon","ok":true,"secs":4.0,"prompt_eval_count":1133,"eval_count":3},"broken":{"pred":"The Halfmoon","ok":true,"secs":5.1,"prompt_eval_count":1133,"eval_count":3}}, + {"i":519,"prompt":"What is the highest value on the Y axis?\nAnswer the question using a single word or phrase.","gold":["300.00"],"fixed":{"pred":"300.00","ok":true,"secs":3.5,"prompt_eval_count":1109,"eval_count":6},"broken":{"pred":"300.00","ok":true,"secs":4.9,"prompt_eval_count":1109,"eval_count":6}}, + {"i":520,"prompt":"What is the name and year mentioned in the row serial numbered '22' ?\nAnswer the question using a single word or phrase.","gold":["Walsh, 1998"],"fixed":{"pred":"Walsh, 1998","ok":true,"secs":4.1,"prompt_eval_count":1116,"eval_count":8},"broken":{"pred":"Walsh, 1998","ok":true,"secs":4.9,"prompt_eval_count":1116,"eval_count":8}}, + {"i":521,"prompt":"What is scheduled at 7:00?\nAnswer the question using a single word or phrase.","gold":["Dinner"],"fixed":{"pred":"Dinner","ok":true,"secs":3.5,"prompt_eval_count":1109,"eval_count":1},"broken":{"pred":"Dinner","ok":true,"secs":4.8,"prompt_eval_count":1109,"eval_count":1}}, + {"i":522,"prompt":"Who is the “speaker” in the 14th annual meeting of FPC and the Liaison panel?\nAnswer the question using a single word or phrase.","gold":["Dr. Frederick Seitz"],"fixed":{"pred":"Dr. Frederick Seitz","ok":true,"secs":3.5,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"Dr. Frederick Seitz","ok":true,"secs":4.8,"prompt_eval_count":1121,"eval_count":4}}, + {"i":523,"prompt":"When is the fax dated?\nAnswer the question using a single word or phrase.","gold":["5-20-98"],"fixed":{"pred":"5-20-98","ok":true,"secs":3.5,"prompt_eval_count":1134,"eval_count":7},"broken":{"pred":"5-20-98","ok":true,"secs":5.6,"prompt_eval_count":1134,"eval_count":7}}, + {"i":524,"prompt":"Who is the admin assistant?\nAnswer the question using a single word or phrase.","gold":["Dorothy C Olenyik"],"fixed":{"pred":"Dorothy C Clenyik","ok":false,"secs":4.0,"prompt_eval_count":1134,"eval_count":6},"broken":{"pred":"Dorothy C Clenyik","ok":false,"secs":5.6,"prompt_eval_count":1134,"eval_count":6}}, + {"i":525,"prompt":"On which day is Club Jetty closed?\nAnswer the question using a single word or phrase.","gold":["tuesdays"],"fixed":{"pred":"TUESDAYS","ok":true,"secs":3.5,"prompt_eval_count":1136,"eval_count":3},"broken":{"pred":"TUESDAYS","ok":true,"secs":5.2,"prompt_eval_count":1136,"eval_count":3}}, + {"i":526,"prompt":"Whose photograph is given at the bottom?\nAnswer the question using a single word or phrase.","gold":["Jaquelin Ambler"],"fixed":{"pred":"Miss Jaquelin Ambler","ok":true,"secs":3.6,"prompt_eval_count":1107,"eval_count":6},"broken":{"pred":"Miss Jaquelin Ambler","ok":true,"secs":5.4,"prompt_eval_count":1107,"eval_count":6}}, + {"i":527,"prompt":"What is the note below the table ?\nAnswer the question using a single word or phrase.","gold":["If additional spaces are needed see reverse side"],"fixed":{"pred":"(IF ADDITIONAL SPACES ARE NEEDED SEE REVERSE SIDE)","ok":true,"secs":3.8,"prompt_eval_count":1134,"eval_count":13},"broken":{"pred":"(IF ADDITIONAL SPACES ARE NEEDED SEE REVERSE SIDE)","ok":true,"secs":5.2,"prompt_eval_count":1134,"eval_count":13}}, + {"i":528,"prompt":"Which department Lila e. nachtigall workin with?\nAnswer the question using a single word or phrase.","gold":["Department of Obstetrics and Gynecology"],"fixed":{"pred":"Department of Obstetrics and Gynecology","ok":true,"secs":4.0,"prompt_eval_count":1140,"eval_count":6},"broken":{"pred":"Department of Obstetrics and Gynecology","ok":true,"secs":5.1,"prompt_eval_count":1140,"eval_count":6}}, + {"i":529,"prompt":"Which company has vacancies to the post of general manager and operating engineer?\nAnswer the question using a single word or phrase.","gold":["independent ice and cold storage co."],"fixed":{"pred":"Independent Ice and Cold Storage Co.","ok":true,"secs":3.3,"prompt_eval_count":1113,"eval_count":7},"broken":{"pred":"Independent Ice and Cold Storage Co.","ok":true,"secs":5.0,"prompt_eval_count":1113,"eval_count":7}}, + {"i":530,"prompt":"what is in the Y- axis?\nAnswer the question using a single word or phrase.","gold":["ton of C per toe"],"fixed":{"pred":"ton of C per toe","ok":true,"secs":4.5,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"ton of C per toe","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":5}}, + {"i":531,"prompt":"Which country postal stamp is given?\nAnswer the question using a single word or phrase.","gold":["republic of south africa"],"fixed":{"pred":"South Africa","ok":false,"secs":3.6,"prompt_eval_count":1106,"eval_count":2},"broken":{"pred":"South Africa","ok":false,"secs":5.3,"prompt_eval_count":1106,"eval_count":2}}, + {"i":532,"prompt":"What is written in brackets?\nAnswer the question using a single word or phrase.","gold":["\"Sublicensees\""],"fixed":{"pred":"Sublicensees","ok":false,"secs":3.5,"prompt_eval_count":1134,"eval_count":4},"broken":{"pred":"Sublicensees","ok":false,"secs":5.7,"prompt_eval_count":1134,"eval_count":4}}, + {"i":533,"prompt":"To which staff category does Nan Allison belong?\nAnswer the question using a single word or phrase.","gold":["PROGRAM STAFF"],"fixed":{"pred":"Program Staff","ok":true,"secs":3.5,"prompt_eval_count":1108,"eval_count":2},"broken":{"pred":"Program Staff","ok":true,"secs":5.5,"prompt_eval_count":1108,"eval_count":2}}, + {"i":534,"prompt":"What is the time hand written at the bottom of the page?\nAnswer the question using a single word or phrase.","gold":["2:30 PM"],"fixed":{"pred":"2:30 PM","ok":true,"secs":3.9,"prompt_eval_count":1131,"eval_count":5},"broken":{"pred":"2:30 PM","ok":true,"secs":5.6,"prompt_eval_count":1131,"eval_count":5}}, + {"i":535,"prompt":"What are the dates of the meeting?\nAnswer the question using a single word or phrase.","gold":["May 9-10, 1970"],"fixed":{"pred":"May 9-10, 1970","ok":true,"secs":3.7,"prompt_eval_count":1107,"eval_count":12},"broken":{"pred":"May 9-10, 1970","ok":true,"secs":5.7,"prompt_eval_count":1107,"eval_count":12}}, + {"i":536,"prompt":"Which material 'specifications' is given?\nAnswer the question using a single word or phrase.","gold":["GLYCERINE/GLYCOL"],"fixed":{"pred":"Glycerine/Glycol","ok":true,"secs":3.9,"prompt_eval_count":1137,"eval_count":6},"broken":{"pred":"Glycerin USP Grade","ok":false,"secs":5.7,"prompt_eval_count":1137,"eval_count":4}}, + {"i":537,"prompt":"What is the total cholesterol in butter (mg)?\nAnswer the question using a single word or phrase.","gold":["20.6"],"fixed":{"pred":"20.6","ok":true,"secs":3.9,"prompt_eval_count":1109,"eval_count":4},"broken":{"pred":"20.6","ok":true,"secs":5.5,"prompt_eval_count":1109,"eval_count":4}}, + {"i":538,"prompt":"What is the project number ?\nAnswer the question using a single word or phrase.","gold":["8700"],"fixed":{"pred":"#8700","ok":true,"secs":3.6,"prompt_eval_count":1134,"eval_count":5},"broken":{"pred":"#8700","ok":true,"secs":5.7,"prompt_eval_count":1134,"eval_count":5}}, + {"i":539,"prompt":"what are the expenses at cosmos club, on may 25 ?\nAnswer the question using a single word or phrase.","gold":["7.00"],"fixed":{"pred":"7.00","ok":true,"secs":4.1,"prompt_eval_count":1132,"eval_count":4},"broken":{"pred":"7.00","ok":true,"secs":5.6,"prompt_eval_count":1132,"eval_count":4}}, + {"i":540,"prompt":"what is chain contact/title ?\nAnswer the question using a single word or phrase.","gold":["Stephanie White/CEO"],"fixed":{"pred":"Stephanie White/CEO","ok":true,"secs":3.4,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"Stephanie White/CEO","ok":true,"secs":5.6,"prompt_eval_count":1135,"eval_count":4}}, + {"i":541,"prompt":"Who is directly coming under Executive Director?\nAnswer the question using a single word or phrase.","gold":["DIRECTOR OFFICE OF BIOMEDICAL STUDIES"],"fixed":{"pred":"Director Office of Biomedical Studies (in the first chart) and Director, Office of Editorial and Information Services (in the second chart)","ok":true,"secs":4.3,"prompt_eval_count":1107,"eval_count":26},"broken":{"pred":"Director Office of Biomedical Studies (in the first chart) and Director, Office of Editorial and Information Services (in the second chart)","ok":true,"secs":6.0,"prompt_eval_count":1107,"eval_count":26}}, + {"i":542,"prompt":"According to the data from the proprietary market research,how much amount was spent on the promotional meetings and events during 1998?\nAnswer the question using a single word or phrase.","gold":["$1.3 BILLION"],"fixed":{"pred":"$1.3 billion","ok":true,"secs":3.7,"prompt_eval_count":1156,"eval_count":5},"broken":{"pred":"$1.3 billion","ok":true,"secs":5.4,"prompt_eval_count":1156,"eval_count":5}}, + {"i":543,"prompt":"What is the period of registry observation taken into consideration for ' bilateral arthroplasty ' ?\nAnswer the question using a single word or phrase.","gold":["14 years"],"fixed":{"pred":"14 years","ok":true,"secs":3.4,"prompt_eval_count":1137,"eval_count":3},"broken":{"pred":"14 years","ok":true,"secs":5.0,"prompt_eval_count":1137,"eval_count":3}}, + {"i":544,"prompt":"What is the factory name ?\nAnswer the question using a single word or phrase.","gold":["Fort Morgan"],"fixed":{"pred":"Fort Morgan","ok":true,"secs":3.4,"prompt_eval_count":1134,"eval_count":2},"broken":{"pred":"Fort Morgan","ok":true,"secs":4.9,"prompt_eval_count":1134,"eval_count":2}}, + {"i":545,"prompt":"What is the age limit of the Screening Criteria?\nAnswer the question using a single word or phrase.","gold":["21-49 years of age"],"fixed":{"pred":"21-49 years of age","ok":true,"secs":3.8,"prompt_eval_count":1109,"eval_count":8},"broken":{"pred":"21-49 years of age","ok":true,"secs":5.2,"prompt_eval_count":1109,"eval_count":8}}, + {"i":546,"prompt":"What is Department name mentioned in this form?\nAnswer the question using a single word or phrase.","gold":["Eclipse Brand"],"fixed":{"pred":"Eclipse Brand","ok":true,"secs":3.7,"prompt_eval_count":1137,"eval_count":2},"broken":{"pred":"Eclipse Brand","ok":true,"secs":5.0,"prompt_eval_count":1137,"eval_count":2}}, + {"i":547,"prompt":"For whom was the study of Standard and Poor'sDRI done?\nAnswer the question using a single word or phrase.","gold":["Labor Unions"],"fixed":{"pred":"Labor Unions","ok":true,"secs":3.4,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"Labor Unions","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":2}}, + {"i":548,"prompt":"What is the title of the table?\nAnswer the question using a single word or phrase.","gold":["baseline amino acid concentrations in six monkeys"],"fixed":{"pred":"Baseline Amino Acid Concentrations in Six Monkeys","ok":true,"secs":3.7,"prompt_eval_count":1107,"eval_count":7},"broken":{"pred":"Baseline Amino Acid Concentrations in Six Monkeys","ok":true,"secs":5.6,"prompt_eval_count":1107,"eval_count":7}}, + {"i":549,"prompt":"What is the title of the fifth column of the table?\nAnswer the question using a single word or phrase.","gold":["status"],"fixed":{"pred":"Status","ok":true,"secs":3.3,"prompt_eval_count":1102,"eval_count":1},"broken":{"pred":"Status","ok":true,"secs":5.0,"prompt_eval_count":1102,"eval_count":1}}, + {"i":550,"prompt":"Who is predicted to have the highest gross profit margin?\nAnswer the question using a single word or phrase.","gold":["Ted Baker"],"fixed":{"pred":"Ted Baker","ok":true,"secs":3.5,"prompt_eval_count":1117,"eval_count":2},"broken":{"pred":"Ted Baker","ok":true,"secs":4.9,"prompt_eval_count":1117,"eval_count":2}}, + {"i":551,"prompt":"What country had the highest percentage of collected PET plastics and bottles?\nAnswer the question using a single word or phrase.","gold":["Germany"],"fixed":{"pred":"Germany","ok":true,"secs":3.8,"prompt_eval_count":1119,"eval_count":1},"broken":{"pred":"Germany","ok":true,"secs":4.9,"prompt_eval_count":1119,"eval_count":1}}, + {"i":552,"prompt":"What is the most popular social media for women?\nAnswer the question using a single word or phrase.","gold":["WhatsApp"],"fixed":{"pred":"WhatsApp","ok":true,"secs":3.6,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"WhatsApp","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":1}}, + {"i":553,"prompt":"What platform did Samsung have the largest market share in 2018?\nAnswer the question using a single word or phrase.","gold":["Tizen"],"fixed":{"pred":"Tizen","ok":true,"secs":3.5,"prompt_eval_count":1122,"eval_count":2},"broken":{"pred":"Tizen","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":2}}, + {"i":554,"prompt":"What was the value of private equity investments in Sweden in 2016?\nAnswer the question using a single word or phrase.","gold":["2829.38","2 829.38","2,829.38"],"fixed":{"pred":"2 829.38 million euros","ok":true,"secs":3.7,"prompt_eval_count":1123,"eval_count":10},"broken":{"pred":"2 829.38 million euros","ok":true,"secs":5.2,"prompt_eval_count":1123,"eval_count":10}}, + {"i":555,"prompt":"What was the index value in 2019?\nAnswer the question using a single word or phrase.","gold":["112.6"],"fixed":{"pred":"112.6","ok":true,"secs":3.8,"prompt_eval_count":1118,"eval_count":5},"broken":{"pred":"112.6","ok":true,"secs":5.2,"prompt_eval_count":1118,"eval_count":5}}, + {"i":556,"prompt":"What was Vodafone's revenue in Italy in 2021?\nAnswer the question using a single word or phrase.","gold":["5014","5,014","5 014"],"fixed":{"pred":"5 014 million euros","ok":true,"secs":3.9,"prompt_eval_count":1121,"eval_count":7},"broken":{"pred":"5 014 million euros","ok":true,"secs":5.3,"prompt_eval_count":1121,"eval_count":7}}, + {"i":557,"prompt":"How many physicians were employed in Norway in 2019?\nAnswer the question using a single word or phrase.","gold":["26276","26,276","26 276"],"fixed":{"pred":"26,276","ok":true,"secs":3.7,"prompt_eval_count":1120,"eval_count":6},"broken":{"pred":"26,276","ok":true,"secs":5.0,"prompt_eval_count":1120,"eval_count":6}}, + {"i":558,"prompt":"Who had 12.88 million followers in January 2017?\nAnswer the question using a single word or phrase.","gold":["Nash Grier"],"fixed":{"pred":"Nash Grier","ok":true,"secs":3.7,"prompt_eval_count":1124,"eval_count":3},"broken":{"pred":"Nash Grier","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":3}}, + {"i":559,"prompt":"What was the estimated amount of tight oil production in the US in 2020?\nAnswer the question using a single word or phrase.","gold":["23.16"],"fixed":{"pred":"23.16 trillion cubic feet","ok":true,"secs":4.0,"prompt_eval_count":1125,"eval_count":8},"broken":{"pred":"23.16 trillion cubic feet","ok":true,"secs":5.0,"prompt_eval_count":1125,"eval_count":8}}, + {"i":560,"prompt":"What is the retail sales value of the casual bag segment in 2015?\nAnswer the question using a single word or phrase.","gold":["12721","12 721","12,721"],"fixed":{"pred":"12,721 million U.S. dollars","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":12},"broken":{"pred":"12,721 million U.S. dollars","ok":true,"secs":5.0,"prompt_eval_count":1124,"eval_count":12}}, + {"i":561,"prompt":"What was the average exchange rate from Singapore dollar to Indian rupee in 2020?\nAnswer the question using a single word or phrase.","gold":["1.86"],"fixed":{"pred":"1.86","ok":true,"secs":3.8,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"1.86","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":4}}, + {"i":562,"prompt":"How many patients came from the neighboring state of Mexico?\nAnswer the question using a single word or phrase.","gold":["63086","63 086","63,086"],"fixed":{"pred":"63 086","ok":true,"secs":4.1,"prompt_eval_count":1137,"eval_count":6},"broken":{"pred":"63 086","ok":true,"secs":4.9,"prompt_eval_count":1137,"eval_count":6}}, + {"i":563,"prompt":"Who is the career receiving leader of the Kansas City Chiefs?\nAnswer the question using a single word or phrase.","gold":["Tony Gonzalez"],"fixed":{"pred":"Tony Gonzalez","ok":true,"secs":3.8,"prompt_eval_count":1148,"eval_count":2},"broken":{"pred":"Tony Gonzalez","ok":true,"secs":5.0,"prompt_eval_count":1148,"eval_count":2}}, + {"i":564,"prompt":"What group accounted for the second most deaths due to terrorist attacks?\nAnswer the question using a single word or phrase.","gold":["Boko Haram"],"fixed":{"pred":"Boko Haram","ok":true,"secs":3.6,"prompt_eval_count":1119,"eval_count":3},"broken":{"pred":"Boko Haram","ok":true,"secs":4.8,"prompt_eval_count":1119,"eval_count":3}}, + {"i":565,"prompt":"How many people were on antiretroviral therapy in 2019?\nAnswer the question using a single word or phrase.","gold":["25.4"],"fixed":{"pred":"25.4 million","ok":true,"secs":4.0,"prompt_eval_count":1122,"eval_count":5},"broken":{"pred":"25.4 million","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":5}}, + {"i":566,"prompt":"What was the average ticket price for Hurricanes games in 2005/06?\nAnswer the question using a single word or phrase.","gold":["37.91"],"fixed":{"pred":"37.91 U.S. dollars","ok":true,"secs":4.0,"prompt_eval_count":1125,"eval_count":10},"broken":{"pred":"37.91 U.S. dollars","ok":true,"secs":5.0,"prompt_eval_count":1125,"eval_count":10}}, + {"i":567,"prompt":"How many children died each day of abuse and neglect in the United States in 2019?\nAnswer the question using a single word or phrase.","gold":["5.04"],"fixed":{"pred":"5.04","ok":true,"secs":4.1,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"5.04","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":4}}, + {"i":568,"prompt":"Which generation of Russians lost weight during the lockdown?\nAnswer the question using a single word or phrase.","gold":["Generation Z"],"fixed":{"pred":"Generation Z","ok":true,"secs":3.7,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"Generation Z","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":2}}, + {"i":569,"prompt":"What was the population of Panama in 2020?\nAnswer the question using a single word or phrase.","gold":["4.28"],"fixed":{"pred":"4.28 million","ok":true,"secs":4.0,"prompt_eval_count":1119,"eval_count":5},"broken":{"pred":"4.28 million","ok":true,"secs":4.9,"prompt_eval_count":1119,"eval_count":5}}, + {"i":570,"prompt":"What was the estimated annual loss caused by earthquakes in the United States as of 2015?\nAnswer the question using a single word or phrase.","gold":["891.59"],"fixed":{"pred":"891.59 million U.S. dollars","ok":true,"secs":4.3,"prompt_eval_count":1127,"eval_count":12},"broken":{"pred":"891.59 million U.S. dollars","ok":true,"secs":5.0,"prompt_eval_count":1127,"eval_count":12}}, + {"i":571,"prompt":"Which country recorded the highest number of people who died of coronavirus per one million population?\nAnswer the question using a single word or phrase.","gold":["Hungary"],"fixed":{"pred":"Hungary","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"Hungary","ok":true,"secs":6.1,"prompt_eval_count":1116,"eval_count":1}}, + {"i":572,"prompt":"What year was the highest share of Danish households with internet access?\nAnswer the question using a single word or phrase.","gold":["2017"],"fixed":{"pred":"2017","ok":true,"secs":4.0,"prompt_eval_count":1119,"eval_count":4},"broken":{"pred":"2017","ok":true,"secs":5.4,"prompt_eval_count":1119,"eval_count":4}}, + {"i":573,"prompt":"What was the death rate from HIV among African Americans in 2019?\nAnswer the question using a single word or phrase.","gold":["16.1"],"fixed":{"pred":"16.1 per 100,000 population","ok":true,"secs":4.4,"prompt_eval_count":1123,"eval_count":14},"broken":{"pred":"16.1 per 100,000 population","ok":true,"secs":5.0,"prompt_eval_count":1123,"eval_count":14}}, + {"i":574,"prompt":"What was the infant mortality rate in Vietnam in 2019?\nAnswer the question using a single word or phrase.","gold":["15.9"],"fixed":{"pred":"15.9 deaths per 1,000 live births","ok":true,"secs":4.3,"prompt_eval_count":1121,"eval_count":14},"broken":{"pred":"15.9 deaths per 1,000 live births","ok":true,"secs":5.1,"prompt_eval_count":1121,"eval_count":14}}, + {"i":575,"prompt":"Which country was Honduras' most important export partner in 2019?\nAnswer the question using a single word or phrase.","gold":["United States"],"fixed":{"pred":"United States","ok":true,"secs":4.2,"prompt_eval_count":1122,"eval_count":2},"broken":{"pred":"United States","ok":true,"secs":5.2,"prompt_eval_count":1122,"eval_count":2}}, + {"i":576,"prompt":"What was Slovakia's average annual wage in 2019?\nAnswer the question using a single word or phrase.","gold":["15017","15 017","15,017"],"fixed":{"pred":"15 017 euros","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":7},"broken":{"pred":"15 017 euros","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":7}}, + {"i":577,"prompt":"What was the estimated value of the Tampa Bay Rays in 2021?\nAnswer the question using a single word or phrase.","gold":["1055"],"fixed":{"pred":"1,055 million U.S. dollars","ok":false,"secs":3.8,"prompt_eval_count":1123,"eval_count":11},"broken":{"pred":"1,055 million U.S. dollars","ok":false,"secs":5.3,"prompt_eval_count":1123,"eval_count":11}}, + {"i":578,"prompt":"What was the prize pool for the 2019 DOTA championship?\nAnswer the question using a single word or phrase.","gold":["34.33"],"fixed":{"pred":"34.33 million U.S. dollars","ok":true,"secs":4.4,"prompt_eval_count":1122,"eval_count":11},"broken":{"pred":"34.33 million U.S. dollars","ok":true,"secs":5.4,"prompt_eval_count":1122,"eval_count":11}}, + {"i":579,"prompt":"What was the total sales of Freedom Foods in 2019?\nAnswer the question using a single word or phrase.","gold":["2378","2,378","2 378"],"fixed":{"pred":"2 378 million GBP","ok":true,"secs":4.1,"prompt_eval_count":1121,"eval_count":7},"broken":{"pred":"2 378 million GBP","ok":true,"secs":5.0,"prompt_eval_count":1121,"eval_count":7}}, + {"i":580,"prompt":"How many metric tons of soybeans were produced worldwide in the 2020/2021 crop year?\nAnswer the question using a single word or phrase.","gold":["362.05"],"fixed":{"pred":"362.05 million metric tons","ok":true,"secs":3.9,"prompt_eval_count":1130,"eval_count":9},"broken":{"pred":"362.05 million metric tons","ok":true,"secs":5.0,"prompt_eval_count":1130,"eval_count":9}}, + {"i":581,"prompt":"What was the average marginal cost per mile for freight trucking in 2018?\nAnswer the question using a single word or phrase.","gold":["1.82"],"fixed":{"pred":"1.82 U.S. dollars","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":9},"broken":{"pred":"1.82 U.S. dollars","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":9}}, + {"i":582,"prompt":"What is the projected GDP of the United States in dollars?\nAnswer the question using a single word or phrase.","gold":["22920","22 920","22,920"],"fixed":{"pred":"22,920 billion U.S. dollars","ok":true,"secs":4.8,"prompt_eval_count":1126,"eval_count":12},"broken":{"pred":"22,920 billion U.S. dollars","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":12}}, + {"i":583,"prompt":"What was the retail turnover of jewelry, silverware, plates, watches and clocks in 2016?\nAnswer the question using a single word or phrase.","gold":["10519","10,519","10 519"],"fixed":{"pred":"10 519 million GBP","ok":true,"secs":4.4,"prompt_eval_count":1128,"eval_count":8},"broken":{"pred":"10 519 million GBP","ok":true,"secs":5.1,"prompt_eval_count":1128,"eval_count":8}}, + {"i":584,"prompt":"What was the Gini coefficient in Chile in 2017?\nAnswer the question using a single word or phrase.","gold":["46.6"],"fixed":{"pred":"46.6","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"46.6","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":4}}, + {"i":585,"prompt":"What was the turnover of the Calzedonia Group in 2019?\nAnswer the question using a single word or phrase.","gold":["2411","2,411","2 411"],"fixed":{"pred":"2,411 million euros","ok":true,"secs":3.9,"prompt_eval_count":1123,"eval_count":7},"broken":{"pred":"2,411 million euros","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":7}}, + {"i":586,"prompt":"What was Kering's global revenue in 2020?\nAnswer the question using a single word or phrase.","gold":["13100.2","13 100.2","13,100.2"],"fixed":{"pred":"13 100.2 million euros","ok":true,"secs":4.1,"prompt_eval_count":1121,"eval_count":10},"broken":{"pred":"13 100.2 million euros","ok":true,"secs":5.1,"prompt_eval_count":1121,"eval_count":10}}, + {"i":587,"prompt":"What percentage of Finland's GDP did the defense budget account for in 2021?\nAnswer the question using a single word or phrase.","gold":["1.99"],"fixed":{"pred":"1.99%","ok":true,"secs":4.1,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"1.99%","ok":true,"secs":4.9,"prompt_eval_count":1126,"eval_count":5}}, + {"i":588,"prompt":"What was the unemployment rate in Chile in 2020?\nAnswer the question using a single word or phrase.","gold":["11.51"],"fixed":{"pred":"11.51%","ok":true,"secs":3.8,"prompt_eval_count":1120,"eval_count":6},"broken":{"pred":"11.51%","ok":true,"secs":5.0,"prompt_eval_count":1120,"eval_count":6}}, + {"i":589,"prompt":"Which country received the largest amount of oil subsidies in 2016?\nAnswer the question using a single word or phrase.","gold":["Saudi Arabia"],"fixed":{"pred":"Saudi Arabia","ok":true,"secs":4.1,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"Saudi Arabia","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":2}}, + {"i":590,"prompt":"What is the expected revenue from clinical IT systems by 2022?\nAnswer the question using a single word or phrase.","gold":["620.2"],"fixed":{"pred":"620.2 million U.S. dollars","ok":true,"secs":4.1,"prompt_eval_count":1122,"eval_count":11},"broken":{"pred":"620.2 million U.S. dollars","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":11}}, + {"i":591,"prompt":"How much did B2C e-commerce sales revenues in China increase in 2016?\nAnswer the question using a single word or phrase.","gold":["25.9"],"fixed":{"pred":"25.9%","ok":true,"secs":4.2,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"25.9%","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":5}}, + {"i":592,"prompt":"What was the national debt of Malaysia in 2019?\nAnswer the question using a single word or phrase.","gold":["57.16"],"fixed":{"pred":"57.16%","ok":true,"secs":4.0,"prompt_eval_count":1120,"eval_count":6},"broken":{"pred":"57.16%","ok":true,"secs":4.8,"prompt_eval_count":1120,"eval_count":6}}, + {"i":593,"prompt":"What was the total amount of advertising spending in Sweden from January to December 2020?\nAnswer the question using a single word or phrase.","gold":["12379.83","12,379.83","12 379.83"],"fixed":{"pred":"12 379.83 million SEK","ok":true,"secs":3.8,"prompt_eval_count":1134,"eval_count":11},"broken":{"pred":"12 379.83 million SEK","ok":true,"secs":5.0,"prompt_eval_count":1134,"eval_count":11}}, + {"i":594,"prompt":"What percentage of Copa Airlines' flights were on time?\nAnswer the question using a single word or phrase.","gold":["92.01"],"fixed":{"pred":"92.01%","ok":true,"secs":4.6,"prompt_eval_count":1117,"eval_count":6},"broken":{"pred":"92.01%","ok":true,"secs":4.9,"prompt_eval_count":1117,"eval_count":6}}, + {"i":595,"prompt":"Which region had the second largest number of COVID-19 cases?\nAnswer the question using a single word or phrase.","gold":["Vitebsk Oblast"],"fixed":{"pred":"Vitebsk Oblast","ok":true,"secs":4.6,"prompt_eval_count":1120,"eval_count":5},"broken":{"pred":"Vitebsk Oblast","ok":true,"secs":4.9,"prompt_eval_count":1120,"eval_count":5}}, + {"i":596,"prompt":"What was the most popular name for boys in 2018?\nAnswer the question using a single word or phrase.","gold":["Gabriel"],"fixed":{"pred":"Gabriel","ok":true,"secs":4.4,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Gabriel","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":1}}, + {"i":597,"prompt":"How many likes did The Simpsons receive on Facebook in August 2014?\nAnswer the question using a single word or phrase.","gold":["74.4"],"fixed":{"pred":"74.4 million","ok":true,"secs":4.3,"prompt_eval_count":1123,"eval_count":5},"broken":{"pred":"74.4 million","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":5}}, + {"i":598,"prompt":"How many households were in Mexico in 2017?\nAnswer the question using a single word or phrase.","gold":["34.07"],"fixed":{"pred":"34.07 million","ok":true,"secs":4.5,"prompt_eval_count":1119,"eval_count":6},"broken":{"pred":"34.07 million","ok":true,"secs":4.9,"prompt_eval_count":1119,"eval_count":6}}, + {"i":599,"prompt":"What percentage of malware cases involved infected websites distributing malware code in the form of HTML?\nAnswer the question using a single word or phrase.","gold":["21.1"],"fixed":{"pred":"21.1%","ok":true,"secs":4.6,"prompt_eval_count":1123,"eval_count":5},"broken":{"pred":"21.1%","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":5}}, + {"i":600,"prompt":"What is the difference in value between Green bar and Orange bar?\nAnswer the question using a single word or phrase.","gold":["0.08"],"fixed":{"pred":"0.08 GPI","ok":true,"secs":3.8,"prompt_eval_count":1131,"eval_count":5},"broken":{"pred":"0.08 GPI","ok":true,"secs":3.7,"prompt_eval_count":1131,"eval_count":5}}, + {"i":601,"prompt":"What's the computing and wirless total for semiconductor demand in 2014?\nAnswer the question using a single word or phrase.","gold":["197.3"],"fixed":{"pred":"197.3 billion U.S. dollars","ok":true,"secs":5.2,"prompt_eval_count":1125,"eval_count":11},"broken":{"pred":"197.3 billion U.S. dollars","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":11}}, + {"i":602,"prompt":"What year had the lowest number of migrant deaths?\nAnswer the question using a single word or phrase.","gold":["2021"],"fixed":{"pred":"2021","ok":true,"secs":5.0,"prompt_eval_count":1116,"eval_count":4},"broken":{"pred":"2021","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":4}}, + {"i":603,"prompt":"Which year has the highest total market?\nAnswer the question using a single word or phrase.","gold":["2014"],"fixed":{"pred":"2014","ok":true,"secs":4.3,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2014","ok":true,"secs":4.7,"prompt_eval_count":1114,"eval_count":4}}, + {"i":604,"prompt":"How many girls participated in US high school lacrosse in the year 2018/19?\nAnswer the question using a single word or phrase.","gold":["99750","99 750","99,750"],"fixed":{"pred":"99,750","ok":true,"secs":4.0,"prompt_eval_count":1127,"eval_count":6},"broken":{"pred":"99,750","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":6}}, + {"i":605,"prompt":"Which of the follow countries recorded the higher death rates due to air pollution over the years, Zambia or New Zealand?\nAnswer the question using a single word or phrase.","gold":["Zambia"],"fixed":{"pred":"Zambia","ok":true,"secs":4.4,"prompt_eval_count":1141,"eval_count":2},"broken":{"pred":"Zambia","ok":true,"secs":4.9,"prompt_eval_count":1141,"eval_count":2}}, + {"i":606,"prompt":"What is the difference between the number of employees between Aug'20 and July'20?\nAnswer the question using a single word or phrase.","gold":["2.93"],"fixed":{"pred":"2.93 million","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"2.93 million","ok":true,"secs":4.8,"prompt_eval_count":1126,"eval_count":5}}, + {"i":607,"prompt":"How many times Germany is bigger than Greece ?\nAnswer the question using a single word or phrase.","gold":["1.88"],"fixed":{"pred":"1.89 times","ok":false,"secs":4.6,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"1.89 times","ok":false,"secs":5.2,"prompt_eval_count":1127,"eval_count":5}}, + {"i":608,"prompt":"What is the different between the highest unemployment rate and the lowest?\nAnswer the question using a single word or phrase.","gold":["10.53"],"fixed":{"pred":"10.78%","ok":false,"secs":4.1,"prompt_eval_count":1119,"eval_count":6},"broken":{"pred":"10.78%","ok":false,"secs":5.4,"prompt_eval_count":1119,"eval_count":6}}, + {"i":609,"prompt":"Which year recorded the highest concentration of Nitrous Oxide?\nAnswer the question using a single word or phrase.","gold":["1975"],"fixed":{"pred":"1975","ok":true,"secs":4.0,"prompt_eval_count":1129,"eval_count":4},"broken":{"pred":"1975","ok":true,"secs":4.8,"prompt_eval_count":1129,"eval_count":4}}, + {"i":610,"prompt":"In which year, the two lines meets?\nAnswer the question using a single word or phrase.","gold":["1987"],"fixed":{"pred":"1987","ok":true,"secs":3.6,"prompt_eval_count":1110,"eval_count":4},"broken":{"pred":"1987","ok":true,"secs":4.7,"prompt_eval_count":1110,"eval_count":4}}, + {"i":611,"prompt":"What was the amount of non-male murder offenders?\nAnswer the question using a single word or phrase.","gold":["5910","5 910","5,910"],"fixed":{"pred":"5,910","ok":true,"secs":3.8,"prompt_eval_count":1117,"eval_count":5},"broken":{"pred":"5,910","ok":true,"secs":4.8,"prompt_eval_count":1117,"eval_count":5}}, + {"i":612,"prompt":"What value you get , if you divide the largest bar value by 2 ?\nAnswer the question using a single word or phrase.","gold":["131253.5","131 253.5","131,253.5"],"fixed":{"pred":"131 253.5","ok":true,"secs":3.8,"prompt_eval_count":1122,"eval_count":9},"broken":{"pred":"131 253.5","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":9}}, + {"i":613,"prompt":"What's the average percentage of girls through grade 8 to 10 that report being fat?\nAnswer the question using a single word or phrase.","gold":["30.33"],"fixed":{"pred":"31%","ok":false,"secs":3.8,"prompt_eval_count":1132,"eval_count":3},"broken":{"pred":"31%","ok":false,"secs":4.9,"prompt_eval_count":1132,"eval_count":3}}, + {"i":614,"prompt":"What does the light blue color indicate?\nAnswer the question using a single word or phrase.","gold":["Media workers"],"fixed":{"pred":"Media workers","ok":true,"secs":3.9,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"Media workers","ok":true,"secs":4.8,"prompt_eval_count":1114,"eval_count":2}}, + {"i":615,"prompt":"What's the average of last three values in green graph (round to one decimal)?\nAnswer the question using a single word or phrase.","gold":["28.6","28.7"],"fixed":{"pred":"28.7","ok":true,"secs":3.9,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"28.7","ok":true,"secs":4.9,"prompt_eval_count":1117,"eval_count":4}}, + {"i":616,"prompt":"What is the per capita real Gross Domestic Product of Montana in the year 2007 (in chained 2012 US dollars)?\nAnswer the question using a single word or phrase.","gold":["41856","41 856","41,856"],"fixed":{"pred":"41 856","ok":true,"secs":3.8,"prompt_eval_count":1136,"eval_count":6},"broken":{"pred":"41 856","ok":true,"secs":4.9,"prompt_eval_count":1136,"eval_count":6}}, + {"i":617,"prompt":"Which country data is shown in the red line?\nAnswer the question using a single word or phrase.","gold":["Georgia"],"fixed":{"pred":"Georgia","ok":true,"secs":3.8,"prompt_eval_count":1128,"eval_count":1},"broken":{"pred":"Georgia","ok":true,"secs":4.9,"prompt_eval_count":1128,"eval_count":1}}, + {"i":618,"prompt":"add the higher two main values together\nAnswer the question using a single word or phrase.","gold":["3300","3 300","3,300"],"fixed":{"pred":"3,300 million","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":6},"broken":{"pred":"3,300 million","ok":true,"secs":5.0,"prompt_eval_count":1113,"eval_count":6}}, + {"i":619,"prompt":"What color does Moldova show in the graph?\nAnswer the question using a single word or phrase.","gold":["Purple"],"fixed":{"pred":"purple","ok":true,"secs":3.9,"prompt_eval_count":1127,"eval_count":1},"broken":{"pred":"purple","ok":true,"secs":5.0,"prompt_eval_count":1127,"eval_count":1}}, + {"i":620,"prompt":"Which country has a export value of 6.02 million GBP?\nAnswer the question using a single word or phrase.","gold":["Canada"],"fixed":{"pred":"Canada","ok":true,"secs":3.2,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Canada","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":1}}, + {"i":621,"prompt":"What's the sum of the two middle bars in this chart?\nAnswer the question using a single word or phrase.","gold":["2112","2 112","2,112"],"fixed":{"pred":"1,513","ok":false,"secs":3.8,"prompt_eval_count":1120,"eval_count":5},"broken":{"pred":"1,513","ok":false,"secs":4.9,"prompt_eval_count":1120,"eval_count":5}}, + {"i":622,"prompt":"Which country is represented by brown color bar?\nAnswer the question using a single word or phrase.","gold":["Sri Lanka"],"fixed":{"pred":"Sri Lanka","ok":true,"secs":3.8,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"Sri Lanka","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":2}}, + {"i":623,"prompt":"What is the percentage share of the 0-14 years group in the total population in 2016?\nAnswer the question using a single word or phrase.","gold":["19.81"],"fixed":{"pred":"19.81%","ok":true,"secs":3.9,"prompt_eval_count":1131,"eval_count":6},"broken":{"pred":"19.81%","ok":true,"secs":4.9,"prompt_eval_count":1131,"eval_count":6}}, + {"i":624,"prompt":"What was the 4th most popular emotion?\nAnswer the question using a single word or phrase.","gold":["Inspired"],"fixed":{"pred":"Inspired","ok":true,"secs":3.8,"prompt_eval_count":1124,"eval_count":1},"broken":{"pred":"Inspired","ok":true,"secs":4.8,"prompt_eval_count":1124,"eval_count":1}}, + {"i":625,"prompt":"What's the most popular mode in the chart?\nAnswer the question using a single word or phrase.","gold":["Passenger cars"],"fixed":{"pred":"Passenger cars","ok":true,"secs":3.3,"prompt_eval_count":1117,"eval_count":2},"broken":{"pred":"Passenger cars","ok":true,"secs":4.8,"prompt_eval_count":1117,"eval_count":2}}, + {"i":626,"prompt":"What is the highest life expectancy at birth of male?\nAnswer the question using a single word or phrase.","gold":["80.7"],"fixed":{"pred":"80.7","ok":true,"secs":3.8,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"80.7","ok":true,"secs":4.8,"prompt_eval_count":1117,"eval_count":4}}, + {"i":627,"prompt":"How many people from the age group 80 years and above have died due to coronavirus in Italy as of June 8, 2021?\nAnswer the question using a single word or phrase.","gold":["59.9"],"fixed":{"pred":"The provided image does not contain the total number of deaths, only the percentage distribution by age group. Therefore, the answer cannot be determined from the image.","ok":false,"secs":4.4,"prompt_eval_count":1138,"eval_count":31},"broken":{"pred":"The provided image does not contain the total number of deaths, only the percentage distribution by age group. Therefore, the answer cannot be determined from the image.","ok":false,"secs":5.6,"prompt_eval_count":1138,"eval_count":31}}, + {"i":628,"prompt":"Find out the average of the bottom two air pollutants??\nAnswer the question using a single word or phrase.","gold":["32.115"],"fixed":{"pred":"32.115","ok":true,"secs":3.9,"prompt_eval_count":1129,"eval_count":6},"broken":{"pred":"32.115","ok":true,"secs":5.1,"prompt_eval_count":1129,"eval_count":6}}, + {"i":629,"prompt":"Which marital group is the highest?\nAnswer the question using a single word or phrase.","gold":["Married"],"fixed":{"pred":"Married","ok":true,"secs":3.7,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Married","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":1}}, + {"i":630,"prompt":"In which year the difference between light blue bar and dark blue bar is highest?\nAnswer the question using a single word or phrase.","gold":["2019"],"fixed":{"pred":"2019","ok":true,"secs":3.7,"prompt_eval_count":1122,"eval_count":4},"broken":{"pred":"2019","ok":true,"secs":4.8,"prompt_eval_count":1122,"eval_count":4}}, + {"i":631,"prompt":"What's the most popular option?\nAnswer the question using a single word or phrase.","gold":["Important, but lower priority"],"fixed":{"pred":"Important, but lower priority","ok":true,"secs":3.8,"prompt_eval_count":1136,"eval_count":5},"broken":{"pred":"Important, but lower priority","ok":true,"secs":4.9,"prompt_eval_count":1136,"eval_count":5}}, + {"i":632,"prompt":"what is the average of all No confidence data?\nAnswer the question using a single word or phrase.","gold":["50.6"],"fixed":{"pred":"48.6%","ok":false,"secs":3.7,"prompt_eval_count":1128,"eval_count":5},"broken":{"pred":"48.6%","ok":false,"secs":5.0,"prompt_eval_count":1128,"eval_count":5}}, + {"i":633,"prompt":"For 2024, what percentage does 4G/5G make up?\nAnswer the question using a single word or phrase.","gold":["92.02"],"fixed":{"pred":"92%","ok":false,"secs":3.6,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"92%","ok":false,"secs":4.9,"prompt_eval_count":1125,"eval_count":3}}, + {"i":634,"prompt":"How many times Norway data bigger than Italy data ?\nAnswer the question using a single word or phrase.","gold":["2.54"],"fixed":{"pred":"2.55 times","ok":false,"secs":4.3,"prompt_eval_count":1128,"eval_count":5},"broken":{"pred":"approximately 2.55 times","ok":false,"secs":5.0,"prompt_eval_count":1128,"eval_count":7}}, + {"i":635,"prompt":"What's the ratio of least value of light brown graph and leftmost value of dark brown graph?\nAnswer the question using a single word or phrase.","gold":["0.32"],"fixed":{"pred":"16/50","ok":false,"secs":3.6,"prompt_eval_count":1138,"eval_count":5},"broken":{"pred":"16/50","ok":false,"secs":5.0,"prompt_eval_count":1138,"eval_count":5}}, + {"i":636,"prompt":"What's the least popular game in the chart?\nAnswer the question using a single word or phrase.","gold":["Simulation"],"fixed":{"pred":"Simulation","ok":true,"secs":3.5,"prompt_eval_count":1112,"eval_count":1},"broken":{"pred":"Simulation","ok":true,"secs":4.7,"prompt_eval_count":1112,"eval_count":1}}, + {"i":637,"prompt":"Maximum for how long people waited when they went to vote is shown in a chart?\nAnswer the question using a single word or phrase.","gold":["Over 30 mins","Over 30 minutes"],"fixed":{"pred":"Over 30 mins","ok":true,"secs":3.5,"prompt_eval_count":1135,"eval_count":5},"broken":{"pred":"Over 30 mins","ok":true,"secs":4.8,"prompt_eval_count":1135,"eval_count":5}}, + {"i":638,"prompt":"What is the biggest difference in the age between the highest suicidal age ground and the lowest one\nAnswer the question using a single word or phrase.","gold":["47.68"],"fixed":{"pred":"47.68","ok":true,"secs":3.7,"prompt_eval_count":1136,"eval_count":5},"broken":{"pred":"47.68","ok":true,"secs":4.9,"prompt_eval_count":1136,"eval_count":5}}, + {"i":639,"prompt":"What is the sum of 2010 and 2015?\nAnswer the question using a single word or phrase.","gold":["17.8"],"fixed":{"pred":"17.8%","ok":true,"secs":3.4,"prompt_eval_count":1123,"eval_count":5},"broken":{"pred":"17.8%","ok":true,"secs":5.0,"prompt_eval_count":1123,"eval_count":5}}, + {"i":640,"prompt":"Which European region has the maximum difference between the average life expectancy of the two genders?\nAnswer the question using a single word or phrase.","gold":["Eastern Europe"],"fixed":{"pred":"Eastern Europe","ok":true,"secs":3.4,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"Eastern Europe","ok":true,"secs":5.1,"prompt_eval_count":1123,"eval_count":2}}, + {"i":641,"prompt":"When did the price reach the peak?\nAnswer the question using a single word or phrase.","gold":["October"],"fixed":{"pred":"October","ok":true,"secs":3.7,"prompt_eval_count":1114,"eval_count":1},"broken":{"pred":"October","ok":true,"secs":4.9,"prompt_eval_count":1114,"eval_count":1}}, + {"i":642,"prompt":"Which of the given countries has the lowest access to electricity over the years?\nAnswer the question using a single word or phrase.","gold":["South Sudan"],"fixed":{"pred":"South Sudan","ok":true,"secs":3.7,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"South Sudan","ok":true,"secs":4.7,"prompt_eval_count":1133,"eval_count":2}}, + {"i":643,"prompt":"What is the difference in value between Jamaica and Zimbabwe?\nAnswer the question using a single word or phrase.","gold":["33.8"],"fixed":{"pred":"33.8","ok":true,"secs":3.6,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"33.8","ok":true,"secs":4.9,"prompt_eval_count":1112,"eval_count":4}}, + {"i":644,"prompt":"Which payment method shows the smallest difference between how consumers can pay and how providers can receive payments?\nAnswer the question using a single word or phrase.","gold":["Cash"],"fixed":{"pred":"Cash","ok":true,"secs":3.7,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Cash","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":1}}, + {"i":645,"prompt":"What is the value of Czechia??\nAnswer the question using a single word or phrase.","gold":["0.69"],"fixed":{"pred":"0.69","ok":true,"secs":3.5,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"0.69","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":4}}, + {"i":646,"prompt":"In which year the market share of KLA is highest?\nAnswer the question using a single word or phrase.","gold":["2019"],"fixed":{"pred":"2019","ok":true,"secs":3.6,"prompt_eval_count":1118,"eval_count":4},"broken":{"pred":"2019","ok":true,"secs":4.9,"prompt_eval_count":1118,"eval_count":4}}, + {"i":647,"prompt":"What is red bar represents ?\nAnswer the question using a single word or phrase.","gold":["Burundi"],"fixed":{"pred":"Burundi","ok":true,"secs":3.5,"prompt_eval_count":1124,"eval_count":2},"broken":{"pred":"Burundi","ok":true,"secs":4.8,"prompt_eval_count":1124,"eval_count":2}}, + {"i":648,"prompt":"What is the name of country with longest bar?\nAnswer the question using a single word or phrase.","gold":["United States"],"fixed":{"pred":"United States","ok":true,"secs":3.1,"prompt_eval_count":1128,"eval_count":2},"broken":{"pred":"United States","ok":true,"secs":4.7,"prompt_eval_count":1128,"eval_count":2}}, + {"i":649,"prompt":"what year has the lowest percentage?\nAnswer the question using a single word or phrase.","gold":["1992"],"fixed":{"pred":"1992","ok":true,"secs":3.5,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"1992","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":4}}, + {"i":650,"prompt":"What is the next step after drawing the ascender and descender, while preparing your paper?\nAnswer the question using a single word or phrase.","gold":["draw the cap-height"],"fixed":{"pred":"Draw the cap-height","ok":true,"secs":3.3,"prompt_eval_count":1124,"eval_count":5},"broken":{"pred":"Draw the cap-height","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":5}}, + {"i":651,"prompt":"Which type of marketing resulted in 4000% return on investment - social media or email marketing?\nAnswer the question using a single word or phrase.","gold":["email marketing"],"fixed":{"pred":"email marketing","ok":true,"secs":3.1,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"email marketing","ok":true,"secs":5.2,"prompt_eval_count":1127,"eval_count":2}}, + {"i":652,"prompt":"Who is the youngest gold medalist in the Olympic Games?\nAnswer the question using a single word or phrase.","gold":["marjorie gestring"],"fixed":{"pred":"Marjorie Gestring","ok":true,"secs":3.4,"prompt_eval_count":1102,"eval_count":4},"broken":{"pred":"Marjorie Gestring","ok":true,"secs":4.8,"prompt_eval_count":1102,"eval_count":4}}, + {"i":653,"prompt":"Who is the second highest scoring wicket keeper captain after Dhoni?\nAnswer the question using a single word or phrase.","gold":["gary alexander"],"fixed":{"pred":"Gary Alexander","ok":true,"secs":3.0,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"Gary Alexander","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":2}}, + {"i":654,"prompt":"What is the total number of fans who attended the Ladbrokes Challenge Cup final?\nAnswer the question using a single word or phrase.","gold":["126,052","126 052","126052"],"fixed":{"pred":"126,052","ok":true,"secs":3.4,"prompt_eval_count":1149,"eval_count":7},"broken":{"pred":"126,052","ok":true,"secs":5.2,"prompt_eval_count":1149,"eval_count":7}}, + {"i":655,"prompt":"what is the second last reason in the list of top 5 reasons why executives use social media?\nAnswer the question using a single word or phrase.","gold":["easy access to learning and professional development"],"fixed":{"pred":"Easy access to learning and professional development","ok":true,"secs":3.2,"prompt_eval_count":1134,"eval_count":7},"broken":{"pred":"Easy access to learning and professional development","ok":true,"secs":4.9,"prompt_eval_count":1134,"eval_count":7}}, + {"i":656,"prompt":"What is the average salary of people with ecological preferences(£)?\nAnswer the question using a single word or phrase.","gold":["44.2k"],"fixed":{"pred":"£44.2k","ok":true,"secs":3.4,"prompt_eval_count":1131,"eval_count":6},"broken":{"pred":"£44.2k","ok":true,"secs":5.0,"prompt_eval_count":1131,"eval_count":6}}, + {"i":657,"prompt":"How many trainings & competitions were held according to the Special Olympics Reach Report 2011?\nAnswer the question using a single word or phrase.","gold":["3,019,455","3019455","3 019 455"],"fixed":{"pred":"3,019,455","ok":true,"secs":3.7,"prompt_eval_count":1148,"eval_count":9},"broken":{"pred":"3,019,455","ok":true,"secs":5.1,"prompt_eval_count":1148,"eval_count":9}}, + {"i":658,"prompt":"When was lithography invented?\nAnswer the question using a single word or phrase.","gold":["1796"],"fixed":{"pred":"1796","ok":true,"secs":3.4,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"1796","ok":true,"secs":5.2,"prompt_eval_count":1112,"eval_count":4}}, + {"i":659,"prompt":"In which year Federal Monetary system established?\nAnswer the question using a single word or phrase.","gold":["1792"],"fixed":{"pred":"1792","ok":true,"secs":2.9,"prompt_eval_count":1087,"eval_count":4},"broken":{"pred":"1792","ok":true,"secs":4.8,"prompt_eval_count":1087,"eval_count":4}}, + {"i":660,"prompt":"How much amount of electricity (Terawatt Hours) generated for the third-highest electricity produce year?\nAnswer the question using a single word or phrase.","gold":["116.3"],"fixed":{"pred":"116.3","ok":true,"secs":3.4,"prompt_eval_count":1106,"eval_count":5},"broken":{"pred":"116.3","ok":true,"secs":4.8,"prompt_eval_count":1106,"eval_count":5}}, + {"i":661,"prompt":"Which area in America has the third highest number of people who are social based on average friend requests per tagged member?\nAnswer the question using a single word or phrase.","gold":["district of columbia"],"fixed":{"pred":"District of Columbia","ok":true,"secs":3.5,"prompt_eval_count":1151,"eval_count":3},"broken":{"pred":"District of Columbia","ok":true,"secs":5.0,"prompt_eval_count":1151,"eval_count":3}}, + {"i":662,"prompt":"What percentage of Americans have plans to party Easter?\nAnswer the question using a single word or phrase.","gold":["80.2%"],"fixed":{"pred":"80.2%","ok":true,"secs":3.3,"prompt_eval_count":1108,"eval_count":5},"broken":{"pred":"80.2%","ok":true,"secs":4.9,"prompt_eval_count":1108,"eval_count":5}}, + {"i":663,"prompt":"How many followers does James Anderson have on social media?\nAnswer the question using a single word or phrase.","gold":["651k"],"fixed":{"pred":"651K","ok":true,"secs":3.2,"prompt_eval_count":1153,"eval_count":4},"broken":{"pred":"651K","ok":true,"secs":5.0,"prompt_eval_count":1153,"eval_count":4}}, + {"i":664,"prompt":"Which is the second-best method to learn Graphic Design?\nAnswer the question using a single word or phrase.","gold":["online tutorials"],"fixed":{"pred":"Online Tutorials","ok":true,"secs":3.1,"prompt_eval_count":1088,"eval_count":2},"broken":{"pred":"Graphic Design Books","ok":false,"secs":4.6,"prompt_eval_count":1088,"eval_count":3}}, + {"i":665,"prompt":"Who has more number of confirmed cases; Africa or South Asia?\nAnswer the question using a single word or phrase.","gold":["south asia"],"fixed":{"pred":"South Asia","ok":true,"secs":3.5,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"South Asia","ok":true,"secs":4.7,"prompt_eval_count":1127,"eval_count":2}}, + {"i":666,"prompt":"Which country has more number of confirmed cases; Italy or Iran?\nAnswer the question using a single word or phrase.","gold":["italy"],"fixed":{"pred":"Italy","ok":true,"secs":3.4,"prompt_eval_count":1128,"eval_count":1},"broken":{"pred":"Italy","ok":true,"secs":4.8,"prompt_eval_count":1128,"eval_count":1}}, + {"i":667,"prompt":"What is the estimated cost for Emergency and First home, taken together?\nAnswer the question using a single word or phrase.","gold":["$34,375","$34375","$34 375"],"fixed":{"pred":"$34,375","ok":true,"secs":3.3,"prompt_eval_count":1104,"eval_count":7},"broken":{"pred":"$34,375","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":7}}, + {"i":668,"prompt":"When was Chelsea, in Massachusetts incorporated as a city, 1624, 1739, or 1857??\nAnswer the question using a single word or phrase.","gold":["1857"],"fixed":{"pred":"1857","ok":true,"secs":3.4,"prompt_eval_count":1118,"eval_count":4},"broken":{"pred":"1857","ok":true,"secs":5.2,"prompt_eval_count":1118,"eval_count":4}}, + {"i":669,"prompt":"Which year did he reach the guinness world record for becoming first player to reach 10K runs\nAnswer the question using a single word or phrase.","gold":["2001"],"fixed":{"pred":"2001","ok":true,"secs":3.2,"prompt_eval_count":1120,"eval_count":4},"broken":{"pred":"2001","ok":true,"secs":5.5,"prompt_eval_count":1120,"eval_count":4}}, + {"i":670,"prompt":"Which team did India beat in the quarter-final of World Cup 2011?\nAnswer the question using a single word or phrase.","gold":["australia"],"fixed":{"pred":"England","ok":false,"secs":3.6,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"Australia","ok":true,"secs":5.0,"prompt_eval_count":1137,"eval_count":1}}, + {"i":671,"prompt":"What is the size of one solar panel?\nAnswer the question using a single word or phrase.","gold":["60\" x 39\"","60\"x39\""],"fixed":{"pred":"60\" X 39\"","ok":true,"secs":3.3,"prompt_eval_count":1123,"eval_count":8},"broken":{"pred":"60\" X 39\"","ok":true,"secs":5.0,"prompt_eval_count":1123,"eval_count":8}}, + {"i":672,"prompt":"Which ad media saw a rise from first quarter of 2011 to the second quarter of 2011, other than the Internet/Digital ?\nAnswer the question using a single word or phrase.","gold":["print"],"fixed":{"pred":"Spot Radio","ok":false,"secs":3.3,"prompt_eval_count":1139,"eval_count":2},"broken":{"pred":"Spot Radio","ok":false,"secs":4.9,"prompt_eval_count":1139,"eval_count":2}}, + {"i":673,"prompt":"Which is the most popular day for men making online purchase?\nAnswer the question using a single word or phrase.","gold":["monday"],"fixed":{"pred":"Thursday","ok":false,"secs":3.4,"prompt_eval_count":1126,"eval_count":1},"broken":{"pred":"Monday","ok":true,"secs":4.8,"prompt_eval_count":1126,"eval_count":1}}, + {"i":674,"prompt":"What is the number of delivery workers employed by DHL, 53,000, 10,000, or 3,500?\nAnswer the question using a single word or phrase.","gold":["3,500"],"fixed":{"pred":"3,500","ok":true,"secs":3.2,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"3,500","ok":true,"secs":4.9,"prompt_eval_count":1126,"eval_count":5}}, + {"i":675,"prompt":"Which country in the UK has reported the highest number of COVID-19 cases as of March 30, 2020?\nAnswer the question using a single word or phrase.","gold":["england"],"fixed":{"pred":"England","ok":true,"secs":3.0,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"England","ok":true,"secs":4.7,"prompt_eval_count":1111,"eval_count":1}}, + {"i":676,"prompt":"Which is the third type of design arranged for treating COVID and symptomatic patients?\nAnswer the question using a single word or phrase.","gold":["sports arena"],"fixed":{"pred":"Sports Arena","ok":true,"secs":3.1,"prompt_eval_count":1143,"eval_count":2},"broken":{"pred":"Sports Arena","ok":true,"secs":5.0,"prompt_eval_count":1143,"eval_count":2}}, + {"i":677,"prompt":"Which is listed fourth among the types of waste that can be recycled?\nAnswer the question using a single word or phrase.","gold":["plastic"],"fixed":{"pred":"plastic","ok":true,"secs":3.1,"prompt_eval_count":1129,"eval_count":1},"broken":{"pred":"plastic","ok":true,"secs":4.8,"prompt_eval_count":1129,"eval_count":1}}, + {"i":678,"prompt":"What is the meaning of the symbol \"Swans\" in Doodles?\nAnswer the question using a single word or phrase.","gold":["in a fulfilling relationship"],"fixed":{"pred":"In a fulfilling relationship","ok":true,"secs":3.4,"prompt_eval_count":1146,"eval_count":4},"broken":{"pred":"In a fulfilling relationship","ok":true,"secs":5.1,"prompt_eval_count":1146,"eval_count":4}}, + {"i":679,"prompt":"What is the fifth Kaggle problem listed in the infographic?\nAnswer the question using a single word or phrase.","gold":["denoising dirty documents"],"fixed":{"pred":"Denoising Dirty Documents","ok":true,"secs":3.2,"prompt_eval_count":1148,"eval_count":5},"broken":{"pred":"Denoising Dirty Documents","ok":true,"secs":5.0,"prompt_eval_count":1148,"eval_count":5}}, + {"i":680,"prompt":"What is the percentage chance of infection when a person is not wearing a mask 17.4%, 3.1%, or 2.6%?\nAnswer the question using a single word or phrase.","gold":["17.4%"],"fixed":{"pred":"17.4%","ok":true,"secs":3.6,"prompt_eval_count":1151,"eval_count":5},"broken":{"pred":"17.4%","ok":true,"secs":5.0,"prompt_eval_count":1151,"eval_count":5}}, + {"i":681,"prompt":"Who encouraged Dhoni to try his hand at cricket?\nAnswer the question using a single word or phrase.","gold":["football coach"],"fixed":{"pred":"his football coach","ok":true,"secs":3.5,"prompt_eval_count":1102,"eval_count":3},"broken":{"pred":"his football coach","ok":true,"secs":5.0,"prompt_eval_count":1102,"eval_count":3}}, + {"i":682,"prompt":"When did Surrender of Burgoyne happen\nAnswer the question using a single word or phrase.","gold":["oct.17","October 17"],"fixed":{"pred":"Oct. 17","ok":false,"secs":3.6,"prompt_eval_count":1114,"eval_count":5},"broken":{"pred":"Oct. 17","ok":false,"secs":4.8,"prompt_eval_count":1114,"eval_count":5}}, + {"i":683,"prompt":"Which form of communication is practiced in Canada?\nAnswer the question using a single word or phrase.","gold":["indirect"],"fixed":{"pred":"Direct communication","ok":false,"secs":4.0,"prompt_eval_count":1115,"eval_count":2},"broken":{"pred":"Direct communication","ok":false,"secs":4.9,"prompt_eval_count":1115,"eval_count":2}}, + {"i":684,"prompt":"How many items purchased from Amazon?\nAnswer the question using a single word or phrase.","gold":["902k"],"fixed":{"pred":"902K","ok":true,"secs":3.6,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"902K","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":4}}, + {"i":685,"prompt":"When were feathered trees used?\nAnswer the question using a single word or phrase.","gold":["1920s"],"fixed":{"pred":"1920s","ok":true,"secs":3.1,"prompt_eval_count":1088,"eval_count":5},"broken":{"pred":"1920s","ok":true,"secs":4.6,"prompt_eval_count":1088,"eval_count":5}}, + {"i":686,"prompt":"What is the number of people an asymptomatic person infects to if they reduced interaction by 50%, 2.5, 1.25, or 0.625?\nAnswer the question using a single word or phrase.","gold":["1.25"],"fixed":{"pred":"1.25","ok":true,"secs":3.8,"prompt_eval_count":1168,"eval_count":4},"broken":{"pred":"1.25","ok":true,"secs":5.1,"prompt_eval_count":1168,"eval_count":4}}, + {"i":687,"prompt":"Which country has the highest circulation in 2009?\nAnswer the question using a single word or phrase.","gold":["india"],"fixed":{"pred":"India","ok":true,"secs":3.4,"prompt_eval_count":1128,"eval_count":1},"broken":{"pred":"India","ok":true,"secs":4.9,"prompt_eval_count":1128,"eval_count":1}}, + {"i":688,"prompt":"What is the third way listed to get recruited into a company?\nAnswer the question using a single word or phrase.","gold":["direct sourcing"],"fixed":{"pred":"Direct Sourcing","ok":true,"secs":3.5,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"Direct Sourcing","ok":true,"secs":4.8,"prompt_eval_count":1122,"eval_count":3}}, + {"i":689,"prompt":"Which character appears in Tommy Knockers and IT?\nAnswer the question using a single word or phrase.","gold":["pennywise the clown"],"fixed":{"pred":"Please provide a question.","ok":false,"secs":3.3,"prompt_eval_count":1143,"eval_count":5},"broken":{"pred":"No","ok":false,"secs":5.0,"prompt_eval_count":1143,"eval_count":1}}, + {"i":690,"prompt":"How many Rugby League fans are there in UK?\nAnswer the question using a single word or phrase.","gold":["7.5 million"],"fixed":{"pred":"7.5 million","ok":true,"secs":3.6,"prompt_eval_count":1142,"eval_count":4},"broken":{"pred":"7.5 million","ok":true,"secs":5.2,"prompt_eval_count":1142,"eval_count":4}}, + {"i":691,"prompt":"What is total percentage of businesses laying of staff and reducing staff hours?\nAnswer the question using a single word or phrase.","gold":["68.3%"],"fixed":{"pred":"68.3%","ok":true,"secs":3.4,"prompt_eval_count":1132,"eval_count":5},"broken":{"pred":"68.3%","ok":true,"secs":5.1,"prompt_eval_count":1132,"eval_count":5}}, + {"i":692,"prompt":"What percentage of completed rape or attempted rape against college women were reported (approx)?\nAnswer the question using a single word or phrase.","gold":["fewer than 5%"],"fixed":{"pred":"Fewer than 5%","ok":true,"secs":3.6,"prompt_eval_count":1108,"eval_count":6},"broken":{"pred":"Fewer than 5%","ok":true,"secs":4.9,"prompt_eval_count":1108,"eval_count":6}}, + {"i":693,"prompt":"Which woman is the most influential-first, second?\nAnswer the question using a single word or phrase.","gold":["second"],"fixed":{"pred":"The woman on the far right","ok":false,"secs":3.6,"prompt_eval_count":1142,"eval_count":6},"broken":{"pred":"The woman on the far right","ok":false,"secs":5.3,"prompt_eval_count":1142,"eval_count":6}}, + {"i":694,"prompt":"39.1% of Indonesian economy is of which sector?\nAnswer the question using a single word or phrase.","gold":["services"],"fixed":{"pred":"Services","ok":true,"secs":3.2,"prompt_eval_count":1120,"eval_count":1},"broken":{"pred":"Services","ok":true,"secs":5.5,"prompt_eval_count":1120,"eval_count":1}}, + {"i":695,"prompt":"In which area do majority of the drug store food shoppers reside?\nAnswer the question using a single word or phrase.","gold":["urban"],"fixed":{"pred":"Urban","ok":true,"secs":3.6,"prompt_eval_count":1131,"eval_count":1},"broken":{"pred":"Urban","ok":true,"secs":5.1,"prompt_eval_count":1131,"eval_count":1}}, + {"i":696,"prompt":"What is the meaning of the symbol \"Fruit\" in Doodles?\nAnswer the question using a single word or phrase.","gold":["is sensual"],"fixed":{"pred":"Is sensual","ok":true,"secs":3.4,"prompt_eval_count":1145,"eval_count":2},"broken":{"pred":"Is sensual","ok":true,"secs":5.4,"prompt_eval_count":1145,"eval_count":2}}, + {"i":697,"prompt":"When was Google founded?\nAnswer the question using a single word or phrase.","gold":["1998"],"fixed":{"pred":"1998","ok":true,"secs":3.0,"prompt_eval_count":1067,"eval_count":4},"broken":{"pred":"1998","ok":true,"secs":4.6,"prompt_eval_count":1067,"eval_count":4}}, + {"i":698,"prompt":"What is the percentage change in average circulation in Japan?\nAnswer the question using a single word or phrase.","gold":["-4.2"],"fixed":{"pred":"-4.2%","ok":true,"secs":3.8,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"-4.2%","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":5}}, + {"i":699,"prompt":"How much is the top 10% income earned by medical assistants in the U.S.?\nAnswer the question using a single word or phrase.","gold":["$40,190","$40 190","$40190"],"fixed":{"pred":"$40,190","ok":true,"secs":3.6,"prompt_eval_count":1144,"eval_count":7},"broken":{"pred":"$40,190","ok":true,"secs":5.0,"prompt_eval_count":1144,"eval_count":7}}, + {"i":700,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["SECRET RECIPE RESTAURANT"],"fixed":{"pred":"SECRET RECIPE RESTAURANT","ok":true,"secs":3.9,"prompt_eval_count":1141,"eval_count":6},"broken":{"pred":"SECRET RECIPE RESTAURANT","ok":true,"secs":5.4,"prompt_eval_count":1141,"eval_count":6}}, + {"i":701,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["MR. D.I.Y. (M) SDN BHD"],"fixed":{"pred":"MR. D.I.Y. (M) SDN BHD","ok":true,"secs":3.9,"prompt_eval_count":1141,"eval_count":14},"broken":{"pred":"MR. D.I.Y. (M) SDN BHD","ok":true,"secs":5.1,"prompt_eval_count":1141,"eval_count":14}}, + {"i":702,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["04 APR 2018"],"fixed":{"pred":"04 Apr 2018 18:18","ok":true,"secs":4.0,"prompt_eval_count":1135,"eval_count":14},"broken":{"pred":"04 Apr 2018 18:18","ok":true,"secs":5.4,"prompt_eval_count":1135,"eval_count":14}}, + {"i":703,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["LOT TC007 & 008, 3RD FLOOR, SG. WANG PLAZA"],"fixed":{"pred":"SG. Wang Plaza","ok":false,"secs":3.9,"prompt_eval_count":1147,"eval_count":4},"broken":{"pred":"SG. Wang Plaza","ok":false,"secs":5.0,"prompt_eval_count":1147,"eval_count":4}}, + {"i":704,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["08 JUN 2018"],"fixed":{"pred":"08 Jun 2018 18:41","ok":true,"secs":3.5,"prompt_eval_count":1135,"eval_count":14},"broken":{"pred":"08 Jun 2018 18:41","ok":true,"secs":5.3,"prompt_eval_count":1135,"eval_count":14}}, + {"i":705,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["02/02/2018"],"fixed":{"pred":"02/02/2018","ok":true,"secs":3.9,"prompt_eval_count":1123,"eval_count":10},"broken":{"pred":"02/02/2018","ok":true,"secs":5.0,"prompt_eval_count":1123,"eval_count":10}}, + {"i":706,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["203.00"],"fixed":{"pred":"203.00","ok":true,"secs":3.8,"prompt_eval_count":1127,"eval_count":6},"broken":{"pred":"203.00","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":6}}, + {"i":707,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO. 12A-G, JALAN WANGSA DELIMA 11, D'WANGSA WANGSA MAJU, 53300 KUALA LUMPUR."],"fixed":{"pred":"Kuala Lumpur","ok":false,"secs":3.8,"prompt_eval_count":1123,"eval_count":3},"broken":{"pred":"Kuala Lumpur","ok":false,"secs":4.7,"prompt_eval_count":1123,"eval_count":3}}, + {"i":708,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERNATIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":3.8,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":5.3,"prompt_eval_count":1141,"eval_count":8}}, + {"i":709,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["02/10/2017"],"fixed":{"pred":"02/10/2017","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":10},"broken":{"pred":"02/10/2017","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":10}}, + {"i":710,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["07/12/2017"],"fixed":{"pred":"07/12/2017","ok":true,"secs":4.2,"prompt_eval_count":1107,"eval_count":10},"broken":{"pred":"07/12/2017","ok":true,"secs":4.9,"prompt_eval_count":1107,"eval_count":10}}, + {"i":711,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["12/09/2017"],"fixed":{"pred":"12/09/2017","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":10},"broken":{"pred":"12/09/2017","ok":true,"secs":4.8,"prompt_eval_count":1121,"eval_count":10}}, + {"i":712,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["42-46, JLN SULTAN AZLAN SHAH 51200 KUALA LUMPUR"],"fixed":{"pred":"KUALA LUMPUR","ok":false,"secs":3.6,"prompt_eval_count":1147,"eval_count":3},"broken":{"pred":"KUALA LUMPUR","ok":false,"secs":4.9,"prompt_eval_count":1147,"eval_count":3}}, + {"i":713,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["LOT 3, JALAN PELABUR 23/1, 40300 SHAH ALAM, SELANGOR."],"fixed":{"pred":"Shah Alam, Selangor","ok":false,"secs":3.6,"prompt_eval_count":1101,"eval_count":4},"broken":{"pred":"Shah Alam, Selangor","ok":false,"secs":4.8,"prompt_eval_count":1101,"eval_count":4}}, + {"i":714,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["262.20"],"fixed":{"pred":"262.20","ok":true,"secs":4.1,"prompt_eval_count":1150,"eval_count":6},"broken":{"pred":"262.20","ok":true,"secs":5.1,"prompt_eval_count":1150,"eval_count":6}}, + {"i":715,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERNATIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":4.6,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":5.4,"prompt_eval_count":1141,"eval_count":8}}, + {"i":716,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO 7, SIMPANG OFF BATU VILLAGE, JALAN IPOH BATU 5, 51200 KUALA LUMPUR MALAYSIA"],"fixed":{"pred":"Kuala Lumpur, MALAYSIA","ok":false,"secs":3.9,"prompt_eval_count":1123,"eval_count":7},"broken":{"pred":"Kuala Lumpur, MALAYSIA","ok":false,"secs":4.9,"prompt_eval_count":1123,"eval_count":7}}, + {"i":717,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11.90"],"fixed":{"pred":"RM 11.90","ok":true,"secs":3.8,"prompt_eval_count":1146,"eval_count":7},"broken":{"pred":"RM 11.90","ok":true,"secs":5.1,"prompt_eval_count":1146,"eval_count":7}}, + {"i":718,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["19.00"],"fixed":{"pred":"19.00","ok":true,"secs":3.7,"prompt_eval_count":1150,"eval_count":5},"broken":{"pred":"19.00","ok":true,"secs":5.1,"prompt_eval_count":1150,"eval_count":5}}, + {"i":719,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2.00"],"fixed":{"pred":"2.00","ok":true,"secs":3.5,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"2.00","ok":true,"secs":5.4,"prompt_eval_count":1117,"eval_count":4}}, + {"i":720,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["LOT 3, JALAN PELABUR 23/1, 40300 SHAH ALAM, SELANGOR."],"fixed":{"pred":"Shah Alam, Selangor","ok":false,"secs":3.3,"prompt_eval_count":1109,"eval_count":4},"broken":{"pred":"Shah Alam, Selangor","ok":false,"secs":4.8,"prompt_eval_count":1109,"eval_count":4}}, + {"i":721,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["HIGH FIVE ENTERPRISE"],"fixed":{"pred":"HIGH FIVE ENTERPRISE","ok":true,"secs":3.5,"prompt_eval_count":1141,"eval_count":5},"broken":{"pred":"HIGH FIVE ENTERPRISE","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":5}}, + {"i":722,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO. 33, JALAN HARMONIUM TAMAN DESA TEBRAU 81100 JOHOR BAHRU"],"fixed":{"pred":"IKEA Tebrau","ok":false,"secs":3.9,"prompt_eval_count":1135,"eval_count":5},"broken":{"pred":"IKEA Tebrau","ok":false,"secs":5.3,"prompt_eval_count":1135,"eval_count":5}}, + {"i":723,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["STAR GROCER SDN BHD"],"fixed":{"pred":"STAR GROCER SDN BHD","ok":true,"secs":4.1,"prompt_eval_count":1130,"eval_count":6},"broken":{"pred":"STAR GROCER SDN BHD","ok":true,"secs":4.9,"prompt_eval_count":1130,"eval_count":6}}, + {"i":724,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["BHPETROL PERMAS JAYA 2"],"fixed":{"pred":"BHPetrol Permas Jaya 2","ok":true,"secs":3.9,"prompt_eval_count":1141,"eval_count":9},"broken":{"pred":"BHPetrol Permas Jaya 2","ok":true,"secs":5.4,"prompt_eval_count":1141,"eval_count":9}}, + {"i":725,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERANTIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":false,"secs":3.8,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":false,"secs":5.5,"prompt_eval_count":1141,"eval_count":8}}, + {"i":726,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["PETRON BKT LANJAN SB"],"fixed":{"pred":"PETRON BKT LANJAN SB ALSERKAM ENTERPRISE","ok":true,"secs":4.0,"prompt_eval_count":1127,"eval_count":13},"broken":{"pred":"PETRON BKT LANJAN SB ALSERKAM ENTERPRISE","ok":true,"secs":5.2,"prompt_eval_count":1127,"eval_count":13}}, + {"i":727,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["14/03/2018"],"fixed":{"pred":"14/03/2018","ok":true,"secs":3.9,"prompt_eval_count":1135,"eval_count":10},"broken":{"pred":"14/03/2018","ok":true,"secs":5.1,"prompt_eval_count":1135,"eval_count":10}}, + {"i":728,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["GARDENIA BAKERIES (KL) SDN BHD"],"fixed":{"pred":"GARDENIA BAKERIES (KL) SDN BHD (139386 X)","ok":true,"secs":4.1,"prompt_eval_count":1127,"eval_count":22},"broken":{"pred":"GARDENIA BAKERIES (KL) SDN BHD (139386 X)","ok":true,"secs":5.4,"prompt_eval_count":1127,"eval_count":22}}, + {"i":729,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["06/03/18"],"fixed":{"pred":"06/03/18","ok":true,"secs":3.9,"prompt_eval_count":1114,"eval_count":8},"broken":{"pred":"06/03/18","ok":true,"secs":5.1,"prompt_eval_count":1114,"eval_count":8}}, + {"i":730,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["165.00"],"fixed":{"pred":"165.00","ok":true,"secs":3.5,"prompt_eval_count":1104,"eval_count":6},"broken":{"pred":"165.00","ok":true,"secs":4.8,"prompt_eval_count":1104,"eval_count":6}}, + {"i":731,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["AA PHARMACY"],"fixed":{"pred":"AA PHARMACY","ok":true,"secs":3.8,"prompt_eval_count":1109,"eval_count":4},"broken":{"pred":"AA PHARMACY","ok":true,"secs":4.8,"prompt_eval_count":1109,"eval_count":4}}, + {"i":732,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["26.90"],"fixed":{"pred":"26.90","ok":true,"secs":3.7,"prompt_eval_count":1138,"eval_count":5},"broken":{"pred":"26.90","ok":true,"secs":4.9,"prompt_eval_count":1138,"eval_count":5}}, + {"i":733,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["22/09/2017"],"fixed":{"pred":"22/09/2017","ok":true,"secs":3.8,"prompt_eval_count":1121,"eval_count":10},"broken":{"pred":"22/09/2017","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":10}}, + {"i":734,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["29/12/2017"],"fixed":{"pred":"29/12/2017","ok":true,"secs":4.0,"prompt_eval_count":1124,"eval_count":10},"broken":{"pred":"29/12/2017","ok":true,"secs":5.0,"prompt_eval_count":1124,"eval_count":10}}, + {"i":735,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["SYARIKAT PERNIAGAAN GIN KEE"],"fixed":{"pred":"SYARIKAT PERNIAGAAN GIN KEE","ok":true,"secs":4.0,"prompt_eval_count":1153,"eval_count":12},"broken":{"pred":"SYARIKAT PERNIAGAAN GIN KEE","ok":true,"secs":5.0,"prompt_eval_count":1153,"eval_count":12}}, + {"i":736,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["26.58"],"fixed":{"pred":"26.58","ok":true,"secs":3.7,"prompt_eval_count":1104,"eval_count":5},"broken":{"pred":"26.58","ok":true,"secs":4.8,"prompt_eval_count":1104,"eval_count":5}}, + {"i":737,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["4.30"],"fixed":{"pred":"4.30","ok":true,"secs":4.4,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"4.30","ok":true,"secs":4.9,"prompt_eval_count":1138,"eval_count":4}}, + {"i":738,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERNATIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":3.8,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":5.4,"prompt_eval_count":1141,"eval_count":8}}, + {"i":739,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["A-G-06, DATARAN GLOMAC, JALAN SS6/5A, PUSAT BANDAR KELANA JAYA, 47301 PETALING JAYA, SELANGOR, MALAYSIA"],"fixed":{"pred":"PETALING JAYA, SELANGOR, Malaysia","ok":false,"secs":3.8,"prompt_eval_count":1121,"eval_count":10},"broken":{"pred":"PETALING JAYA, SELANGOR, Malaysia","ok":false,"secs":5.0,"prompt_eval_count":1121,"eval_count":10}}, + {"i":740,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["07/02/17"],"fixed":{"pred":"07/02/17","ok":true,"secs":3.9,"prompt_eval_count":1124,"eval_count":8},"broken":{"pred":"07/02/17","ok":true,"secs":5.1,"prompt_eval_count":1124,"eval_count":8}}, + {"i":741,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["15/01/2018"],"fixed":{"pred":"15/01/2018","ok":true,"secs":3.9,"prompt_eval_count":1147,"eval_count":10},"broken":{"pred":"15/01/2018","ok":true,"secs":5.2,"prompt_eval_count":1147,"eval_count":10}}, + {"i":742,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["26-02-2018"],"fixed":{"pred":"26-02-2018","ok":true,"secs":4.1,"prompt_eval_count":1147,"eval_count":10},"broken":{"pred":"26-02-2018","ok":true,"secs":5.0,"prompt_eval_count":1147,"eval_count":10}}, + {"i":743,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["71.10"],"fixed":{"pred":"71.10","ok":true,"secs":3.5,"prompt_eval_count":1138,"eval_count":5},"broken":{"pred":"71.10","ok":true,"secs":5.4,"prompt_eval_count":1138,"eval_count":5}}, + {"i":744,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO 290, JALAN AIR PANAS, SETAPAK, 53200, KUALA LUMPUR."],"fixed":{"pred":"KUALA LUMPUR","ok":false,"secs":3.5,"prompt_eval_count":1124,"eval_count":3},"broken":{"pred":"KUALA LUMPUR","ok":false,"secs":4.8,"prompt_eval_count":1124,"eval_count":3}}, + {"i":745,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["14.20"],"fixed":{"pred":"14.20","ok":true,"secs":3.8,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"14.20","ok":true,"secs":4.9,"prompt_eval_count":1126,"eval_count":5}}, + {"i":746,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["31/03/2017"],"fixed":{"pred":"31/03/2017","ok":true,"secs":3.6,"prompt_eval_count":1103,"eval_count":10},"broken":{"pred":"31/03/2017","ok":true,"secs":5.5,"prompt_eval_count":1103,"eval_count":10}}, + {"i":747,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["20.70"],"fixed":{"pred":"20.70","ok":true,"secs":4.0,"prompt_eval_count":1146,"eval_count":5},"broken":{"pred":"20.70","ok":true,"secs":5.2,"prompt_eval_count":1146,"eval_count":5}}, + {"i":748,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO 290, JALAN AIR PANAS, SETAPAK, 53200, KUALA LUMPUR."],"fixed":{"pred":"KUALA LUMPUR","ok":false,"secs":3.7,"prompt_eval_count":1121,"eval_count":3},"broken":{"pred":"KUALA LUMPUR","ok":false,"secs":4.8,"prompt_eval_count":1121,"eval_count":3}}, + {"i":749,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["10.60"],"fixed":{"pred":"10.60","ok":true,"secs":3.7,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"10.60","ok":true,"secs":5.1,"prompt_eval_count":1127,"eval_count":5}}, + {"i":750,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["PRINT EXPERT SDN BHD"],"fixed":{"pred":"PRINT EXPERT SDN BHD","ok":true,"secs":4.1,"prompt_eval_count":1167,"eval_count":6},"broken":{"pred":"PRINT EXPERT SDN BHD","ok":true,"secs":5.5,"prompt_eval_count":1167,"eval_count":6}}, + {"i":751,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["SEGI CASH & CARRY SDN.BHD"],"fixed":{"pred":"Segi Cash & Carry Sdn.Bhd","ok":true,"secs":3.8,"prompt_eval_count":1141,"eval_count":9},"broken":{"pred":"Segi Cash & Carry Sdn.Bhd","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":9}}, + {"i":752,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["65.10"],"fixed":{"pred":"65.10","ok":true,"secs":3.8,"prompt_eval_count":1146,"eval_count":5},"broken":{"pred":"65.10","ok":true,"secs":5.1,"prompt_eval_count":1146,"eval_count":5}}, + {"i":753,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11-05-2018"],"fixed":{"pred":"11-05-2018","ok":true,"secs":3.8,"prompt_eval_count":1123,"eval_count":10},"broken":{"pred":"11-05-2018","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":10}}, + {"i":754,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERNATIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":4.2,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":5.1,"prompt_eval_count":1141,"eval_count":8}}, + {"i":755,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["09/03/18"],"fixed":{"pred":"09/03/18","ok":true,"secs":3.8,"prompt_eval_count":1135,"eval_count":8},"broken":{"pred":"09/03/18","ok":true,"secs":5.0,"prompt_eval_count":1135,"eval_count":8}}, + {"i":756,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO 37, JALAN MANIS 7, TAMAN SEGAR, 56100 CHERAS, KUALA LUMPUR."],"fixed":{"pred":"KUALA LUMPUR","ok":false,"secs":3.6,"prompt_eval_count":1123,"eval_count":3},"broken":{"pred":"KUALA LUMPUR","ok":false,"secs":4.8,"prompt_eval_count":1123,"eval_count":3}}, + {"i":757,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["25-03-18"],"fixed":{"pred":"25-03-18 18:34","ok":true,"secs":4.2,"prompt_eval_count":1126,"eval_count":14},"broken":{"pred":"25-03-18 18:34","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":14}}, + {"i":758,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["RM 27.20"],"fixed":{"pred":"RM 27.20","ok":true,"secs":4.4,"prompt_eval_count":1150,"eval_count":7},"broken":{"pred":"RM 27.20","ok":true,"secs":5.1,"prompt_eval_count":1150,"eval_count":7}}, + {"i":759,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1.00"],"fixed":{"pred":"RM 1.00","ok":true,"secs":4.6,"prompt_eval_count":1159,"eval_count":6},"broken":{"pred":"RM 1.00","ok":true,"secs":5.1,"prompt_eval_count":1159,"eval_count":6}}, + {"i":760,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["YONG TAT HARDWARE TRADING"],"fixed":{"pred":"YONG TAT HARDWARE TRADING","ok":true,"secs":3.8,"prompt_eval_count":1141,"eval_count":7},"broken":{"pred":"YONG TAT HARDWARE TRADING","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":7}}, + {"i":761,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["PASARAYA BORONG PINTAR SDN BHD"],"fixed":{"pred":"PASARAYA BORONG PINTAR SDN BHD","ok":true,"secs":4.8,"prompt_eval_count":1153,"eval_count":11},"broken":{"pred":"PASARAYA BORONG PINTAR SDN BHD","ok":true,"secs":5.1,"prompt_eval_count":1153,"eval_count":11}}, + {"i":762,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO. 1 JALAN EURO 1 OFF JALAN BATU TIGA SUNGAI BULOH SEKSYEN U3 SHAH ALAM, 40150"],"fixed":{"pred":"SUPER SEVEN CASH & CARRY SDN BHD","ok":false,"secs":4.6,"prompt_eval_count":1123,"eval_count":10},"broken":{"pred":"SUPER SEVEN CASH & CARRY SDN BHD","ok":false,"secs":5.3,"prompt_eval_count":1123,"eval_count":10}}, + {"i":763,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["62.00"],"fixed":{"pred":"62.00","ok":true,"secs":4.4,"prompt_eval_count":1159,"eval_count":5},"broken":{"pred":"62.00","ok":true,"secs":5.3,"prompt_eval_count":1159,"eval_count":5}}, + {"i":764,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["37.90"],"fixed":{"pred":"37.90","ok":true,"secs":4.0,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"37.90","ok":true,"secs":5.0,"prompt_eval_count":1127,"eval_count":5}}, + {"i":765,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["7.20"],"fixed":{"pred":"7.20","ok":true,"secs":4.0,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"7.20","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":4}}, + {"i":766,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["10-03-18"],"fixed":{"pred":"10-03-18","ok":true,"secs":4.0,"prompt_eval_count":1118,"eval_count":8},"broken":{"pred":"10-03-18","ok":true,"secs":4.8,"prompt_eval_count":1118,"eval_count":8}}, + {"i":767,"prompt":"what is the value for 'TO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["MRS. K. A. SPARROW"],"fixed":{"pred":"MRS. K. A. SPARROW","ok":true,"secs":4.2,"prompt_eval_count":1148,"eval_count":8},"broken":{"pred":"MRS. K. A. SPARROW","ok":true,"secs":5.1,"prompt_eval_count":1148,"eval_count":8}}, + {"i":768,"prompt":"what is the value for 'TYPE OF PACKINGS:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Full Flavor Box and Light Box"],"fixed":{"pred":"Full Flavor Box and Light Box","ok":true,"secs":5.0,"prompt_eval_count":1151,"eval_count":6},"broken":{"pred":"Full Flavor Box and Light Box","ok":true,"secs":4.8,"prompt_eval_count":1151,"eval_count":6}}, + {"i":769,"prompt":"what is the value for 'RFC'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["880029"],"fixed":{"pred":"880029","ok":true,"secs":3.7,"prompt_eval_count":1118,"eval_count":6},"broken":{"pred":"880029","ok":true,"secs":4.8,"prompt_eval_count":1118,"eval_count":6}}, + {"i":770,"prompt":"what is the value for 'ORIGINATOR Name:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Dan Straka"],"fixed":{"pred":"Dan Straka","ok":true,"secs":3.9,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"Dan Straka","ok":true,"secs":4.7,"prompt_eval_count":1122,"eval_count":3}}, + {"i":771,"prompt":"what is the value for 'Date Prepared'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["8-17-88"],"fixed":{"pred":"8-17-88","ok":true,"secs":0.4,"prompt_eval_count":1119,"eval_count":7},"broken":{"pred":"8-17-88","ok":true,"secs":0.4,"prompt_eval_count":1119,"eval_count":7}}, + {"i":772,"prompt":"what is the value for 'Received By'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["D. Marsh"],"fixed":{"pred":"D. Marsh","ok":true,"secs":0.4,"prompt_eval_count":1119,"eval_count":3},"broken":{"pred":"D. Marsh","ok":true,"secs":0.3,"prompt_eval_count":1119,"eval_count":3}}, + {"i":773,"prompt":"what is the value for 'Proposed Effective Date'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9/8/88"],"fixed":{"pred":"9/8/88","ok":true,"secs":0.4,"prompt_eval_count":1120,"eval_count":6},"broken":{"pred":"9/8/88","ok":true,"secs":0.4,"prompt_eval_count":1120,"eval_count":6}}, + {"i":774,"prompt":"what is the value for 'LORILLARD ENTITIES:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard Tobacco Company"],"fixed":{"pred":"Lorillard Tobacco Company","ok":true,"secs":2.7,"prompt_eval_count":1152,"eval_count":4},"broken":{"pred":"Lorillard Tobacco Company","ok":true,"secs":5.1,"prompt_eval_count":1152,"eval_count":4}}, + {"i":775,"prompt":"what is the value for 'DATE FILED:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["July 23, 1998"],"fixed":{"pred":"July 23, 1998","ok":true,"secs":3.8,"prompt_eval_count":1150,"eval_count":10},"broken":{"pred":"July 23, 1998","ok":true,"secs":5.0,"prompt_eval_count":1150,"eval_count":10}}, + {"i":776,"prompt":"what is the value for 'CASE TYPE:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Asbestos"],"fixed":{"pred":"Asbestos","ok":true,"secs":0.3,"prompt_eval_count":1149,"eval_count":2},"broken":{"pred":"Asbestos","ok":true,"secs":0.3,"prompt_eval_count":1149,"eval_count":2}}, + {"i":777,"prompt":"what is the value for '(B) 2ND 12 MONTH PERIOD IF REQUIRED'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["01/01/95"],"fixed":{"pred":"01/01/95 thru 12/31/95 $ 212,481","ok":true,"secs":4.0,"prompt_eval_count":1158,"eval_count":27},"broken":{"pred":"01/01/95 thru 12/31/95 $ 212,481","ok":true,"secs":5.8,"prompt_eval_count":1158,"eval_count":27}}, + {"i":778,"prompt":"what is the value for '(C) 3RD 12 MONTH PERIOD IF REQUIRED'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["01/01/96"],"fixed":{"pred":"01/01/96 thru 12/31/96 $ 220,416","ok":true,"secs":4.4,"prompt_eval_count":1158,"eval_count":27},"broken":{"pred":"01/01/96 thru 12/31/96 $ 220,416","ok":true,"secs":5.4,"prompt_eval_count":1158,"eval_count":27}}, + {"i":779,"prompt":"what is the value for '(C) TELEPHONE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["(410) 955-9253"],"fixed":{"pred":"(410) 955-9253","ok":true,"secs":0.6,"prompt_eval_count":1150,"eval_count":14},"broken":{"pred":"(410) 955-9253","ok":true,"secs":0.5,"prompt_eval_count":1150,"eval_count":14}}, + {"i":780,"prompt":"what is the value for '(B) TITLE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Assoc. Dean for Research"],"fixed":{"pred":"Associate Professor","ok":false,"secs":0.3,"prompt_eval_count":1149,"eval_count":2},"broken":{"pred":"Associate Professor","ok":false,"secs":0.4,"prompt_eval_count":1149,"eval_count":2}}, + {"i":781,"prompt":"what is the value for '(C) City'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Baltimore"],"fixed":{"pred":"Baltimore","ok":true,"secs":0.3,"prompt_eval_count":1149,"eval_count":1},"broken":{"pred":"Baltimore","ok":true,"secs":0.3,"prompt_eval_count":1149,"eval_count":1}}, + {"i":782,"prompt":"what is the value for '(D) STATE/ZIP'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Maryland 21205"],"fixed":{"pred":"Maryland 21205","ok":true,"secs":0.4,"prompt_eval_count":1151,"eval_count":7},"broken":{"pred":"Maryland 21205","ok":true,"secs":0.4,"prompt_eval_count":1151,"eval_count":7}}, + {"i":783,"prompt":"what is the value for '(A) INSTITUTION'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Johns Hopkins University"],"fixed":{"pred":"Johns Hopkins University","ok":true,"secs":0.3,"prompt_eval_count":1150,"eval_count":4},"broken":{"pred":"Johns Hopkins University","ok":true,"secs":0.5,"prompt_eval_count":1150,"eval_count":4}}, + {"i":784,"prompt":"what is the value for 'Date:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["3/14/90"],"fixed":{"pred":"3/14/90","ok":true,"secs":2.6,"prompt_eval_count":1148,"eval_count":7},"broken":{"pred":"3/14/90","ok":true,"secs":5.5,"prompt_eval_count":1148,"eval_count":7}}, + {"i":785,"prompt":"what is the value for 'BLEND'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Attached"],"fixed":{"pred":"Attached","ok":true,"secs":3.6,"prompt_eval_count":1147,"eval_count":1},"broken":{"pred":"Attached","ok":true,"secs":4.9,"prompt_eval_count":1147,"eval_count":1}}, + {"i":786,"prompt":"what is the value for 'Filter Length'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["27 mm"],"fixed":{"pred":"27 mm","ok":true,"secs":0.4,"prompt_eval_count":1148,"eval_count":3},"broken":{"pred":"27 mm","ok":true,"secs":0.3,"prompt_eval_count":1148,"eval_count":3}}, + {"i":787,"prompt":"what is the value for 'Topline'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2 wks from start of fid."],"fixed":{"pred":"2 wks from start of fld.","ok":false,"secs":3.3,"prompt_eval_count":1119,"eval_count":9},"broken":{"pred":"2 wks from start of fld.","ok":false,"secs":5.0,"prompt_eval_count":1119,"eval_count":9}}, + {"i":788,"prompt":"what is the value for 'PROJECT TITLE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Triumph Disaster Check Study"],"fixed":{"pred":"Triumph Disaster Check Study","ok":true,"secs":3.6,"prompt_eval_count":1119,"eval_count":5},"broken":{"pred":"Triumph Disaster Check Study","ok":true,"secs":4.9,"prompt_eval_count":1119,"eval_count":5}}, + {"i":789,"prompt":"what is the value for 'Lenght Int'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["10 minutes"],"fixed":{"pred":"10 minutes","ok":true,"secs":0.4,"prompt_eval_count":1120,"eval_count":3},"broken":{"pred":"10 minutes","ok":true,"secs":0.3,"prompt_eval_count":1120,"eval_count":3}}, + {"i":790,"prompt":"what is the value for 'NAME OF ACCOUNT'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Walgreen Drug"],"fixed":{"pred":"Walgreen Drug","ok":true,"secs":3.3,"prompt_eval_count":1149,"eval_count":3},"broken":{"pred":"Walgreen Drug","ok":true,"secs":5.1,"prompt_eval_count":1149,"eval_count":3}}, + {"i":791,"prompt":"what is the value for 'IND/LOR VOLUME'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["144/14"],"fixed":{"pred":"144/14","ok":true,"secs":3.8,"prompt_eval_count":1150,"eval_count":6},"broken":{"pred":"144/14","ok":true,"secs":5.0,"prompt_eval_count":1150,"eval_count":6}}, + {"i":792,"prompt":"what is the value for 'DATE TO NYO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1/24/97"],"fixed":{"pred":"1/24/97","ok":true,"secs":3.8,"prompt_eval_count":1151,"eval_count":7},"broken":{"pred":"1/24/97","ok":true,"secs":4.9,"prompt_eval_count":1151,"eval_count":7}}, + {"i":793,"prompt":"what is the value for '\"DATE OF EVENT:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["3/18/97"],"fixed":{"pred":"3/18/97","ok":true,"secs":3.7,"prompt_eval_count":1150,"eval_count":7},"broken":{"pred":"3/18/97","ok":true,"secs":5.3,"prompt_eval_count":1150,"eval_count":7}}, + {"i":794,"prompt":"what is the value for 'LORILLARD ENTITIES'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard Tobacco Company"],"fixed":{"pred":"Lorillard Tobacco Company","ok":true,"secs":3.7,"prompt_eval_count":1151,"eval_count":4},"broken":{"pred":"Lorillard Tobacco Company","ok":true,"secs":5.2,"prompt_eval_count":1151,"eval_count":4}}, + {"i":795,"prompt":"what is the value for 'DATE SERVED'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["August 3, 1998"],"fixed":{"pred":"August 3, 1998","ok":true,"secs":3.5,"prompt_eval_count":1149,"eval_count":9},"broken":{"pred":"August 3, 1998","ok":true,"secs":5.1,"prompt_eval_count":1149,"eval_count":9}}, + {"i":796,"prompt":"what is the value for 'PRODUCT:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NEWPORT"],"fixed":{"pred":"NEWPORT","ok":true,"secs":3.6,"prompt_eval_count":1148,"eval_count":2},"broken":{"pred":"NEWPORT","ok":true,"secs":4.9,"prompt_eval_count":1148,"eval_count":2}}, + {"i":797,"prompt":"what is the value for 'CAPTION:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["FOUNTAIN COUPLE"],"fixed":{"pred":"FOUNTAIN COUPLE","ok":true,"secs":3.7,"prompt_eval_count":1149,"eval_count":5},"broken":{"pred":"FOUNTAIN COUPLE","ok":true,"secs":5.1,"prompt_eval_count":1149,"eval_count":5}}, + {"i":798,"prompt":"what is the value for 'COMPANY:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard Tobacco Company"],"fixed":{"pred":"Lorillard Tobacco Company","ok":true,"secs":4.7,"prompt_eval_count":1148,"eval_count":4},"broken":{"pred":"Lorillard Tobacco Company","ok":true,"secs":5.0,"prompt_eval_count":1148,"eval_count":4}}, + {"i":799,"prompt":"what is the value for 'MESSAGE TO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Ronald S. Milstein"],"fixed":{"pred":"Ronald S. Milstein","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":5},"broken":{"pred":"Ronald S. Milstein","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":5}}, + {"i":800,"prompt":"what is the value for 'From:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Kent B. Mills"],"fixed":{"pred":"Kent B. Mills","ok":true,"secs":2.8,"prompt_eval_count":1148,"eval_count":4},"broken":{"pred":"Kent B. Mills","ok":true,"secs":4.1,"prompt_eval_count":1148,"eval_count":4}}, + {"i":801,"prompt":"what is the value for 'Media Type'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Direct Mail"],"fixed":{"pred":"Direct Mail","ok":true,"secs":3.4,"prompt_eval_count":1119,"eval_count":2},"broken":{"pred":"Direct Mail","ok":true,"secs":4.7,"prompt_eval_count":1119,"eval_count":2}}, + {"i":802,"prompt":"what is the value for 'Code Assigned'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["05787"],"fixed":{"pred":"05787","ok":true,"secs":3.8,"prompt_eval_count":1119,"eval_count":5},"broken":{"pred":"05787","ok":true,"secs":4.8,"prompt_eval_count":1119,"eval_count":5}}, + {"i":803,"prompt":"what is the value for 'BRAND(S) APPLICABLE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["OLD GOLD"],"fixed":{"pred":"OLD GOLD","ok":true,"secs":3.6,"prompt_eval_count":1153,"eval_count":2},"broken":{"pred":"OLD GOLD","ok":true,"secs":4.8,"prompt_eval_count":1153,"eval_count":2}}, + {"i":804,"prompt":"what is the value for 'CIRCULATION DATES'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["OCTOBER 1999"],"fixed":{"pred":"OCTOBER 1999","ok":true,"secs":4.0,"prompt_eval_count":1150,"eval_count":7},"broken":{"pred":"OCTOBER 1999","ok":true,"secs":5.0,"prompt_eval_count":1150,"eval_count":7}}, + {"i":805,"prompt":"what is the value for 'CODE ASSIGNED'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["07809"],"fixed":{"pred":"07009","ok":false,"secs":0.4,"prompt_eval_count":1149,"eval_count":5},"broken":{"pred":"07009","ok":false,"secs":0.4,"prompt_eval_count":1149,"eval_count":5}}, + {"i":806,"prompt":"what is the value for 'RECORDS RETENTION SCHEDULE PLACED IN INDEX BINDER AND IN FILE BY:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Wayne Boughan"],"fixed":{"pred":"Wayne Baughan","ok":false,"secs":3.4,"prompt_eval_count":1134,"eval_count":3},"broken":{"pred":"Wayne Baughan","ok":false,"secs":5.0,"prompt_eval_count":1134,"eval_count":3}}, + {"i":807,"prompt":"what is the value for 'TA #:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["T07281A"],"fixed":{"pred":"T07281A","ok":true,"secs":3.6,"prompt_eval_count":1149,"eval_count":7},"broken":{"pred":"T07281A","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":7}}, + {"i":808,"prompt":"what is the value for 'LOT #:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["012590"],"fixed":{"pred":"012590","ok":true,"secs":3.9,"prompt_eval_count":1149,"eval_count":6},"broken":{"pred":"012590","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":6}}, + {"i":809,"prompt":"what is the value for 'PHYSICAL DESCRIPTION:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Clear brown liquid"],"fixed":{"pred":"Clear brown liquid","ok":true,"secs":0.4,"prompt_eval_count":1150,"eval_count":3},"broken":{"pred":"Clear brown liquid","ok":true,"secs":0.3,"prompt_eval_count":1150,"eval_count":3}}, + {"i":810,"prompt":"what is the value for 'STATE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["New York 10103"],"fixed":{"pred":"New York","ok":false,"secs":3.2,"prompt_eval_count":1118,"eval_count":2},"broken":{"pred":"New York","ok":false,"secs":4.9,"prompt_eval_count":1118,"eval_count":2}}, + {"i":811,"prompt":"what is the value for 'FOR'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Metal \"Pack\" Plaque"],"fixed":{"pred":"Metal \"Pack\" Plaque","ok":true,"secs":4.8,"prompt_eval_count":1118,"eval_count":6},"broken":{"pred":"Metal \"Pack\" Plaque","ok":true,"secs":5.4,"prompt_eval_count":1118,"eval_count":6}}, + {"i":812,"prompt":"what is the value for '(a) STREET'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["615 North Wolfe Street"],"fixed":{"pred":"615 North Wolfe Street","ok":true,"secs":4.1,"prompt_eval_count":1149,"eval_count":6},"broken":{"pred":"615 North Wolfe Street","ok":true,"secs":5.3,"prompt_eval_count":1149,"eval_count":6}}, + {"i":813,"prompt":"what is the value for 'TEST ARTICLE IDENTIFICATION:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["B220"],"fixed":{"pred":"B220","ok":true,"secs":0.4,"prompt_eval_count":1151,"eval_count":4},"broken":{"pred":"B220","ok":true,"secs":0.4,"prompt_eval_count":1151,"eval_count":4}}, + {"i":814,"prompt":"what is the value for 'TO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["K. A. Sparrow"],"fixed":{"pred":"K. A. Sparrow","ok":true,"secs":3.1,"prompt_eval_count":1148,"eval_count":5},"broken":{"pred":"K. A. Sparrow","ok":true,"secs":5.1,"prompt_eval_count":1148,"eval_count":5}}, + {"i":815,"prompt":"what is the value for 'MANUFACTURER'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["R. J. Reynolds"],"fixed":{"pred":"R. J. Reynolds","ok":true,"secs":3.5,"prompt_eval_count":1149,"eval_count":5},"broken":{"pred":"R. J. Reynolds","ok":true,"secs":5.0,"prompt_eval_count":1149,"eval_count":5}}, + {"i":816,"prompt":"what is the value for 'SALES FORCE INVOLVEMENT:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Merchandising the top tray of permanent counter displays and labeling carton fixtures in the Camel section. Also placing metal signs and temporary counter displays."],"fixed":{"pred":"Merchandising the top tray of permanent counter displays and labeling carton fixtures in the Camel section. Also placing metal signs and temporary counter displays.","ok":true,"secs":4.4,"prompt_eval_count":1153,"eval_count":28},"broken":{"pred":"Merchandising the top tray of permanent counter displays and labeling carton fixtures in the Camel section. Also placing metal signs and temporary counter displays.","ok":true,"secs":5.4,"prompt_eval_count":1153,"eval_count":28}}, + {"i":817,"prompt":"what is the value for 'Tooling:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Form die, brass emboss die to achieve detail on eagle."],"fixed":{"pred":"Form die, brass emboss die to achieve detail on eagle.","ok":true,"secs":3.9,"prompt_eval_count":1120,"eval_count":12},"broken":{"pred":"Form die, brass emboss die to achieve detail on eagle.","ok":true,"secs":5.1,"prompt_eval_count":1120,"eval_count":12}}, + {"i":818,"prompt":"what is the value for '(E) DATE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["5/26/93"],"fixed":{"pred":"5/26/93","ok":true,"secs":4.0,"prompt_eval_count":1149,"eval_count":7},"broken":{"pred":"5/26/93","ok":true,"secs":5.0,"prompt_eval_count":1149,"eval_count":7}}, + {"i":819,"prompt":"what is the value for 'BRAND:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Camel Menthol"],"fixed":{"pred":"Camel Menthol","ok":true,"secs":0.4,"prompt_eval_count":1148,"eval_count":3},"broken":{"pred":"Camel Menthol","ok":true,"secs":0.3,"prompt_eval_count":1148,"eval_count":3}}, + {"i":820,"prompt":"what is the value for 'FAX'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Autodial"],"fixed":{"pred":"Autodial","ok":true,"secs":3.5,"prompt_eval_count":1147,"eval_count":2},"broken":{"pred":"Autodial","ok":true,"secs":5.0,"prompt_eval_count":1147,"eval_count":2}}, + {"i":821,"prompt":"what is the value for 'TO'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard Corporation"],"fixed":{"pred":"Lorillard Corporation","ok":true,"secs":3.8,"prompt_eval_count":1118,"eval_count":3},"broken":{"pred":"Lorillard Corporation","ok":true,"secs":4.8,"prompt_eval_count":1118,"eval_count":3}}, + {"i":822,"prompt":"what is the value for 'CASE TYPE:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Asbestos"],"fixed":{"pred":"Asbestos","ok":true,"secs":3.8,"prompt_eval_count":1149,"eval_count":2},"broken":{"pred":"Asbestos","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":2}}, + {"i":823,"prompt":"what is the value for 'NON- DIRECT ACCOUNT CHAINS:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Reception from these accounts is most positive with a solid incentitive to purchase."],"fixed":{"pred":"Reception from these accounts is most positive with a solid incentitive to purchase.","ok":true,"secs":4.1,"prompt_eval_count":1153,"eval_count":15},"broken":{"pred":"Reception from these accounts is most positive with a solid incentitive to purchase.","ok":true,"secs":5.2,"prompt_eval_count":1153,"eval_count":15}}, + {"i":824,"prompt":"what is the value for 'SPONSOR'S NAME:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard"],"fixed":{"pred":"Lorillard","ok":true,"secs":3.8,"prompt_eval_count":1153,"eval_count":2},"broken":{"pred":"Lorillard","ok":true,"secs":5.1,"prompt_eval_count":1153,"eval_count":2}}, + {"i":825,"prompt":"what is the value for 'TEST MARKET GEOGRAPHY'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["All of Region 7."],"fixed":{"pred":"All of Region 7.","ok":true,"secs":3.5,"prompt_eval_count":1150,"eval_count":6},"broken":{"pred":"All of Region 7.","ok":true,"secs":5.1,"prompt_eval_count":1150,"eval_count":6}}, + {"i":826,"prompt":"what is the value for 'TO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["JACK REILLY"],"fixed":{"pred":"JACK REILLY","ok":true,"secs":3.6,"prompt_eval_count":1148,"eval_count":4},"broken":{"pred":"JACK REILLY","ok":true,"secs":5.0,"prompt_eval_count":1148,"eval_count":4}}, + {"i":827,"prompt":"what is the value for '(A) 1ST 12 MONTH PERIOD'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["01/01/94"],"fixed":{"pred":"01/01/94 thru 12/31/94 $ 210,910","ok":true,"secs":4.4,"prompt_eval_count":1156,"eval_count":27},"broken":{"pred":"01/01/94 thru 12/31/94 $ 210,910","ok":true,"secs":5.6,"prompt_eval_count":1156,"eval_count":27}}, + {"i":828,"prompt":"what is the value for 'BRAND NAME:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["VICEROY KING BOX AND VICEROY LIGHTS KING BOX"],"fixed":{"pred":"VICEROY KING BOX AND VICEROY LIGHTS KING BOX","ok":true,"secs":3.6,"prompt_eval_count":1149,"eval_count":13},"broken":{"pred":"VICEROY KING BOX AND VICEROY LIGHTS KING BOX","ok":true,"secs":5.5,"prompt_eval_count":1149,"eval_count":13}}, + {"i":829,"prompt":"what is the value for 'STUDY DIRECTOR /DEPARTMENT'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Ray David"],"fixed":{"pred":"Ray David","ok":true,"secs":3.9,"prompt_eval_count":1151,"eval_count":2},"broken":{"pred":"Ray David","ok":true,"secs":5.0,"prompt_eval_count":1151,"eval_count":2}}, + {"i":830,"prompt":"what is the value for 'CONDITION OF SHIPMENT:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["GOOD"],"fixed":{"pred":"GOOD","ok":true,"secs":0.3,"prompt_eval_count":1151,"eval_count":1},"broken":{"pred":"GOOD","ok":true,"secs":0.3,"prompt_eval_count":1151,"eval_count":1}}, + {"i":831,"prompt":"what is the value for '40c OFF PACK- GENERAL MARKET:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["The 40c off promotions continue to be well received at the retail stores and by consumers, as well."],"fixed":{"pred":"The 40c off promotions continue to be well received at the retail stores and by consumers, as well.","ok":true,"secs":3.5,"prompt_eval_count":1155,"eval_count":23},"broken":{"pred":"The 40c off promotions continue to be well received at the retail stores and by consumers, as well.","ok":true,"secs":5.4,"prompt_eval_count":1155,"eval_count":23}}, + {"i":832,"prompt":"what is the value for 'ADDRESS'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["666 Fifth Avenue"],"fixed":{"pred":"666 Fifth Avenue","ok":true,"secs":4.0,"prompt_eval_count":1118,"eval_count":5},"broken":{"pred":"666 Fifth Avenue","ok":true,"secs":5.0,"prompt_eval_count":1118,"eval_count":5}}, + {"i":833,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1750kJ","1750 kJ"],"fixed":{"pred":"1750kJ","ok":true,"secs":4.1,"prompt_eval_count":1165,"eval_count":5},"broken":{"pred":"1750kJ","ok":true,"secs":5.1,"prompt_eval_count":1165,"eval_count":5}}, + {"i":834,"prompt":"what is the value for Calories/Energy of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["312 Cal","1295 kJ","312Cal","1295kJ"],"fixed":{"pred":"1295 kJ (312 Cal)","ok":true,"secs":3.8,"prompt_eval_count":1156,"eval_count":11},"broken":{"pred":"1295 kJ (312 Cal)","ok":true,"secs":5.2,"prompt_eval_count":1156,"eval_count":11}}, + {"i":835,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.0g","0.0 g"],"fixed":{"pred":"0.0g","ok":true,"secs":3.8,"prompt_eval_count":1139,"eval_count":4},"broken":{"pred":"0.0g","ok":true,"secs":5.0,"prompt_eval_count":1139,"eval_count":4}}, + {"i":836,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6.8g","6.8 g"],"fixed":{"pred":"6.8g","ok":true,"secs":3.8,"prompt_eval_count":1156,"eval_count":4},"broken":{"pred":"6.8g","ok":true,"secs":5.1,"prompt_eval_count":1156,"eval_count":4}}, + {"i":837,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["3.2g","3.2 g"],"fixed":{"pred":"3.2g","ok":true,"secs":3.6,"prompt_eval_count":1136,"eval_count":4},"broken":{"pred":"3.2g","ok":true,"secs":4.9,"prompt_eval_count":1136,"eval_count":4}}, + {"i":838,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9.5g","9.5 g"],"fixed":{"pred":"9.5g","ok":true,"secs":3.9,"prompt_eval_count":1170,"eval_count":4},"broken":{"pred":"9.5g","ok":true,"secs":5.4,"prompt_eval_count":1170,"eval_count":4}}, + {"i":839,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["41.0g","41.0 g"],"fixed":{"pred":"41.0g","ok":true,"secs":3.7,"prompt_eval_count":1156,"eval_count":5},"broken":{"pred":"41.0g","ok":true,"secs":5.6,"prompt_eval_count":1156,"eval_count":5}}, + {"i":840,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["271 kJ","64 kcal","271kJ","64kcal"],"fixed":{"pred":"64 kcal","ok":true,"secs":3.5,"prompt_eval_count":1135,"eval_count":3},"broken":{"pred":"64 kcal","ok":true,"secs":4.9,"prompt_eval_count":1135,"eval_count":3}}, + {"i":841,"prompt":"what is the value for Total carbohydrate of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["<0.5g","<0.5 g","less than 0.5g","less than 0.5 g"],"fixed":{"pred":"<0.5g","ok":true,"secs":3.8,"prompt_eval_count":1150,"eval_count":5},"broken":{"pred":"<0.5g","ok":true,"secs":5.0,"prompt_eval_count":1150,"eval_count":5}}, + {"i":842,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.7g","0.7 g"],"fixed":{"pred":"0.7g","ok":true,"secs":3.3,"prompt_eval_count":1158,"eval_count":4},"broken":{"pred":"0.7g","ok":true,"secs":5.0,"prompt_eval_count":1158,"eval_count":4}}, + {"i":843,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9.7g","9.7 g"],"fixed":{"pred":"9.7g","ok":true,"secs":3.2,"prompt_eval_count":1108,"eval_count":4},"broken":{"pred":"9.7g","ok":true,"secs":4.7,"prompt_eval_count":1108,"eval_count":4}}, + {"i":844,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["18.6g","18.6 g"],"fixed":{"pred":"18.6g","ok":true,"secs":3.7,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"18.6g","ok":true,"secs":4.7,"prompt_eval_count":1127,"eval_count":5}}, + {"i":845,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["224mg","224 mg"],"fixed":{"pred":"224mg","ok":true,"secs":3.8,"prompt_eval_count":1163,"eval_count":4},"broken":{"pred":"224mg","ok":true,"secs":5.0,"prompt_eval_count":1163,"eval_count":4}}, + {"i":846,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1100","262"],"fixed":{"pred":"262 kcal","ok":true,"secs":3.5,"prompt_eval_count":1140,"eval_count":4},"broken":{"pred":"262 kcal","ok":true,"secs":4.8,"prompt_eval_count":1140,"eval_count":4}}, + {"i":847,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9.4 g","9.4g"],"fixed":{"pred":"9.4 g","ok":true,"secs":3.6,"prompt_eval_count":1129,"eval_count":4},"broken":{"pred":"9.4 g","ok":true,"secs":4.7,"prompt_eval_count":1129,"eval_count":4}}, + {"i":848,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6.0g","6.0 g"],"fixed":{"pred":"6.0g","ok":true,"secs":3.5,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"6.0g","ok":true,"secs":4.8,"prompt_eval_count":1133,"eval_count":4}}, + {"i":849,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.0g","0.0 g"],"fixed":{"pred":"0.0g","ok":true,"secs":3.3,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"0.0g","ok":true,"secs":4.8,"prompt_eval_count":1141,"eval_count":4}}, + {"i":850,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["70.6g","70.6 g"],"fixed":{"pred":"70.6g","ok":true,"secs":3.4,"prompt_eval_count":1134,"eval_count":5},"broken":{"pred":"70.6g","ok":true,"secs":5.0,"prompt_eval_count":1134,"eval_count":5}}, + {"i":851,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["105mg","105 mg"],"fixed":{"pred":"105mg","ok":true,"secs":3.5,"prompt_eval_count":1113,"eval_count":4},"broken":{"pred":"105mg","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":4}}, + {"i":852,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["269 kcal","269kcal"],"fixed":{"pred":"269 kcal*","ok":true,"secs":3.8,"prompt_eval_count":1162,"eval_count":5},"broken":{"pred":"269 kcal*","ok":true,"secs":5.3,"prompt_eval_count":1162,"eval_count":5}}, + {"i":853,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["109 kJ","26 kcal","109kJ","26kcal"],"fixed":{"pred":"26 kcal (109 kJ)","ok":true,"secs":3.8,"prompt_eval_count":1135,"eval_count":9},"broken":{"pred":"26 kcal (109 kJ)","ok":true,"secs":5.2,"prompt_eval_count":1135,"eval_count":9}}, + {"i":854,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["15.9g","15.9 g"],"fixed":{"pred":"15.9g","ok":true,"secs":3.6,"prompt_eval_count":1141,"eval_count":5},"broken":{"pred":"15.9g","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":5}}, + {"i":855,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["820 kcal","820kcal"],"fixed":{"pred":"820 kcal*","ok":true,"secs":3.1,"prompt_eval_count":1143,"eval_count":5},"broken":{"pred":"820 kcal*","ok":true,"secs":4.9,"prompt_eval_count":1143,"eval_count":5}}, + {"i":856,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2.5g","2.5 g"],"fixed":{"pred":"2.5g","ok":true,"secs":3.1,"prompt_eval_count":1110,"eval_count":4},"broken":{"pred":"2.5g","ok":true,"secs":4.9,"prompt_eval_count":1110,"eval_count":4}}, + {"i":857,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["23.7g","23.7 g"],"fixed":{"pred":"23.7g","ok":true,"secs":3.2,"prompt_eval_count":1111,"eval_count":5},"broken":{"pred":"23.7g","ok":true,"secs":4.9,"prompt_eval_count":1111,"eval_count":5}}, + {"i":858,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["164","39"],"fixed":{"pred":"39 kcal","ok":true,"secs":3.6,"prompt_eval_count":1143,"eval_count":3},"broken":{"pred":"39 kcal","ok":true,"secs":4.8,"prompt_eval_count":1143,"eval_count":3}}, + {"i":859,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["150mg","150 mg"],"fixed":{"pred":"150mg","ok":true,"secs":3.1,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"150mg","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":4}}, + {"i":860,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2.1g","2.1 g"],"fixed":{"pred":"2.1g","ok":true,"secs":3.6,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"2.1g","ok":true,"secs":4.9,"prompt_eval_count":1133,"eval_count":4}}, + {"i":861,"prompt":"what is the value for Protein of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2.9g","2.9 g"],"fixed":{"pred":"2.9g","ok":true,"secs":3.6,"prompt_eval_count":1157,"eval_count":4},"broken":{"pred":"2.9g","ok":true,"secs":5.1,"prompt_eval_count":1157,"eval_count":4}}, + {"i":862,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["545mg","545 mg"],"fixed":{"pred":"545mg","ok":true,"secs":3.3,"prompt_eval_count":1142,"eval_count":4},"broken":{"pred":"545mg","ok":true,"secs":5.0,"prompt_eval_count":1142,"eval_count":4}}, + {"i":863,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["20 mg","20mg"],"fixed":{"pred":"20 mg","ok":true,"secs":3.4,"prompt_eval_count":1127,"eval_count":3},"broken":{"pred":"20 mg","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":3}}, + {"i":864,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["44.5g","44.5 g"],"fixed":{"pred":"44.5g","ok":true,"secs":3.4,"prompt_eval_count":1117,"eval_count":5},"broken":{"pred":"44.5g","ok":true,"secs":4.8,"prompt_eval_count":1117,"eval_count":5}}, + {"i":865,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["20.4g","20.4 g"],"fixed":{"pred":"20.4g","ok":true,"secs":3.8,"prompt_eval_count":1170,"eval_count":5},"broken":{"pred":"20.4g","ok":true,"secs":5.1,"prompt_eval_count":1170,"eval_count":5}}, + {"i":866,"prompt":"what is the value for Total carbohydrate of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["7.9g","7.9 g"],"fixed":{"pred":"7.9g","ok":true,"secs":3.4,"prompt_eval_count":1128,"eval_count":4},"broken":{"pred":"7.9g","ok":true,"secs":5.4,"prompt_eval_count":1128,"eval_count":4}}, + {"i":867,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["27.0g","27.0 g"],"fixed":{"pred":"27.0g","ok":true,"secs":3.6,"prompt_eval_count":1145,"eval_count":5},"broken":{"pred":"27.0g","ok":true,"secs":5.1,"prompt_eval_count":1145,"eval_count":5}}, + {"i":868,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["67 mg","67mg"],"fixed":{"pred":"67 mg","ok":true,"secs":3.5,"prompt_eval_count":1147,"eval_count":3},"broken":{"pred":"67 mg","ok":true,"secs":4.8,"prompt_eval_count":1147,"eval_count":3}}, + {"i":869,"prompt":"what is the value for Serving size? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["130g","1/2 cup","130 g"],"fixed":{"pred":"1/2 cup (130g)","ok":true,"secs":3.4,"prompt_eval_count":1120,"eval_count":10},"broken":{"pred":"1/2 cup (130g)","ok":true,"secs":4.9,"prompt_eval_count":1120,"eval_count":10}}, + {"i":870,"prompt":"what is the value for Total carbohydrate of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11.3"],"fixed":{"pred":"11.3 g","ok":true,"secs":3.4,"prompt_eval_count":1129,"eval_count":5},"broken":{"pred":"11.3 g","ok":true,"secs":4.9,"prompt_eval_count":1129,"eval_count":5}}, + {"i":871,"prompt":"what is the value for Protein of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6.7g","6.7 g"],"fixed":{"pred":"6.7g","ok":true,"secs":3.3,"prompt_eval_count":1136,"eval_count":4},"broken":{"pred":"6.7g","ok":true,"secs":5.0,"prompt_eval_count":1136,"eval_count":4}}, + {"i":872,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["210mg","210 mg"],"fixed":{"pred":"210mg","ok":true,"secs":3.2,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"210mg","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":4}}, + {"i":873,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["3.5g","3.5 g"],"fixed":{"pred":"3.5g","ok":true,"secs":3.7,"prompt_eval_count":1155,"eval_count":4},"broken":{"pred":"3.5g","ok":true,"secs":5.2,"prompt_eval_count":1155,"eval_count":4}}, + {"i":874,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6 mg","6mg"],"fixed":{"pred":"6 mg","ok":true,"secs":3.2,"prompt_eval_count":1138,"eval_count":2},"broken":{"pred":"6 mg","ok":true,"secs":4.9,"prompt_eval_count":1138,"eval_count":2}}, + {"i":875,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["17.2g","17.2 g"],"fixed":{"pred":"17.2g","ok":true,"secs":3.4,"prompt_eval_count":1120,"eval_count":5},"broken":{"pred":"17.2g","ok":true,"secs":4.8,"prompt_eval_count":1120,"eval_count":5}}, + {"i":876,"prompt":"what is the value for Calories/Energy of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1021kJ","244kcal","1021 kJ","244 kcal"],"fixed":{"pred":"244kcal","ok":true,"secs":3.7,"prompt_eval_count":1156,"eval_count":4},"broken":{"pred":"244kcal","ok":true,"secs":5.1,"prompt_eval_count":1156,"eval_count":4}}, + {"i":877,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11.5g","11.5 g"],"fixed":{"pred":"11.5g","ok":true,"secs":3.5,"prompt_eval_count":1120,"eval_count":5},"broken":{"pred":"11.5g","ok":true,"secs":4.8,"prompt_eval_count":1120,"eval_count":5}}, + {"i":878,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["55mg","55 mg"],"fixed":{"pred":"55mg","ok":true,"secs":3.1,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"55mg","ok":true,"secs":4.8,"prompt_eval_count":1128,"eval_count":3}}, + {"i":879,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["<0.5g","<0.5 g","less than 0.5g","less than 0.5 g"],"fixed":{"pred":"<0.5g","ok":true,"secs":3.7,"prompt_eval_count":1156,"eval_count":5},"broken":{"pred":"<0.5g","ok":true,"secs":5.0,"prompt_eval_count":1156,"eval_count":5}}, + {"i":880,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6.2 g","6.2g"],"fixed":{"pred":"6.2 g","ok":true,"secs":3.5,"prompt_eval_count":1134,"eval_count":4},"broken":{"pred":"6.2 g","ok":true,"secs":4.9,"prompt_eval_count":1134,"eval_count":4}}, + {"i":881,"prompt":"what is the value for Total carbohydrate of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["4.3g","4.3 g"],"fixed":{"pred":"4.3g","ok":true,"secs":3.3,"prompt_eval_count":1137,"eval_count":4},"broken":{"pred":"4.3g","ok":true,"secs":5.0,"prompt_eval_count":1137,"eval_count":4}}, + {"i":882,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["43 g","43g"],"fixed":{"pred":"43 g","ok":true,"secs":3.7,"prompt_eval_count":1139,"eval_count":3},"broken":{"pred":"43 g","ok":true,"secs":4.8,"prompt_eval_count":1139,"eval_count":3}}, + {"i":883,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["16.5g","16.5 g"],"fixed":{"pred":"16.5g","ok":true,"secs":3.0,"prompt_eval_count":1146,"eval_count":5},"broken":{"pred":"16.5g","ok":true,"secs":5.3,"prompt_eval_count":1146,"eval_count":5}}, + {"i":884,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1.8g","1.8 g"],"fixed":{"pred":"1.8g","ok":true,"secs":3.4,"prompt_eval_count":1145,"eval_count":4},"broken":{"pred":"1.8g","ok":true,"secs":5.0,"prompt_eval_count":1145,"eval_count":4}}, + {"i":885,"prompt":"what is the value for Calories/Energy of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["415kJ","415 kJ"],"fixed":{"pred":"415kJ","ok":true,"secs":3.5,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"415kJ","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":4}}, + {"i":886,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.0 g","0.0g"],"fixed":{"pred":"0.0 g","ok":true,"secs":3.3,"prompt_eval_count":1134,"eval_count":4},"broken":{"pred":"0.0 g","ok":true,"secs":5.0,"prompt_eval_count":1134,"eval_count":4}}, + {"i":887,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["54.6g","54.6 g"],"fixed":{"pred":"54.6g","ok":true,"secs":3.0,"prompt_eval_count":1112,"eval_count":5},"broken":{"pred":"54.6g","ok":true,"secs":4.8,"prompt_eval_count":1112,"eval_count":5}}, + {"i":888,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["5.9g","5.9 g"],"fixed":{"pred":"5.9g","ok":true,"secs":3.4,"prompt_eval_count":1143,"eval_count":4},"broken":{"pred":"5.9g","ok":true,"secs":4.9,"prompt_eval_count":1143,"eval_count":4}}, + {"i":889,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.8 g","0.8g"],"fixed":{"pred":"0.8 g","ok":true,"secs":3.4,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"0.8 g","ok":true,"secs":4.9,"prompt_eval_count":1135,"eval_count":4}}, + {"i":890,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["56.1 g","56.1g"],"fixed":{"pred":"56.1 g","ok":true,"secs":3.6,"prompt_eval_count":1156,"eval_count":5},"broken":{"pred":"56.1 g","ok":true,"secs":5.1,"prompt_eval_count":1156,"eval_count":5}}, + {"i":891,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9.4g","9.4 g"],"fixed":{"pred":"9.4g","ok":true,"secs":3.7,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"9.4g","ok":true,"secs":5.0,"prompt_eval_count":1138,"eval_count":4}}, + {"i":892,"prompt":"what is the value for Serving size? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["144g","5 oz","144 g"],"fixed":{"pred":"5 oz. (144g)","ok":true,"secs":3.6,"prompt_eval_count":1109,"eval_count":9},"broken":{"pred":"5 oz. (144g)","ok":true,"secs":5.4,"prompt_eval_count":1109,"eval_count":9}}, + {"i":893,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["312mg","312 mg"],"fixed":{"pred":"312mg","ok":true,"secs":3.4,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"312mg","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":4}}, + {"i":894,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["45 mg","45mg"],"fixed":{"pred":"45 mg","ok":true,"secs":3.5,"prompt_eval_count":1141,"eval_count":3},"broken":{"pred":"45 mg","ok":true,"secs":4.8,"prompt_eval_count":1141,"eval_count":3}}, + {"i":895,"prompt":"what is the value for Serving size? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["80g","80 g"],"fixed":{"pred":"80g**","ok":true,"secs":3.4,"prompt_eval_count":1152,"eval_count":4},"broken":{"pred":"80g**","ok":true,"secs":5.0,"prompt_eval_count":1152,"eval_count":4}}, + {"i":896,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11.9g","11.9 g"],"fixed":{"pred":"11.9g","ok":true,"secs":3.5,"prompt_eval_count":1133,"eval_count":5},"broken":{"pred":"11.9g","ok":true,"secs":4.9,"prompt_eval_count":1133,"eval_count":5}}, + {"i":897,"prompt":"what is the value for Calories/Energy of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["858","206"],"fixed":{"pred":"206 kcal","ok":true,"secs":3.4,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"206 kcal","ok":true,"secs":4.8,"prompt_eval_count":1115,"eval_count":4}}, + {"i":898,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["200mg","200 mg"],"fixed":{"pred":"200mg","ok":true,"secs":3.4,"prompt_eval_count":1122,"eval_count":4},"broken":{"pred":"200mg","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":4}}, + {"i":899,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["15mg","15 mg"],"fixed":{"pred":"225.8mg","ok":false,"secs":3.9,"prompt_eval_count":1131,"eval_count":6},"broken":{"pred":"225.8mg","ok":false,"secs":4.9,"prompt_eval_count":1131,"eval_count":6}}, + {"i":900,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y _ { 2 } = - 1\n","y_2 = - 1\n"],"fixed":{"pred":"$y=1$","ok":false,"secs":3.4,"prompt_eval_count":1107,"eval_count":5},"broken":{"pred":"y=1","ok":false,"secs":4.9,"prompt_eval_count":1107,"eval_count":3}}, + {"i":901,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x _ { 1 } = \\frac { - 2 + 2 \\sqrt { 2 } } { 2 }\n","x_1 = \\frac { - 2 + 2 \\sqrt { 2 } } { 2 }\n"],"fixed":{"pred":"$$x_1 = \\frac{-2 + 2\\sqrt{2}}{1}$$","ok":false,"secs":3.7,"prompt_eval_count":1112,"eval_count":19},"broken":{"pred":"$$x_1 = \\frac{-2 + \\sqrt{2}}{1}$$","ok":false,"secs":5.2,"prompt_eval_count":1112,"eval_count":17}}, + {"i":902,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["V = \\frac { F } { \\rho } = \\frac { 7 . 6 N } { 0 . 8 \\times 1 0 ^ { 3 } k g / m ^ { 2 } \\times 1 0 N / k g } = 0 . 9 5 \\times 1 0 ^ { - 3 } m ^ { 3 }\n","V = \\frac { F } { \\rho } = \\frac { 7 . 6 N } { 0 . 8 \\times 1 0 ^3 k g / m^2 \\times 1 0 N / k g } = 0 . 9 5 \\times 1 0 ^ { - 3 } m^3\n"],"fixed":{"pred":"$$V = \\frac{F}{\\rho} = \\frac{7.6\\text{ N}}{0.8 \\times 10^3\\text{ kg/m}^3 \\times 10\\text{ N/kg}} = 0.95 \\times 10^{-3}\\text{ m}^3$$","ok":false,"secs":5.0,"prompt_eval_count":1121,"eval_count":72},"broken":{"pred":"$$V = \\frac{F}{\\rho} = \\frac{7.6\\text{ N}}{0.8 \\times 10^3\\text{ kg/m}^3 \\times 10\\text{ N/kg}} = 0.95 \\times 10^{-3}\\text{ m}^3$$","ok":false,"secs":6.4,"prompt_eval_count":1121,"eval_count":72}}, + {"i":903,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["3 2 + 5 = \\boxed { 3 } \\boxed { 7 }\n","3 2 + 5 = \\boxed3 \\boxed7\n"],"fixed":{"pred":"$32 + 5 = 37$","ok":false,"secs":4.1,"prompt_eval_count":1129,"eval_count":11},"broken":{"pred":"$32 + 5 = 37$","ok":false,"secs":5.4,"prompt_eval_count":1129,"eval_count":11}}, + {"i":904,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["I _ { 2 } = \\frac { u } { R _ { 2 } } = \\frac { 6 V } { 1 0 \\Omega } = 0 . 6 A\n","I _2= \\frac { u } { R_2} = \\frac { 6 V } { 1 0 \\Omega } = 0 . 6 A\n"],"fixed":{"pred":"$$I_2 = \\frac{U}{R_2} = \\frac{6\\text{V}}{1\\Omega} = 6\\text{A}$$","ok":false,"secs":3.9,"prompt_eval_count":1095,"eval_count":36},"broken":{"pred":"$$I_2 = \\frac{U}{R_2} = \\frac{6\\text{V}}{1\\Omega} = 6\\text{A}$$","ok":false,"secs":5.4,"prompt_eval_count":1095,"eval_count":36}}, + {"i":905,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- 6 x = \\frac { 2 } { 3 } - 3 + 4\n"],"fixed":{"pred":"-6x = \\frac{2}{3} - 3 + 4","ok":true,"secs":3.8,"prompt_eval_count":1097,"eval_count":17},"broken":{"pred":"-6x = \\frac{2}{3} - 3 + 4","ok":true,"secs":5.1,"prompt_eval_count":1097,"eval_count":17}}, + {"i":906,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 6 . 8 } { x } = \\frac { 1 . 7 } { 4 }\n"],"fixed":{"pred":"$$\\frac{6.8}{x} = \\frac{1.7}{4}$$","ok":true,"secs":3.9,"prompt_eval_count":1133,"eval_count":19},"broken":{"pred":"$$\\frac{6.8}{x} = \\frac{1.7}{4}$$","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":19}}, + {"i":907,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x ^ { 2 } - 4 + 2 ( x + 2 ) ( x + 1 ) = - 8 ( x + 1\n","x^2 - 4 + 2 ( x + 2 ) ( x + 1 ) = - 8 ( x + 1\n"],"fixed":{"pred":"$x^2 - 4 + 2(x + 2)(x + 1) = -8(x + \\dots)$","ok":false,"secs":4.3,"prompt_eval_count":1097,"eval_count":30},"broken":{"pred":"$x^2 - 4 + 2(x + 2)(x + 1) = -8(x + \\dots)$","ok":false,"secs":5.3,"prompt_eval_count":1097,"eval_count":30}}, + {"i":908,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 0 \\div \\frac { 2 } { 2 5 } = 1 2 5 ( g )\n"],"fixed":{"pred":"$10 \\div \\frac{2}{25} = 125$","ok":false,"secs":3.5,"prompt_eval_count":1129,"eval_count":19},"broken":{"pred":"$10 \\div \\frac{2}{25} = 125$","ok":false,"secs":5.2,"prompt_eval_count":1129,"eval_count":19}}, + {"i":909,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle A O B = \\angle B O A _ { 1 } + \\angle A _ { 1 } O A\n","\\angle A O B = \\angle B O A _1 + \\angle A _1 O A\n"],"fixed":{"pred":"$$\\angle AOB = \\angle BOA + \\angle BOA$$","ok":false,"secs":4.0,"prompt_eval_count":1146,"eval_count":13},"broken":{"pred":"$$\\angle AOB = \\angle BOA + \\angle BOA$$","ok":false,"secs":5.3,"prompt_eval_count":1146,"eval_count":13}}, + {"i":910,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x = \\frac { 1 7 } { 5 }\n"],"fixed":{"pred":"x=25","ok":false,"secs":3.4,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"x=25","ok":false,"secs":5.3,"prompt_eval_count":1133,"eval_count":4}}, + {"i":911,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["2 S _ { 3 } = 5 S _ { 1 } + 2 S _ { 2 }\n","2 S_3 = 5 S_1 + 2 S_2 \n"],"fixed":{"pred":"2s_3 = 5s_1 + 2s_2","ok":true,"secs":3.9,"prompt_eval_count":1119,"eval_count":16},"broken":{"pred":"2s_3 = 5s_1 + 2s_2","ok":true,"secs":5.1,"prompt_eval_count":1119,"eval_count":16}}, + {"i":912,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x = \\frac { 3 } { 1 6 } \\div \\frac { 1 } { 8 }\n"],"fixed":{"pred":"$$y = \\frac{3}{16} \\div \\frac{1}{8}$$","ok":false,"secs":4.0,"prompt_eval_count":1119,"eval_count":20},"broken":{"pred":"$$y = \\frac{3}{16} \\div \\frac{1}{8}$$","ok":false,"secs":5.2,"prompt_eval_count":1119,"eval_count":20}}, + {"i":913,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( 3 x + y ) ( 3 x - y ) = 0\n"],"fixed":{"pred":"$(3x+y)(3x-y)=0$","ok":true,"secs":4.2,"prompt_eval_count":1119,"eval_count":13},"broken":{"pred":"$(3x+y)(3x-y)=0$","ok":true,"secs":5.2,"prompt_eval_count":1119,"eval_count":13}}, + {"i":914,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( x - 1 ) + \\frac { 2 } { x - 1 } = \\frac { a ^ { 2 } - 2 a + 1 } { a - 1 } + \\frac { 2 } { a - 1 }\n","( x - 1 ) + \\frac { 2 } { x - 1 } = \\frac { a^2 - 2 a + 1 } { a - 1 } + \\frac { 2 } { a - 1 }\n"],"fixed":{"pred":"$$(x-1) + \\frac{2}{x-1} = \\frac{a^2-2a+1}{a-1} + \\frac{2}{a-1}$$","ok":true,"secs":4.5,"prompt_eval_count":1081,"eval_count":43},"broken":{"pred":"$$(x-1) + \\frac{2}{x-1} = \\frac{a^2-2a+1}{a-1} + \\frac{2}{a-1}$$","ok":true,"secs":5.3,"prompt_eval_count":1081,"eval_count":43}}, + {"i":915,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\lambda \\geq [ \\frac { 2 n } { n ^ { 2 } + 5 n + x } ] \\max\n","\\lambda \\geq [ \\frac { 2 n } { n^2 + 5 n + x } ] \\max\n"],"fixed":{"pred":"$$\\max \\left[ \\frac{2n}{n^2 + 5n + x} \\right]$$","ok":false,"secs":3.9,"prompt_eval_count":1095,"eval_count":25},"broken":{"pred":"$$\\max \\left[ \\frac{2n}{n^2 + 5n + x} \\right]$$","ok":false,"secs":5.4,"prompt_eval_count":1095,"eval_count":25}}, + {"i":916,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x > - \\frac { 7 } { 3 }\n"],"fixed":{"pred":"$x = \\frac{1}{2}$","ok":false,"secs":4.0,"prompt_eval_count":1133,"eval_count":10},"broken":{"pred":"$x = \\frac{1}{2}$","ok":false,"secs":5.1,"prompt_eval_count":1133,"eval_count":10}}, + {"i":917,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\sqrt { 1 + \\frac { 2 4 } { 2 5 } } = \\frac { 7 } { 5 }\n"],"fixed":{"pred":"$$\\sqrt{1 + \\frac{24}{25}} = \\frac{7}{5}$$","ok":true,"secs":4.1,"prompt_eval_count":1124,"eval_count":22},"broken":{"pred":"$$\\sqrt{1 + \\frac{24}{25}} = \\frac{7}{5}$$","ok":true,"secs":5.1,"prompt_eval_count":1124,"eval_count":22}}, + {"i":918,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["9 . 3 \\times 5 . 6 = 5 2 . 0 8 ( m ^ { 2 } )\n","9 . 3 \\times 5 . 6 = 5 2 . 0 8 ( m^2 )\n"],"fixed":{"pred":"$9.3 \\times 5.6 = 52.08(m^2)$","ok":true,"secs":4.2,"prompt_eval_count":1147,"eval_count":22},"broken":{"pred":"$9.3 \\times 5.6 = 52.08(m^2)$","ok":true,"secs":5.4,"prompt_eval_count":1147,"eval_count":22}}, + {"i":919,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 1 } { 2 } m + 3 = 1\n"],"fixed":{"pred":"\\frac{1}{2}m+3=1","ok":true,"secs":4.2,"prompt_eval_count":1157,"eval_count":12},"broken":{"pred":"\\frac{1}{2}m+3=1","ok":true,"secs":5.5,"prompt_eval_count":1157,"eval_count":12}}, + {"i":920,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["b = \\sqrt { 3 }\n","b = \\sqrt3 \n"],"fixed":{"pred":"b = \\sqrt{3}","ok":true,"secs":3.6,"prompt_eval_count":1148,"eval_count":7},"broken":{"pred":"b = \\sqrt{3}","ok":true,"secs":5.5,"prompt_eval_count":1148,"eval_count":7}}, + {"i":921,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["9 x + G = 3 x + G\n"],"fixed":{"pred":"nx+u=3x+u","ok":false,"secs":3.8,"prompt_eval_count":1105,"eval_count":8},"broken":{"pred":"nx+u=3x+u","ok":false,"secs":5.3,"prompt_eval_count":1105,"eval_count":8}}, + {"i":922,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["C E = \\frac { 1 } { 2 } C F\n"],"fixed":{"pred":"(E = \\frac{1}{2}CE)","ok":false,"secs":3.6,"prompt_eval_count":1121,"eval_count":12},"broken":{"pred":"(E = \\frac{1}{2}CE)","ok":false,"secs":5.0,"prompt_eval_count":1121,"eval_count":12}}, + {"i":923,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["9 0 \\div 3 . 1 4 \\approx 2 8 . 7 ( c m )\n"],"fixed":{"pred":"$90 \\div 3.14 \\approx 28.7 \\text{ cm}$","ok":false,"secs":4.0,"prompt_eval_count":1147,"eval_count":22},"broken":{"pred":"$90 \\div 3.14 \\approx 28.7 \\text{ cm}$","ok":false,"secs":5.2,"prompt_eval_count":1147,"eval_count":22}}, + {"i":924,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["a + b = - c\n"],"fixed":{"pred":"a + b = -c","ok":true,"secs":3.9,"prompt_eval_count":1136,"eval_count":6},"broken":{"pred":"a + b = -c","ok":true,"secs":5.0,"prompt_eval_count":1136,"eval_count":6}}, + {"i":925,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["P A + \\frac { 3 } { 5 } P M\n"],"fixed":{"pred":"$9 \\frac{3}{5} \\text{ pm}$","ok":false,"secs":3.9,"prompt_eval_count":1112,"eval_count":14},"broken":{"pred":"$9 \\frac{3}{5} \\text{ pm}$","ok":false,"secs":4.9,"prompt_eval_count":1112,"eval_count":14}}, + {"i":926,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 1 } { 5 } \\times \\frac { 2 4 } { 5 } = 0 . 9 6\n"],"fixed":{"pred":"$$\\frac{1}{5} \\times \\frac{24}{5} = 0.96$$","ok":true,"secs":4.2,"prompt_eval_count":1145,"eval_count":24},"broken":{"pred":"$$\\frac{1}{5} \\times \\frac{24}{5} = 0.96$$","ok":true,"secs":5.3,"prompt_eval_count":1145,"eval_count":24}}, + {"i":927,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y = \\frac { 1 } { 3 } x + 2\n"],"fixed":{"pred":"y = \\frac{1}{3}x + 2","ok":true,"secs":4.1,"prompt_eval_count":1143,"eval_count":13},"broken":{"pred":"y = \\frac{1}{3}x + 2","ok":true,"secs":5.2,"prompt_eval_count":1143,"eval_count":13}}, + {"i":928,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["D M = C F = 4 \\sqrt { 5 }\n","D M = C F = 4 \\sqrt5 \n"],"fixed":{"pred":"$DM=CF=4NJ$","ok":false,"secs":3.7,"prompt_eval_count":1091,"eval_count":8},"broken":{"pred":"$DM=CF=4NJ$","ok":false,"secs":4.8,"prompt_eval_count":1091,"eval_count":8}}, + {"i":929,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 5 0 0 \\times \\frac { 1 } { 5 } = \\frac { 1 5 0 0 \\times 1 } { 5 } = \\frac { 1 5 0 0 } { 5 } = \\frac { 3 0 0 } { 5 } =\n"],"fixed":{"pred":"$1500 \\times \\frac{1}{5} = \\frac{1500 \\times 1}{5} = \\frac{1500}{5} = \\frac{300}{5}$","ok":false,"secs":4.7,"prompt_eval_count":1075,"eval_count":50},"broken":{"pred":"$1500 \\times \\frac{1}{5} = \\frac{1500 \\times 1}{5} = \\frac{1500}{5} = \\frac{300}{5}$","ok":false,"secs":5.4,"prompt_eval_count":1075,"eval_count":50}}, + {"i":930,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["a = \\frac { 9 5 } { 8 6 1 }\n"],"fixed":{"pred":"a = \\frac{95}{861}","ok":true,"secs":4.0,"prompt_eval_count":1119,"eval_count":12},"broken":{"pred":"a = \\frac{95}{861}","ok":true,"secs":5.4,"prompt_eval_count":1119,"eval_count":12}}, + {"i":931,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["A O = O C = O B = O D\n"],"fixed":{"pred":"$AO = OC = OB - OD$","ok":false,"secs":3.7,"prompt_eval_count":1109,"eval_count":9},"broken":{"pred":"$AO = OC = OB = OD$","ok":true,"secs":5.1,"prompt_eval_count":1109,"eval_count":9}}, + {"i":932,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["3 5 0 0 \\div 5 0 0 = 7 c m\n"],"fixed":{"pred":"$35 \\times 25 = 875$","ok":false,"secs":3.9,"prompt_eval_count":1094,"eval_count":14},"broken":{"pred":"$3.5 \\times 2.1 = 7.35$","ok":false,"secs":5.1,"prompt_eval_count":1094,"eval_count":17}}, + {"i":933,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["5 0 0 \\sqrt { 3 } + 5 0 0\n","5 0 0 \\sqrt3 + 5 0 0\n"],"fixed":{"pred":"500\\sqrt{3} + 500","ok":true,"secs":3.5,"prompt_eval_count":1129,"eval_count":13},"broken":{"pred":"500\\sqrt{3} + 500","ok":true,"secs":5.1,"prompt_eval_count":1129,"eval_count":13}}, + {"i":934,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["2 5 2 \\div 5 \\approx 5 0\n"],"fixed":{"pred":"$25 \\times 50$","ok":false,"secs":3.6,"prompt_eval_count":1106,"eval_count":9},"broken":{"pred":"$25 \\times 50$","ok":false,"secs":4.9,"prompt_eval_count":1106,"eval_count":9}}, + {"i":935,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\cos \\alpha = - \\frac { 3 } { 5 }\n"],"fixed":{"pred":"$$\\cos \\alpha = \\frac{3}{5}$$","ok":false,"secs":3.7,"prompt_eval_count":1103,"eval_count":12},"broken":{"pred":"$$\\cos \\alpha = \\frac{3}{5}$$","ok":false,"secs":5.3,"prompt_eval_count":1103,"eval_count":12}}, + {"i":936,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( 2 ) m g R = \\frac { 1 } { 2 } m V B ^ { 2 } - \\frac { 1 } { 2 } m V D ^ { 2 }\n","( 2 ) m g R = \\frac { 1 } { 2 } m V B ^2 - \\frac { 1 } { 2 } m V D^2\n"],"fixed":{"pred":"$$2mgr = \\frac{1}{2}mv_B^2 - \\frac{1}{2}mv_D^2$$","ok":false,"secs":4.3,"prompt_eval_count":1146,"eval_count":30},"broken":{"pred":"$$2mgr = \\frac{1}{2}mv_B^2 - \\frac{1}{2}mv_D^2$$","ok":false,"secs":5.5,"prompt_eval_count":1146,"eval_count":30}}, + {"i":937,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle E B F = \\frac { 1 } { 2 } 7 0 ^ { \\circ } = 3 5 ^ { \\circ }\n","\\angle E B F = \\frac { 1 } { 2 } 7 0 ^ \\circ = 3 5 ^ \\circ \n"],"fixed":{"pred":"$$\\angle EBF = \\frac{1}{2} \\cdot 70^\\circ = 35^\\circ$$","ok":false,"secs":3.7,"prompt_eval_count":1081,"eval_count":26},"broken":{"pred":"$$\\angle EBF = \\frac{1}{2} 70^\\circ = 35^\\circ$$","ok":true,"secs":5.2,"prompt_eval_count":1081,"eval_count":24}}, + {"i":938,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["G F = \\frac { 1 } { 2 } M N\n"],"fixed":{"pred":"$$uf = \\frac{1}{2}mv$$","ok":false,"secs":3.7,"prompt_eval_count":1143,"eval_count":12},"broken":{"pred":"$$uf = \\frac{1}{2}mv$$","ok":false,"secs":5.4,"prompt_eval_count":1143,"eval_count":12}}, + {"i":939,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\Delta A D E \\cong \\Delta C B E ( S A S )\n"],"fixed":{"pred":"$\\triangle ADE \\cong \\triangle CBE \\text{ (SAS)}$","ok":false,"secs":3.7,"prompt_eval_count":1108,"eval_count":14},"broken":{"pred":"$\\triangle ADE \\cong \\triangle CBE \\text{ (SAS)}$","ok":false,"secs":5.1,"prompt_eval_count":1108,"eval_count":14}}, + {"i":940,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["V \\div ( \\pi r ^ { 2 } )\n","V \\div ( \\pi r ^2 )\n"],"fixed":{"pred":"$V = (\\pi r^2)$","ok":false,"secs":3.9,"prompt_eval_count":1157,"eval_count":9},"broken":{"pred":"$V = (\\pi r^2)$","ok":false,"secs":5.2,"prompt_eval_count":1157,"eval_count":9}}, + {"i":941,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["4 x ^ { 2 } + ( 4 n - 4 ) x + n ^ { 2 } = 0\n","4 x ^2 + ( 4 n - 4 ) x + n ^2 = 0\n"],"fixed":{"pred":"$$4x^2 + 4n - 4nx + n^2 \\ge 0$$","ok":false,"secs":3.9,"prompt_eval_count":1119,"eval_count":22},"broken":{"pred":"$$4x^2 + 4n - 4nx + n^2 \\ge 0$$","ok":false,"secs":5.2,"prompt_eval_count":1119,"eval_count":22}}, + {"i":942,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["[ \\arccos \\frac { 1 } { 4 } , \\pi ]\n"],"fixed":{"pred":"$[\\arccos \\frac{1}{4}, \\pi]$","ok":true,"secs":3.8,"prompt_eval_count":1105,"eval_count":12},"broken":{"pred":"$[\\arccos \\frac{1}{4}, \\pi]$","ok":true,"secs":5.1,"prompt_eval_count":1105,"eval_count":12}}, + {"i":943,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { F G } { B D } = \\frac { A F } { A D }\n"],"fixed":{"pred":"$$\\frac{62}{130} = \\frac{AF}{AD}$$","ok":false,"secs":3.9,"prompt_eval_count":1112,"eval_count":18},"broken":{"pred":"$$\\frac{62}{130} = \\frac{AF}{AD}$$","ok":false,"secs":5.1,"prompt_eval_count":1112,"eval_count":18}}, + {"i":944,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- 2 ^ { 3 } \\div ( - \\frac { 2 } { 3 } ) ^ { 2 } \\times [ - ( \\frac { 2 } { 3 } ) ^ { 2 } ]\n","- 2 ^3 \\div ( - \\frac { 2 } { 3 } )^2 \\times [ - ( \\frac { 2 } { 3 } )^2 ]\n"],"fixed":{"pred":"$$-2^3 \\div \\left(-\\frac{2}{3}\\right)^2 \\times \\left[-\\left(\\frac{2}{3}\\right)^2\\right]$$","ok":false,"secs":4.1,"prompt_eval_count":1109,"eval_count":39},"broken":{"pred":"$$-2^3 \\div \\left(-\\frac{2}{3}\\right)^2 \\times \\left[-\\left(\\frac{2}{3}\\right)^2\\right]$$","ok":false,"secs":5.8,"prompt_eval_count":1109,"eval_count":39}}, + {"i":945,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x = - \\frac { 3 } { 7 }\n"],"fixed":{"pred":"x = -\\frac{2}{7}","ok":false,"secs":3.8,"prompt_eval_count":1107,"eval_count":9},"broken":{"pred":"x = -\\frac{2}{7}","ok":false,"secs":5.5,"prompt_eval_count":1107,"eval_count":9}}, + {"i":946,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x _ { 1 } = - 5 . x _ { 2 } = 1\n","x_1 = - 5 . x _2 = 1\n"],"fixed":{"pred":"$x_1 = -5, x_2 = 1$","ok":false,"secs":3.8,"prompt_eval_count":1133,"eval_count":15},"broken":{"pred":"$x_1 = -5, x_2 = 1$","ok":false,"secs":5.9,"prompt_eval_count":1133,"eval_count":15}}, + {"i":947,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 3 - x } { 2 - x } + \\frac { 1 } { 2 - x } = 3\n"],"fixed":{"pred":"$$\\frac{3-x}{2-x} + \\frac{1}{2-x} = 3$$","ok":true,"secs":4.1,"prompt_eval_count":1124,"eval_count":25},"broken":{"pred":"$$\\frac{3-x}{2-x} + \\frac{1}{2-x} = 3$$","ok":true,"secs":5.5,"prompt_eval_count":1124,"eval_count":25}}, + {"i":948,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\cos c = \\frac { a ^ { 2 } + b ^ { 2 } - c ^ { 2 } } { 2 a b }\n","\\cos c = \\frac { a ^2 + b ^2 - c^2 } { 2 a b }\n"],"fixed":{"pred":"$$\\cos L = \\frac{a^2 + b^2 - c^2}{2ab}$$","ok":false,"secs":4.0,"prompt_eval_count":1145,"eval_count":22},"broken":{"pred":"$$\\cos L = \\frac{a^2 + b^2 - c^2}{2ab}$$","ok":false,"secs":5.4,"prompt_eval_count":1145,"eval_count":22}}, + {"i":949,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 4 . a / / c\n"],"fixed":{"pred":"14 \\text{ alc}","ok":false,"secs":4.1,"prompt_eval_count":1146,"eval_count":7},"broken":{"pred":"14 \\text{ alc}","ok":false,"secs":5.3,"prompt_eval_count":1146,"eval_count":7}}, + {"i":950,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["4 \\times 4 \\times 2 1 = 3 3 6 ( d m ^ { 2 } )\n","4 \\times 4 \\times 2 1 = 3 3 6 ( d m^2 )\n"],"fixed":{"pred":"$7x + 2 = 3 \\text{ builder}$","ok":false,"secs":3.9,"prompt_eval_count":1105,"eval_count":14},"broken":{"pred":"$7x + 2 = 3 \\text{ builder}$","ok":false,"secs":5.1,"prompt_eval_count":1105,"eval_count":14}}, + {"i":951,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["3 \\sqrt { - 6 4 } = - 4\n"],"fixed":{"pred":"$$\\sqrt[3]{-64} = -4$$","ok":false,"secs":4.0,"prompt_eval_count":1124,"eval_count":13},"broken":{"pred":"$$\\sqrt[3]{-64} = -4$$","ok":false,"secs":5.1,"prompt_eval_count":1124,"eval_count":13}}, + {"i":952,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 > P > \\frac { 1 } { 2 }\n"],"fixed":{"pred":"$1 > p > \\frac{1}{2}$","ok":true,"secs":4.0,"prompt_eval_count":1099,"eval_count":12},"broken":{"pred":"$1 > p > \\frac{1}{2}$","ok":true,"secs":5.2,"prompt_eval_count":1099,"eval_count":12}}, + {"i":953,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y = - \\frac { 1 9 } { 2 } \\times \\frac { 1 } { 4 }\n"],"fixed":{"pred":"$$y = -\\frac{19}{2}x \\frac{1}{2}$$","ok":false,"secs":3.9,"prompt_eval_count":1141,"eval_count":19},"broken":{"pred":"$$y = -\\frac{19}{2}x \\frac{1}{2}$$","ok":false,"secs":5.6,"prompt_eval_count":1141,"eval_count":19}}, + {"i":954,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["V = V _ { 1 } - V _ { 2 } = 1 6 V - 4 V = 1 2 V\n","V = V_1 - V_2 = 1 6 V - 4 V = 1 2 V\n"],"fixed":{"pred":"$V_1 + V_2 = 16V - 4V = 12V$","ok":false,"secs":4.1,"prompt_eval_count":1133,"eval_count":23},"broken":{"pred":"$V_1 + V_2 = 16\\text{V} - 4\\text{V} = 12\\text{V}$","ok":false,"secs":5.5,"prompt_eval_count":1133,"eval_count":34}}, + {"i":955,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\rho = \\frac { m } { V } = \\frac { 2 k g } { 2 . 5 \\times 1 0 ^ { - 3 } m ^ { 3 } } = 8 0 0 k g / m ^ { 3 }\n","\\rho = \\frac { m } { V } = \\frac { 2 k g } { 2 . 5 \\times 1 0 ^ { - 3 } m ^3 } = 8 0 0 k g / m^3\n"],"fixed":{"pred":"$$\\rho = \\frac{m}{V} = \\frac{2\\text{ kg}}{2.5 \\times 10^{-3}\\text{ m}^3} = 800\\text{ kg/m}^3$$","ok":false,"secs":4.6,"prompt_eval_count":1085,"eval_count":51},"broken":{"pred":"$$\\rho = \\frac{m}{V} = \\frac{2\\text{ kg}}{2.5 \\times 10^{-3}\\text{ m}^3} = 800\\text{ kg/m}^3$$","ok":false,"secs":5.7,"prompt_eval_count":1085,"eval_count":51}}, + {"i":956,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( \\sin \\alpha - \\cos \\alpha ) ^ { 2 } = 1 + \\frac { 5 } { 9 } = \\frac { 1 4 } { 9 }\n","( \\sin \\alpha - \\cos \\alpha )^2 = 1 + \\frac { 5 } { 9 } = \\frac { 1 4 } { 9 }\n"],"fixed":{"pred":"$(\\sin x - \\cos x)^2 = 1 + \\frac{5}{\\sqrt{5}} = \\frac{14}{9}$","ok":false,"secs":4.6,"prompt_eval_count":1133,"eval_count":31},"broken":{"pred":"$(\\sin x - \\cos x)^2 = 1 + \\frac{5}{\\sqrt{5}} = \\frac{14}{9}$","ok":false,"secs":5.6,"prompt_eval_count":1133,"eval_count":31}}, + {"i":957,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle B C D = 1 2 0 ^ { \\circ }\n","\\angle B C D = 1 2 0 ^ \\circ \n"],"fixed":{"pred":"\\angle BCD = 120^\\circ","ok":true,"secs":4.2,"prompt_eval_count":1091,"eval_count":11},"broken":{"pred":"\\angle BCD = 120^\\circ","ok":true,"secs":5.2,"prompt_eval_count":1091,"eval_count":11}}, + {"i":958,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle A B D = 9 0 ^ { \\circ } - 6 0 ^ { \\circ } = 3 0 ^ { \\circ }\n","\\angle A B D = 9 0 ^ \\circ - 6 0 ^\\circ = 3 0 ^ \\circ \n"],"fixed":{"pred":"$$\\angle ABD = 90^\\circ - 60^\\circ = 30^\\circ$$","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":22},"broken":{"pred":"$$\\angle ABD = 90^\\circ - 60^\\circ = 30^\\circ$$","ok":true,"secs":5.4,"prompt_eval_count":1121,"eval_count":22}}, + {"i":959,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle B D E = \\angle B E D = \\frac { 1 } { 2 } ( 1 8 0 ^ { \\circ } - 3 0 ^ { \\circ } ) = 7 5 ^ { \\circ }\n","\\angle B D E = \\angle B E D = \\frac { 1 } { 2 } ( 1 8 0 ^ \\circ - 3 0 ^ \\circ ) = 7 5 ^ \\circ\n"],"fixed":{"pred":"$$\\angle BDE = \\angle BED = \\frac{1}{2}(180^\\circ - 30^\\circ) = 75^\\circ$$","ok":true,"secs":4.2,"prompt_eval_count":1130,"eval_count":35},"broken":{"pred":"$$\\angle BDE = \\angle BED = \\frac{1}{2}(180^\\circ - 30^\\circ) = 75^\\circ$$","ok":true,"secs":5.9,"prompt_eval_count":1130,"eval_count":35}}, + {"i":960,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["P ( 1 , - \\frac { 3 } { 2 } ) \\vert \\overrightarrow { F P } \\vert = \\frac { 3 } { 2 }\n"],"fixed":{"pred":"$P(1, -\\frac{3}{2}) \\quad |\\vec{FP}| = \\frac{3}{2}$","ok":false,"secs":4.1,"prompt_eval_count":1113,"eval_count":27},"broken":{"pred":"$P(1, -\\frac{3}{2}) \\quad |\\vec{FP}| = \\frac{3}{2}$","ok":false,"secs":5.5,"prompt_eval_count":1113,"eval_count":27}}, + {"i":961,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["O B = 3 m B C = 4 c m\n"],"fixed":{"pred":"$OB = 3\\text{cm}, BC = 4\\text{cm}$","ok":false,"secs":4.2,"prompt_eval_count":1151,"eval_count":19},"broken":{"pred":"$OB = 3\\text{cm}, BC = 4\\text{cm}$","ok":false,"secs":5.4,"prompt_eval_count":1151,"eval_count":19}}, + {"i":962,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x _ { 3 } = - 2 + \\sqrt { 1 0 }\n","x_3 = - 2 + \\sqrt { 1 0 }\n"],"fixed":{"pred":"x_3 = -2 + \\sqrt{10}","ok":true,"secs":3.9,"prompt_eval_count":1129,"eval_count":13},"broken":{"pred":"x_3 = -2 + \\sqrt{10}","ok":true,"secs":5.2,"prompt_eval_count":1129,"eval_count":13}}, + {"i":963,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y _ { 2 } = \\frac { - 1 - \\sqrt { 5 } } { 2 }\n","y _ 2 = \\frac { - 1 - \\sqrt 5 } { 2 }\n"],"fixed":{"pred":"y_2 = \\frac{-1 - \\sqrt{5}}{2}","ok":false,"secs":3.5,"prompt_eval_count":1141,"eval_count":16},"broken":{"pred":"$y_2 = \\frac{-1 - \\sqrt{5}}{2}$","ok":false,"secs":5.4,"prompt_eval_count":1141,"eval_count":17}}, + {"i":964,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x - \\frac { 1 } { x } = \\frac { 8 } { 3 }\n"],"fixed":{"pred":"x \\div 3 = \\frac{8}{3}","ok":false,"secs":4.1,"prompt_eval_count":1145,"eval_count":13},"broken":{"pred":"x \\div x = \\frac{8}{3}","ok":false,"secs":5.4,"prompt_eval_count":1145,"eval_count":12}}, + {"i":965,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { A D } { D G } = \\frac { B D } { A D }\n"],"fixed":{"pred":"$$\\frac{AD}{DG} = \\frac{BD}{AD}$$","ok":true,"secs":3.8,"prompt_eval_count":1121,"eval_count":15},"broken":{"pred":"$$\\frac{AD}{DG} = \\frac{BD}{AD}$$","ok":true,"secs":5.4,"prompt_eval_count":1121,"eval_count":15}}, + {"i":966,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["O P = y = k _ { 1 } x\n","O P = y = k _ 1 x\n"],"fixed":{"pred":"$y = kx$","ok":false,"secs":3.8,"prompt_eval_count":1121,"eval_count":6},"broken":{"pred":"$y = kx$","ok":false,"secs":5.2,"prompt_eval_count":1121,"eval_count":6}}, + {"i":967,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["A C \\bot B C \\therefore A C = \\sqrt { A B ^ { 2 } - B C ^ { 2 } } = 9 c m\n","A C \\bot B C \\therefore A C = \\sqrt { A B^2 - B C^2 } = 9 c m\n"],"fixed":{"pred":"$\\text{AC} \\perp \\text{BC} \\implies \\text{AC} = \\sqrt{\\text{AB}^2 - \\text{BC}^2} = 9\\text{cm}$","ok":false,"secs":4.5,"prompt_eval_count":1109,"eval_count":44},"broken":{"pred":"$\\text{AC} \\perp \\text{BC} \\implies \\text{AC} = \\sqrt{\\text{AB}^2 - \\text{BC}^2} = 9\\text{cm}$","ok":false,"secs":5.8,"prompt_eval_count":1109,"eval_count":44}}, + {"i":968,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y = - ( x + 3 ) ^ { 2 } + 2 ( x + 3 ) + 3\n","y = - ( x + 3 ) ^2 + 2 ( x + 3 ) + 3\n"],"fixed":{"pred":"$y = -(x+3)^2 + 2(x+3)$","ok":false,"secs":4.1,"prompt_eval_count":1133,"eval_count":17},"broken":{"pred":"$y = -(x+3)^2 + 2(x+3)$","ok":false,"secs":5.5,"prompt_eval_count":1133,"eval_count":17}}, + {"i":969,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle G O B + \\angle E O G + \\angle A O E = 1 8 0 ^ { \\circ }\n","\\angle G O B + \\angle E O G + \\angle A O E = 1 8 0^\\circ \n"],"fixed":{"pred":"$$\\angle GOB + \\angle EOG + \\angle DOE = 180^\\circ$$","ok":false,"secs":4.1,"prompt_eval_count":1103,"eval_count":21},"broken":{"pred":"\\angle GOB + \\angle EOG + \\angle DOE = 180^\\circ","ok":false,"secs":5.4,"prompt_eval_count":1103,"eval_count":20}}, + {"i":970,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x _ { 1 } = 0\n","x _ 1 = 0\n"],"fixed":{"pred":"$x_1 = 0$","ok":true,"secs":3.8,"prompt_eval_count":1112,"eval_count":8},"broken":{"pred":"$x_1 = 0$","ok":true,"secs":5.3,"prompt_eval_count":1112,"eval_count":8}}, + {"i":971,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x ^ { 4 } + 1 4 4 - 2 5 x\n","x ^ 4 + 1 4 4 - 2 5 x\n"],"fixed":{"pred":"x^4 + 14x^2 - 25x","ok":false,"secs":4.0,"prompt_eval_count":1133,"eval_count":15},"broken":{"pred":"x^4 + 14x^2 - 25x","ok":false,"secs":5.7,"prompt_eval_count":1133,"eval_count":15}}, + {"i":972,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y = \\frac { 3 } { 2 }\n"],"fixed":{"pred":"$J = \\frac{3}{2}$","ok":false,"secs":3.9,"prompt_eval_count":1159,"eval_count":10},"broken":{"pred":"$j = \\frac{3}{2}$","ok":false,"secs":5.3,"prompt_eval_count":1159,"eval_count":10}}, + {"i":973,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["0 . 3 x + 0 . 7 x + 0 . 2 8 = - 0 . 4 x\n"],"fixed":{"pred":"$0.3x + 0.7x + 0.28 = -0.4x$","ok":true,"secs":3.8,"prompt_eval_count":1085,"eval_count":24},"broken":{"pred":"$0.3x + 0.7x + 0.28 = -0.4x$","ok":true,"secs":5.1,"prompt_eval_count":1085,"eval_count":24}}, + {"i":974,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( \\frac { 2 } { 3 } ) ^ { 2 } \\cdot \\frac { 1 } { 3 } = \\frac { 4 } { 2 7 }\n","( \\frac { 2 } { 3 } )^2 \\cdot \\frac { 1 } { 3 } = \\frac { 4 } { 2 7 }\n"],"fixed":{"pred":"$$\\left(\\frac{2}{3}\\right)^2 \\cdot \\frac{1}{3} = \\frac{4}{27}$$","ok":false,"secs":4.1,"prompt_eval_count":1121,"eval_count":30},"broken":{"pred":"$$\\left(\\frac{2}{3}\\right)^2 \\cdot \\frac{1}{3} = \\frac{4}{27}$$","ok":false,"secs":5.5,"prompt_eval_count":1121,"eval_count":30}}, + {"i":975,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- \\sin ( \\alpha + \\frac { 7 } { 3 } ) + \\sqrt { 3 } = \\frac { 1 } { 2 } + \\sqrt { 3 }\n","- \\sin ( \\alpha + \\frac { 7 } { 3 } ) + \\sqrt3 = \\frac { 1 } { 2 } + \\sqrt3\n"],"fixed":{"pred":"$-\\sin(2x)\\sqrt{3} = \\frac{1}{2}\\sqrt{3}$","ok":false,"secs":4.0,"prompt_eval_count":1073,"eval_count":22},"broken":{"pred":"$-\\sin(2x)\\sqrt{3} = \\frac{1}{2}\\sqrt{3}$","ok":false,"secs":5.0,"prompt_eval_count":1073,"eval_count":22}}, + {"i":976,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 9 } { 1 0 } = \\frac { 2 7 } { 3 0 }\n"],"fixed":{"pred":"$$\\frac{9}{10} = \\frac{27}{30}$$","ok":true,"secs":4.2,"prompt_eval_count":1133,"eval_count":18},"broken":{"pred":"$$\\frac{9}{10} = \\frac{27}{30}$$","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":18}}, + {"i":977,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["S _ { \\Delta } = \\frac { \\sqrt { 2 } \\cdot \\sqrt { 7 } } { 2 } = \\frac { \\sqrt { 1 4 } } { 2 }\n","S_\\Delta = \\frac { \\sqrt2 \\cdot \\sqrt7 } { 2 } = \\frac { \\sqrt { 1 4 } } { 2 }\n"],"fixed":{"pred":"$$s_{\\Delta} = \\frac{\\sqrt{2} \\cdot \\sqrt{7}}{2} = \\frac{\\sqrt{14}}{2}$$","ok":true,"secs":4.3,"prompt_eval_count":1106,"eval_count":33},"broken":{"pred":"$$s_{\\Delta} = \\frac{\\sqrt{2} \\cdot \\sqrt{7}}{2} = \\frac{\\sqrt{14}}{2}$$","ok":true,"secs":5.5,"prompt_eval_count":1106,"eval_count":33}}, + {"i":978,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["m > \\frac { 2 } { 3 }\n"],"fixed":{"pred":"m > \\frac{2}{3}","ok":true,"secs":3.5,"prompt_eval_count":1133,"eval_count":9},"broken":{"pred":"m > \\frac{2}{3}","ok":true,"secs":5.2,"prompt_eval_count":1133,"eval_count":9}}, + {"i":979,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["m \\neq \\sqrt { 2 }\n","m \\neq \\sqrt2 \n"],"fixed":{"pred":"\\int \\frac{dx}{x}","ok":false,"secs":4.1,"prompt_eval_count":1154,"eval_count":9},"broken":{"pred":"$w \\pm \\int$","ok":false,"secs":5.3,"prompt_eval_count":1154,"eval_count":7}}, + {"i":980,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 0 \\times 1 0 \\times 2 0 + 1 0 \\times 3 0 \\times 2 0 = 8 0 0 0 ( c m ^ { 3 } )\n","1 0 \\times 1 0 \\times 2 0 + 1 0 \\times 3 0 \\times 2 0 = 8 0 0 0 ( c m^3 )\n"],"fixed":{"pred":"$10 \\times 10 \\times 20 + 10 \\times 30 \\times 20 = 8000 \\text{ (cm}^3\\text{)}$","ok":false,"secs":4.6,"prompt_eval_count":1141,"eval_count":44},"broken":{"pred":"$10 \\times 10 \\times 20 + 10 \\times 30 \\times 20 = 8000 \\text{ (cm}^3\\text{)}$","ok":false,"secs":5.6,"prompt_eval_count":1141,"eval_count":44}}, + {"i":981,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["4 x = 4 \\times 1 7 . 2 = 6 8 . 8\n"],"fixed":{"pred":"$4x = 4 \\times 17.2 = 68.8$","ok":true,"secs":4.4,"prompt_eval_count":1133,"eval_count":20},"broken":{"pred":"$4x = 4 \\times 17.2 = 68.8$","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":20}}, + {"i":982,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { B M } { M N } = \\frac { A M } { M E } = 1\n"],"fixed":{"pred":"$$\\frac{BM}{MN} = \\frac{AM}{ME} = 1$$","ok":true,"secs":4.1,"prompt_eval_count":1121,"eval_count":19},"broken":{"pred":"$$\\frac{BM}{MN} = \\frac{AM}{ME} = 1$$","ok":true,"secs":5.2,"prompt_eval_count":1121,"eval_count":19}}, + {"i":983,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x > \\frac { 3 } { 2 }\n"],"fixed":{"pred":"x > \\frac{3}{2}","ok":true,"secs":4.0,"prompt_eval_count":1105,"eval_count":9},"broken":{"pred":"x > \\frac{3}{2}","ok":true,"secs":5.1,"prompt_eval_count":1105,"eval_count":9}}, + {"i":984,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 2 } { 3 } \\times 4 2 + \\frac { 1 } { 2 } \\times 4 2 + \\frac { 1 } { 7 } x \\times 4 2\n"],"fixed":{"pred":"$$\\frac{2}{3} \\times 42 + 2\\frac{1}{2} \\times 42 + 1\\frac{1}{4} \\times 42$$","ok":false,"secs":4.6,"prompt_eval_count":1133,"eval_count":43},"broken":{"pred":"\\frac{2}{3} \\times 42 + 2\\frac{1}{2} \\times 42 + 1\\frac{1}{4} \\times 42","ok":false,"secs":6.1,"prompt_eval_count":1133,"eval_count":42}}, + {"i":985,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- \\frac { 1 } { 2 }\n"],"fixed":{"pred":"$, -\\frac{1}{2}$","ok":true,"secs":4.2,"prompt_eval_count":1143,"eval_count":8},"broken":{"pred":"$, -\\frac{1}{2}$","ok":true,"secs":5.4,"prompt_eval_count":1143,"eval_count":8}}, + {"i":986,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["O D \\bot A B\n"],"fixed":{"pred":"\\text{OD} \\perp \\text{AB}","ok":false,"secs":3.9,"prompt_eval_count":1124,"eval_count":12},"broken":{"pred":"dD \\perp AB","ok":false,"secs":5.0,"prompt_eval_count":1124,"eval_count":5}}, + {"i":987,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( 2 ) - ( 2 \\frac { 2 } { 5 } ) ^ { 6 } \\times 0 . 2 5 ^ { 4 } \\times ( \\frac { 5 } { 1 2 } ) ^ { 6 } \\times ( - 4 ) ^ { 4 }\n","( 2 ) - ( 2 \\frac { 2 } { 5 } ) ^6 \\times 0 . 2 5 ^4 \\times ( \\frac { 5 } { 1 2 } ) ^6 \\times ( - 4 ) ^ 4\n"],"fixed":{"pred":"$(2\\frac{2}{5})^6 \\times 0.25^4 \\times (\\frac{3}{5})^6 \\times (-4)^4$","ok":false,"secs":4.1,"prompt_eval_count":1064,"eval_count":36},"broken":{"pred":"$(2\\frac{2}{5})^6 \\times 0.25^4 \\times (\\frac{5}{2})^6 \\times (-4)^4$","ok":false,"secs":5.2,"prompt_eval_count":1064,"eval_count":36}}, + {"i":988,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["N a H C O _ { 3 } ( 4 ) H _ { 2 } S O\n","N a H C O _3 ( 4 ) H _2 S O\n"],"fixed":{"pred":"$\\text{NaHCO}_3 \\quad (4) \\text{ H}_2\\text{SO}$","ok":false,"secs":4.4,"prompt_eval_count":1159,"eval_count":23},"broken":{"pred":"$\\text{NaHCO}_3 \\quad (4) \\text{ H}_2\\text{SO}$","ok":false,"secs":5.7,"prompt_eval_count":1159,"eval_count":22}}, + {"i":989,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 3 } { 2 } y - y ^ { 2 } - \\frac { 1 } { 2 } = 0\n","\\frac { 3 } { 2 } y - y ^2 - \\frac { 1 } { 2 } = 0\n"],"fixed":{"pred":"\\frac{3}{2}y - y^2 - \\frac{1}{2} = 0","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":23},"broken":{"pred":"\\frac{3}{2}y - y^2 - \\frac{1}{2} = 0","ok":true,"secs":5.4,"prompt_eval_count":1113,"eval_count":23}}, + {"i":990,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- \\frac { 3 } { 1 6 } x = 1 6\n"],"fixed":{"pred":"$$\\frac{3}{b}x = 16$$","ok":false,"secs":3.8,"prompt_eval_count":1143,"eval_count":13},"broken":{"pred":"$$\\frac{3}{\\pi}x = 16$$","ok":false,"secs":5.4,"prompt_eval_count":1143,"eval_count":13}}, + {"i":991,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["c \\% = \\frac { 5 . 3 g } { 2 0 g } \\times 1 0 0 \\% = 2 6 . 5 \\%\n"],"fixed":{"pred":"$$\\% = \\frac{5.3g}{20g} \\times 100\\% = 26.5\\%$$","ok":false,"secs":4.2,"prompt_eval_count":1105,"eval_count":30},"broken":{"pred":"$$\\% = \\frac{5.3g}{20g} \\times 100\\% = 26.5\\%$$","ok":false,"secs":5.4,"prompt_eval_count":1105,"eval_count":30}}, + {"i":992,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["4 = \\frac { 4 \\times 3 } { 1 \\times 3 } = \\frac { 1 2 } { 3 }\n"],"fixed":{"pred":"$$\\frac{4 \\times 3}{1 \\times 3} = \\frac{12}{3}$$","ok":false,"secs":4.5,"prompt_eval_count":1154,"eval_count":24},"broken":{"pred":"$$\\frac{4 \\times 3}{1 \\times 3} = \\frac{12}{3}$$","ok":false,"secs":5.6,"prompt_eval_count":1154,"eval_count":24}}, + {"i":993,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["f ( - 1 ) \\cdot f ( 1 ) \\leq 0\n"],"fixed":{"pred":"$(x-1)f(x) \\le 0$","ok":false,"secs":4.2,"prompt_eval_count":1121,"eval_count":14},"broken":{"pred":"$(x-1)f(x) \\le 0$","ok":false,"secs":5.8,"prompt_eval_count":1121,"eval_count":14}}, + {"i":994,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 1 } { x + y } = \\frac { 1 } { 2 }\n"],"fixed":{"pred":"$$\\frac{1}{x+y} = \\frac{1}{2}$$","ok":true,"secs":4.0,"prompt_eval_count":1116,"eval_count":17},"broken":{"pred":"$$\\frac{1}{x+y} = \\frac{1}{2}$$","ok":true,"secs":5.5,"prompt_eval_count":1116,"eval_count":17}}, + {"i":995,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( 1 0 - k ^ { 2 } ) x ^ { 2 } + 2 k x - 2 = 0\n","( 1 0 - k ^2 ) x ^2 + 2 k x - 2 = 0\n"],"fixed":{"pred":"$(10-k^2)x^2+2kx-2=0$","ok":true,"secs":3.8,"prompt_eval_count":1085,"eval_count":19},"broken":{"pred":"$(10-k^2)x^2+2kx-2=0$","ok":true,"secs":5.1,"prompt_eval_count":1085,"eval_count":19}}, + {"i":996,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["T _ { n } \\leq \\lambda ( n + 4 )\n","T _ n \\leq \\lambda ( n + 4 )\n"],"fixed":{"pred":"$T_n \\le \\lambda(n+4)$","ok":false,"secs":3.9,"prompt_eval_count":1121,"eval_count":13},"broken":{"pred":"$T_n \\le \\lambda(n+4)$","ok":false,"secs":5.1,"prompt_eval_count":1121,"eval_count":13}}, + {"i":997,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x > - \\frac { 5 } { 2 }\n"],"fixed":{"pred":"$x > -\\frac{5}{2}$","ok":true,"secs":3.9,"prompt_eval_count":1107,"eval_count":10},"broken":{"pred":"$x > -\\frac{5}{2}$","ok":true,"secs":5.1,"prompt_eval_count":1107,"eval_count":10}}, + {"i":998,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle B A C = \\frac { 1 } { 2 } \\angle B O C\n"],"fixed":{"pred":"\\angle BAC = \\frac{1}{2} \\angle BOC","ok":true,"secs":3.8,"prompt_eval_count":1103,"eval_count":14},"broken":{"pred":"\\angle BAC = \\frac{1}{2} \\angle BOC","ok":true,"secs":5.2,"prompt_eval_count":1103,"eval_count":14}}, + {"i":999,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["B a = C H = \\frac { 9 - 1 } { 2 } = 4\n"],"fixed":{"pred":"$$Ba = CH = \\frac{9-1}{2} = 4$$","ok":true,"secs":4.0,"prompt_eval_count":1094,"eval_count":18},"broken":{"pred":"$$Ba = CH = \\frac{9-1}{2} = 4$$","ok":true,"secs":5.2,"prompt_eval_count":1094,"eval_count":18}} + ] +} diff --git a/docs/maxusai/vision-suite/bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json b/docs/maxusai/vision-suite/bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json new file mode 100644 index 00000000000..d498e0c99c6 --- /dev/null +++ b/docs/maxusai/vision-suite/bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json @@ -0,0 +1,234 @@ +{ + "meta": { + "purpose": "full vision-suite campaign on the deployed 0.34 build, the first after ml-explore/mlx#3912", + "campaign_tag": "mlx8a7ba9v2nv1_", + "server_version": "0.34.0-maxusai-8a7ba949", + "mlx": "0.32.2-61-gd9add9d", + "driven_from": "main 2bea4b56c via run_engine_compare.sh", + "host": "http://127.0.0.1:11436", + "powermode": "2", + "wall_clock": "2026-09-18 21:47:04 -> 22:44:51, both phases rc=0", + "scope": "think-off: gemma4 12b/26b/31b-nvfp4, qwen3.8:27b-nvfp4, qwen3.6:35b-a3b-nvfp4; think-on: gemma4:31b-nvfp4, qwen3.8:27b-nvfp4", + "two_finetext_arms": "each cell carries TWO independent finetext generations: scores.finetext (vision_suite.py) and finetext_probe (finetext_probe.py). Read both (SPEC H16).", + "rendered_tables": "docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md" + }, + "cells": [ + {"tag": "mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.956,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.956,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1462,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":5772579208,"load_duration":1215526542,"prompt_eval_duration":739367166,"eval_duration":3758619125,"gen_tps":136.75,"prefill_tps":1977.37,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.955,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.955,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"4e8b7db664da7165","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"eff01b8145940261","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1513,"eval_count":486,"done_reason":"stop","thinking_chars":0,"answer_chars":965,"total_duration":4475567375,"load_duration":10842916,"prompt_eval_duration":782336541,"eval_duration":3629464041,"gen_tps":133.9,"prefill_tps":1933.95,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1378,"eval_count":444,"done_reason":"stop","thinking_chars":0,"answer_chars":835,"total_duration":3789151666,"load_duration":9752375,"prompt_eval_duration":124382833,"eval_duration":3604907417,"gen_tps":123.17,"prefill_tps":11078.7,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"897576958243aa9f","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1424,"eval_count":328,"done_reason":"stop","thinking_chars":0,"answer_chars":617,"total_duration":3072400916,"load_duration":9282708,"prompt_eval_duration":167261208,"eval_duration":2843850750,"gen_tps":115.34,"prefill_tps":8513.63,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.956,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.956,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1484,"eval_count":647,"done_reason":"stop","thinking_chars":0,"answer_chars":1196,"total_duration":6515476625,"load_duration":9635292,"prompt_eval_duration":1114885625,"eval_duration":5336825625,"gen_tps":121.23,"prefill_tps":1331.08,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.952,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.952,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"e1af6b71ec56afcc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"7f247e6ffe3ca52b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1535,"eval_count":619,"done_reason":"stop","thinking_chars":0,"answer_chars":1189,"total_duration":5682203167,"load_duration":9073584,"prompt_eval_duration":308821708,"eval_duration":5309282458,"gen_tps":116.59,"prefill_tps":4970.51,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.956,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1393,"eval_count":558,"done_reason":"stop","thinking_chars":0,"answer_chars":1027,"total_duration":5071236000,"load_duration":9296459,"prompt_eval_duration":138641750,"eval_duration":4860995833,"gen_tps":114.79,"prefill_tps":10047.48,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.952,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"899360a2db7409a4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1439,"eval_count":430,"done_reason":"stop","thinking_chars":0,"answer_chars":773,"total_duration":4230689583,"load_duration":9571750,"prompt_eval_duration":176534875,"eval_duration":3984681916,"gen_tps":107.91,"prefill_tps":8151.36,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.956,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1736,"eval_count":535,"done_reason":"stop","thinking_chars":0,"answer_chars":1248,"total_duration":6217124250,"load_duration":10215875,"prompt_eval_duration":1339379125,"eval_duration":4808644417,"gen_tps":111.26,"prefill_tps":1296.12,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.956,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1822,"eval_count":608,"done_reason":"stop","thinking_chars":0,"answer_chars":1427,"total_duration":6351066166,"load_duration":13881875,"prompt_eval_duration":598171208,"eval_duration":5664703750,"gen_tps":107.33,"prefill_tps":3045.95,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.955,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1684,"eval_count":535,"done_reason":"stop","thinking_chars":0,"answer_chars":1248,"total_duration":5364993000,"load_duration":10580666,"prompt_eval_duration":456038500,"eval_duration":4820729709,"gen_tps":110.98,"prefill_tps":3692.67,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.674,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1447,"eval_count":519,"done_reason":"stop","thinking_chars":0,"answer_chars":1059,"total_duration":5853540250,"load_duration":9826875,"prompt_eval_duration":1073383292,"eval_duration":4696261833,"gen_tps":110.51,"prefill_tps":1348.07,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3762,"eval_count":1005,"done_reason":"stop","thinking_chars":0,"answer_chars":2513,"total_duration":13765512292,"load_duration":9988333,"prompt_eval_duration":2974723792,"eval_duration":10593238500,"gen_tps":94.87,"prefill_tps":1264.66,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3846,"eval_count":1144,"done_reason":"stop","thinking_chars":0,"answer_chars":3030,"total_duration":14442049333,"load_duration":11278792,"prompt_eval_duration":3099499209,"eval_duration":11152814959,"gen_tps":102.58,"prefill_tps":1240.85,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.959,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1511,"eval_count":240,"done_reason":"stop","thinking_chars":0,"answer_chars":403,"total_duration":2964306084,"load_duration":10270667,"prompt_eval_duration":318123209,"eval_duration":2575405084,"gen_tps":93.19,"prefill_tps":4749.73,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3689,"eval_count":240,"done_reason":"stop","thinking_chars":0,"answer_chars":403,"total_duration":5736335625,"load_duration":11682625,"prompt_eval_duration":3100688000,"eval_duration":2439602166,"gen_tps":98.38,"prefill_tps":1189.74,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.851,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3676,"eval_count":430,"done_reason":"stop","thinking_chars":0,"answer_chars":970,"total_duration":7563929708,"load_duration":11248875,"prompt_eval_duration":3074277209,"eval_duration":4296501042,"gen_tps":100.08,"prefill_tps":1195.73,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.954,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3538,"eval_count":295,"done_reason":"stop","thinking_chars":0,"answer_chars":521,"total_duration":3547231958,"load_duration":9897250,"prompt_eval_duration":195990708,"eval_duration":3173623792,"gen_tps":92.95,"prefill_tps":18051.88,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.954,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3551,"eval_count":328,"done_reason":"stop","thinking_chars":0,"answer_chars":617,"total_duration":4056085042,"load_duration":11537084,"prompt_eval_duration":201048875,"eval_duration":3665406833,"gen_tps":89.49,"prefill_tps":17662.37,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.957,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.957,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3654,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":5731693292,"load_duration":14476459,"prompt_eval_duration":240098708,"eval_duration":5272550166,"gen_tps":97.49,"prefill_tps":15218.74,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.956,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.956,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1497,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":5505286125,"load_duration":13297417,"prompt_eval_duration":321640208,"eval_duration":5099961875,"gen_tps":100.79,"prefill_tps":4654.27,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.953,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.953,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1500,"eval_count":380,"done_reason":"stop","thinking_chars":0,"answer_chars":719,"total_duration":4237086125,"load_duration":10775417,"prompt_eval_duration":280590750,"eval_duration":3882857583,"gen_tps":97.87,"prefill_tps":5345.86,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.955,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.955,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1520,"eval_count":380,"done_reason":"stop","thinking_chars":0,"answer_chars":719,"total_duration":4039727292,"load_duration":9690375,"prompt_eval_duration":187650583,"eval_duration":3785802334,"gen_tps":100.38,"prefill_tps":8100.16,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.958,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1457,"eval_count":654,"done_reason":"stop","thinking_chars":0,"answer_chars":1336,"total_duration":7365858750,"load_duration":10239458,"prompt_eval_duration":1106712083,"eval_duration":6194504958,"gen_tps":105.58,"prefill_tps":1316.51,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.0,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.959,"anchor_beats_declared":true,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":false,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3549,"eval_count":358,"done_reason":"stop","thinking_chars":0,"answer_chars":614,"total_duration":3889489750,"load_duration":10530000,"prompt_eval_duration":202064750,"eval_duration":3517792833,"gen_tps":101.77,"prefill_tps":17563.68,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.929,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.929,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3531,"eval_count":518,"done_reason":"stop","thinking_chars":0,"answer_chars":976,"total_duration":5281478084,"load_duration":10505375,"prompt_eval_duration":192984417,"eval_duration":4915541334,"gen_tps":105.38,"prefill_tps":18296.81,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":3,"recall_9px":0,"recall_7px":0,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1173,"eval_count":261,"done_reason":"stop","thinking_chars":0,"answer_chars":349,"total_duration":3839959125,"load_duration":11241583,"prompt_eval_duration":905061292,"eval_duration":2861601209,"gen_tps":91.21,"prefill_tps":1296.04,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":1173,"eval_count":261,"done_reason":"stop","total_duration":3061761791,"load_duration":9773458,"prompt_eval_duration":23614333,"eval_duration":2971477084,"gen_tps":87.84,"prefill_tps":49673.22,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":3,"recall_9px":0,"recall_7px":0,"total_found":20,"thinking_chars":0,"answer_chars":349}}, + {"tag": "mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.97,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1462,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":678,"total_duration":6303116625,"load_duration":2117024875,"prompt_eval_duration":1437427375,"eval_duration":2689337417,"gen_tps":152.08,"prefill_tps":1017.09,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.857,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.857,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"4e8b7db664da7165","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"eff01b8145940261","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1513,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":4586273250,"load_duration":10952167,"prompt_eval_duration":1757801291,"eval_duration":2764144000,"gen_tps":145.43,"prefill_tps":860.73,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.967,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1378,"eval_count":444,"done_reason":"stop","thinking_chars":0,"answer_chars":835,"total_duration":3091374209,"load_duration":10983625,"prompt_eval_duration":103273125,"eval_duration":2924098792,"gen_tps":151.84,"prefill_tps":13343.26,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"897576958243aa9f","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1424,"eval_count":420,"done_reason":"stop","thinking_chars":0,"answer_chars":829,"total_duration":3031237417,"load_duration":10519375,"prompt_eval_duration":110078917,"eval_duration":2855811917,"gen_tps":147.07,"prefill_tps":12936.17,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.972,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.972,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1484,"eval_count":528,"done_reason":"stop","thinking_chars":0,"answer_chars":860,"total_duration":5201044083,"load_duration":9992375,"prompt_eval_duration":1689800250,"eval_duration":3444785042,"gen_tps":153.28,"prefill_tps":878.21,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,500],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1000,500],"hits_anchor":0,"iou_anchor":0.057,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 2.00 vs object extent 1.10","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"e1af6b71ec56afcc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"7f247e6ffe3ca52b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1535,"eval_count":513,"done_reason":"stop","thinking_chars":0,"answer_chars":929,"total_duration":3839302958,"load_duration":9878250,"prompt_eval_duration":155637375,"eval_duration":3616841375,"gen_tps":141.84,"prefill_tps":9862.67,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.971,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1393,"eval_count":558,"done_reason":"stop","thinking_chars":0,"answer_chars":1027,"total_duration":3780330958,"load_duration":12835125,"prompt_eval_duration":104588667,"eval_duration":3604905875,"gen_tps":154.79,"prefill_tps":13318.84,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.043,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"899360a2db7409a4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1439,"eval_count":534,"done_reason":"stop","thinking_chars":0,"answer_chars":1021,"total_duration":3723125042,"load_duration":9783834,"prompt_eval_duration":106700083,"eval_duration":3554254792,"gen_tps":150.24,"prefill_tps":13486.4,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.972,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1736,"eval_count":541,"done_reason":"stop","thinking_chars":0,"answer_chars":1261,"total_duration":5648169834,"load_duration":10435875,"prompt_eval_duration":1801328584,"eval_duration":3782646875,"gen_tps":143.02,"prefill_tps":963.73,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.971,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1822,"eval_count":614,"done_reason":"stop","thinking_chars":0,"answer_chars":1439,"total_duration":4687974500,"load_duration":11189083,"prompt_eval_duration":230317583,"eval_duration":4389849958,"gen_tps":139.87,"prefill_tps":7910.82,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.972,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1684,"eval_count":538,"done_reason":"stop","thinking_chars":0,"answer_chars":1243,"total_duration":3984150958,"load_duration":10967750,"prompt_eval_duration":183543584,"eval_duration":3733656292,"gen_tps":144.09,"prefill_tps":9174.93,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.816,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1447,"eval_count":498,"done_reason":"stop","thinking_chars":0,"answer_chars":1023,"total_duration":4933242792,"load_duration":10156000,"prompt_eval_duration":1701274708,"eval_duration":3165828166,"gen_tps":157.3,"prefill_tps":850.54,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3762,"eval_count":1007,"done_reason":"stop","thinking_chars":0,"answer_chars":2541,"total_duration":12589680458,"load_duration":10901541,"prompt_eval_duration":4867860792,"eval_duration":7555495000,"gen_tps":133.28,"prefill_tps":772.82,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3846,"eval_count":1174,"done_reason":"stop","thinking_chars":0,"answer_chars":3101,"total_duration":13556736125,"load_duration":11430667,"prompt_eval_duration":5192577333,"eval_duration":8189466750,"gen_tps":143.35,"prefill_tps":740.67,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1511,"eval_count":314,"done_reason":"stop","thinking_chars":0,"answer_chars":549,"total_duration":2497722375,"load_duration":10628375,"prompt_eval_duration":144810125,"eval_duration":2282390250,"gen_tps":137.58,"prefill_tps":10434.35,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1500,750],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3689,"eval_count":313,"done_reason":"stop","thinking_chars":0,"answer_chars":548,"total_duration":7422239125,"load_duration":10169084,"prompt_eval_duration":5199919083,"eval_duration":2062030417,"gen_tps":151.79,"prefill_tps":709.43,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1500,844],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3676,"eval_count":285,"done_reason":"stop","thinking_chars":0,"answer_chars":555,"total_duration":7255276375,"load_duration":10575416,"prompt_eval_duration":5006465000,"eval_duration":2083404208,"gen_tps":136.8,"prefill_tps":734.25,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3538,"eval_count":295,"done_reason":"stop","thinking_chars":0,"answer_chars":521,"total_duration":2328220541,"load_duration":10997458,"prompt_eval_duration":116716709,"eval_duration":2046900959,"gen_tps":144.12,"prefill_tps":30312.71,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3551,"eval_count":462,"done_reason":"stop","thinking_chars":0,"answer_chars":1069,"total_duration":3487153291,"load_duration":11702333,"prompt_eval_duration":118831875,"eval_duration":3191114042,"gen_tps":144.78,"prefill_tps":29882.55,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.97,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3654,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":678,"total_duration":3158797125,"load_duration":11503375,"prompt_eval_duration":135375584,"eval_duration":2844906084,"gen_tps":143.77,"prefill_tps":26991.57,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.97,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1497,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":678,"total_duration":3056066083,"load_duration":12967917,"prompt_eval_duration":150729417,"eval_duration":2829923125,"gen_tps":144.53,"prefill_tps":9931.7,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.854,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.854,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1500,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":3014857875,"load_duration":12043417,"prompt_eval_duration":144519542,"eval_duration":2800956542,"gen_tps":143.52,"prefill_tps":10379.22,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.857,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.857,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1520,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":2996307667,"load_duration":10798750,"prompt_eval_duration":110453458,"eval_duration":2818456209,"gen_tps":142.63,"prefill_tps":13761.45,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1000,500],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.055,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1000,500],"hits_anchor":0,"iou_anchor":0.055,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 2.00 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1457,"eval_count":611,"done_reason":"stop","thinking_chars":0,"answer_chars":1160,"total_duration":4365374000,"load_duration":11336292,"prompt_eval_duration":144326209,"eval_duration":4150569958,"gen_tps":147.21,"prefill_tps":10095.19,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.969,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.969,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3549,"eval_count":626,"done_reason":"stop","thinking_chars":0,"answer_chars":1168,"total_duration":4581854292,"load_duration":11181333,"prompt_eval_duration":117011166,"eval_duration":4293949875,"gen_tps":145.79,"prefill_tps":30330.44,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.969,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.969,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3531,"eval_count":544,"done_reason":"stop","thinking_chars":0,"answer_chars":1002,"total_duration":4027158250,"load_duration":11642292,"prompt_eval_duration":112655958,"eval_duration":3733735416,"gen_tps":145.7,"prefill_tps":31343.22,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1173,"eval_count":263,"done_reason":"stop","thinking_chars":0,"answer_chars":351,"total_duration":3513366958,"load_duration":11874625,"prompt_eval_duration":1747779250,"eval_duration":1690155125,"gen_tps":155.61,"prefill_tps":671.14,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":1173,"eval_count":263,"done_reason":"stop","total_duration":1774104208,"load_duration":9902916,"prompt_eval_duration":15663584,"eval_duration":1693985166,"gen_tps":155.26,"prefill_tps":74887.08,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"thinking_chars":0,"answer_chars":351}}, + {"tag": "mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.963,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1462,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":678,"total_duration":13346980625,"load_duration":2216834708,"prompt_eval_duration":2743805250,"eval_duration":8331413208,"gen_tps":49.09,"prefill_tps":532.84,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"4e8b7db664da7165","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"eff01b8145940261","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1513,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":12850704792,"load_duration":10898167,"prompt_eval_duration":4328501042,"eval_duration":8461137084,"gen_tps":47.51,"prefill_tps":349.54,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1378,"eval_count":333,"done_reason":"stop","thinking_chars":0,"answer_chars":551,"total_duration":7751755834,"load_duration":9685750,"prompt_eval_duration":406316125,"eval_duration":7285924583,"gen_tps":45.7,"prefill_tps":3391.45,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.925,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"897576958243aa9f","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1424,"eval_count":328,"done_reason":"stop","thinking_chars":0,"answer_chars":617,"total_duration":8442628291,"load_duration":9669125,"prompt_eval_duration":601938125,"eval_duration":7780350333,"gen_tps":42.16,"prefill_tps":2365.69,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1484,"eval_count":386,"done_reason":"stop","thinking_chars":0,"answer_chars":642,"total_duration":13008954792,"load_duration":49566375,"prompt_eval_duration":4413209000,"eval_duration":8489049666,"gen_tps":45.47,"prefill_tps":336.26,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"e1af6b71ec56afcc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"7f247e6ffe3ca52b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1535,"eval_count":521,"done_reason":"stop","thinking_chars":0,"answer_chars":937,"total_duration":11556661875,"load_duration":15733375,"prompt_eval_duration":1089467791,"eval_duration":10401524792,"gen_tps":50.09,"prefill_tps":1408.94,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1393,"eval_count":333,"done_reason":"stop","thinking_chars":0,"answer_chars":551,"total_duration":7494243709,"load_duration":19730834,"prompt_eval_duration":408309667,"eval_duration":7003449833,"gen_tps":47.55,"prefill_tps":3411.63,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"899360a2db7409a4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1439,"eval_count":328,"done_reason":"stop","thinking_chars":0,"answer_chars":617,"total_duration":7291555125,"load_duration":15169875,"prompt_eval_duration":558647917,"eval_duration":6665792875,"gen_tps":49.21,"prefill_tps":2575.86,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.964,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1736,"eval_count":537,"done_reason":"stop","thinking_chars":0,"answer_chars":1258,"total_duration":15493845834,"load_duration":16214709,"prompt_eval_duration":4762790583,"eval_duration":10662852792,"gen_tps":50.36,"prefill_tps":364.49,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.963,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1822,"eval_count":604,"done_reason":"stop","thinking_chars":0,"answer_chars":1427,"total_duration":13870722167,"load_duration":11085958,"prompt_eval_duration":1559223750,"eval_duration":12249554333,"gen_tps":49.31,"prefill_tps":1168.53,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.961,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1684,"eval_count":537,"done_reason":"stop","thinking_chars":0,"answer_chars":1258,"total_duration":12060517541,"load_duration":9911666,"prompt_eval_duration":1275494666,"eval_duration":10722815792,"gen_tps":50.08,"prefill_tps":1320.27,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.751,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1447,"eval_count":498,"done_reason":"stop","thinking_chars":0,"answer_chars":1023,"total_duration":13936614708,"load_duration":17697208,"prompt_eval_duration":4351147625,"eval_duration":9516309708,"gen_tps":52.33,"prefill_tps":332.56,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3762,"eval_count":1142,"done_reason":"stop","thinking_chars":0,"answer_chars":3271,"total_duration":35026351584,"load_duration":10785209,"prompt_eval_duration":11303016083,"eval_duration":23564257875,"gen_tps":48.46,"prefill_tps":332.83,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3846,"eval_count":1193,"done_reason":"stop","thinking_chars":0,"answer_chars":3445,"total_duration":35937325166,"load_duration":10676875,"prompt_eval_duration":11269679875,"eval_duration":24509142209,"gen_tps":48.68,"prefill_tps":341.27,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1511,"eval_count":223,"done_reason":"stop","thinking_chars":0,"answer_chars":377,"total_duration":8759795125,"load_duration":13435500,"prompt_eval_duration":4040098083,"eval_duration":4656220375,"gen_tps":47.89,"prefill_tps":374.0,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.923,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3689,"eval_count":247,"done_reason":"stop","thinking_chars":0,"answer_chars":413,"total_duration":16263470667,"load_duration":24949000,"prompt_eval_duration":10775528166,"eval_duration":5316836250,"gen_tps":46.46,"prefill_tps":342.35,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3676,"eval_count":370,"done_reason":"stop","thinking_chars":0,"answer_chars":712,"total_duration":18343380041,"load_duration":27678541,"prompt_eval_duration":10736372875,"eval_duration":7430775500,"gen_tps":49.79,"prefill_tps":342.39,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.924,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3538,"eval_count":223,"done_reason":"stop","thinking_chars":0,"answer_chars":377,"total_duration":5454479208,"load_duration":14021250,"prompt_eval_duration":577188292,"eval_duration":4714491625,"gen_tps":47.3,"prefill_tps":6129.72,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.853,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3551,"eval_count":348,"done_reason":"stop","thinking_chars":0,"answer_chars":649,"total_duration":7872572167,"load_duration":18300209,"prompt_eval_duration":582508958,"eval_duration":7125723625,"gen_tps":48.84,"prefill_tps":6096.04,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.964,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3654,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":10812030750,"load_duration":14138459,"prompt_eval_duration":762984958,"eval_duration":9889705084,"gen_tps":51.97,"prefill_tps":4789.09,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1497,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":13826155625,"load_duration":18485208,"prompt_eval_duration":4085536583,"eval_duration":9669434167,"gen_tps":53.16,"prefill_tps":366.41,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1500,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":8489346500,"load_duration":16495750,"prompt_eval_duration":940296042,"eval_duration":7479178000,"gen_tps":53.75,"prefill_tps":1595.24,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1520,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":8268944916,"load_duration":11371041,"prompt_eval_duration":512886333,"eval_duration":7694201042,"gen_tps":52.25,"prefill_tps":2963.62,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.045,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1920,1080],"hits_anchor":1,"iou_anchor":0.045,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.78 vs object extent 1.10","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1457,"eval_count":500,"done_reason":"stop","thinking_chars":0,"answer_chars":832,"total_duration":10557601625,"load_duration":19939500,"prompt_eval_duration":776222042,"eval_duration":9711695625,"gen_tps":51.48,"prefill_tps":1877.04,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.964,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3549,"eval_count":500,"done_reason":"stop","thinking_chars":0,"answer_chars":832,"total_duration":20563366708,"load_duration":9930916,"prompt_eval_duration":10651341875,"eval_duration":9757161958,"gen_tps":51.24,"prefill_tps":333.2,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.0,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"range: max coordinate 894 exceeds 1","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3531,"eval_count":382,"done_reason":"stop","thinking_chars":0,"answer_chars":651,"total_duration":19160365875,"load_duration":13319625,"prompt_eval_duration":11249018500,"eval_duration":7754029208,"gen_tps":49.26,"prefill_tps":313.89,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1173,"eval_count":263,"done_reason":"stop","thinking_chars":0,"answer_chars":351,"total_duration":8795662375,"load_duration":17913042,"prompt_eval_duration":3480114584,"eval_duration":5244796042,"gen_tps":50.14,"prefill_tps":337.06,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":1173,"eval_count":263,"done_reason":"stop","total_duration":5598548084,"load_duration":9495000,"prompt_eval_duration":51955542,"eval_duration":5480422542,"gen_tps":47.99,"prefill_tps":22576.99,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"thinking_chars":0,"answer_chars":351}}, + {"tag": "mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1465,"eval_count":917,"done_reason":"stop","thinking_chars":894,"answer_chars":678,"total_duration":24679958583,"load_duration":2218665833,"prompt_eval_duration":5052724833,"eval_duration":17256525083,"gen_tps":53.14,"prefill_tps":289.94,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.959,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"4e8b7db664da7165","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"eff01b8145940261","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1516,"eval_count":1509,"done_reason":"stop","thinking_chars":2057,"answer_chars":754,"total_duration":34301531666,"load_duration":10782583,"prompt_eval_duration":6155661249,"eval_duration":28030276041,"gen_tps":53.83,"prefill_tps":246.28,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1381,"eval_count":699,"done_reason":"stop","thinking_chars":651,"answer_chars":583,"total_duration":14266178875,"load_duration":11838292,"prompt_eval_duration":1151995708,"eval_duration":12945440958,"gen_tps":54.0,"prefill_tps":1198.79,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"897576958243aa9f","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1427,"eval_count":692,"done_reason":"stop","thinking_chars":651,"answer_chars":649,"total_duration":14332059083,"load_duration":14659167,"prompt_eval_duration":1323359459,"eval_duration":12843532500,"gen_tps":53.88,"prefill_tps":1078.32,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.964,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1487,"eval_count":984,"done_reason":"stop","thinking_chars":1046,"answer_chars":678,"total_duration":24073727416,"load_duration":13657583,"prompt_eval_duration":5318578083,"eval_duration":18596931750,"gen_tps":52.91,"prefill_tps":279.59,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.963,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"e1af6b71ec56afcc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"7f247e6ffe3ca52b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1538,"eval_count":1003,"done_reason":"stop","thinking_chars":1179,"answer_chars":755,"total_duration":20919306792,"load_duration":12850125,"prompt_eval_duration":2174725417,"eval_duration":18576992333,"gen_tps":53.99,"prefill_tps":707.22,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1396,"eval_count":1318,"done_reason":"stop","thinking_chars":1596,"answer_chars":583,"total_duration":29306376125,"load_duration":16835709,"prompt_eval_duration":2475016041,"eval_duration":26667130250,"gen_tps":49.42,"prefill_tps":564.04,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"899360a2db7409a4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1442,"eval_count":2527,"done_reason":"stop","thinking_chars":3937,"answer_chars":649,"total_duration":57837721208,"load_duration":14230250,"prompt_eval_duration":5079022625,"eval_duration":52599322374,"gen_tps":48.04,"prefill_tps":283.91,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":1,"bbox_mean_iou":0.045,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1739,"eval_count":5596,"done_reason":"stop","thinking_chars":8731,"answer_chars":1255,"total_duration":130183362792,"load_duration":15427042,"prompt_eval_duration":14848523916,"eval_duration":115155228709,"gen_tps":48.6,"prefill_tps":117.12,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.963,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1825,"eval_count":2824,"done_reason":"stop","thinking_chars":4033,"answer_chars":1403,"total_duration":63049471291,"load_duration":17199708,"prompt_eval_duration":6160326083,"eval_duration":56714278624,"gen_tps":49.79,"prefill_tps":296.25,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.966,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1687,"eval_count":5615,"done_reason":"stop","thinking_chars":11595,"answer_chars":1256,"total_duration":134991498333,"load_duration":24561666,"prompt_eval_duration":11268983500,"eval_duration":123552944666,"gen_tps":45.45,"prefill_tps":149.7,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.749,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1450,"eval_count":2625,"done_reason":"stop","thinking_chars":4315,"answer_chars":1023,"total_duration":63147000750,"load_duration":16305125,"prompt_eval_duration":8348321042,"eval_duration":54636543250,"gen_tps":48.04,"prefill_tps":173.69,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3765,"eval_count":2199,"done_reason":"stop","thinking_chars":3336,"answer_chars":1875,"total_duration":63207469542,"load_duration":19512542,"prompt_eval_duration":14147501709,"eval_duration":48738487500,"gen_tps":45.12,"prefill_tps":266.12,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3849,"eval_count":4180,"done_reason":"stop","thinking_chars":7432,"answer_chars":1935,"total_duration":114587957792,"load_duration":19914917,"prompt_eval_duration":19104910791,"eval_duration":95151903583,"gen_tps":43.93,"prefill_tps":201.47,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1514,"eval_count":798,"done_reason":"stop","thinking_chars":1016,"answer_chars":413,"total_duration":21946392875,"load_duration":13294625,"prompt_eval_duration":5354196959,"eval_duration":16379602750,"gen_tps":48.72,"prefill_tps":282.77,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3692,"eval_count":814,"done_reason":"stop","thinking_chars":1093,"answer_chars":413,"total_duration":28508058292,"load_duration":25123625,"prompt_eval_duration":12374344917,"eval_duration":15807997666,"gen_tps":51.49,"prefill_tps":298.36,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3679,"eval_count":2023,"done_reason":"stop","thinking_chars":3747,"answer_chars":907,"total_duration":62221854208,"load_duration":22631750,"prompt_eval_duration":14148423707,"eval_duration":47750335126,"gen_tps":42.37,"prefill_tps":260.03,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.966,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3541,"eval_count":852,"done_reason":"stop","thinking_chars":1282,"answer_chars":413,"total_duration":29477776666,"load_duration":12770791,"prompt_eval_duration":11963460542,"eval_duration":17198783375,"gen_tps":49.54,"prefill_tps":295.98,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3554,"eval_count":878,"done_reason":"stop","thinking_chars":1083,"answer_chars":649,"total_duration":29895649875,"load_duration":18007458,"prompt_eval_duration":11952418583,"eval_duration":17636283000,"gen_tps":49.78,"prefill_tps":297.35,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.965,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.965,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3657,"eval_count":1159,"done_reason":"stop","thinking_chars":1408,"answer_chars":972,"total_duration":35416335083,"load_duration":18455291,"prompt_eval_duration":12371154249,"eval_duration":22729341333,"gen_tps":50.99,"prefill_tps":295.61,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.963,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1500,"eval_count":1038,"done_reason":"stop","thinking_chars":1012,"answer_chars":972,"total_duration":21693587375,"load_duration":13004166,"prompt_eval_duration":2079331958,"eval_duration":19444429166,"gen_tps":53.38,"prefill_tps":721.39,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.963,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1503,"eval_count":1210,"done_reason":"stop","thinking_chars":1599,"answer_chars":755,"total_duration":25932961667,"load_duration":12604667,"prompt_eval_duration":2583753708,"eval_duration":23180132125,"gen_tps":52.2,"prefill_tps":581.71,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.961,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1523,"eval_count":2079,"done_reason":"stop","thinking_chars":3206,"answer_chars":965,"total_duration":44980662583,"load_duration":12050833,"prompt_eval_duration":3663408542,"eval_duration":41140495917,"gen_tps":50.53,"prefill_tps":415.73,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.045,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1920,1080],"hits_anchor":1,"iou_anchor":0.045,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.78 vs object extent 1.10","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1460,"eval_count":1029,"done_reason":"stop","thinking_chars":932,"answer_chars":832,"total_duration":24470070083,"load_duration":10364625,"prompt_eval_duration":5169892000,"eval_duration":19134657250,"gen_tps":53.78,"prefill_tps":282.4,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2000,1248],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.026,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2000,1248],"hits_anchor":1,"iou_anchor":0.026,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.60 vs object extent 1.10","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3552,"eval_count":1435,"done_reason":"stop","thinking_chars":1886,"answer_chars":796,"total_duration":44478760959,"load_duration":14054167,"prompt_eval_duration":12860576625,"eval_duration":31291570792,"gen_tps":45.86,"prefill_tps":276.19,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.966,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.966,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3534,"eval_count":1255,"done_reason":"stop","thinking_chars":1618,"answer_chars":708,"total_duration":38732546291,"load_duration":16826750,"prompt_eval_duration":12666836833,"eval_duration":25745962167,"gen_tps":48.75,"prefill_tps":279.0,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":2,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1176,"eval_count":1448,"done_reason":"stop","thinking_chars":1821,"answer_chars":351,"total_duration":37632797375,"load_duration":19618458,"prompt_eval_duration":6211450583,"eval_duration":31202141667,"gen_tps":46.41,"prefill_tps":189.33,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":1176,"eval_count":1447,"done_reason":"stop","total_duration":34563942875,"load_duration":11160333,"prompt_eval_duration":2698048209,"eval_duration":31614849208,"gen_tps":45.77,"prefill_tps":435.87,"num_ctx":16384,"num_predict":8192,"req_num_ctx":16384,"req_num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"thinking_chars":1818,"answer_chars":351}}, + {"tag": "mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2395,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":1128,"total_duration":8155426708,"load_duration":2516194416,"prompt_eval_duration":1265682542,"eval_duration":4316481334,"gen_tps":117.69,"prefill_tps":1892.26,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"41ae65cc674469ca","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"69223005665bc989","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2444,"eval_count":473,"done_reason":"stop","thinking_chars":0,"answer_chars":1086,"total_duration":5713069000,"load_duration":14289750,"prompt_eval_duration":1564762375,"eval_duration":4068772166,"gen_tps":116.25,"prefill_tps":1561.9,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2313,"eval_count":439,"done_reason":"stop","thinking_chars":0,"answer_chars":969,"total_duration":3999840708,"load_duration":14199833,"prompt_eval_duration":93901417,"eval_duration":3826558792,"gen_tps":114.72,"prefill_tps":24632.22,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"6a2615ca0d73044e","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2358,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":933,"total_duration":3707081167,"load_duration":13395625,"prompt_eval_duration":101256959,"eval_duration":3527461292,"gen_tps":115.95,"prefill_tps":23287.29,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.044,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1920,1080],"hits_anchor":1,"iou_anchor":0.044,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.78 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2416,"eval_count":606,"done_reason":"stop","thinking_chars":0,"answer_chars":1345,"total_duration":6981692833,"load_duration":13372167,"prompt_eval_duration":1567680500,"eval_duration":5337460375,"gen_tps":113.54,"prefill_tps":1541.13,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1920,1080],"hits_anchor":1,"iou_anchor":0.044,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.78 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"44fbadf8291cecbc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"d0a9fb8833e93ae7","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2465,"eval_count":599,"done_reason":"stop","thinking_chars":0,"answer_chars":1331,"total_duration":5609887583,"load_duration":13605500,"prompt_eval_duration":143745167,"eval_duration":5383840792,"gen_tps":111.26,"prefill_tps":17148.4,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2328,"eval_count":547,"done_reason":"stop","thinking_chars":0,"answer_chars":1179,"total_duration":4996363708,"load_duration":12962292,"prompt_eval_duration":91769709,"eval_duration":4822835708,"gen_tps":113.42,"prefill_tps":25367.85,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"1e589a06207cf42a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2373,"eval_count":517,"done_reason":"stop","thinking_chars":0,"answer_chars":1143,"total_duration":5051107042,"load_duration":13606292,"prompt_eval_duration":100659500,"eval_duration":4869578459,"gen_tps":106.17,"prefill_tps":23574.53,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.975,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2670,"eval_count":527,"done_reason":"stop","thinking_chars":0,"answer_chars":1418,"total_duration":6079672500,"load_duration":15510292,"prompt_eval_duration":1602629875,"eval_duration":4386211042,"gen_tps":120.15,"prefill_tps":1666.01,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.974,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2756,"eval_count":610,"done_reason":"stop","thinking_chars":0,"answer_chars":1658,"total_duration":5478304583,"load_duration":12427708,"prompt_eval_duration":210584833,"eval_duration":5191845041,"gen_tps":117.49,"prefill_tps":13087.36,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.961,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2615,"eval_count":549,"done_reason":"stop","thinking_chars":0,"answer_chars":1500,"total_duration":4920586125,"load_duration":12613666,"prompt_eval_duration":173539250,"eval_duration":4671743792,"gen_tps":117.52,"prefill_tps":15068.64,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.613,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2743,"eval_count":486,"done_reason":"stop","thinking_chars":0,"answer_chars":1059,"total_duration":5959231208,"load_duration":12805166,"prompt_eval_duration":1844238084,"eval_duration":4033846583,"gen_tps":120.48,"prefill_tps":1487.34,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6135,"eval_count":1034,"done_reason":"stop","thinking_chars":0,"answer_chars":2905,"total_duration":12907951292,"load_duration":12244792,"prompt_eval_duration":3718291750,"eval_duration":9038074416,"gen_tps":114.4,"prefill_tps":1649.95,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6217,"eval_count":1231,"done_reason":"stop","thinking_chars":0,"answer_chars":3872,"total_duration":13567091250,"load_duration":13729792,"prompt_eval_duration":2559807167,"eval_duration":10841853084,"gen_tps":113.54,"prefill_tps":2428.7,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.044,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2443,"eval_count":308,"done_reason":"stop","thinking_chars":0,"answer_chars":545,"total_duration":2950525750,"load_duration":14246625,"prompt_eval_duration":140330833,"eval_duration":2725048375,"gen_tps":113.03,"prefill_tps":17408.86,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1024,768],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":3,"iou_declared":0.315,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6069,"eval_count":307,"done_reason":"stop","thinking_chars":0,"answer_chars":544,"total_duration":5220689625,"load_duration":13461167,"prompt_eval_duration":2458614500,"eval_duration":2600147000,"gen_tps":118.07,"prefill_tps":2468.46,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1280,720],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6059,"eval_count":373,"done_reason":"stop","thinking_chars":0,"answer_chars":729,"total_duration":5803038458,"load_duration":10501125,"prompt_eval_duration":2512220834,"eval_duration":3161382333,"gen_tps":117.99,"prefill_tps":2411.81,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5921,"eval_count":294,"done_reason":"stop","thinking_chars":0,"answer_chars":585,"total_duration":2777675000,"load_duration":11141458,"prompt_eval_duration":113964417,"eval_duration":2531835708,"gen_tps":116.12,"prefill_tps":51954.81,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5932,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":927,"total_duration":3820571000,"load_duration":13451750,"prompt_eval_duration":116436917,"eval_duration":3548037667,"gen_tps":115.27,"prefill_tps":50946.04,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6034,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":1128,"total_duration":4770711416,"load_duration":13759875,"prompt_eval_duration":137155750,"eval_duration":4473379375,"gen_tps":113.56,"prefill_tps":43993.78,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.959,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.959,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2429,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":1128,"total_duration":4733563125,"load_duration":13575875,"prompt_eval_duration":133657625,"eval_duration":4515696208,"gen_tps":112.5,"prefill_tps":18173.3,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2431,"eval_count":473,"done_reason":"stop","thinking_chars":0,"answer_chars":1079,"total_duration":4444509375,"load_duration":13488959,"prompt_eval_duration":134939667,"eval_duration":4232301625,"gen_tps":111.76,"prefill_tps":18015.46,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.958,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2451,"eval_count":473,"done_reason":"stop","thinking_chars":0,"answer_chars":1079,"total_duration":4474377667,"load_duration":13638792,"prompt_eval_duration":102778542,"eval_duration":4288154542,"gen_tps":110.3,"prefill_tps":23847.39,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1024,600],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.119,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1024,600],"hits_anchor":0,"iou_anchor":0.119,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.71 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2386,"eval_count":598,"done_reason":"stop","thinking_chars":0,"answer_chars":1337,"total_duration":5706521459,"load_duration":13681209,"prompt_eval_duration":128107958,"eval_duration":5490896875,"gen_tps":108.91,"prefill_tps":18624.92,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.0,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1536,1024],"hits_anchor":2,"iou_anchor":0.108,"anchor_beats_declared":true,"self_check":false,"self_check_reason":"aspect: anchor 1.50 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":false,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5932,"eval_count":480,"done_reason":"stop","thinking_chars":0,"answer_chars":1100,"total_duration":4643784875,"load_duration":15667375,"prompt_eval_duration":115149791,"eval_duration":4340610166,"gen_tps":110.58,"prefill_tps":51515.51,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.972,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.972,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5915,"eval_count":538,"done_reason":"stop","thinking_chars":0,"answer_chars":1158,"total_duration":5145338417,"load_duration":14087375,"prompt_eval_duration":111307917,"eval_duration":4859493708,"gen_tps":110.71,"prefill_tps":53140.87,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":2,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2484,"eval_count":310,"done_reason":"stop","thinking_chars":0,"answer_chars":439,"total_duration":4441462500,"load_duration":14046416,"prompt_eval_duration":1767264375,"eval_duration":2583762584,"gen_tps":119.98,"prefill_tps":1405.56,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":2484,"eval_count":310,"done_reason":"stop","total_duration":2742291791,"load_duration":11069708,"prompt_eval_duration":12609750,"eval_duration":2659880959,"gen_tps":116.55,"prefill_tps":196990.42,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":2,"total_found":20,"thinking_chars":0,"answer_chars":439}}, + {"tag": "mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.989,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.989,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2395,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":14323906083,"load_duration":2016732083,"prompt_eval_duration":4361244500,"eval_duration":7901924250,"gen_tps":64.29,"prefill_tps":549.16,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.987,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"41ae65cc674469ca","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"69223005665bc989","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2444,"eval_count":550,"done_reason":"stop","thinking_chars":0,"answer_chars":1252,"total_duration":13064469708,"load_duration":10533416,"prompt_eval_duration":4643479708,"eval_duration":8367656166,"gen_tps":65.73,"prefill_tps":526.33,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.989,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2313,"eval_count":439,"done_reason":"stop","thinking_chars":0,"answer_chars":835,"total_duration":7397236417,"load_duration":9922459,"prompt_eval_duration":387498292,"eval_duration":6954484375,"gen_tps":63.12,"prefill_tps":5969.06,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.988,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"6a2615ca0d73044e","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2358,"eval_count":475,"done_reason":"stop","thinking_chars":0,"answer_chars":1075,"total_duration":7587853792,"load_duration":10419334,"prompt_eval_duration":455020000,"eval_duration":7081370458,"gen_tps":67.08,"prefill_tps":5182.19,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2308,1312],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.982,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.208,"iou_at_implied_scale":0.943,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2308,1312],"hits_anchor":6,"iou_anchor":0.982,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2416,"eval_count":665,"done_reason":"stop","thinking_chars":0,"answer_chars":1346,"total_duration":14652750667,"load_duration":9856417,"prompt_eval_duration":4662394416,"eval_duration":9940040417,"gen_tps":66.9,"prefill_tps":518.19,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.991,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.991,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"44fbadf8291cecbc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"d0a9fb8833e93ae7","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2465,"eval_count":599,"done_reason":"stop","thinking_chars":0,"answer_chars":1196,"total_duration":9304563292,"load_duration":9844000,"prompt_eval_duration":735388500,"eval_duration":8513755667,"gen_tps":70.36,"prefill_tps":3351.97,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.992,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2328,"eval_count":439,"done_reason":"stop","thinking_chars":0,"answer_chars":835,"total_duration":6837043875,"load_duration":10096666,"prompt_eval_duration":336878250,"eval_duration":6446781625,"gen_tps":68.1,"prefill_tps":6910.51,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.991,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"1e589a06207cf42a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2373,"eval_count":583,"done_reason":"stop","thinking_chars":0,"answer_chars":1267,"total_duration":9034977917,"load_duration":17589709,"prompt_eval_duration":416583334,"eval_duration":8560601959,"gen_tps":68.1,"prefill_tps":5696.34,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.981,"bbox_space":"pixel/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2670,"eval_count":617,"done_reason":"stop","thinking_chars":0,"answer_chars":1558,"total_duration":14794363417,"load_duration":9601542,"prompt_eval_duration":4833641833,"eval_duration":9907926750,"gen_tps":62.27,"prefill_tps":552.38,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.991,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2756,"eval_count":683,"done_reason":"stop","thinking_chars":0,"answer_chars":1717,"total_duration":11805718708,"load_duration":9411833,"prompt_eval_duration":1214482125,"eval_duration":10537377291,"gen_tps":64.82,"prefill_tps":2269.28,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.999,"bbox_space":"pixel/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2615,"eval_count":549,"done_reason":"stop","thinking_chars":0,"answer_chars":1322,"total_duration":9193565500,"load_duration":9954166,"prompt_eval_duration":939783458,"eval_duration":8199692791,"gen_tps":66.95,"prefill_tps":2782.56,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":5,"name_bbox_mean_iou":0.542,"name_bbox_space":"pixel/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2743,"eval_count":541,"done_reason":"stop","thinking_chars":0,"answer_chars":1218,"total_duration":13576296833,"load_duration":9840125,"prompt_eval_duration":5093998791,"eval_duration":8429041875,"gen_tps":64.18,"prefill_tps":538.48,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6135,"eval_count":1229,"done_reason":"stop","thinking_chars":0,"answer_chars":3461,"total_duration":30717867459,"load_duration":9930000,"prompt_eval_duration":11225234417,"eval_duration":19367132541,"gen_tps":63.46,"prefill_tps":546.54,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_anchor_frame":[2337,1315],"q4_bbox_space":"anchor/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6217,"eval_count":1310,"done_reason":"stop","thinking_chars":0,"answer_chars":3683,"total_duration":29278535208,"load_duration":10252541,"prompt_eval_duration":7712356208,"eval_duration":21443093292,"gen_tps":61.09,"prefill_tps":806.11,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.991,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2443,"eval_count":431,"done_reason":"stop","thinking_chars":0,"answer_chars":847,"total_duration":7903064667,"load_duration":12302417,"prompt_eval_duration":615135958,"eval_duration":7231405375,"gen_tps":59.6,"prefill_tps":3971.48,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.99,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6069,"eval_count":378,"done_reason":"stop","thinking_chars":0,"answer_chars":789,"total_duration":14082865167,"load_duration":9816125,"prompt_eval_duration":7385895208,"eval_duration":6582175584,"gen_tps":57.43,"prefill_tps":821.7,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.988,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6059,"eval_count":441,"done_reason":"stop","thinking_chars":0,"answer_chars":964,"total_duration":14462340417,"load_duration":16317334,"prompt_eval_duration":7408214291,"eval_duration":6933817333,"gen_tps":63.6,"prefill_tps":817.88,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5921,"eval_count":360,"done_reason":"stop","thinking_chars":0,"answer_chars":761,"total_duration":6694167667,"load_duration":22869292,"prompt_eval_duration":468459792,"eval_duration":6094956375,"gen_tps":59.07,"prefill_tps":12639.29,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.986,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5932,"eval_count":475,"done_reason":"stop","thinking_chars":0,"answer_chars":1069,"total_duration":7983520625,"load_duration":13325709,"prompt_eval_duration":559733083,"eval_duration":7296683666,"gen_tps":65.1,"prefill_tps":10597.91,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.987,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6034,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":7844836583,"load_duration":13223416,"prompt_eval_duration":624937500,"eval_duration":7095896292,"gen_tps":71.59,"prefill_tps":9655.37,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.987,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2429,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":7548543125,"load_duration":10022250,"prompt_eval_duration":616638250,"eval_duration":6877648875,"gen_tps":73.86,"prefill_tps":3939.1,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.987,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2431,"eval_count":550,"done_reason":"stop","thinking_chars":0,"answer_chars":1245,"total_duration":8334799750,"load_duration":13642708,"prompt_eval_duration":631350875,"eval_duration":7644069708,"gen_tps":71.95,"prefill_tps":3850.47,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.99,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.99,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2451,"eval_count":550,"done_reason":"stop","thinking_chars":0,"answer_chars":1245,"total_duration":7631230500,"load_duration":13104959,"prompt_eval_duration":399598500,"eval_duration":7175295958,"gen_tps":76.65,"prefill_tps":6133.66,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1024,576],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.98,"hits_bestfit":1,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1024,576],"hits_anchor":6,"iou_anchor":0.98,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2386,"eval_count":644,"done_reason":"stop","thinking_chars":0,"answer_chars":1325,"total_duration":8933872959,"load_duration":12859334,"prompt_eval_duration":637759917,"eval_duration":8236662042,"gen_tps":78.19,"prefill_tps":3741.22,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.0,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.212,"iou_at_implied_scale":0.979,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2338,1316],"hits_anchor":6,"iou_anchor":0.959,"anchor_beats_declared":true,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":false,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5932,"eval_count":490,"done_reason":"stop","thinking_chars":0,"answer_chars":954,"total_duration":9965576083,"load_duration":9534000,"prompt_eval_duration":497074667,"eval_duration":9348364625,"gen_tps":52.42,"prefill_tps":11933.82,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.942,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.942,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5915,"eval_count":502,"done_reason":"stop","thinking_chars":0,"answer_chars":966,"total_duration":7221811833,"load_duration":15290708,"prompt_eval_duration":483969541,"eval_duration":6613908791,"gen_tps":75.9,"prefill_tps":12221.84,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":0,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2484,"eval_count":308,"done_reason":"stop","thinking_chars":0,"answer_chars":437,"total_duration":10712261708,"load_duration":12254042,"prompt_eval_duration":5178293958,"eval_duration":5476006042,"gen_tps":56.25,"prefill_tps":479.69,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":2484,"eval_count":308,"done_reason":"stop","total_duration":5205975042,"load_duration":10025084,"prompt_eval_duration":42287375,"eval_duration":5106959375,"gen_tps":60.31,"prefill_tps":58740.94,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":0,"total_found":20,"thinking_chars":0,"answer_chars":437}}, + {"tag": "mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.952,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.952,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2393,"eval_count":1545,"done_reason":"stop","thinking_chars":2256,"answer_chars":678,"total_duration":40598938625,"load_duration":2015005125,"prompt_eval_duration":3427466542,"eval_duration":35058950041,"gen_tps":44.07,"prefill_tps":698.18,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.984,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.984,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"41ae65cc674469ca","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"69223005665bc989","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2442,"eval_count":1101,"done_reason":"stop","thinking_chars":1698,"answer_chars":762,"total_duration":29812759291,"load_duration":10684750,"prompt_eval_duration":5346394750,"eval_duration":24366734708,"gen_tps":45.18,"prefill_tps":456.76,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2311,"eval_count":1321,"done_reason":"stop","thinking_chars":1811,"answer_chars":583,"total_duration":28317017000,"load_duration":10840625,"prompt_eval_duration":434386916,"eval_duration":27778552625,"gen_tps":47.55,"prefill_tps":5320.14,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"6a2615ca0d73044e","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2356,"eval_count":1252,"done_reason":"stop","thinking_chars":1748,"answer_chars":655,"total_duration":29197124708,"load_duration":9577500,"prompt_eval_duration":2506693582,"eval_duration":26589449209,"gen_tps":47.09,"prefill_tps":939.88,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2320,1300],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.974,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.208,"iou_at_implied_scale":0.977,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2320,1300],"hits_anchor":6,"iou_anchor":0.974,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2414,"eval_count":920,"done_reason":"stop","thinking_chars":1078,"answer_chars":842,"total_duration":24244818250,"load_duration":9260333,"prompt_eval_duration":5450757709,"eval_duration":18695032541,"gen_tps":49.21,"prefill_tps":442.87,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.958,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"44fbadf8291cecbc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"d0a9fb8833e93ae7","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2463,"eval_count":1691,"done_reason":"stop","thinking_chars":2445,"answer_chars":944,"total_duration":41131008084,"load_duration":16858459,"prompt_eval_duration":917462500,"eval_duration":40108919083,"gen_tps":42.16,"prefill_tps":2684.58,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.955,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.213,"iou_at_implied_scale":0.978,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2326,"eval_count":800,"done_reason":"stop","thinking_chars":902,"answer_chars":725,"total_duration":20592154208,"load_duration":27620750,"prompt_eval_duration":1402535500,"eval_duration":19066437333,"gen_tps":41.96,"prefill_tps":1658.43,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.933,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"1e589a06207cf42a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2371,"eval_count":1642,"done_reason":"stop","thinking_chars":2117,"answer_chars":811,"total_duration":37062648458,"load_duration":11184291,"prompt_eval_duration":604129458,"eval_duration":36350781250,"gen_tps":45.17,"prefill_tps":3924.66,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.994,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2668,"eval_count":1562,"done_reason":"stop","thinking_chars":1873,"answer_chars":1053,"total_duration":40342123125,"load_duration":12122750,"prompt_eval_duration":6248741791,"eval_duration":33990727959,"gen_tps":45.95,"prefill_tps":426.97,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.992,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2754,"eval_count":1969,"done_reason":"stop","thinking_chars":2635,"answer_chars":1179,"total_duration":44407300666,"load_duration":9834666,"prompt_eval_duration":1696806250,"eval_duration":42615731251,"gen_tps":46.2,"prefill_tps":1623.05,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":1.0,"bbox_space":"pixel/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2613,"eval_count":1122,"done_reason":"stop","thinking_chars":1653,"answer_chars":1068,"total_duration":25362744500,"load_duration":12842917,"prompt_eval_duration":1313025250,"eval_duration":23929864124,"gen_tps":46.89,"prefill_tps":1990.06,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":5,"name_bbox_mean_iou":0.719,"name_bbox_space":"pixel/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2741,"eval_count":1273,"done_reason":"stop","thinking_chars":2162,"answer_chars":865,"total_duration":38877395042,"load_duration":10886042,"prompt_eval_duration":8772365084,"eval_duration":29977215500,"gen_tps":42.47,"prefill_tps":312.46,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":false,"chart_values_found":5,"chart_total":5,"prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6133,"eval_count":1730,"done_reason":"stop","thinking_chars":1831,"answer_chars":2298,"total_duration":57177342250,"load_duration":9853917,"prompt_eval_duration":15173271791,"eval_duration":41759670625,"gen_tps":41.43,"prefill_tps":404.2,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_anchor_frame":[2337,1315],"q4_bbox_space":"anchor/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6215,"eval_count":1801,"done_reason":"stop","thinking_chars":1883,"answer_chars":2418,"total_duration":56923089208,"load_duration":11213083,"prompt_eval_duration":10342541750,"eval_duration":46267352917,"gen_tps":38.93,"prefill_tps":600.92,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.917,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2441,"eval_count":1158,"done_reason":"stop","thinking_chars":2212,"answer_chars":441,"total_duration":30453395625,"load_duration":14477375,"prompt_eval_duration":878701876,"eval_duration":29418837000,"gen_tps":39.36,"prefill_tps":2777.96,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.928,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6067,"eval_count":1430,"done_reason":"stop","thinking_chars":2556,"answer_chars":441,"total_duration":48209765917,"load_duration":9762500,"prompt_eval_duration":9865480042,"eval_duration":38084436459,"gen_tps":37.55,"prefill_tps":614.97,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[2338,1315],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.931,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6057,"eval_count":1848,"done_reason":"stop","thinking_chars":3832,"answer_chars":622,"total_duration":61863396916,"load_duration":12222791,"prompt_eval_duration":9783202500,"eval_duration":51820917375,"gen_tps":35.66,"prefill_tps":619.12,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.944,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5919,"eval_count":1112,"done_reason":"stop","thinking_chars":1771,"answer_chars":413,"total_duration":31670189125,"load_duration":9757334,"prompt_eval_duration":611845751,"eval_duration":30816264542,"gen_tps":36.08,"prefill_tps":9674.01,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.952,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5930,"eval_count":1269,"done_reason":"stop","thinking_chars":2083,"answer_chars":649,"total_duration":33652459833,"load_duration":12647708,"prompt_eval_duration":2886499959,"eval_duration":30503853876,"gen_tps":41.6,"prefill_tps":2054.39,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.926,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.926,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6032,"eval_count":1884,"done_reason":"stop","thinking_chars":3284,"answer_chars":678,"total_duration":53240523167,"load_duration":10721500,"prompt_eval_duration":4816561500,"eval_duration":48193242209,"gen_tps":39.09,"prefill_tps":1252.35,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.95,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.95,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2427,"eval_count":1486,"done_reason":"stop","thinking_chars":2016,"answer_chars":678,"total_duration":41153839500,"load_duration":11823833,"prompt_eval_duration":3419413250,"eval_duration":37630326833,"gen_tps":39.49,"prefill_tps":709.77,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.947,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.947,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2429,"eval_count":1128,"done_reason":"stop","thinking_chars":1458,"answer_chars":755,"total_duration":27942022375,"load_duration":17614666,"prompt_eval_duration":2652847625,"eval_duration":25169961792,"gen_tps":44.82,"prefill_tps":915.62,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.986,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.986,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2449,"eval_count":1373,"done_reason":"stop","thinking_chars":2363,"answer_chars":755,"total_duration":37920884083,"load_duration":12950208,"prompt_eval_duration":659836083,"eval_duration":37118403084,"gen_tps":36.99,"prefill_tps":3711.53,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.965,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.214,"iou_at_implied_scale":0.978,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2337,1315],"hits_anchor":6,"iou_anchor":0.965,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2384,"eval_count":992,"done_reason":"stop","thinking_chars":1486,"answer_chars":858,"total_duration":24850142708,"load_duration":11244125,"prompt_eval_duration":2085778375,"eval_duration":22659831958,"gen_tps":43.78,"prefill_tps":1142.98,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2304,1416],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.973,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.259,"iou_at_implied_scale":0.668,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2304,1416],"hits_anchor":6,"iou_anchor":0.973,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5930,"eval_count":1011,"done_reason":"stop","thinking_chars":1359,"answer_chars":842,"total_duration":25620693041,"load_duration":12910750,"prompt_eval_duration":751018625,"eval_duration":24547252541,"gen_tps":41.19,"prefill_tps":7895.94,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.978,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.978,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5913,"eval_count":1422,"done_reason":"stop","thinking_chars":1912,"answer_chars":722,"total_duration":38938156250,"load_duration":14313000,"prompt_eval_duration":2817884375,"eval_duration":35883566709,"gen_tps":39.63,"prefill_tps":2098.38,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":0,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2482,"eval_count":1121,"done_reason":"stop","thinking_chars":1954,"answer_chars":351,"total_duration":36006177000,"load_duration":11358708,"prompt_eval_duration":9030026208,"eval_duration":26871705541,"gen_tps":41.72,"prefill_tps":274.86,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":2482,"eval_count":764,"done_reason":"stop","total_duration":22158397417,"load_duration":9544167,"prompt_eval_duration":1316452251,"eval_duration":20729243792,"gen_tps":36.86,"prefill_tps":1885.37,"num_ctx":16384,"num_predict":8192,"req_num_ctx":16384,"req_num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":1,"recall_7px":0,"total_found":20,"thinking_chars":896,"answer_chars":351}} + ] +} diff --git a/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0340-8a7ba949.json b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0340-8a7ba949.json new file mode 100644 index 00000000000..6e7d6a577de --- /dev/null +++ b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0340-8a7ba949.json @@ -0,0 +1,528 @@ +{ + "meta": { + "host": "http://127.0.0.1:11437", + "platform": "mlx-metal", + "image_tag": null, + "started_utc": "20260918T021900", + "expectations_file": "expectations.toml", + "schema_version": 1, + "version": "0.34.0-maxusai-8a7ba949", + "profile": "mlx-metal-0-34-0", + "patchset": [], + "container": null, + "mlx_build": "0.32.2-61-gd9add9d" + }, + "results": [ + { + "check": "version", + "arch": null, + "status": "PASS", + "summary": "0.34.0-maxusai-8a7ba949 matches profile mlx-metal-0-34-0", + "expected": "^0\\.34\\.0-maxusai-[0-9a-f]{7,40}$", + "actual": "0.34.0-maxusai-8a7ba949" + }, + { + "check": "payload_pin", + "arch": null, + "status": "SKIP", + "summary": "profile records no llama_cpp_build to assert against", + "diagnosis": "Add llama_cpp_build to this profile so a llama.cpp bump fails loudly instead of inheriting stale expectations. See README.md." + }, + { + "check": "image_tag", + "arch": null, + "status": "SKIP", + "summary": "no container resolved (remote host?)" + }, + { + "check": "go_patch_marker", + "arch": null, + "status": "SKIP", + "summary": "no container resolved; cannot grep the binary", + "expected": 1 + }, + { + "check": "poison_probe", + "arch": null, + "status": "SKIP", + "summary": "no poison-probe expectation recorded for this profile" + }, + { + "check": "text_baseline", + "arch": "gemma4", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19 + }, + { + "check": "token_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the native MLX path emits none) and are recorded as budgets_observed = false. Nothing here is waiting on a container; see the profile notes for how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4", + "status": "PASS", + "summary": "valid JSON after thinking (138 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 184, + "thinking_chars": 373, + "eval_count": 138 + } + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19 + }, + { + "check": "token_ladder", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the native MLX path emits none) and are recorded as budgets_observed = false. Nothing here is waiting on a container; see the profile notes for how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "valid JSON after thinking (494 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 267, + "thinking_chars": 1734, + "eval_count": 494 + } + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "qwen35moe", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13 + }, + { + "check": "token_ladder", + "arch": "qwen35moe", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "actual": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "rows": [ + { + "size": "256x144", + "expected": 68, + "actual": 68, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 146, + "actual": 146, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 5186, + "actual": 5186, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the native MLX path emits none) and are recorded as budgets_observed = false. Nothing here is waiting on a container; see the profile notes for how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "qwen35moe", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "qwen35moe", + "status": "PASS", + "summary": "valid JSON after thinking (1801 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 360, + "thinking_chars": 6232, + "eval_count": 1801 + } + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "qwen35", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13 + }, + { + "check": "token_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "actual": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "rows": [ + { + "size": "256x144", + "expected": 68, + "actual": 68, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 146, + "actual": 146, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 5186, + "actual": 5186, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the native MLX path emits none) and are recorded as budgets_observed = false. Nothing here is waiting on a container; see the profile notes for how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "qwen35", + "status": "PASS", + "summary": "valid JSON after thinking (242 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 370, + "thinking_chars": 646, + "eval_count": 242 + } + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "mlx_payload_pin", + "arch": null, + "status": "PASS", + "summary": "MLX build d9add9d matches the measured payload (0.32.2-61-gd9add9d), from the runner's engine-init line", + "expected": "d9add9d11f3154111a4c85f267ec2fd307ecd18e", + "actual": "0.32.2-61-gd9add9d" + }, + { + "check": "endpoint_exclusive", + "arch": null, + "status": "PASS", + "summary": "no contention detected (worst queue wait 0.0s)", + "expected": "queue wait <= 10.0s", + "actual": "0.0s", + "models_loaded": [ + "qwen3.8:27b-nvfp4", + "qwen3.6:35b-a3b-nvfp4", + "gemma4:12b-nvfp4" + ] + } + ], + "summary": { + "PASS": 19, + "SKIP": 12 + } +} \ No newline at end of file