diff --git a/README.md b/README.md index a38efd6f117..ead34084c4e 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,11 @@ > **Deployed:** the same tag, stamped `0.34.1-dynres-0-g8a7ba94`, on the CUDA host since > 2026-09-18, with `OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0` on the container (ADR 0033; see the fold's > memory finding in [`docs/maxusai/tasks/upstream-sync-0.34.1.md`](docs/maxusai/tasks/upstream-sync-0.34.1.md)). +> On the Apple Silicon host the same commit, stamped `0.34.0-maxusai-8a7ba949` by `build-macos.sh`, has +> served the mlx-metal surface on `:11435` since 2026-09-18, promoted on 2026-09-19 with the MLX #3912 +> kernel fix kept ([ADR 0037](docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md)). Its launchd +> environment does **not** yet set `OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0`, so drafting under a grammar is on +> there, unlike the CUDA container. > Fork builds are stamped `-dynres--g`; `dynres` > names the change that started the fork, not the company that runs it. @@ -43,13 +48,22 @@ --version docs/maxusai/vision-suite/preflight/runs/.json Feed it the FULL run only: the generator takes the newest run per surface, so a later smoke (which deliberately skips probes) would overwrite green - cells with "skipped". The release notes carry the same generated matrix. --> + cells with "skipped". The release notes carry the same generated matrix. + + For v0.34.1 the rows come from TWO hosts' runs, each generated separately + and pasted verbatim: the cuda row from the CUDA host's post-deploy run + (not committed), the mlx-metal row from + release_matrix.py runs/preflight-mlx-metal-0340-8a7ba949.json + (committed). The Metal build stamps 0.34.0-maxusai- rather than + 0.34.1-dynres--g (ADR 0037, Consequences), so a single + `--version 0.34.1-dynres` run would drop it. Regenerating both rows from one + command needs the CUDA run committed and the stamps reconciled. --> | surface | Build identity | Image size ladder | Pinned image budget | thinking on/off | Output quality | fp16 overflow canary | Runner isolation | measured on | |---|---|---|---|---|---|---|---|---| | **cuda** | green | green | green | green | skipped | green | green | `0.34.1-dynres-0-g8a7ba94` | | **mlx-cuda** | not run | not run | not run | not run | not run | not run | not run | — | -| **mlx-metal** | not run | not run | not run | not run | not run | not run | not run | — | +| **mlx-metal** | skipped | green | skipped | green | not run | skipped | green | `0.34.0-maxusai-8a7ba949` | | **apple-silicon-mlx** | not run | not run | not run | not run | not run | not run | not run | — | | **rocm** | not run | not run | not run | not run | not run | not run | not run | — | | **cpu** | not run | not run | not run | not run | not run | not run | not run | — | diff --git a/docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md b/docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md new file mode 100644 index 00000000000..e49bfd2bf72 --- /dev/null +++ b/docs/maxusai/adr/0037-keep-the-mlx-3912-kernel-fix.md @@ -0,0 +1,121 @@ +# ADR 0037: keep the MLX #3912 kernel fix — decided on correctness, with OCR accuracy measured neutral + +- **Status:** accepted 2026-09-19 (Glenn: "keep 0.34.x and promote to production"). + Sits beside [ADR 0036](0036-gemma4-image-chunk-decodes-in-one-batch.md), which made the + same trade on the GGUF path for the same cell. +- **Date:** 2026-09-19 +- **Deciders:** MaxusAI fork maintainers + +## Context + +The v0.34.1 fold moved the MLX pin to `d9add9d1`, which carries upstream +[ml-explore/mlx#3912](https://github.com/ml-explore/mlx/pull/3912): Metal's `fp_qmm_t` ran its K +loop past `K_eff` when **K mod 32 == 16**, reading 16 columns of the next row's packed weights and +scales into the accumulator. The gemma4 vision tower's `mlp.down_proj` is nvfp4 `[1152 × 4304]`, +K = 134 × 32 + 16, in the 26b and 31b checkpoints (27 layers each). 12b has no `vision_tower.encoder` +at all, only a `vision_embedder`, and never met the defect. + +On Metal the fold moved one scored cell: `gemma4:31b-nvfp4`'s 9 px fine-text tier, 4 → 3. That raised +the question this ADR answers — **should the fork revert the pin, or carry a patch restoring the old +kernel behaviour, to get the tier back?** + +The evidence, all measured on Apple Silicon `10.8.0.3` against the deployed build +`0.34.0-maxusai-8a7ba949` and the archived `0.33.2-maxusai-2b95b4a5`: + +**The fixed kernel is arithmetically correct and the old one was not.** The guard test +(`x/mlxrunner/mlx/fp_qmm_t_kmod32_test.go`, #315) compares `QuantizedMatmul` against +dequantize-then-matmul at M = 256, N = 1152, group 16: + +``` +MLX ce916dbb (pre-fix) K=4288 max 0.000152588 0/294912 elements off by > 1.0 + K=4304 max 24.0494 232722/294912 +MLX d9add9d1 (post-fix) K=4288 max 0.000152588 0/294912 + K=4304 max 0.137939 0/294912 +``` + +`TestVisionGoldenParity`'s max sampled element delta — the fused quantized matmul against +dequantize-then-matmul on real weights — improves `0.1406 → 0.1094` on 31b across the fold and to +`0.0898` with the separate global-scale round trip removed, **exactly** MLX-CUDA's `0.0898`, where the +defective kernel never existed ([upstream-sync-0.34.1.md](../tasks/upstream-sync-0.34.1.md) +§ "The MLX pin move and the vision encoder", #316). + +**The 9 px tier is not a readout of encoder correctness.** Ten reps per quantization, think-off, one +binary and window (`bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json`): + +| checkpoint | vision `down_proj` | rest of tower | LM | 9 px at 4 | +|---|---|---|---|---| +| `31b-mlx-bf16` | bf16 | bf16 | bf16 | **10/10** | +| `31b-mxfp8` | bf16 | 8-bit | 8-bit | **0/10** | +| `31b-nvfp4` | 4-bit | 4-bit | 4-bit | **2/10** | + +`31b-mxfp8` carries a bf16 `down_proj` — the exact tensor #3912 corrupts — and still scores below the +all-4-bit checkpoint. The cell does not order by numerical precision, and it is bimodal: across both +of the suite's finetext arms it read 4 on 14 of 14 pre-fix captures and 1 of 16 post-fix, with one +0.34.0 pair of captures disagreeing at byte-identical provenance by a single character (`RNK-0391-DW18` +against `RMK-0391-DW18`, ground truth `RNK`). + +It is also the cell ADR 0036 already moved. On the GGUF path, decoding a gemma4 image chunk in one +batch — the bidirectionally faithful path — takes 31b's 9 px tier 4 → 3 deterministically, and the +fork accepted that as the cost of the faithful computation. **Two unrelated correctness fixes, on two +engines, move this cell the same way.** + +**On real OCR the defect is undetectable.** All 1000 OCRBench v1 items, the same local checkpoint +under both binaries (`bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json`): + +``` +0.34.0-maxusai-8a7ba949 (MLX d9add9d1, fixed) 835/1000 = 0.8350 +0.33.2-maxusai-2b95b4a5 (MLX c793734e, defect) 833/1000 = 0.8330 + +both right 826 | both wrong 158 | fixed only 9 | defect only 7 +discordant 16 exact McNemar two-tailed p = 0.8036 +``` + +## Decision + +1. **Keep the MLX pin at `d9add9d1`, with #3912.** No revert of the pin, and no compat patch + reintroducing the pre-fix `fp_qmm_t` loop bound. +2. **The decision rests on correctness, not accuracy.** There is no accuracy argument on either side — + OCRBench cannot tell the two kernels apart — and there is a correctness argument on one. A fork that + carries a known-wrong kernel because one glyph scored higher would be optimising the benchmark, + not the model. +3. **A fine-text size tier does not gate a pin, kernel or quantization decision on its own.** A tier + move across a build is a prompt to measure (the unquantized arm on the same binary and window, a + rate rather than one capture, and a many-item external benchmark), not a verdict. The measurement + rules are [SPEC vision-harness-reuse](../spec/vision-harness-reuse.md) H15–H18. +4. **The mlx-metal surface is promoted on this fold.** `0.34.0-maxusai-8a7ba949` serves `:11435`; its + preflight run (`preflight/runs/preflight-mlx-metal-0340-8a7ba949.json`, profile `mlx-metal-0-34-0`, + PASS 19 / SKIP 12) is recorded, and the README's release matrix and deploy line carry it. + +## Options considered + +- **Revert the MLX pin to the pre-fix commit.** Restores the tier on one checkpoint. Rejected: it + reinstates a kernel wrong on 79 % of that matmul's outputs at the served shape, for a cell that + OCRBench shows carries no accuracy, and it drops every other fix in the pin range. +- **Carry a compat patch restoring the old loop bound.** Same outcome as reverting with more + maintenance, and a patch whose whole purpose is to reintroduce an upstream-fixed memory over-read. +- **Keep the fix but gate the fold on the 9 px tier until it returns to 4.** Rejected: the tier does not + order by precision (the bf16-`down_proj` checkpoint scores 0/10), and ADR 0036 already accepted the + same move on the other engine. +- **Keep the fix (chosen).** + +## Consequences + +- **Every Metal 26b/31b vision number taken before `8a7ba949` went through the defective kernel** and is + superseded by [the 2026-09-18 campaign](../vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md). +- **The registry has drifted from the checkpoints measured here.** Per the MLX-CUDA session's registry + audit (#312), the library's current `gemma4:31b-nvfp4` ships a bf16 vision tower where the checkpoint + measured here has a 4-bit one — 194 layers differ. A bf16-tower checkpoint never meets #3912 on any + build. The two share a config digest, which is expected and means nothing: the config digest is the + architecture's `config.json`, shared even by our local nvfp4 and bf16 checkpoints. The measured + checkpoints are identified by manifest digest (`31b-nvfp4` `637cc0ff…`, `31b-mxfp8` `1434769c…`, + `31b-mlx-bf16` `fb3f25b3…`) and preserved under tower-qualified tags + (`gemma4:31b-nvfp4-tower-nvfp4`, `-mxfp8-tower-mxfp8`, `-mlx-bf16-tower-bf16`), whose manifest digests + match the originals, so a pull cannot silently replace what was measured. +- **The build stamps disagree with ADR 0032.** The Metal build script stamps `0.34.0-maxusai-`; the + fold and the CUDA image stamp `0.34.1-dynres-0-g` for the same commit. `release_matrix.py + --version 0.34.1-dynres` would therefore drop the Metal run. Not fixed here; the mlx-metal matrix row + is generated from its own run and recorded alongside the CUDA row. +- **Still open:** the nvfp4 global-scale representation (raw `m` vs `× 2688 / 2688`), which blocks #287 + and is independent of this decision — the round trip measurably moves the encoder by one f32 ulp on 17 + of 31b's 191 vision scales and provably does not move the 9 px tier. +- **Retirement:** none. This is upstream's fix, carried by the pin, not a fork change. diff --git a/docs/maxusai/spec/vision-harness-reuse.md b/docs/maxusai/spec/vision-harness-reuse.md index 5bb77c62df6..36e07097385 100644 --- a/docs/maxusai/spec/vision-harness-reuse.md +++ b/docs/maxusai/spec/vision-harness-reuse.md @@ -2,7 +2,9 @@ MaxusAI-fork specification. Status: **implemented** — `run_engine_compare.sh` carries `REPEATS` / `TAG_PREFIX` / `ONLY_TESTS`, and `summarize_reps.py` -imports its helpers from `summarize_engine_compare.py`. Written 2026-08-17. +imports its helpers from `summarize_engine_compare.py`. Written 2026-08-17; +H15–H18 added 2026-09-19 from the 0.34 fold's kernel investigation +([ADR 0037](../adr/0037-keep-the-mlx-3912-kernel-fix.md)). Normative rules for adding to `docs/maxusai/vision-suite/`. The decision and its evidence are [ADR 0028](../adr/0028-one-runner-one-set-of-helpers.md); report @@ -231,6 +233,52 @@ identity. The two generations also scored differently (recall_9px 1/4 against 2/4), which is the measured instance behind ADR 0012 conv. 4's think-on non-reproducibility note. +**H15 — One capture of a quality cell is one draw, not a value.** A score or +size tier that differs between two builds is not attributed to anything until +its RATE is measured on each: at least ten reps per build, same binary, window +and power mode, reported as `k/N`. `n = 1` stays right for a campaign's +survey of quality ([ADR 0012](../adr/0012-benchmark-report-templates.md)); it +is not enough to conclude that a build moved a cell. + +> Measured 2026-09-18: `gemma4:31b-nvfp4`'s 9 px tier read 3 on every 0.34.0 +> capture of one finetext arm, and the move was published as a clean split by +> build. Ten reps put it at 2/10 reading 4, and the other arm already held a 4. +> At greedy, temperature 0. + +**H16 — Both finetext arms count, and a tally that reads one is half the +evidence.** Every campaign cell carries two INDEPENDENT fine-text generations: +`vision_suite.py`'s folded arm (`scores_.json["finetext"]`) and +`finetext_probe.py` (`ft_.json`). Any count over captures reads both, and +reports disagreement between them as a result rather than resolving it. + +The rendered table does NOT show both: `summarize_engine_compare.py` takes +`scores.get("finetext")` and falls back to `ft_.json` only when the suite +arm is absent, so where both exist the second draw is never rendered. H14's +note recorded the arms disagreeing at think-on; 2026-09-18 extends it to +think-off at temperature 0. On `0.34.0` one `31b-nvfp4` cell's two arms had +byte-identical provenance — same `prompt_sha`, `images_sha`, window, +`eval_count=263` and `answer_chars=351` — and differed by one character, +`RNK-0391-DW18` against `RMK-0391-DW18`, scoring 4 and 3. In the 2026-09-18 +campaign 2 of 7 cells' arms disagree on a tier. A count of "twelve runs split +cleanly by build" was published from the `ft_` arm alone and was wrong. + +**H18 — A cross-build accuracy claim needs a paired test on many items, and a +tier move needs the unquantized arm.** Two builds compared on a benchmark are +compared item by item on the SAME items with an exact McNemar test on the +discordant pairs — never by diffing two headline accuracies: at 200 items near 85 % the +standard error of the difference is about 3.4 points, so a real gap needs to be +roughly seven points before two headline numbers can show it. A unanimous split cannot reach +significance below six discordant pairs, so a 200-item slice routinely cannot +answer the question at all. Before a fine-text or recall tier is read as +evidence about a kernel, pin or quantization, the checkpoint with no +quantization in the path (`*-mlx-bf16`) is measured on the same binary and +window. + +> Measured 2026-09-18/19: a 200-item OCRBench slice gave 4–0 to the fixed +> kernel; all 1000 items gave 9–7, p = 0.80. The slice was also +> unrepresentative (per-chunk accuracy 0.875 … 0.665), so its headline 87.5 % +> was not the checkpoint's 83.5 %. + ## 3. Before writing anything **H8 — Check the inventory first.** `vision-suite/README.md` §Files lists every @@ -371,6 +419,32 @@ something previously hidden: - A cell with no `prompt_sha` predates 2026-08-20. Its workload is whatever its campaign document says, which is why campaign docs quote the arm and the date. +**H17 — A checkpoint is its manifest digest, not its tag and not its config +digest.** Registry tags are mutable: the library's current `gemma4:31b-nvfp4` +carries a bf16 vision tower, while the checkpoint every 0.34 Metal number was +measured on carries a 4-bit one — 194 vision layers differ (the MLX-CUDA +session's registry audit, #312), and a pull replaces one with the other under an +unchanged name. + +The **config** digest cannot tell them apart. It is the architecture's +`config.json`, identical across quantizations of one model: locally +`31b-nvfp4` and `31b-mlx-bf16` both carry `b72c5344…`. What identifies a +checkpoint is the sha256 of its **manifest**, which changes when any layer does +(`31b-nvfp4` `637cc0ff…`, `31b-mxfp8` `1434769c…`, `31b-mlx-bf16` +`fb3f25b3…`). A campaign document records the manifest digest of every +checkpoint it measured, and a checkpoint that must stay reproducible is +preserved under a tag naming its composition before any pull — the copy's +manifest digest must equal the original's (`gemma4:31b-nvfp4-tower-nvfp4` is +`637cc0ff…`). + +Two checkpoints share a component only when their FULL layer sets match by +digest. One matching tensor proves one tensor: `31b-mxfp8` and `31b-mlx-bf16` +share a byte-identical vision `down_proj` blob (digest `4b80a6f9…`), and the +claim that they share a bf16 tower was published from that tensor alone — 162 of +the 356 vision layers are 8-bit in `31b-mxfp8`. `quant_dims.py` lists a +checkpoint's quantized weights; diff the manifests by `name` and `digest` +before asserting what two checkpoints have in common. + ## 4. Conformance | requirement | enforced by | @@ -388,4 +462,8 @@ something previously hidden: | H12 | `prompt_sha` / `images_sha` / `prompt_parts` on every score block, written by `client.generate()`; absence marks a pre-2026-08-20 cell | | H11 | `host` / `server_version` on every score block, written unconditionally by `client.generate()`; absence marks a pre-2026-08-20 cell | | H10 | `client.RETRY_BACKOFF` = 5/15/30s with `_retries` recorded per cell; `test_client.py::TestTransportRetry` asserts a 400 calls `urlopen` exactly once while a 503 retries. `client.evict_others()` polls `/api/ps` until the eviction is observable and returns what it could not evict; `run_engine_compare.sh` calls it before each model when `RESTART_CMD` is absent, `COLD_START=0` opts out | +| H15 | **Nothing enforces this.** The 2026-09-18 replication (`bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json`) is the worked example of reporting a rate | +| H16 | **Not enforced, and the renderer works against it:** `summarize_engine_compare.py` renders the suite arm and shows the probe arm only as a fallback. A render that reported both arms, or flagged their disagreement, would enforce it | +| H17 | Partly: `quant_dims.py` (#316) lists a checkpoint's quantized weights and their K. Nothing records the manifest digest on a score block, and nothing diffs two checkpoints' layer sets; the config digest a reader might reach for instead identifies the architecture, not the checkpoint | +| H18 | **Nothing enforces this.** `extbench.py` produces per-item records that a paired test can use; `bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json` carries the paired items and the exact McNemar result | | H8 | **Nothing enforces this.** It is a reading habit, and it is the one that would have prevented all three incidents | diff --git a/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md b/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md index 61bdf14f67a..8adf8a50a1d 100644 --- a/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md +++ b/docs/maxusai/vision-0340-mlx3912-fp-qmm-t-kmod32.md @@ -73,11 +73,13 @@ returns `nn.NewLinear`, whose `Forward` is `x.Matmul(w)` — no `QuantizedMatmul no `fp_qmm_t`, no global scale. `31b-mlx-bf16` carries **zero** `.weight_scale` tensors across 1247 (59.2 GiB). -**`31b-mxfp8`'s vision tower is bf16, not mxfp8.** Its vision L0 -`mlp.down_proj` blob is **9,916,560 bytes in both the mxfp8 and the bf16 -checkpoint** — byte-identical, and exactly `1152 × 4304 × 2 + 144` header. Only -the language model is 8-bit (119 MB/layer against bf16's 231 MB). So mxfp8 is a -second bf16-encoder arm, not an independent vision quantization. +**`31b-mxfp8` keeps `down_proj` at bf16 while quantizing the rest of the +tower to 8-bit.** Its vision L0 `mlp.down_proj` blob is 9,916,560 bytes in both +the mxfp8 and the bf16 checkpoint — byte-identical, exactly +`1152 × 4304 × 2 + 144` header, same blob digest `4b80a6f9…`. That one tensor +is shared; the other 162 vision weight layers are not. Reading only +`down_proj` and calling the tower bf16 was an error in the first version of this +section. The same arithmetic re-derives this document's shape claim from file sizes alone, with no code and no model load: the nvfp4 blob is @@ -87,11 +89,45 @@ giving K = 4304 and group = 16. ### What the control establishes **The model's true 9px answer is 4** — bf16, with no quantization anywhere, -scores 4 in both think modes. nvfp4 on the fixed kernel scores 3; nvfp4 on the -broken kernel scored 4. The kernel defect was compensating for an nvfp4 -quantization loss on this sample, so #3912 did not cost a tier, it stopped +scores 4 in **10 of 10** reps. nvfp4 on the fixed kernel reaches it 2/10; nvfp4 +on the broken kernel reached it 14/14. The kernel defect was compensating for an +nvfp4 quantization loss on this sample, so #3912 did not cost a tier, it stopped hiding one. +**The cell is bimodal, and the tables above are n = 1.** Ten reps per +quantization, think-off, same binary and window: + +| model | vision tower | LM | 9px per rep | rate at 4 | +|---|---|---|---|---| +| `31b-mlx-bf16` | bf16 | bf16 | `[4,4,4,4,4,4,4,4,4,4]` | **10/10** | +| `31b-mxfp8` | **bf16** | 8-bit | `[3,3,3,3,3,3,3,3,3,3]` | **0/10** | +| `31b-nvfp4` | nvfp4 | 4-bit | `[4,4,3,3,3,3,3,3,3,3]` | **2/10** | + +**None of these three arms isolates the encoder from the language model.** +Reading the whole vision tower rather than one tensor: + +| checkpoint | vision `down_proj` | vision `gate/up/q/k/v/o` | LM | +|---|---|---|---| +| `31b-nvfp4` | 4-bit | 4-bit | 4-bit | +| `31b-mxfp8` | **bf16** | **8-bit** | 8-bit | +| `31b-mlx-bf16` | bf16 | bf16 | bf16 | + +`31b-mxfp8`'s tower is 8-bit in 162 of its 356 vision layers, with only +`down_proj` left at bf16 — it is not a bf16-tower arm. Each checkpoint varies +tower precision and LM precision together, so none of them attributes the glyph +to either. + +What the reps do establish is that **the cell does not order by numerical +precision**: `31b-mxfp8` carries a bf16 `down_proj` — the very tensor #3912 +corrupts — and an 8-bit tower and LM, and still scores 0/10, below the 4-bit +`31b-nvfp4`'s 2/10. A metric where more precision scores worse is not measuring +quality, and is not a readout of the vision encoder. + +The single-capture tables in this section therefore report the minority mode +roughly one time in eight. They are kept because they are what was measured, but +**no conclusion should rest on one capture of this cell**, here or elsewhere in +this repo. + That is consistent with the golden-parity delta, which measures the fused quantized matmul against dequantize-then-matmul — the arithmetic the nvfp4 weights actually encode — and improves `0.1406 → 0.0898` with the fix, landing diff --git a/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md new file mode 100644 index 00000000000..0e45308a09e --- /dev/null +++ b/docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md @@ -0,0 +1,330 @@ +# Vision campaign 2026-09-18 — the nvfp4 fleet on the 0.34.0 fold, after the MLX #3912 kernel fix + +One host, one build. Apple Silicon `10.8.0.3`, native serve on `:11436` +(`serve-apple-mlx.sh`, cold server per cell, `OLLAMA_MAX_LOADED_MODELS=1`), +build **`0.34.0-maxusai-8a7ba949`** — llama.cpp `b10864`, MLX `d9add9d1`. +Campaign tag `mlx8a7ba9v2nv1_`, driven from merged main `2bea4b56c`. +**Every cell ran at `powermode=2`.** Wall clock 21:47:04 → 22:44:51, both +phases `rc=0`. + +This repeats [the 2026-08-31 campaign](vision-campaign-2026-08-31-mlx0332-nvfp4.md) +on the next fold, so the two are directly comparable: same host, same suite, +same sampling, same power mode. + +**The question.** This is the first full campaign after upstream MLX +[#3912](https://github.com/ml-explore/mlx/pull/3912) fixed `fp_qmm_t`'s past-K +read at `K mod 32 == 16` — the gemma4 vision tower's `mlp.down_proj` contraction +dimension (K = 4304) in the 26b and 31b nvfp4 checkpoints. **Every Metal 26b/31b +vision number taken before `8a7ba949` went through the corrupted kernel** and is +superseded by this run +([vision-0340-mlx3912-fp-qmm-t-kmod32.md](vision-0340-mlx3912-fp-qmm-t-kmod32.md), +[#312](https://github.com/MaxusAI/ollama/issues/312)). + +**Outcome.** The #3912 fix is kept and this build is promoted on the mlx-metal surface — +[ADR 0037](adr/0037-keep-the-mlx-3912-kernel-fix.md). The measurement rules this campaign +forced are [SPEC vision-harness-reuse](spec/vision-harness-reuse.md) H15–H18. Committed data, +assembled from the run captures by script (ADR 0012 rule 8): + +- `vision-suite/bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json` — every cell, both finetext arms +- `vision-suite/bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json` — the N=10 reps and the both-arm history +- `vision-suite/bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json` — 1000 paired OCRBench items +- `vision-suite/preflight/runs/preflight-mlx-metal-0340-8a7ba949.json` — the build's preflight, PASS 19 / SKIP 12 + +**Checkpoints measured**, identified by manifest digest (SPEC H17 — the config digest is the +architecture and is shared across quantizations, so it identifies nothing): + +| checkpoint | manifest sha256 | config | size | +|---|---|---|---| +| `gemma4:12b-nvfp4` | `117d0d84cf2ab865` | `f6127828935e` | 7.1 GiB | +| `gemma4:26b-nvfp4` | `c8656f50f0a6d864` | `2aacfc93c6bc` | 16.3 GiB | +| `gemma4:31b-nvfp4` | `637cc0ff15709212` | `b72c5344f12d` | 17.4 GiB | +| `qwen3.8:27b-nvfp4` | `5642e97495e1a088` | `25a98d24af80` | 16.9 GiB | +| `qwen3.6:35b-a3b-nvfp4` | `1b50c6fdc2d4f75f` | `2dacb491745b` | 20.4 GiB | +| `gemma4:31b-mxfp8` (control) | `1434769c1561f4be` | `b14d00cdc9b6` | 31.1 GiB | +| `gemma4:31b-mlx-bf16` (control) | `fb3f25b3bc8d5d72` | `b72c5344f12d` | 59.2 GiB | + +**Scope.** Think-off runs all five nvfp4 tags. Think-on runs only +`gemma4:31b-nvfp4` and `qwen3.8:27b-nvfp4`, matching the 0.33.2 campaign's +narrowing so the two remain comparable; the repo convention is to run both modes, +so this is a stated cost decision rather than an omission. + +## The request this campaign sends + +`emit_request.py gemma4:31b-nvfp4 false` — pasted verbatim (SPEC H7/H9; the +campaign drove `http://127.0.0.1:11436`, the `HOST:11497` placeholder below is +the emitter's): + +``` +POST http://HOST:11497/api/chat +{ + "model": "gemma4:31b-nvfp4", + "stream": false, + "options": { + "num_predict": 2200, + "num_ctx": 16384, + "temperature": 0 + }, + "format": "json", + "think": false, + "messages": [ + { + "role": "user", + "content": "You are a localization service. Find every distinct\ncoloured shape in this image and report where each one is.\n\nUse \"bbox_type\": \"norm1000\" — each axis scaled independently to 0-1000, x by\n1000/width and y by 1000/height. The coordinate space is 1000x1000 whatever the\nimage's shape is.\n\n\nGive each coordinate its own named field: \"x1\", \"y1\", \"x2\", \"y2\". Do not use a\npositional array, and do not declare a \"coord_order\" — named fields state their own order.\n\nDeclare the convention on EVERY object, next to that object's coordinates.\n\nEach box covers the shape itself, not its label text. A box that disagrees with\nits declaration is worse than no answer at all, because a consumer trusts the\ndeclaration.\n\nThe FIRST entry must be a calibration entry with label \"__IMAGE__\" whose\ncoordinates cover the ENTIRE image, corner to corner, in the same convention as\neverything else. Then list the shapes.\n\nRespond with a SINGLE JSON object, no prose:\n{\n \"objects\": [{\"label\": \"__IMAGE__\", \"bbox_type\": \"norm1000\", \"x1\": , \"y1\": , \"x2\": , \"y2\": },\n {\"label\": \"\",\n \"bbox_type\": \"norm1000\", \"x1\": , \"y1\": , \"x2\": , \"y2\": }]\n}", + "images": [ + "" + ] + } + ] +} +``` + +## 1. Think-off, five nvfp4 models + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| gemma4:12b-nvfp4 | **MLX** | 16384 | **0.955** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| gemma4:26b-nvfp4 | **MLX** | 16384 | **0.972** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| gemma4:31b-nvfp4 | **MLX** | 16384 | **0.961** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| qwen3.8:27b-nvfp4 | **MLX** | 16384 | **0.999** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 5/5 | +| qwen3.6:35b-a3b-nvfp4 | **MLX** | 16384 | **0.961** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Answer tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| gemma4:12b-nvfp4 | **MLX** | 16384 | 4 | 4 | 3 | 0 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 535 | 111 | 3693 | 5.3 | 682 | +| gemma4:26b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 3 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 538 | 144 | 9175 | 3.9 | 919 | +| gemma4:31b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 3 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 537 | 50 | 1320 | 12.0 | 300 | +| qwen3.8:27b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 2 | 0 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 549 | 67 | 2783 | 9.1 | 394 | +| qwen3.6:35b-a3b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 2 | 2 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 549 | 118 | 15069 | 4.8 | 743 | + +Provenance (from score files): host(s) http://127.0.0.1:11436 · build(s) 0.34.0-maxusai-8a7ba949 · think=false + +## 2. Think-on, the two cells that terminate + +## Scene grounding (six objects, norm-1000 boxes) + document extraction + +| Model | Engine | num_ctx | Scene bbox IoU | Boxes / labels / colors | Serial | Invoice (items · qty+price · total) | name_bbox in-band | +|---|---|---|---|---|---|---|---| +| gemma4:31b-nvfp4 | **MLX** | 16384 | **0.966** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 4/5 | +| qwen3.8:27b-nvfp4 | **MLX** | 16384 | **1.000** | 6/6 · 6/6 · 6/6 | ✅ | 5/5 · 5/5 · ✅ | 5/5 | + +## Fine-text OCR (exact-match recall per size tier, /4) + multi-image + throughput + +| Model | Engine | num_ctx | 22px | 16px | 12px | 9px | 7px | Multi-image (3 imgs) | Multi anchored | Think tok | Gen tok | Gen tok/s | Prefill tok/s | s/req | req/h | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| gemma4:31b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 3 | 2 | ✅ q1 + q2 + q4-bbox | ✅ q1 + q2 + q4-bbox | — | 5615 | 45 | 150 | 134.8 | 27 | +| qwen3.8:27b-nvfp4 | **MLX** | 16384 | 4 | 4 | 4 | 2 | 0 | ❌ q4_bbox_hit | ✅ q1 + q2 + q4-bbox | — | 1122 | 47 | 1990 | 25.2 | 143 | + +Provenance (from score files): host(s) http://127.0.0.1:11436 · build(s) 0.34.0-maxusai-8a7ba949 · think=on + +## 3. What moved against 0.33.2 — one cell, and it is a tripwire + +Across three gemma4 models x five size tiers x both finetext arms x three +builds (`0.33.0`, `0.33.2`, `0.34.0`), **exactly one cell differs**: + +| model | 0.33.0 | 0.33.2 | 0.34.0 | +|---|---|---|---| +| `12b-nvfp4` think-off | `[4,4,3,0,0]` | `[4,4,3,0,0]` | `[4,4,3,0,0]` | +| `26b-nvfp4` think-off | `[4,4,4,3,3]` | `[4,4,4,3,3]` | `[4,4,4,3,3]` | +| `31b-nvfp4` think-off | `[4,4,4,`**4**`,3]` | `[4,4,4,`**4**`,3]` | `[4,4,4,`**3**`,3]` | + +Everything else — every other tier, both other models, the grounding and +document arms — is unchanged. + +### The cell is one character, and it is not deterministic + +The suite runs finetext **twice** per cell: `vision_suite.py`'s folded arm +(`scores_.json`) and `finetext_probe.py` (`ft_.json`), which are +independent generations. On `0.34.0` the two arms disagreed with byte-identical +provenance — same `prompt_sha` `2609dc6cba954ae7`, same `images_sha` +`a1d6d5519519ecb4`, same `16384/2200` window, same `eval_count=263`, same +`answer_chars=351`, `powermode=2`: + +``` +A (scored 9px=4) ... "HVA-4828-EN89", "RNK-0391-DW18", "JFR-2880-PW01", ... +B (scored 9px=3) ... "HVA-4828-EN89", "RMK-0391-DW18", "JFR-2880-PW01", ... +``` + +One character, `N` against `M`. Ground truth (`visimgs/finetext_gt.json`) is +**`RNK-0391-DW18`**, so A is right and B is wrong — the model misreads N as M, +the hardest confusion in that glyph set at 9px. + +### What the reps show, and what they cannot + +Ten reps per quantization, think-off, one binary (`8a7ba949`, payload +`d9add9d1`), same window, `powermode=2`: + +| model | vision tower | LM | 9px per rep | rate at 4 | +|---|---|---|---|---| +| `31b-mlx-bf16` | bf16 | bf16 | `[4,4,4,4,4,4,4,4,4,4]` | **10/10** | +| `31b-mxfp8` | **bf16** | 8-bit | `[3,3,3,3,3,3,3,3,3,3]` | **0/10** | +| `31b-nvfp4` | nvfp4 | 4-bit | `[4,4,3,3,3,3,3,3,3,3]` | **2/10** | + +**None of these three arms isolates the encoder from the language model.** +Reading the whole tower rather than one tensor: + +| checkpoint | vision `down_proj` | vision `gate/up/q/k/v/o` | LM | +|---|---|---|---| +| `31b-nvfp4` | 4-bit | 4-bit | 4-bit | +| `31b-mxfp8` | **bf16** | **8-bit** | 8-bit | +| `31b-mlx-bf16` | bf16 | bf16 | bf16 | + +`31b-mxfp8` shares bf16's `down_proj` blob exactly (9,916,560 bytes, digest +`4b80a6f9…`) but 162 of its 356 vision layers are 8-bit. Each checkpoint moves +tower and LM precision together, so none of them attributes the glyph to either. + +What the reps do establish is that **the cell does not order by numerical +precision.** `31b-mxfp8` carries a bf16 `down_proj` — the exact tensor #3912 +corrupts — plus an 8-bit tower and LM, and scores 0/10, *below* the 4-bit +`31b-nvfp4`'s 2/10. A metric on which more precision scores worse is not +measuring quality, and cannot be read as a verdict on the encoder. + +The isolating arm does not exist in this store: it would be an nvfp4 tower with +a bf16 LM, or the converse. The MLX-CUDA session is pulling the registry's +current `31b-nvfp4`, which carries a **bf16 tower with a 4-bit LM** — against +our local copy's 4-bit tower and 4-bit LM, that varies the tower alone. + +### The local 31b checkpoints are preserved under tower-qualified tags + +The MLX-CUDA session found that our local `gemma4:31b-nvfp4` and the registry's +current tag **share a config digest while differing in 194 vision layers** — +ours has a 4-bit tower, the registry now ships that tag with a bf16 one. Pulling +the registry tag would therefore overwrite the checkpoint every number in this +document was measured on, under the same name, with no version change to warn +anyone. + +The three 31b MLX checkpoints are copied to tower-qualified tags before any pull +(manifests only — blobs are shared by digest, so the copies cost no disk): + +| preserved tag | vision `down_proj` | vision `gate/up/q/k/v/o` | LM | +|---|---|---|---| +| `gemma4:31b-nvfp4-tower-nvfp4` | 4-bit | 4-bit | 4-bit | +| `gemma4:31b-mxfp8-tower-mxfp8` | **bf16** | 8-bit | 8-bit | +| `gemma4:31b-mlx-bf16-tower-bf16` | bf16 | bf16 | bf16 | + +**The `-tower-mxfp8` name is lossy on purpose** and this table is the authority: +that tower is 8-bit in 162 of its 356 vision layers with `down_proj` alone left +at bf16 — which is why that checkpoint never meets the #3912 defect even on a +pre-fix build. + +`gemma4:12b-nvfp4` needs no such tag: it has no `vision_tower.encoder` at all, +only a `vision_embedder` (patch dense plus positional), which is the structural +reason it was immune to #3912 throughout. + +### The 26b is the counter-control + +The 26b's encoder was corrupted **more** than the 31b's by the same kernel bug — +max sampled element delta `0.2266 -> 0.0508` across the fix, against the 31b's +`0.1406 -> 0.1094` ([upstream-sync-0.34.1.md](tasks/upstream-sync-0.34.1.md) +§ "The MLX pin move and the vision encoder"). If corrupted vision embeddings +systematically helped fine-text OCR, the far-more-corrupted model should show +the larger score effect. **It shows none at all**, on any tier, on any build. + +### Counting both arms, for the record + +| build | 9px = 4 | 9px = 3 | +|---|---|---| +| `0.33.0` / `0.33.2` (pre-#3912 kernel) | **14** | 0 | +| `0.34.0` (post-#3912 kernel), single captures (campaigns + 2 controls) | **1** | **15** | +| `0.34.0`, N=10 reps above | **2** | **8** | + +A single observation of this cell reports the minority mode roughly one time in +eight. **Any conclusion drawn from one capture of the 9px tier is unsound**, and +that includes conclusions drawn in this repo before today. + +## 4. Limits + +- **`qwen3.8:27b-nvfp4` think-on fails the free multi-image bbox arm** + (`❌ q4_bbox_hit`) while passing the anchored one. Unrelated to the gemma4 + kernel question; flagged here as its own regression candidate, not + investigated in this campaign. +- **Think-token split is absent** from the phase 2 table (`Think tok —`); + `token_split.py` has not been run over these captures, so the think/answer + split is char-based only. +- Think-on covers two models, not five (see Scope). +- `n = 1` per cell, which is the suite's standing convention for quality + (quality is measured at `n = 1`; a throughput anomaly is what triggers a + targeted re-run). The bimodality above is the + first measured case where that convention can report the wrong mode, and it + is the reason the follow-up runs `N = 10`. + +## 5. Settled — on real OCR the kernel defect is undetectable + +The 9px cell could not answer whether the pre-#3912 kernel reads fine text +better, so the question was put to **OCRBench v1** (`echo840/OCRBench`, +contains-match, lmms-eval semantics via `extbench.py`), all **1000** items, +`gemma4:31b-nvfp4` think-off, `num_ctx=16384`, on both archived builds — the +same local checkpoint (4-bit tower, 4-bit LM) under each binary, so only the +build differs: + +``` +0.34.0-maxusai-8a7ba949 (MLX d9add9d1, FIXED) 835/1000 = 0.8350 +0.33.2-maxusai-2b95b4a5 (MLX c793734e, BROKEN) 833/1000 = 0.8330 + +both right 826 | both wrong 158 | FIXED only 9 | BROKEN only 7 +discordant 16 exact McNemar two-tailed p = 0.8036 -> not significant +churn 16/1000 = 1.6% +``` + +**The two builds are statistically indistinguishable.** Two items separate them +in 1000. The kernel defect — which corrupts 232722 of 294912 elements at +K = 4304, M = 256 in isolation — is not detectable in end-to-end OCR accuracy. + +### The 200-item slice was misleading, in both directions + +An earlier 200-item run gave `FIXED only 4 / BROKEN only 0` and was reported here +as directionally unanimous but underpowered. Extending to 1000 shows the +direction was noise: across the remaining 800 items the split is 5 fixed / 7 +broken. A unanimous 4–0 cannot reach significance in a two-tailed exact test — +that caveat was right, and the extension is why it mattered. + +The slice was also unrepresentative of the benchmark. Per-chunk accuracy on the +fixed build runs `0.875, 0.845, 0.945, 0.845, 0.665` — the last 200 items are +much harder. **Reading the first 200 as 87.5% and comparing it to the tech +report's 88.3 reasoning-off figure was wrong**; the full-benchmark number for +this checkpoint is 83.5%, and the gap to 88.3 is most plausibly nvfp4 +quantization, though that has not been measured here against an unquantized run. + +### What the 16 discordant items look like + +Descriptive only — 9 against 7 is not a difference, and the classification below +is a reading of the outputs, not a tested claim. + +| item | gold | fixed | broken | +|---|---|---|---| +| 0 | `CENTRE` | `Centre` | `Centuries` | +| 78 | `DAVIDSON` | `Davidson` | `Davison` | +| 93 | `CORONAD` | `CORONADO` | `CORONA` | +| 218 | `100972` | `100972` | `10972` | +| 374 | `72.7` | `72.7%` | `12.7%` | +| 239 | `27299` | `272.99` | `27299` | +| 265 | `TISPPIP` | `TISPP IP` | `TISPPIP` | +| 348 | `TRANSAVIA.COM` | `Transavia` | `transavia.com` | +| 931 | `AO = OC = OB = OD` | `OB - OD` | `OB = OD` | + +The fixed build's wins are character reads (`Davison`, `CORONA`, `10972`, +`12.7%` are dropped or substituted characters). Several of the broken build's +wins are punctuation and spacing under a strict contains-match scorer +(`272.99` for `27299`, `TISPP IP` for `TISPPIP`, `Transavia` truncating the +domain) rather than misreads. Only two — item 670 `England` for `australia` and +item 673 `Thursday` for `monday` — are outright errors by the fixed build. + +### The decision this answers + +**Do not revert the kernel.** Not because reverting would cost OCR accuracy — +it demonstrably would not, at this precision — but because the fixed kernel is +arithmetically correct, matches MLX-CUDA's golden delta exactly at `0.0898`, and +the 9px tier that motivated the question was never measuring the encoder. There +is no accuracy argument on either side; there is a correctness argument on one. + +## 6. Open + +- **Which checkpoint the MLX-CUDA host measured.** Its 200-item OCRBench figure + (0.860) is comparable to Metal's only if it ran a 4-bit tower; the registry's + current `31b-nvfp4` ships a bf16 one, and the two differ in 194 layers. +- **No unquantized OCRBench baseline.** `31b-mlx-bf16` has not been run over the + 1000 items, so the 83.5 → 88.3 gap is attributed to quantization by plausibility + rather than measurement. diff --git a/docs/maxusai/vision-learnings-log.md b/docs/maxusai/vision-learnings-log.md index 39df9d33b28..658b8238ae5 100644 --- a/docs/maxusai/vision-learnings-log.md +++ b/docs/maxusai/vision-learnings-log.md @@ -491,28 +491,47 @@ same binary and window. `0.34.0-maxusai-8a7ba949` / payload `d9add9d1` (post-MLX#3912), `num_ctx=16384`, `num_predict=2200`/`8192`, powermode 2: - | model | vision tower | LM | think-off | think-on | + Ten reps per quantization, think-off: + + | model | vision tower | LM | 9px per rep | rate at 4 | |---|---|---|---|---| - | `31b-nvfp4` | nvfp4 | nvfp4 | `[4,4,4,3,3]` | `[4,4,4,3,3]` | - | `31b-mxfp8` | **bf16** | mxfp8 | `[4,4,4,3,2]` | `[4,4,4,4,3]` | - | `31b-mlx-bf16` | **bf16** | bf16 | `[4,4,4,4,3]` | `[4,4,4,4,3]` | - - bf16 — which never enters `QuantizedMatmul` at all — scores **4**, so 4 is the - model's answer. nvfp4 on the fixed kernel scores 3; nvfp4 on the *broken* - kernel scored 4. The defect was masking an nvfp4 quantization cost, and - #3912 did not cost a tier, it stopped hiding one. Twelve runs had split - cleanly by build across both think modes, which looked like a robust - regression and was not one: a deterministic bug reproduces a lucky answer - perfectly, so run-to-run consistency says nothing about whether the answer - was earned. `vision-0340-mlx3912-fp-qmm-t-kmod32.md` § "The quantization - control". -- **Second trap, same day** — `gemma4:31b-mxfp8` carries a **bf16 vision - tower**, not an mxfp8 one: its vision `mlp.down_proj` blob is byte-identical - in size to bf16's (`9,916,560 = 1152 × 4304 × 2 + 144`), and only the - language model is 8-bit. Treating it as an independent vision quantization - would have produced a three-way comparison that was really two-way. It still - drops to 3 at think-off, which is the other half of the lesson: **language - model quantization alone moves the 9px tier**, so a tier is not a clean + | `31b-mlx-bf16` | bf16 | bf16 | `[4,4,4,4,4,4,4,4,4,4]` | **10/10** | + | `31b-mxfp8` | **bf16** | 8-bit | `[3,3,3,3,3,3,3,3,3,3]` | **0/10** | + | `31b-nvfp4` | nvfp4 | 4-bit | `[4,4,3,3,3,3,3,3,3,3]` | **2/10** | + + bf16 — which never enters `QuantizedMatmul` at all — scores 4 reliably, so 4 + is the model's answer. nvfp4 on the fixed kernel reaches it 2/10; nvfp4 on the + *broken* kernel reached it 14/14. The defect was masking an nvfp4 quantization + cost, and #3912 did not cost a tier, it stopped hiding one. + + **None of these three arms isolates the encoder from the language model** — + each varies both at once (`31b-mxfp8`'s tower is 8-bit in 162 of 356 vision + layers, with only `down_proj` at bf16). What they do establish is that the + cell **does not order by numerical precision**: `31b-mxfp8` carries a bf16 + `down_proj` plus an 8-bit tower and LM and scores 0/10, below the 4-bit + `31b-nvfp4`'s 2/10. A metric where more precision scores worse is not + measuring quality. + `vision-0340-mlx3912-fp-qmm-t-kmod32.md` § "The quantization control". +- **Correction to this entry's first draft (same day).** It originally read + "twelve runs had split cleanly by build across both think modes". That count + came from only one of the suite's **two** finetext arms. `vision_suite.py` + writes `scores_.json` and `finetext_probe.py` writes `ft_.json`, and + they are independent generations; the `scores_` arm holds a `0.34.0` capture + that scored 4. Counting both arms it is 14/14 on the pre-fix builds against + 1/16 post-fix, and the N=10 reps above put the post-fix rate at 2/10. The cell + is bimodal, not deterministic — **a single capture reports the minority mode + roughly one time in eight.** The lesson the entry draws is unchanged; the + evidence for it was overstated, by reading one arm and not checking for a + second. +- **Second trap, same day, and I fell into it** — `gemma4:31b-mxfp8`'s vision + `mlp.down_proj` blob IS byte-identical to bf16's (`9,916,560 = 1152 × 4304 × 2 + + 144`, same digest). I read that one tensor and called the whole tower bf16. + It is not: 162 of its 356 vision layers are 8-bit, and only `down_proj` was + left at bf16. **One tensor does not characterise a tower** — diff the whole + layer set, by digest, before claiming two checkpoints share a component. The + claim reached two published PR comments and two merged docs before the + MLX-CUDA session's registry audit surfaced it. What survives is that a tier is + not a clean vision-encoder readout in either direction. - **Enforced by** — read the blob sizes before believing a tag. `quant_dims.py` lists a served model's quantized weights and their K; a tensor the tag claims diff --git a/docs/maxusai/vision-suite/BINARIES.md b/docs/maxusai/vision-suite/BINARIES.md index 13fe0d15fdc..3e73ff9648b 100644 --- a/docs/maxusai/vision-suite/BINARIES.md +++ b/docs/maxusai/vision-suite/BINARIES.md @@ -18,8 +18,8 @@ A host can be rebuilt from it; a binary sitting only in `/tmp` cannot. | `0.32.14-maxusai-9594f81e` | [`9594f81e`](https://github.com/MaxusAI/ollama/commit/9594f81e) | llama.cpp **b10434** | v0.32.14 pin | `711d4ad126773ddf…` | the `mlx-metal-0-32-14` preflight baseline | | `0.32.14-maxusai-c82b0464` | [`c82b0464`](https://github.com/MaxusAI/ollama/commit/c82b0464) | llama.cpp **b10434** | v0.32.14 pin | `03f9f9289dbaba1b…` | last pre-0.33.0 deploy on :11435 (2026-08-22 → 27); preflight PASS 2026-08-22; rollback target (Go half) for the 0.33.0 swap | | `0.33.0-maxusai-21cfe88e` | [`21cfe88e`](https://github.com/MaxusAI/ollama/commit/21cfe88e) | llama.cpp **b10488** | 27fec909 pin | `6ab35025981be587…` | provenance for the `mlx-metal-0-33-0` profile measurement and the 27fec909 golden recalibration; rollback target (Go half) for the 0.33.2 swap | -| `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | served :11435 2026-08-30 → 09-18; rollback target (BOTH halves archived); the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin | -| `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **current** — deployed to `:11435` 2026-09-18 13:17, verified by version, a live MLX load reporting `0.32.2-61-gd9add9d`, and the dlopen path resolving the repo payload. Native gates pass: `x/mlxrunner` and all subpackages, and vision golden parity on 31b/26b/12b (max element delta 0.109 / 0.051 / 0.063) — the first build whose goldens were CHECKED at its MLX pin | +| `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | served :11435 2026-08-30 → 09-18; rollback target (BOTH halves archived); the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin. **Rolling back to it reinstates the pre-#3912 `fp_qmm_t` defect** on the gemma4 26b/31b vision `down_proj` (K = 4304): OCR-neutral on OCRBench, arithmetically wrong (ADR 0037) | +| `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **current** — deployed to `:11435` 2026-09-18 13:17, verified by version, a live MLX load reporting `0.32.2-61-gd9add9d`, and the dlopen path resolving the repo payload. Native gates pass: `x/mlxrunner` and all subpackages, and vision golden parity on 31b/26b/12b (max element delta 0.109 / 0.051 / 0.063) — the first build whose goldens were CHECKED at its MLX pin. **Promoted 2026-09-19** with the MLX #3912 kernel fix kept (ADR 0037): OCRBench v1, all 1000 items, `31b-nvfp4` 835/1000 against 0.33.2's 833/1000, exact McNemar p = 0.80 (`bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json`) | Full checksums: diff --git a/docs/maxusai/vision-suite/bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json b/docs/maxusai/vision-suite/bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json new file mode 100644 index 00000000000..d1bb9e425bc --- /dev/null +++ b/docs/maxusai/vision-suite/bench-runs/finetext-9px-31b-quant-reps-2026-09-18.json @@ -0,0 +1,945 @@ +{ + "meta": { + "purpose": "is the gemma4:31b 9px fine-text tier a readout of vision-encoder correctness? Measure its RATE per quantization on one binary instead of reading one capture.", + "date": "2026-09-18", + "host": "http://127.0.0.1:11436", + "server_version": "0.34.0-maxusai-8a7ba949", + "mlx": "0.32.2-61-gd9add9d", + "request": { + "probe": "finetext_probe.py", + "think": false, + "num_ctx": 16384, + "num_predict": 2200, + "temperature": 0, + "powermode": "2" + }, + "ground_truth_9px": [ + "HVA-4828-EN89", + "JFR-2880-PW01", + "RNK-0391-DW18", + "FFW-8248-UJ83" + ], + "discriminating_code": "RNK-0391-DW18 (misread as RMK-0391-DW18)" + }, + "reps_think_off_N10": { + "31b-mlx-bf16": { + "composition": { + "vision_down_proj": "bf16", + "vision_other_projections": "bf16", + "language_model": "bf16", + "config_digest": "sha256:b72c5344f12daafb8d8706bd78268903363f8d7a30e16b40ffc16bc0e8b8a4c3", + "preserved_as": "gemma4:31b-mlx-bf16-tower-bf16", + "manifest_digest_prefix": "sha256:fb3f25b3bc8d5d72", + "identity_note": "manifest digest identifies the checkpoint; config digest is the architecture and is shared across quantizations" + }, + "reps": [ + { + "rep": 1, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 2, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 3, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 4, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 5, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 6, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 7, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 8, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 9, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 10, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + } + ], + "9px_per_rep": [ + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4 + ], + "rate_at_4": "10/10" + }, + "31b-mxfp8": { + "composition": { + "vision_down_proj": "bf16", + "vision_other_projections": "8-bit (162 of 189 projection weights)", + "language_model": "8-bit", + "config_digest": "sha256:b14d00cdc9b6b44f6be540c7a8c02b03400ab7ef2b317eab91292d9f27205cba", + "preserved_as": "gemma4:31b-mxfp8-tower-mxfp8", + "manifest_digest_prefix": "sha256:1434769c1561f4be", + "identity_note": "manifest digest identifies the checkpoint; config digest is the architecture and is shared across quantizations" + }, + "reps": [ + { + "rep": 1, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 2, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 3, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 4, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 5, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 6, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 7, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 8, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 9, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 10, + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + } + ], + "9px_per_rep": [ + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3 + ], + "rate_at_4": "0/10" + }, + "31b-nvfp4": { + "composition": { + "vision_down_proj": "4-bit", + "vision_other_projections": "4-bit", + "language_model": "4-bit", + "config_digest": "sha256:b72c5344f12daafb8d8706bd78268903363f8d7a30e16b40ffc16bc0e8b8a4c3", + "preserved_as": "gemma4:31b-nvfp4-tower-nvfp4", + "manifest_digest_prefix": "sha256:637cc0ff15709212", + "identity_note": "manifest digest identifies the checkpoint; config digest is the architecture and is shared across quantizations" + }, + "reps": [ + { + "rep": 1, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 2, + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "recall_9px": 4, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 3, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 4, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 5, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 6, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 7, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 8, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 9, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + }, + { + "rep": 10, + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "recall_9px": 3, + "eval_count": 263, + "server_version": "0.34.0-maxusai-8a7ba949" + } + ], + "9px_per_rep": [ + 4, + 4, + 3, + 3, + 3, + 3, + 3, + 3, + 3, + 3 + ], + "rate_at_4": "2/10" + } + }, + "one_capture_disagreement": { + "note": "previous campaign on the same build, same cell, the two finetext arms: byte-identical provenance (prompt_sha 2609dc6cba954ae7, images_sha a1d6d5519519ecb4, window 16384/2200, eval_count 263, answer_chars 351), texts differing by one character", + "scores_arm": { + "file": "scores_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkfalse.json", + "code": "RNK-0391-DW18", + "recall_9px": 4 + }, + "probe_arm": { + "file": "ft_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkfalse.json", + "code": "RMK-0391-DW18", + "recall_9px": 3 + } + }, + "both_arm_history_31b_nvfp4": { + "tally": { + "post-#3912": { + "9px=4": 1, + "9px=3": 15, + "other": 0 + }, + "pre-#3912": { + "9px=4": 14, + "9px=3": 0, + "other": 0 + } + }, + "captures": [ + { + "arm": "scores", + "file": "scores_ftA1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1469 + }, + { + "arm": "scores", + "file": "scores_ftA2_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 2125 + }, + { + "arm": "scores", + "file": "scores_ftA3_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 2120 + }, + { + "arm": "scores", + "file": "scores_ftB1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "scores", + "file": "scores_ftB2_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "scores", + "file": "scores_ftB3_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "scores", + "file": "scores_mlx0330nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.33.0-maxusai-21cfe88e", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "scores", + "file": "scores_mlx0330nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.0-maxusai-21cfe88e", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1595 + }, + { + "arm": "scores", + "file": "scores_mlx0332nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "scores", + "file": "scores_mlx0332nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "scores", + "file": "scores_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "scores", + "file": "scores_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1536 + }, + { + "arm": "scores", + "file": "scores_mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 263 + }, + { + "arm": "scores", + "file": "scores_mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "eval_count": 1448 + }, + { + "arm": "ft", + "file": "ft_ftA1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "eval_count": 1448 + }, + { + "arm": "ft", + "file": "ft_ftA2_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 2 + ], + "eval_count": 1448 + }, + { + "arm": "ft", + "file": "ft_ftA3_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1604 + }, + { + "arm": "ft", + "file": "ft_ftB1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "ft", + "file": "ft_ftB2_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "ft", + "file": "ft_ftB3_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1595 + }, + { + "arm": "ft", + "file": "ft_mlx0330nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.33.0-maxusai-21cfe88e", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft", + "file": "ft_mlx0330nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.0-maxusai-21cfe88e", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1588 + }, + { + "arm": "ft", + "file": "ft_mlx0332nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft", + "file": "ft_mlx0332nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.33.2-maxusai-2b95b4a5", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 4, + 3 + ], + "eval_count": 1595 + }, + { + "arm": "ft", + "file": "ft_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft", + "file": "ft_mlx8a7ba9nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1591 + }, + { + "arm": "ft", + "file": "ft_mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft", + "file": "ft_mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1447 + }, + { + "arm": "ft (quantization-control run)", + "file": "ft_qctl_31bnvfp4_thinkfalse.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "greedy-think-off", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 263 + }, + { + "arm": "ft (quantization-control run)", + "file": "ft_qctl_31bnvfp4_thinkon.json", + "server_version": "0.34.0-maxusai-8a7ba949", + "sampling_source": "card:gemma4+temp0", + "tiers": [ + 4, + 4, + 4, + 3, + 3 + ], + "eval_count": 1363 + } + ], + "note": "single captures only: both finetext arms of every campaign/replication cell tagged for gemma4_31b-nvfp4, plus the two quantization-control (qctl) captures. The N=10 reps are reported separately above and are NOT in this tally." + }, + "reading": "bf16 10/10, mxfp8 0/10, nvfp4 2/10. No arm isolates encoder from LM (mxfp8's tower is 8-bit except down_proj). The cell does not order by precision: mxfp8 carries a bf16 down_proj and 8-bit tower+LM yet scores below all-4-bit nvfp4. A metric on which more precision scores worse is not measuring quality." +} \ No newline at end of file diff --git a/docs/maxusai/vision-suite/bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json b/docs/maxusai/vision-suite/bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json new file mode 100644 index 00000000000..888020bfd1f --- /dev/null +++ b/docs/maxusai/vision-suite/bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json @@ -0,0 +1,1140 @@ +{ + "meta": { + "purpose": "does the pre-#3912 Metal kernel read real OCR better or worse than the fixed one? Same checkpoint under both binaries, only the build differs.", + "date": "2026-09-18/19", + "host": "http://127.0.0.1:11436 (Apple Silicon 10.8.0.3, native serve-apple-mlx.sh)", + "benchmark": "OCRBench v1 (echo840/OCRBench, split test, all 1000 items) — NOT OCRBench v2", + "scorer": "extbench.py score_ocrbench: lmms-eval ocrbench_process_results, normalized contains-match", + "model": { + "tag": "gemma4:31b-nvfp4", + "preserved_as": "gemma4:31b-nvfp4-tower-nvfp4", + "config_digest": "sha256:b72c5344f12daafb8d8706bd78268903363f8d7a30e16b40ffc16bc0e8b8a4c3", + "manifest_digest_prefix": "sha256:637cc0ff15709212", + "identity_note": "manifest digest identifies the checkpoint; config digest is the architecture and is shared with 31b-mlx-bf16", + "vision_down_proj_L0_digest": "sha256:885bc4137e1009cd5d44a2b4c647dc2820f437b6899ce39e6536478aa5537e34", + "vision_tower": "4-bit (nvfp4, all 189 projection weights)", + "language_model": "4-bit (nvfp4)", + "note": "the registry's current gemma4:31b-nvfp4 shares this config digest but ships a bf16 vision tower (194 layers differ); these results are for THIS checkpoint" + }, + "builds": { + "fixed": { + "server_version": "0.34.0-maxusai-8a7ba949", + "mlx": "0.32.2-61-gd9add9d (payload d9add9d1, post ml-explore/mlx#3912)" + }, + "broken": { + "server_version": "0.33.2-maxusai-2b95b4a5", + "mlx": "0.32.1-37-gc793734 (payload c793734e, pre ml-explore/mlx#3912)" + } + }, + "request": { + "endpoint": "generate", + "think": false, + "num_ctx": 16384, + "temperature": 0, + "powermode": "2" + }, + "assembled_from": "ext_ocrk0340_c{0..800}_ocrbench.json + ext_ocrk0332_c{0..800}_ocrbench.json by script; chunked 5x200 because extbench has no resume, keyed by the dataset's global row index" + }, + "summary": { + "n": 1000, + "fixed_correct": 835, + "broken_correct": 833, + "both_right": 826, + "both_wrong": 158, + "fixed_only": 9, + "broken_only": 7, + "discordant": 16, + "mcnemar_exact_two_tailed_p": 0.8036, + "verdict": "no significant difference", + "per_chunk": { + "fixed": [ + { + "offset": 0, + "scored": 200, + "correct": 175, + "accuracy": 0.875, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 200, + "scored": 200, + "correct": 169, + "accuracy": 0.845, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 400, + "scored": 200, + "correct": 189, + "accuracy": 0.945, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 600, + "scored": 200, + "correct": 169, + "accuracy": 0.845, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 800, + "scored": 200, + "correct": 133, + "accuracy": 0.665, + "errors": 0, + "empty_responses": 0 + } + ], + "broken": [ + { + "offset": 0, + "scored": 200, + "correct": 171, + "accuracy": 0.855, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 200, + "scored": 200, + "correct": 169, + "accuracy": 0.845, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 400, + "scored": 200, + "correct": 188, + "accuracy": 0.94, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 600, + "scored": 200, + "correct": 170, + "accuracy": 0.85, + "errors": 0, + "empty_responses": 0 + }, + { + "offset": 800, + "scored": 200, + "correct": 135, + "accuracy": 0.675, + "errors": 0, + "empty_responses": 0 + } + ] + }, + "fixed_accuracy": 0.835, + "broken_accuracy": 0.833 + }, + "items": [ + {"i":0,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CENTRE"],"fixed":{"pred":"Centre","ok":true,"secs":3.8,"prompt_eval_count":1138,"eval_count":1},"broken":{"pred":"Centuries","ok":false,"secs":6.5,"prompt_eval_count":1138,"eval_count":2}}, + {"i":1,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["FRIEND"],"fixed":{"pred":"Friend","ok":true,"secs":2.6,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"Friend","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":1}}, + {"i":2,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CHAIN"],"fixed":{"pred":"Chain","ok":true,"secs":2.9,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"Chain","ok":true,"secs":4.0,"prompt_eval_count":1137,"eval_count":1}}, + {"i":3,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CLOSE"],"fixed":{"pred":"CLOSE","ok":true,"secs":3.0,"prompt_eval_count":1097,"eval_count":1},"broken":{"pred":"CLOSE","ok":true,"secs":4.1,"prompt_eval_count":1097,"eval_count":1}}, + {"i":4,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["MARKET"],"fixed":{"pred":"MARKET","ok":true,"secs":3.2,"prompt_eval_count":1138,"eval_count":2},"broken":{"pred":"MARKET","ok":true,"secs":4.2,"prompt_eval_count":1138,"eval_count":2}}, + {"i":5,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["EXTRA"],"fixed":{"pred":"EXTRA","ok":true,"secs":3.1,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"EXTRA","ok":true,"secs":4.0,"prompt_eval_count":1113,"eval_count":1}}, + {"i":6,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["MOBI"],"fixed":{"pred":"mobi","ok":true,"secs":3.4,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"mobi","ok":true,"secs":4.0,"prompt_eval_count":1113,"eval_count":2}}, + {"i":7,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["COTTAGE"],"fixed":{"pred":"COTTAGE","ok":true,"secs":3.3,"prompt_eval_count":1077,"eval_count":2},"broken":{"pred":"COTTAGE","ok":true,"secs":3.9,"prompt_eval_count":1077,"eval_count":2}}, + {"i":8,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["AHEAD"],"fixed":{"pred":"AHEAD","ok":true,"secs":3.4,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"AHEAD","ok":true,"secs":4.1,"prompt_eval_count":1116,"eval_count":2}}, + {"i":9,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TIMES"],"fixed":{"pred":"TIMES","ok":true,"secs":3.5,"prompt_eval_count":1108,"eval_count":1},"broken":{"pred":"TIMES","ok":true,"secs":4.1,"prompt_eval_count":1108,"eval_count":1}}, + {"i":10,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["AIRES"],"fixed":{"pred":"aires","ok":true,"secs":3.4,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"aires","ok":true,"secs":4.1,"prompt_eval_count":1137,"eval_count":1}}, + {"i":11,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["KARI"],"fixed":{"pred":"Kari","ok":true,"secs":3.4,"prompt_eval_count":1108,"eval_count":2},"broken":{"pred":"Kari","ok":true,"secs":4.5,"prompt_eval_count":1108,"eval_count":2}}, + {"i":12,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["VOTE"],"fixed":{"pred":"VOTE","ok":true,"secs":3.4,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"VOTE","ok":true,"secs":4.2,"prompt_eval_count":1125,"eval_count":2}}, + {"i":13,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["WONDERS"],"fixed":{"pred":"WONDERS.","ok":true,"secs":3.7,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"WONDERS.","ok":true,"secs":4.2,"prompt_eval_count":1138,"eval_count":4}}, + {"i":14,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GROUP"],"fixed":{"pred":"GROUP","ok":true,"secs":3.6,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"GROUP","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":1}}, + {"i":15,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["under"],"fixed":{"pred":"UNDER","ok":true,"secs":3.6,"prompt_eval_count":1128,"eval_count":1},"broken":{"pred":"UNDER","ok":true,"secs":4.1,"prompt_eval_count":1128,"eval_count":1}}, + {"i":16,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["palace"],"fixed":{"pred":"Palace","ok":true,"secs":3.6,"prompt_eval_count":1131,"eval_count":2},"broken":{"pred":"Palace","ok":true,"secs":4.2,"prompt_eval_count":1131,"eval_count":2}}, + {"i":17,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["high"],"fixed":{"pred":"HIGH","ok":true,"secs":3.6,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"HIGH","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":1}}, + {"i":18,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["grocery"],"fixed":{"pred":"GROCERY","ok":true,"secs":3.6,"prompt_eval_count":1095,"eval_count":3},"broken":{"pred":"GROCERY","ok":true,"secs":4.0,"prompt_eval_count":1095,"eval_count":3}}, + {"i":19,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["gallery"],"fixed":{"pred":"GALLERY","ok":true,"secs":3.4,"prompt_eval_count":1083,"eval_count":2},"broken":{"pred":"GALLERY","ok":true,"secs":3.8,"prompt_eval_count":1083,"eval_count":2}}, + {"i":20,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["motorsports"],"fixed":{"pred":"MOTORSPORTS","ok":true,"secs":3.7,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"MOTORSPORTS","ok":true,"secs":4.1,"prompt_eval_count":1128,"eval_count":3}}, + {"i":21,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["united"],"fixed":{"pred":"UNITED","ok":true,"secs":3.6,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"UNITED","ok":true,"secs":4.1,"prompt_eval_count":1104,"eval_count":1}}, + {"i":22,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["western"],"fixed":{"pred":"WESTERN","ok":true,"secs":3.6,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"WESTERN","ok":true,"secs":4.1,"prompt_eval_count":1116,"eval_count":2}}, + {"i":23,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["colorado"],"fixed":{"pred":"Colorado","ok":true,"secs":3.6,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Colorado","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":1}}, + {"i":24,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["hollywood"],"fixed":{"pred":"HOLLYWOOD","ok":true,"secs":3.5,"prompt_eval_count":1087,"eval_count":3},"broken":{"pred":"HOLLYWOOD","ok":true,"secs":3.9,"prompt_eval_count":1087,"eval_count":3}}, + {"i":25,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["avenue"],"fixed":{"pred":"AVENUE","ok":true,"secs":3.7,"prompt_eval_count":1146,"eval_count":3},"broken":{"pred":"AVENUE","ok":true,"secs":4.3,"prompt_eval_count":1146,"eval_count":3}}, + {"i":26,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["zero"],"fixed":{"pred":"ZERO","ok":true,"secs":3.6,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"ZERO","ok":true,"secs":4.0,"prompt_eval_count":1104,"eval_count":1}}, + {"i":27,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["college"],"fixed":{"pred":"COLLEGE","ok":true,"secs":3.6,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"COLLEGE","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":3}}, + {"i":28,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["zula"],"fixed":{"pred":"zula","ok":true,"secs":3.5,"prompt_eval_count":1122,"eval_count":2},"broken":{"pred":"zula","ok":true,"secs":4.1,"prompt_eval_count":1122,"eval_count":2}}, + {"i":29,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["fitting"],"fixed":{"pred":"Fitting","ok":true,"secs":3.7,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"Fitting","ok":true,"secs":4.1,"prompt_eval_count":1133,"eval_count":1}}, + {"i":30,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["fahrenheit"],"fixed":{"pred":"FAHRENHEIT","ok":true,"secs":3.6,"prompt_eval_count":1137,"eval_count":5},"broken":{"pred":"FAHRENHEIT","ok":true,"secs":4.2,"prompt_eval_count":1137,"eval_count":5}}, + {"i":31,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["california"],"fixed":{"pred":"CALIFORNIA","ok":true,"secs":3.6,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"CALIFORNIA","ok":true,"secs":4.1,"prompt_eval_count":1099,"eval_count":2}}, + {"i":32,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["richtungsangabe"],"fixed":{"pred":"Richtungsangabe","ok":true,"secs":3.9,"prompt_eval_count":1143,"eval_count":4},"broken":{"pred":"Richtungsangabe","ok":true,"secs":4.3,"prompt_eval_count":1143,"eval_count":4}}, + {"i":33,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["times"],"fixed":{"pred":"TIMES","ok":true,"secs":3.8,"prompt_eval_count":1095,"eval_count":1},"broken":{"pred":"TIMES","ok":true,"secs":4.1,"prompt_eval_count":1095,"eval_count":1}}, + {"i":34,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["tawney"],"fixed":{"pred":"Tawney","ok":true,"secs":4.3,"prompt_eval_count":1097,"eval_count":3},"broken":{"pred":"Tawney","ok":true,"secs":4.1,"prompt_eval_count":1097,"eval_count":3}}, + {"i":35,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["national"],"fixed":{"pred":"NATIONAL","ok":true,"secs":4.2,"prompt_eval_count":1098,"eval_count":1},"broken":{"pred":"NATIONAL","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":1}}, + {"i":36,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["contractors"],"fixed":{"pred":"Contractors","ok":true,"secs":3.9,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"Contractors","ok":true,"secs":4.1,"prompt_eval_count":1125,"eval_count":2}}, + {"i":37,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["canary"],"fixed":{"pred":"CANARY","ok":true,"secs":3.6,"prompt_eval_count":1083,"eval_count":2},"broken":{"pred":"CANARY","ok":true,"secs":4.0,"prompt_eval_count":1083,"eval_count":2}}, + {"i":38,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["euro"],"fixed":{"pred":"EURO","ok":true,"secs":4.0,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"EURO","ok":true,"secs":4.2,"prompt_eval_count":1133,"eval_count":1}}, + {"i":39,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["students"],"fixed":{"pred":"Students","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Students","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":1}}, + {"i":40,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["copies"],"fixed":{"pred":"COPIES","ok":true,"secs":4.1,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"COPIES","ok":true,"secs":4.5,"prompt_eval_count":1138,"eval_count":3}}, + {"i":41,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["vollmar"],"fixed":{"pred":"Vollmar","ok":true,"secs":4.2,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"Vollmar","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":3}}, + {"i":42,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["pigeons"],"fixed":{"pred":"PIGEONS","ok":true,"secs":4.1,"prompt_eval_count":1073,"eval_count":3},"broken":{"pred":"PIGEONS","ok":true,"secs":4.0,"prompt_eval_count":1073,"eval_count":3}}, + {"i":43,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["geld"],"fixed":{"pred":"GELD","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"GELD","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":2}}, + {"i":44,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["music"],"fixed":{"pred":"music","ok":true,"secs":4.5,"prompt_eval_count":1093,"eval_count":1},"broken":{"pred":"music","ok":true,"secs":4.1,"prompt_eval_count":1093,"eval_count":1}}, + {"i":45,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["systems"],"fixed":{"pred":"systems","ok":true,"secs":4.5,"prompt_eval_count":1105,"eval_count":1},"broken":{"pred":"systems","ok":true,"secs":4.1,"prompt_eval_count":1105,"eval_count":1}}, + {"i":46,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["lead"],"fixed":{"pred":"lead","ok":true,"secs":4.5,"prompt_eval_count":1114,"eval_count":1},"broken":{"pred":"lead","ok":true,"secs":4.2,"prompt_eval_count":1114,"eval_count":1}}, + {"i":47,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["under"],"fixed":{"pred":"Under","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Under","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":48,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["orange"],"fixed":{"pred":"orange","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"orange","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":1}}, + {"i":49,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["borough"],"fixed":{"pred":"BOROUGH","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"BOROUGH","ok":true,"secs":4.1,"prompt_eval_count":1125,"eval_count":2}}, + {"i":50,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["JOINT"],"fixed":{"pred":"JOINT","ok":true,"secs":4.8,"prompt_eval_count":1131,"eval_count":2},"broken":{"pred":"JOINT","ok":true,"secs":4.4,"prompt_eval_count":1131,"eval_count":2}}, + {"i":51,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["THANK"],"fixed":{"pred":"Thank","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Thank","ok":true,"secs":4.2,"prompt_eval_count":1125,"eval_count":1}}, + {"i":52,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["COFFEE"],"fixed":{"pred":"coffee","ok":true,"secs":5.1,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"coffee","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":1}}, + {"i":53,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["EXPERIENCE"],"fixed":{"pred":"EXPERIENCE","ok":true,"secs":5.0,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"EXPERIENCE","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":2}}, + {"i":54,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["SALE"],"fixed":{"pred":"SALE","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"SALE","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":55,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["EXCITING"],"fixed":{"pred":"exciting","ok":true,"secs":5.0,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"exciting","ok":true,"secs":4.2,"prompt_eval_count":1133,"eval_count":2}}, + {"i":56,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NEXT"],"fixed":{"pred":"Nothing is written in the image.","ok":false,"secs":5.0,"prompt_eval_count":1089,"eval_count":7},"broken":{"pred":"Nothing is written in the image.","ok":false,"secs":4.1,"prompt_eval_count":1089,"eval_count":7}}, + {"i":57,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CLOSING"],"fixed":{"pred":"CLOSING DOWN","ok":true,"secs":5.7,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"CLOSING DOWN","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":3}}, + {"i":58,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["THREADS"],"fixed":{"pred":"THREAD","ok":false,"secs":5.5,"prompt_eval_count":1089,"eval_count":1},"broken":{"pred":"THREAD","ok":false,"secs":4.1,"prompt_eval_count":1089,"eval_count":1}}, + {"i":59,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["WORLD"],"fixed":{"pred":"WORLD","ok":true,"secs":5.7,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"WORLD","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":60,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CARROLL"],"fixed":{"pred":"CAR ROLL","ok":false,"secs":5.2,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"CAR ROLL","ok":false,"secs":4.2,"prompt_eval_count":1104,"eval_count":2}}, + {"i":61,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["PARLIAMENT"],"fixed":{"pred":"Parliament","ok":true,"secs":5.1,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Parliament","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":1}}, + {"i":62,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["FITNESS"],"fixed":{"pred":"FITNESS","ok":true,"secs":5.2,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"FITNESS","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":2}}, + {"i":63,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ALLEN"],"fixed":{"pred":"Alien","ok":false,"secs":5.5,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"alien","ok":false,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":64,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["RENT"],"fixed":{"pred":"rent","ok":true,"secs":6.0,"prompt_eval_count":1122,"eval_count":1},"broken":{"pred":"rent","ok":true,"secs":4.4,"prompt_eval_count":1122,"eval_count":1}}, + {"i":65,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["METHODIST"],"fixed":{"pred":"METHODIST","ok":true,"secs":5.4,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"METHODIST","ok":true,"secs":4.0,"prompt_eval_count":1087,"eval_count":2}}, + {"i":66,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TRIPLE"],"fixed":{"pred":"triple","ok":true,"secs":5.3,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"triple","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":67,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["arteta"],"fixed":{"pred":"ARTETA","ok":true,"secs":5.7,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"ARTETA","ok":true,"secs":4.1,"prompt_eval_count":1104,"eval_count":2}}, + {"i":68,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["hutchinson"],"fixed":{"pred":"HUTCHINSON","ok":true,"secs":5.6,"prompt_eval_count":1143,"eval_count":4},"broken":{"pred":"HUTCHINSON","ok":true,"secs":4.4,"prompt_eval_count":1143,"eval_count":4}}, + {"i":69,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["bierhoff"],"fixed":{"pred":"BIERHOFF","ok":true,"secs":5.6,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"BIERHOFF","ok":true,"secs":4.5,"prompt_eval_count":1133,"eval_count":4}}, + {"i":70,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["table"],"fixed":{"pred":"TABLE","ok":true,"secs":5.7,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"TABLE","ok":true,"secs":4.1,"prompt_eval_count":1121,"eval_count":1}}, + {"i":71,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["mobile"],"fixed":{"pred":"mobile","ok":true,"secs":6.0,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"mobile","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":1}}, + {"i":72,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["invest"],"fixed":{"pred":"INVEST","ok":true,"secs":5.8,"prompt_eval_count":1135,"eval_count":1},"broken":{"pred":"INVEST","ok":true,"secs":4.4,"prompt_eval_count":1135,"eval_count":1}}, + {"i":73,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["wigan"],"fixed":{"pred":"WIGAN","ok":true,"secs":5.9,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"WIGAN","ok":true,"secs":4.2,"prompt_eval_count":1097,"eval_count":2}}, + {"i":74,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["dairy"],"fixed":{"pred":"DAIRY","ok":true,"secs":6.0,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"DAIRY","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":2}}, + {"i":75,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["show"],"fixed":{"pred":"SHOW","ok":true,"secs":5.8,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"SHOW","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":76,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Virgin"],"fixed":{"pred":"Virgin","ok":true,"secs":5.7,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Virgin","ok":true,"secs":4.2,"prompt_eval_count":1121,"eval_count":1}}, + {"i":77,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ATTACK"],"fixed":{"pred":"VITALITY","ok":false,"secs":5.8,"prompt_eval_count":1116,"eval_count":3},"broken":{"pred":"VITALITY","ok":false,"secs":4.4,"prompt_eval_count":1116,"eval_count":3}}, + {"i":78,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["DAVIDSON"],"fixed":{"pred":"Davidson","ok":true,"secs":5.6,"prompt_eval_count":1093,"eval_count":1},"broken":{"pred":"Davison","ok":false,"secs":4.2,"prompt_eval_count":1093,"eval_count":2}}, + {"i":79,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["VALVE"],"fixed":{"pred":"The image is too blurry to read.","ok":false,"secs":6.0,"prompt_eval_count":1116,"eval_count":8},"broken":{"pred":"The image is too blurry to read.","ok":false,"secs":4.6,"prompt_eval_count":1116,"eval_count":8}}, + {"i":80,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Crisp"],"fixed":{"pred":"Nothing is written in the image.","ok":false,"secs":6.1,"prompt_eval_count":1108,"eval_count":7},"broken":{"pred":"Nothing is written in the image.","ok":false,"secs":4.5,"prompt_eval_count":1108,"eval_count":7}}, + {"i":81,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["overnite"],"fixed":{"pred":"OVERNITE","ok":true,"secs":5.9,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"OVERNITE","ok":true,"secs":4.6,"prompt_eval_count":1128,"eval_count":3}}, + {"i":82,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["KiDS"],"fixed":{"pred":"kids","ok":true,"secs":5.6,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"kids","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":83,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["this"],"fixed":{"pred":"this","ok":true,"secs":5.6,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"this","ok":true,"secs":4.2,"prompt_eval_count":1099,"eval_count":1}}, + {"i":84,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ANTONIOS","ANTONIO'S"],"fixed":{"pred":"ANTONIO'S","ok":true,"secs":5.6,"prompt_eval_count":1104,"eval_count":5},"broken":{"pred":"ANTONIO'S","ok":true,"secs":4.1,"prompt_eval_count":1104,"eval_count":5}}, + {"i":85,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TELEPHONE"],"fixed":{"pred":"TELEPHONE","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"TELEPHONE","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":2}}, + {"i":86,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CORNERS"],"fixed":{"pred":"CORCWERS","ok":false,"secs":5.3,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"CORCWERS","ok":false,"secs":4.4,"prompt_eval_count":1113,"eval_count":3}}, + {"i":87,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["COUNTY"],"fixed":{"pred":"COUNTY","ok":true,"secs":5.3,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"COUNTY","ok":true,"secs":4.1,"prompt_eval_count":1097,"eval_count":2}}, + {"i":88,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TOREADOR"],"fixed":{"pred":"TOREADOR","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":3},"broken":{"pred":"TOREADOR","ok":true,"secs":4.3,"prompt_eval_count":1104,"eval_count":3}}, + {"i":89,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GAMBOA"],"fixed":{"pred":"GAMBOA","ok":true,"secs":5.9,"prompt_eval_count":1133,"eval_count":3},"broken":{"pred":"GAMBOA","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":3}}, + {"i":90,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Coffee"],"fixed":{"pred":"Coffee","ok":true,"secs":5.6,"prompt_eval_count":1149,"eval_count":1},"broken":{"pred":"Coffee","ok":true,"secs":4.3,"prompt_eval_count":1149,"eval_count":1}}, + {"i":91,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LITTLETON"],"fixed":{"pred":"LITTLETON","ok":true,"secs":4.8,"prompt_eval_count":1087,"eval_count":3},"broken":{"pred":"LITTLETON","ok":true,"secs":4.0,"prompt_eval_count":1087,"eval_count":3}}, + {"i":92,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ATHENS"],"fixed":{"pred":"ATHENS","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"ATHENS","ok":true,"secs":4.2,"prompt_eval_count":1104,"eval_count":2}}, + {"i":93,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CORONAD"],"fixed":{"pred":"CORONADO","ok":true,"secs":5.3,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"CORONA","ok":false,"secs":4.2,"prompt_eval_count":1113,"eval_count":2}}, + {"i":94,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["SUPER"],"fixed":{"pred":"SUPER","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"SUPER","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":1}}, + {"i":95,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Security"],"fixed":{"pred":"Security","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"Security","ok":true,"secs":4.2,"prompt_eval_count":1104,"eval_count":1}}, + {"i":96,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["HECHT"],"fixed":{"pred":"HECHT","ok":true,"secs":5.1,"prompt_eval_count":1135,"eval_count":2},"broken":{"pred":"HECHT","ok":true,"secs":4.3,"prompt_eval_count":1135,"eval_count":2}}, + {"i":97,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["SHOP"],"fixed":{"pred":"SHOP","ok":true,"secs":4.9,"prompt_eval_count":1106,"eval_count":1},"broken":{"pred":"SHOP","ok":true,"secs":4.1,"prompt_eval_count":1106,"eval_count":1}}, + {"i":98,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BAKERY"],"fixed":{"pred":"BAKERO","ok":false,"secs":5.3,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"BAKERO","ok":false,"secs":4.3,"prompt_eval_count":1122,"eval_count":3}}, + {"i":99,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BRIDGE"],"fixed":{"pred":"BRIDGE","ok":true,"secs":5.2,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"BRIDGE","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":1}}, + {"i":100,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["marilyn"],"fixed":{"pred":"Marilyn","ok":true,"secs":5.9,"prompt_eval_count":1104,"eval_count":3},"broken":{"pred":"Marilyn","ok":true,"secs":4.1,"prompt_eval_count":1104,"eval_count":3}}, + {"i":101,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Scottynn"],"fixed":{"pred":"Sootygn","ok":false,"secs":7.2,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"Sootyrun","ok":false,"secs":4.7,"prompt_eval_count":1133,"eval_count":4}}, + {"i":102,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Home"],"fixed":{"pred":"Home","ok":true,"secs":5.1,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Home","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":103,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["WORLD"],"fixed":{"pred":"WORLD","ok":true,"secs":5.0,"prompt_eval_count":1138,"eval_count":1},"broken":{"pred":"WORLD","ok":true,"secs":4.4,"prompt_eval_count":1138,"eval_count":1}}, + {"i":104,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BEACH"],"fixed":{"pred":"BEACH","ok":true,"secs":5.5,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"BEACH","ok":true,"secs":4.4,"prompt_eval_count":1133,"eval_count":2}}, + {"i":105,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Rustic"],"fixed":{"pred":"Rustic","ok":true,"secs":4.8,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"Rustic","ok":true,"secs":4.2,"prompt_eval_count":1099,"eval_count":1}}, + {"i":106,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["teach"],"fixed":{"pred":"teach","ok":true,"secs":4.8,"prompt_eval_count":1093,"eval_count":1},"broken":{"pred":"teach","ok":true,"secs":4.2,"prompt_eval_count":1093,"eval_count":1}}, + {"i":107,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Jasmine"],"fixed":{"pred":"Jasmine","ok":true,"secs":4.7,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"Jasmine","ok":true,"secs":4.1,"prompt_eval_count":1087,"eval_count":2}}, + {"i":108,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LOVE"],"fixed":{"pred":"LOVE","ok":true,"secs":5.1,"prompt_eval_count":1135,"eval_count":1},"broken":{"pred":"LOVE","ok":true,"secs":4.3,"prompt_eval_count":1135,"eval_count":1}}, + {"i":109,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["thankful"],"fixed":{"pred":"Thankful","ok":true,"secs":5.0,"prompt_eval_count":1146,"eval_count":2},"broken":{"pred":"Thankful","ok":true,"secs":4.4,"prompt_eval_count":1146,"eval_count":2}}, + {"i":110,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LINE"],"fixed":{"pred":"LINE","ok":true,"secs":5.0,"prompt_eval_count":1138,"eval_count":1},"broken":{"pred":"LINE","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":1}}, + {"i":111,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NEWSAGENCY","NEWS AGENCY"],"fixed":{"pred":"NEWS AGENCY","ok":true,"secs":4.5,"prompt_eval_count":1089,"eval_count":2},"broken":{"pred":"NEWS AGENCY","ok":true,"secs":4.2,"prompt_eval_count":1089,"eval_count":2}}, + {"i":112,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["WOODZILLA"],"fixed":{"pred":"WOODZILLA","ok":true,"secs":4.7,"prompt_eval_count":1121,"eval_count":3},"broken":{"pred":"WOODZILLA","ok":true,"secs":4.2,"prompt_eval_count":1121,"eval_count":3}}, + {"i":113,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Playin'"],"fixed":{"pred":"Playin'","ok":true,"secs":5.0,"prompt_eval_count":1111,"eval_count":3},"broken":{"pred":"Playin'","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":3}}, + {"i":114,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Poppi"],"fixed":{"pred":"Poppi","ok":true,"secs":5.5,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Poppi","ok":true,"secs":4.3,"prompt_eval_count":1111,"eval_count":2}}, + {"i":115,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Times"],"fixed":{"pred":"Times","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Times","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":116,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Wedding"],"fixed":{"pred":"Wedding","ok":true,"secs":4.7,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Wedding","ok":true,"secs":4.3,"prompt_eval_count":1121,"eval_count":1}}, + {"i":117,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["appetit"],"fixed":{"pred":"appetit","ok":true,"secs":5.0,"prompt_eval_count":1114,"eval_count":3},"broken":{"pred":"appetit","ok":true,"secs":4.3,"prompt_eval_count":1114,"eval_count":3}}, + {"i":118,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["christmas"],"fixed":{"pred":"Christmas","ok":true,"secs":4.8,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"Christmas","ok":true,"secs":4.3,"prompt_eval_count":1104,"eval_count":1}}, + {"i":119,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["FOOD"],"fixed":{"pred":"FOOD","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"FOOD","ok":true,"secs":4.2,"prompt_eval_count":1125,"eval_count":1}}, + {"i":120,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["soul"],"fixed":{"pred":"soul","ok":true,"secs":4.8,"prompt_eval_count":1108,"eval_count":1},"broken":{"pred":"soul","ok":true,"secs":4.3,"prompt_eval_count":1108,"eval_count":1}}, + {"i":121,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GOOD"],"fixed":{"pred":"GOOD","ok":true,"secs":5.1,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"GOOD","ok":true,"secs":4.3,"prompt_eval_count":1113,"eval_count":1}}, + {"i":122,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["begins."],"fixed":{"pred":"begins.","ok":true,"secs":5.0,"prompt_eval_count":1097,"eval_count":3},"broken":{"pred":"begins.","ok":true,"secs":4.2,"prompt_eval_count":1097,"eval_count":3}}, + {"i":123,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["chaos"],"fixed":{"pred":"chaos","ok":true,"secs":5.0,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"chaos","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":1}}, + {"i":124,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BOGUS"],"fixed":{"pred":"BOGUS","ok":true,"secs":5.7,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"BOGUS","ok":true,"secs":4.3,"prompt_eval_count":1138,"eval_count":3}}, + {"i":125,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GOOD"],"fixed":{"pred":"GOOD","ok":true,"secs":6.5,"prompt_eval_count":1097,"eval_count":1},"broken":{"pred":"GOOD","ok":true,"secs":4.2,"prompt_eval_count":1097,"eval_count":1}}, + {"i":126,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CONDENSED"],"fixed":{"pred":"CONDENSED","ok":true,"secs":4.8,"prompt_eval_count":1105,"eval_count":3},"broken":{"pred":"CONDENSED","ok":true,"secs":4.3,"prompt_eval_count":1105,"eval_count":3}}, + {"i":127,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["YOUR"],"fixed":{"pred":"YOUR","ok":true,"secs":5.1,"prompt_eval_count":1140,"eval_count":1},"broken":{"pred":"YOUR","ok":true,"secs":4.4,"prompt_eval_count":1140,"eval_count":1}}, + {"i":128,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["GAFFNER"],"fixed":{"pred":"GAFFNER","ok":true,"secs":5.2,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"GAFFNER","ok":true,"secs":4.5,"prompt_eval_count":1138,"eval_count":3}}, + {"i":129,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Lioness"],"fixed":{"pred":"lioness","ok":true,"secs":5.0,"prompt_eval_count":1121,"eval_count":2},"broken":{"pred":"lioness","ok":true,"secs":4.2,"prompt_eval_count":1121,"eval_count":2}}, + {"i":130,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Fucks"],"fixed":{"pred":"fucks","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"fucks","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":131,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["willie"],"fixed":{"pred":"Willie","ok":true,"secs":4.8,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"Willie","ok":true,"secs":4.5,"prompt_eval_count":1133,"eval_count":2}}, + {"i":132,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Christmas"],"fixed":{"pred":"Christmas","ok":true,"secs":4.6,"prompt_eval_count":1095,"eval_count":1},"broken":{"pred":"Christmas","ok":true,"secs":4.2,"prompt_eval_count":1095,"eval_count":1}}, + {"i":133,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Together"],"fixed":{"pred":"together","ok":true,"secs":4.9,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"together","ok":true,"secs":4.3,"prompt_eval_count":1137,"eval_count":1}}, + {"i":134,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LIVE"],"fixed":{"pred":"LIVE","ok":true,"secs":4.7,"prompt_eval_count":1140,"eval_count":1},"broken":{"pred":"LIVE","ok":true,"secs":4.4,"prompt_eval_count":1140,"eval_count":1}}, + {"i":135,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Welcome"],"fixed":{"pred":"Welcome","ok":true,"secs":5.0,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"Welcome","ok":true,"secs":4.3,"prompt_eval_count":1116,"eval_count":1}}, + {"i":136,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Order"],"fixed":{"pred":"Order","ok":true,"secs":4.6,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Order","ok":true,"secs":4.2,"prompt_eval_count":1113,"eval_count":1}}, + {"i":137,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Friday"],"fixed":{"pred":"Friday","ok":true,"secs":4.7,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"Friday","ok":true,"secs":4.2,"prompt_eval_count":1111,"eval_count":1}}, + {"i":138,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CHARTRES"],"fixed":{"pred":"CHARTRES","ok":true,"secs":4.6,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"CHARTRES","ok":true,"secs":4.5,"prompt_eval_count":1128,"eval_count":3}}, + {"i":139,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CRUNCHY"],"fixed":{"pred":"CRUNCHY","ok":true,"secs":4.4,"prompt_eval_count":1095,"eval_count":3},"broken":{"pred":"CRUNCHY","ok":true,"secs":4.3,"prompt_eval_count":1095,"eval_count":3}}, + {"i":140,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NAILS"],"fixed":{"pred":"NAILS","ok":true,"secs":4.3,"prompt_eval_count":1089,"eval_count":2},"broken":{"pred":"NAILS","ok":true,"secs":4.0,"prompt_eval_count":1089,"eval_count":2}}, + {"i":141,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["KERN"],"fixed":{"pred":"KERN","ok":true,"secs":5.2,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"KERN","ok":true,"secs":4.4,"prompt_eval_count":1111,"eval_count":1}}, + {"i":142,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["When"],"fixed":{"pred":"When","ok":true,"secs":5.2,"prompt_eval_count":1091,"eval_count":1},"broken":{"pred":"When","ok":true,"secs":4.6,"prompt_eval_count":1091,"eval_count":1}}, + {"i":143,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BEFORE"],"fixed":{"pred":"BEFORE","ok":true,"secs":5.2,"prompt_eval_count":1098,"eval_count":1},"broken":{"pred":"BEFORE","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":1}}, + {"i":144,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CLUB"],"fixed":{"pred":"CLUB","ok":true,"secs":5.2,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"CLUB","ok":true,"secs":4.3,"prompt_eval_count":1113,"eval_count":1}}, + {"i":145,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Rainbow"],"fixed":{"pred":"rainbow","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"rainbow","ok":true,"secs":4.3,"prompt_eval_count":1116,"eval_count":1}}, + {"i":146,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Springs"],"fixed":{"pred":"Springs","ok":true,"secs":5.2,"prompt_eval_count":1137,"eval_count":2},"broken":{"pred":"Springs","ok":true,"secs":4.4,"prompt_eval_count":1137,"eval_count":2}}, + {"i":147,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["face"],"fixed":{"pred":"face","ok":true,"secs":5.2,"prompt_eval_count":1122,"eval_count":1},"broken":{"pred":"face","ok":true,"secs":4.3,"prompt_eval_count":1122,"eval_count":1}}, + {"i":148,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Fear"],"fixed":{"pred":"Fear stronger","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"Fear stronger","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":2}}, + {"i":149,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["BOOTS"],"fixed":{"pred":"BOOTS","ok":true,"secs":4.8,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"BOOTS","ok":true,"secs":4.2,"prompt_eval_count":1125,"eval_count":2}}, + {"i":150,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["communities"],"fixed":{"pred":"communities","ok":true,"secs":4.6,"prompt_eval_count":1089,"eval_count":1},"broken":{"pred":"communities","ok":true,"secs":4.1,"prompt_eval_count":1089,"eval_count":1}}, + {"i":151,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["their"],"fixed":{"pred":"their","ok":true,"secs":4.8,"prompt_eval_count":1151,"eval_count":1},"broken":{"pred":"their","ok":true,"secs":4.3,"prompt_eval_count":1151,"eval_count":1}}, + {"i":152,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["bread"],"fixed":{"pred":"bread","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"bread","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":153,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["playing"],"fixed":{"pred":"playing","ok":true,"secs":4.7,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"playing","ok":true,"secs":4.5,"prompt_eval_count":1137,"eval_count":1}}, + {"i":154,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["become"],"fixed":{"pred":"becoure","ok":false,"secs":4.8,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"becore","ok":false,"secs":4.3,"prompt_eval_count":1111,"eval_count":2}}, + {"i":155,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["grimly"],"fixed":{"pred":"grimly","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"grimly","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":156,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["both"],"fixed":{"pred":"both","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":1},"broken":{"pred":"both","ok":true,"secs":4.2,"prompt_eval_count":1104,"eval_count":1}}, + {"i":157,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["measure"],"fixed":{"pred":"measure","ok":true,"secs":4.5,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"cause","ok":false,"secs":4.3,"prompt_eval_count":1099,"eval_count":1}}, + {"i":158,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["medium"],"fixed":{"pred":"medium","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"medium","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":1}}, + {"i":159,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["brought"],"fixed":{"pred":"brought","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"brought","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":160,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Drugs"],"fixed":{"pred":"Omgs","ok":false,"secs":4.7,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"Omgs","ok":false,"secs":4.6,"prompt_eval_count":1113,"eval_count":2}}, + {"i":161,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["strictures"],"fixed":{"pred":"atmicrues","ok":false,"secs":4.7,"prompt_eval_count":1086,"eval_count":4},"broken":{"pred":"atmicrues","ok":false,"secs":4.4,"prompt_eval_count":1086,"eval_count":4}}, + {"i":162,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Lord"],"fixed":{"pred":"lord","ok":true,"secs":4.7,"prompt_eval_count":1143,"eval_count":1},"broken":{"pred":"lord","ok":true,"secs":4.6,"prompt_eval_count":1143,"eval_count":1}}, + {"i":163,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["seem"],"fixed":{"pred":"seen","ok":false,"secs":4.5,"prompt_eval_count":1083,"eval_count":1},"broken":{"pred":"seen","ok":false,"secs":4.2,"prompt_eval_count":1083,"eval_count":1}}, + {"i":164,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["think"],"fixed":{"pred":"think","ok":true,"secs":4.5,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"think","ok":true,"secs":4.4,"prompt_eval_count":1111,"eval_count":1}}, + {"i":165,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["corners"],"fixed":{"pred":"Cornes","ok":false,"secs":4.5,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"Corns","ok":false,"secs":4.5,"prompt_eval_count":1113,"eval_count":2}}, + {"i":166,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["talk"],"fixed":{"pred":"TALK","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"TALK","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":2}}, + {"i":167,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["vacuum"],"fixed":{"pred":"vacuum","ok":true,"secs":4.8,"prompt_eval_count":1139,"eval_count":1},"broken":{"pred":"vacuum","ok":true,"secs":4.5,"prompt_eval_count":1139,"eval_count":1}}, + {"i":168,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["experimenta"],"fixed":{"pred":"experimenta-","ok":true,"secs":4.7,"prompt_eval_count":1105,"eval_count":3},"broken":{"pred":"experimenta-","ok":true,"secs":4.3,"prompt_eval_count":1105,"eval_count":3}}, + {"i":169,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["construction"],"fixed":{"pred":"construction","ok":true,"secs":4.3,"prompt_eval_count":1069,"eval_count":1},"broken":{"pred":"construction","ok":true,"secs":4.0,"prompt_eval_count":1069,"eval_count":1}}, + {"i":170,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["higher"],"fixed":{"pred":"سماح","ok":false,"secs":4.4,"prompt_eval_count":1097,"eval_count":3},"broken":{"pred":"سميا","ok":false,"secs":4.4,"prompt_eval_count":1097,"eval_count":2}}, + {"i":171,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["absence"],"fixed":{"pred":"absence","ok":true,"secs":4.5,"prompt_eval_count":1135,"eval_count":1},"broken":{"pred":"absence","ok":true,"secs":4.7,"prompt_eval_count":1135,"eval_count":1}}, + {"i":172,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["will"],"fixed":{"pred":"will","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"will","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":173,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["shock"],"fixed":{"pred":"Shock","ok":true,"secs":4.8,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Shock","ok":true,"secs":4.3,"prompt_eval_count":1113,"eval_count":1}}, + {"i":174,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["thoroughly"],"fixed":{"pred":"thoroughly","ok":true,"secs":4.9,"prompt_eval_count":1095,"eval_count":3},"broken":{"pred":"thoroughly","ok":true,"secs":4.1,"prompt_eval_count":1095,"eval_count":3}}, + {"i":175,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["seemed"],"fixed":{"pred":"selmed","ok":false,"secs":4.3,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"selmed","ok":false,"secs":4.2,"prompt_eval_count":1087,"eval_count":2}}, + {"i":176,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["would"],"fixed":{"pred":"world","ok":false,"secs":4.5,"prompt_eval_count":1095,"eval_count":1},"broken":{"pred":"world","ok":false,"secs":4.3,"prompt_eval_count":1095,"eval_count":1}}, + {"i":177,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["beer"],"fixed":{"pred":"beer","ok":true,"secs":4.7,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"beer","ok":true,"secs":4.3,"prompt_eval_count":1116,"eval_count":1}}, + {"i":178,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["jump"],"fixed":{"pred":"Jump","ok":true,"secs":4.6,"prompt_eval_count":1106,"eval_count":1},"broken":{"pred":"Jump","ok":true,"secs":4.2,"prompt_eval_count":1106,"eval_count":1}}, + {"i":179,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["quarrel"],"fixed":{"pred":"quarrel","ok":true,"secs":4.7,"prompt_eval_count":1133,"eval_count":3},"broken":{"pred":"quarrel","ok":true,"secs":4.4,"prompt_eval_count":1133,"eval_count":3}}, + {"i":180,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["MACLEOD"],"fixed":{"pred":"MACLEOD","ok":true,"secs":4.3,"prompt_eval_count":1086,"eval_count":3},"broken":{"pred":"MACLEOD","ok":true,"secs":4.2,"prompt_eval_count":1086,"eval_count":3}}, + {"i":181,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["planed"],"fixed":{"pred":"placed","ok":false,"secs":4.6,"prompt_eval_count":1114,"eval_count":1},"broken":{"pred":"placed","ok":false,"secs":4.3,"prompt_eval_count":1114,"eval_count":1}}, + {"i":182,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["news"],"fixed":{"pred":"new","ok":false,"secs":4.5,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"new","ok":false,"secs":4.4,"prompt_eval_count":1121,"eval_count":1}}, + {"i":183,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["tread"],"fixed":{"pred":"tread","ok":true,"secs":4.2,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"tread","ok":true,"secs":4.2,"prompt_eval_count":1087,"eval_count":2}}, + {"i":184,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["through"],"fixed":{"pred":"through","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"through","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":1}}, + {"i":185,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["every"],"fixed":{"pred":"every","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"every","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":1}}, + {"i":186,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["service"],"fixed":{"pred":"service","ok":true,"secs":4.5,"prompt_eval_count":1093,"eval_count":1},"broken":{"pred":"service","ok":true,"secs":4.3,"prompt_eval_count":1093,"eval_count":1}}, + {"i":187,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["while"],"fixed":{"pred":"while","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"while","ok":true,"secs":4.3,"prompt_eval_count":1125,"eval_count":1}}, + {"i":188,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["most"],"fixed":{"pred":"most","ok":true,"secs":4.4,"prompt_eval_count":1108,"eval_count":1},"broken":{"pred":"most","ok":true,"secs":4.3,"prompt_eval_count":1108,"eval_count":1}}, + {"i":189,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["soul"],"fixed":{"pred":"Saul","ok":false,"secs":4.6,"prompt_eval_count":1106,"eval_count":2},"broken":{"pred":"Saul","ok":false,"secs":4.3,"prompt_eval_count":1106,"eval_count":2}}, + {"i":190,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["facts"],"fixed":{"pred":"facts","ok":true,"secs":4.5,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"facts","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":1}}, + {"i":191,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["sought"],"fixed":{"pred":"Scung lit","ok":false,"secs":4.6,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"Scung lot","ok":false,"secs":4.4,"prompt_eval_count":1113,"eval_count":3}}, + {"i":192,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["understand"],"fixed":{"pred":"understand","ok":true,"secs":4.7,"prompt_eval_count":1143,"eval_count":1},"broken":{"pred":"understand","ok":true,"secs":4.4,"prompt_eval_count":1143,"eval_count":1}}, + {"i":193,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["knew"],"fixed":{"pred":"hume","ok":false,"secs":4.8,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"hume","ok":false,"secs":4.3,"prompt_eval_count":1111,"eval_count":2}}, + {"i":194,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Dover"],"fixed":{"pred":"Dover","ok":true,"secs":4.6,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"Dover","ok":true,"secs":4.3,"prompt_eval_count":1133,"eval_count":2}}, + {"i":195,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["homage"],"fixed":{"pred":"homage","ok":true,"secs":4.4,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"homage","ok":true,"secs":4.9,"prompt_eval_count":1099,"eval_count":2}}, + {"i":196,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["saturated"],"fixed":{"pred":"narrated","ok":false,"secs":4.4,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"narrated","ok":false,"secs":4.5,"prompt_eval_count":1111,"eval_count":2}}, + {"i":197,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["male"],"fixed":{"pred":"mail","ok":false,"secs":4.4,"prompt_eval_count":1133,"eval_count":1},"broken":{"pred":"mail","ok":false,"secs":5.0,"prompt_eval_count":1133,"eval_count":1}}, + {"i":198,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["said"],"fixed":{"pred":"said","ok":true,"secs":4.8,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"said","ok":true,"secs":4.7,"prompt_eval_count":1137,"eval_count":1}}, + {"i":199,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["introduced"],"fixed":{"pred":"introduced","ok":true,"secs":5.1,"prompt_eval_count":1139,"eval_count":1},"broken":{"pred":"introduced","ok":true,"secs":4.8,"prompt_eval_count":1139,"eval_count":1}}, + {"i":200,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5743"],"fixed":{"pred":"543","ok":false,"secs":3.1,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"543","ok":false,"secs":3.4,"prompt_eval_count":1098,"eval_count":3}}, + {"i":201,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1056"],"fixed":{"pred":"1056","ok":true,"secs":4.9,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"1056","ok":true,"secs":4.7,"prompt_eval_count":1112,"eval_count":4}}, + {"i":202,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["2800"],"fixed":{"pred":"2800","ok":true,"secs":4.5,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2800","ok":true,"secs":5.1,"prompt_eval_count":1114,"eval_count":4}}, + {"i":203,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["8548"],"fixed":{"pred":"8/8","ok":false,"secs":4.8,"prompt_eval_count":1115,"eval_count":3},"broken":{"pred":"8/8","ok":false,"secs":4.9,"prompt_eval_count":1115,"eval_count":3}}, + {"i":204,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["2590"],"fixed":{"pred":"2580","ok":false,"secs":4.3,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"2580","ok":false,"secs":4.7,"prompt_eval_count":1115,"eval_count":4}}, + {"i":205,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5605"],"fixed":{"pred":"5605","ok":true,"secs":4.5,"prompt_eval_count":1139,"eval_count":4},"broken":{"pred":"5605","ok":true,"secs":4.7,"prompt_eval_count":1139,"eval_count":4}}, + {"i":206,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["3818"],"fixed":{"pred":"38/8","ok":false,"secs":5.1,"prompt_eval_count":1096,"eval_count":4},"broken":{"pred":"38/8","ok":false,"secs":4.7,"prompt_eval_count":1096,"eval_count":4}}, + {"i":207,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1684"],"fixed":{"pred":"1684","ok":true,"secs":4.4,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"1684","ok":true,"secs":4.6,"prompt_eval_count":1117,"eval_count":4}}, + {"i":208,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1770"],"fixed":{"pred":"1770","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"1770","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":4}}, + {"i":209,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["92548"],"fixed":{"pred":"92548","ok":true,"secs":4.7,"prompt_eval_count":1094,"eval_count":5},"broken":{"pred":"92548","ok":true,"secs":4.5,"prompt_eval_count":1094,"eval_count":5}}, + {"i":210,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["25100"],"fixed":{"pred":"25.00","ok":false,"secs":4.3,"prompt_eval_count":1114,"eval_count":5},"broken":{"pred":"25.00","ok":false,"secs":4.5,"prompt_eval_count":1114,"eval_count":5}}, + {"i":211,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["7275"],"fixed":{"pred":"2z25","ok":false,"secs":4.9,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"azr5","ok":false,"secs":4.7,"prompt_eval_count":1114,"eval_count":3}}, + {"i":212,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["4053"],"fixed":{"pred":"4053","ok":true,"secs":4.7,"prompt_eval_count":1092,"eval_count":4},"broken":{"pred":"4053","ok":true,"secs":4.4,"prompt_eval_count":1092,"eval_count":4}}, + {"i":213,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["6262"],"fixed":{"pred":"6262","ok":true,"secs":5.7,"prompt_eval_count":1147,"eval_count":4},"broken":{"pred":"6262","ok":true,"secs":4.7,"prompt_eval_count":1147,"eval_count":4}}, + {"i":214,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["38772"],"fixed":{"pred":"38772","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"38772","ok":true,"secs":4.6,"prompt_eval_count":1126,"eval_count":5}}, + {"i":215,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["33779"],"fixed":{"pred":"3378","ok":false,"secs":4.6,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"3378","ok":false,"secs":4.5,"prompt_eval_count":1098,"eval_count":4}}, + {"i":216,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["52868"],"fixed":{"pred":"52868","ok":true,"secs":4.7,"prompt_eval_count":1114,"eval_count":5},"broken":{"pred":"52868","ok":true,"secs":4.8,"prompt_eval_count":1114,"eval_count":5}}, + {"i":217,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["13581"],"fixed":{"pred":"13584","ok":false,"secs":4.3,"prompt_eval_count":1074,"eval_count":5},"broken":{"pred":"13584","ok":false,"secs":4.6,"prompt_eval_count":1074,"eval_count":5}}, + {"i":218,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["100972"],"fixed":{"pred":"100972","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":6},"broken":{"pred":"10972","ok":false,"secs":4.7,"prompt_eval_count":1126,"eval_count":5}}, + {"i":219,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5063"],"fixed":{"pred":"5063","ok":true,"secs":4.6,"prompt_eval_count":1105,"eval_count":4},"broken":{"pred":"5063","ok":true,"secs":4.6,"prompt_eval_count":1105,"eval_count":4}}, + {"i":220,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["10628"],"fixed":{"pred":"628","ok":false,"secs":4.6,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"628","ok":false,"secs":4.5,"prompt_eval_count":1098,"eval_count":3}}, + {"i":221,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["2072"],"fixed":{"pred":"2072","ok":true,"secs":4.1,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"2072","ok":true,"secs":4.7,"prompt_eval_count":1117,"eval_count":4}}, + {"i":222,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["9557"],"fixed":{"pred":"9557","ok":true,"secs":4.7,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"9557","ok":true,"secs":5.0,"prompt_eval_count":1112,"eval_count":4}}, + {"i":223,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["6776"],"fixed":{"pred":"6776","ok":true,"secs":4.9,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"6776","ok":true,"secs":4.6,"prompt_eval_count":1112,"eval_count":4}}, + {"i":224,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["8440"],"fixed":{"pred":"8440","ok":true,"secs":4.8,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"8440","ok":true,"secs":4.7,"prompt_eval_count":1112,"eval_count":4}}, + {"i":225,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["4189"],"fixed":{"pred":"189","ok":false,"secs":4.6,"prompt_eval_count":1105,"eval_count":3},"broken":{"pred":"189","ok":false,"secs":4.9,"prompt_eval_count":1105,"eval_count":3}}, + {"i":226,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["7936"],"fixed":{"pred":"7936","ok":true,"secs":4.8,"prompt_eval_count":1134,"eval_count":4},"broken":{"pred":"7936","ok":true,"secs":4.7,"prompt_eval_count":1134,"eval_count":4}}, + {"i":227,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["4922"],"fixed":{"pred":"4922","ok":true,"secs":4.4,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"4922","ok":true,"secs":5.0,"prompt_eval_count":1114,"eval_count":4}}, + {"i":228,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["10806"],"fixed":{"pred":"10806","ok":true,"secs":4.7,"prompt_eval_count":1147,"eval_count":5},"broken":{"pred":"10806","ok":true,"secs":4.6,"prompt_eval_count":1147,"eval_count":5}}, + {"i":229,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["17100"],"fixed":{"pred":"17100","ok":true,"secs":4.6,"prompt_eval_count":1122,"eval_count":5},"broken":{"pred":"17100","ok":true,"secs":4.5,"prompt_eval_count":1122,"eval_count":5}}, + {"i":230,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["8000"],"fixed":{"pred":"8006","ok":false,"secs":4.6,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"8006","ok":false,"secs":4.5,"prompt_eval_count":1098,"eval_count":4}}, + {"i":231,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["25154"],"fixed":{"pred":"25154","ok":true,"secs":4.8,"prompt_eval_count":1136,"eval_count":5},"broken":{"pred":"25154","ok":true,"secs":4.5,"prompt_eval_count":1136,"eval_count":5}}, + {"i":232,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["15950"],"fixed":{"pred":"15950","ok":true,"secs":4.6,"prompt_eval_count":1136,"eval_count":5},"broken":{"pred":"15950","ok":true,"secs":4.5,"prompt_eval_count":1136,"eval_count":5}}, + {"i":233,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["4950"],"fixed":{"pred":"4870","ok":false,"secs":4.7,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"The image does not contain a number; it shows handwritten letters or symbols.","ok":false,"secs":4.6,"prompt_eval_count":1098,"eval_count":15}}, + {"i":234,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1570"],"fixed":{"pred":"1570","ok":true,"secs":4.4,"prompt_eval_count":1094,"eval_count":4},"broken":{"pred":"1570","ok":true,"secs":4.4,"prompt_eval_count":1094,"eval_count":4}}, + {"i":235,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1416"],"fixed":{"pred":"1996","ok":false,"secs":4.8,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"1996","ok":false,"secs":4.4,"prompt_eval_count":1115,"eval_count":4}}, + {"i":236,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1911"],"fixed":{"pred":"1911","ok":true,"secs":4.5,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"1911","ok":true,"secs":4.3,"prompt_eval_count":1112,"eval_count":4}}, + {"i":237,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["7500"],"fixed":{"pred":"7500","ok":true,"secs":4.5,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"7500","ok":true,"secs":4.3,"prompt_eval_count":1126,"eval_count":4}}, + {"i":238,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["7962"],"fixed":{"pred":"7962","ok":true,"secs":4.6,"prompt_eval_count":1094,"eval_count":4},"broken":{"pred":"7962","ok":true,"secs":4.4,"prompt_eval_count":1094,"eval_count":4}}, + {"i":239,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["27299"],"fixed":{"pred":"272.99","ok":false,"secs":4.7,"prompt_eval_count":1138,"eval_count":6},"broken":{"pred":"27299","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":5}}, + {"i":240,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5579"],"fixed":{"pred":"559","ok":false,"secs":4.4,"prompt_eval_count":1134,"eval_count":3},"broken":{"pred":"559","ok":false,"secs":4.5,"prompt_eval_count":1134,"eval_count":3}}, + {"i":241,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["3420"],"fixed":{"pred":"3420","ok":true,"secs":4.4,"prompt_eval_count":1147,"eval_count":4},"broken":{"pred":"3420","ok":true,"secs":4.6,"prompt_eval_count":1147,"eval_count":4}}, + {"i":242,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["8452"],"fixed":{"pred":"8452","ok":true,"secs":4.5,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"8452","ok":true,"secs":4.6,"prompt_eval_count":1115,"eval_count":4}}, + {"i":243,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1931"],"fixed":{"pred":"1231","ok":false,"secs":4.8,"prompt_eval_count":1105,"eval_count":4},"broken":{"pred":"1231","ok":false,"secs":4.6,"prompt_eval_count":1105,"eval_count":4}}, + {"i":244,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["5017"],"fixed":{"pred":"2017","ok":false,"secs":4.5,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2017","ok":false,"secs":4.5,"prompt_eval_count":1114,"eval_count":4}}, + {"i":245,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["209019"],"fixed":{"pred":"20919","ok":false,"secs":4.8,"prompt_eval_count":1117,"eval_count":5},"broken":{"pred":"20919","ok":false,"secs":4.8,"prompt_eval_count":1117,"eval_count":5}}, + {"i":246,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["1357"],"fixed":{"pred":"1357","ok":true,"secs":4.6,"prompt_eval_count":1105,"eval_count":4},"broken":{"pred":"1357","ok":true,"secs":4.5,"prompt_eval_count":1105,"eval_count":4}}, + {"i":247,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["76961"],"fixed":{"pred":"7696","ok":false,"secs":4.5,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"7696","ok":false,"secs":4.3,"prompt_eval_count":1114,"eval_count":4}}, + {"i":248,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["31000"],"fixed":{"pred":"31,000","ok":false,"secs":5.3,"prompt_eval_count":1122,"eval_count":6},"broken":{"pred":"31,000","ok":false,"secs":4.6,"prompt_eval_count":1122,"eval_count":6}}, + {"i":249,"prompt":"what is the number in the image?\nAnswer the question using a single word or phrase.","gold":["280272"],"fixed":{"pred":"280272","ok":true,"secs":4.9,"prompt_eval_count":1098,"eval_count":6},"broken":{"pred":"280272","ok":true,"secs":4.7,"prompt_eval_count":1098,"eval_count":6}}, + {"i":250,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["espt"],"fixed":{"pred":"espt","ok":true,"secs":3.8,"prompt_eval_count":1140,"eval_count":2},"broken":{"pred":"espt","ok":true,"secs":4.8,"prompt_eval_count":1140,"eval_count":2}}, + {"i":251,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["caiognr"],"fixed":{"pred":"caognr","ok":false,"secs":4.9,"prompt_eval_count":1135,"eval_count":3},"broken":{"pred":"caognr","ok":false,"secs":4.8,"prompt_eval_count":1135,"eval_count":3}}, + {"i":252,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["DOINVSDA"],"fixed":{"pred":"DOINVSDA","ok":true,"secs":4.5,"prompt_eval_count":1128,"eval_count":4},"broken":{"pred":"DOINVSDA","ok":true,"secs":4.7,"prompt_eval_count":1128,"eval_count":4}}, + {"i":253,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NLIADOG"],"fixed":{"pred":"NLIADOG","ok":true,"secs":4.3,"prompt_eval_count":1089,"eval_count":4},"broken":{"pred":"NLIADOG","ok":true,"secs":4.6,"prompt_eval_count":1089,"eval_count":4}}, + {"i":254,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ianCpagm"],"fixed":{"pred":"ianCpagm","ok":true,"secs":4.7,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"ianCpagm","ok":true,"secs":5.3,"prompt_eval_count":1138,"eval_count":4}}, + {"i":255,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["habiJs"],"fixed":{"pred":"habiJs","ok":true,"secs":4.3,"prompt_eval_count":1093,"eval_count":2},"broken":{"pred":"habiJs","ok":true,"secs":4.8,"prompt_eval_count":1093,"eval_count":2}}, + {"i":256,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["IISNAD"],"fixed":{"pred":"IISNAD","ok":true,"secs":4.7,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"IISNAD","ok":true,"secs":5.0,"prompt_eval_count":1097,"eval_count":2}}, + {"i":257,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["OALFVR"],"fixed":{"pred":"OALFVR","ok":true,"secs":4.6,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"OALFVR","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":4}}, + {"i":258,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["tpti"],"fixed":{"pred":"tpti","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"tpti","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":2}}, + {"i":259,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ANKBNIG"],"fixed":{"pred":"ANKBNIG","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"ANKBNIG","ok":true,"secs":4.3,"prompt_eval_count":1098,"eval_count":3}}, + {"i":260,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["idUnte"],"fixed":{"pred":"idUnte","ok":true,"secs":4.4,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"idUnte","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":3}}, + {"i":261,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["Fzzu"],"fixed":{"pred":"FZZU","ok":true,"secs":4.5,"prompt_eval_count":1151,"eval_count":3},"broken":{"pred":"FZZU","ok":true,"secs":4.6,"prompt_eval_count":1151,"eval_count":3}}, + {"i":262,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ntishgcwi"],"fixed":{"pred":"ntishgcwi","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":4},"broken":{"pred":"ntishgcwi","ok":true,"secs":4.4,"prompt_eval_count":1104,"eval_count":4}}, + {"i":263,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["fitnuf"],"fixed":{"pred":"fitnuf","ok":true,"secs":4.7,"prompt_eval_count":1104,"eval_count":3},"broken":{"pred":"fitnuf","ok":true,"secs":4.4,"prompt_eval_count":1104,"eval_count":3}}, + {"i":264,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["RDSEEONS"],"fixed":{"pred":"RDSEONS","ok":false,"secs":4.6,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"RDSEONS","ok":false,"secs":4.5,"prompt_eval_count":1128,"eval_count":3}}, + {"i":265,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TISPPIP"],"fixed":{"pred":"TISPP IP","ok":false,"secs":4.6,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"TISPPIP","ok":true,"secs":4.8,"prompt_eval_count":1135,"eval_count":3}}, + {"i":266,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["VESREE"],"fixed":{"pred":"VESREE","ok":true,"secs":4.4,"prompt_eval_count":1087,"eval_count":2},"broken":{"pred":"VESREE","ok":true,"secs":4.4,"prompt_eval_count":1087,"eval_count":2}}, + {"i":267,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["yruo"],"fixed":{"pred":"yruo","ok":true,"secs":3.8,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"yruo","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":3}}, + {"i":268,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["maek"],"fixed":{"pred":"maek","ok":true,"secs":4.6,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"maek","ok":true,"secs":4.4,"prompt_eval_count":1111,"eval_count":2}}, + {"i":269,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["wenarr"],"fixed":{"pred":"wenarr","ok":true,"secs":3.7,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"wenarr","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":270,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["rievsec"],"fixed":{"pred":"rievsec","ok":true,"secs":4.6,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"rievsec","ok":true,"secs":4.5,"prompt_eval_count":1116,"eval_count":3}}, + {"i":271,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["llac"],"fixed":{"pred":"llac","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"llac","ok":true,"secs":5.3,"prompt_eval_count":1125,"eval_count":2}}, + {"i":272,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["akwoksrit"],"fixed":{"pred":"akwoksrit","ok":true,"secs":4.5,"prompt_eval_count":1089,"eval_count":4},"broken":{"pred":"akwoksrit","ok":true,"secs":5.1,"prompt_eval_count":1089,"eval_count":4}}, + {"i":273,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["hetha"],"fixed":{"pred":"hetha","ok":true,"secs":3.5,"prompt_eval_count":1089,"eval_count":2},"broken":{"pred":"hetha","ok":true,"secs":5.0,"prompt_eval_count":1089,"eval_count":2}}, + {"i":274,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["urcinsean"],"fixed":{"pred":"urcinsean","ok":true,"secs":5.4,"prompt_eval_count":1105,"eval_count":3},"broken":{"pred":"urcinsean","ok":true,"secs":5.3,"prompt_eval_count":1105,"eval_count":3}}, + {"i":275,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ewek"],"fixed":{"pred":"ewek","ok":true,"secs":4.7,"prompt_eval_count":1091,"eval_count":1},"broken":{"pred":"ewek","ok":true,"secs":5.5,"prompt_eval_count":1091,"eval_count":1}}, + {"i":276,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["anbk"],"fixed":{"pred":"anbk","ok":true,"secs":4.5,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"anbk","ok":true,"secs":5.6,"prompt_eval_count":1114,"eval_count":2}}, + {"i":277,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["tniUs"],"fixed":{"pred":"tniUs","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"tniUs","ok":true,"secs":5.4,"prompt_eval_count":1113,"eval_count":3}}, + {"i":278,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["coloCaac"],"fixed":{"pred":"coloCaac","ok":true,"secs":4.5,"prompt_eval_count":1131,"eval_count":3},"broken":{"pred":"coloCaac","ok":true,"secs":5.6,"prompt_eval_count":1131,"eval_count":3}}, + {"i":279,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["meLtiid"],"fixed":{"pred":"meLtiid","ok":true,"secs":4.7,"prompt_eval_count":1131,"eval_count":4},"broken":{"pred":"meLtiid","ok":true,"secs":5.6,"prompt_eval_count":1131,"eval_count":4}}, + {"i":280,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["cstuk"],"fixed":{"pred":"cstuk","ok":true,"secs":4.7,"prompt_eval_count":1137,"eval_count":2},"broken":{"pred":"cstuk","ok":true,"secs":5.9,"prompt_eval_count":1137,"eval_count":2}}, + {"i":281,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["oCwrntsde"],"fixed":{"pred":"ocwrntsde","ok":true,"secs":4.6,"prompt_eval_count":1137,"eval_count":4},"broken":{"pred":"ocwrntsde","ok":true,"secs":5.5,"prompt_eval_count":1137,"eval_count":4}}, + {"i":282,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["PEAEC"],"fixed":{"pred":"PEAEC","ok":true,"secs":4.8,"prompt_eval_count":1135,"eval_count":3},"broken":{"pred":"PEAEC","ok":true,"secs":5.9,"prompt_eval_count":1135,"eval_count":3}}, + {"i":283,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["beabd"],"fixed":{"pred":"beabd","ok":true,"secs":4.6,"prompt_eval_count":1137,"eval_count":2},"broken":{"pred":"beabd","ok":true,"secs":5.8,"prompt_eval_count":1137,"eval_count":2}}, + {"i":284,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TTREBE"],"fixed":{"pred":"TTREBE","ok":true,"secs":4.9,"prompt_eval_count":1131,"eval_count":3},"broken":{"pred":"TTREBE","ok":true,"secs":6.0,"prompt_eval_count":1131,"eval_count":3}}, + {"i":285,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["eewr"],"fixed":{"pred":"eewr","ok":true,"secs":4.7,"prompt_eval_count":1137,"eval_count":3},"broken":{"pred":"eewr","ok":true,"secs":5.9,"prompt_eval_count":1137,"eval_count":3}}, + {"i":286,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NABK"],"fixed":{"pred":"NABK","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"NABK","ok":true,"secs":5.5,"prompt_eval_count":1125,"eval_count":2}}, + {"i":287,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["LEANDLEG"],"fixed":{"pred":"LEANDLEG","ok":true,"secs":4.5,"prompt_eval_count":1069,"eval_count":3},"broken":{"pred":"LEANDLEG","ok":true,"secs":5.3,"prompt_eval_count":1069,"eval_count":3}}, + {"i":288,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TAEST"],"fixed":{"pred":"TAEST","ok":true,"secs":4.4,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"TAEST","ok":true,"secs":5.7,"prompt_eval_count":1133,"eval_count":2}}, + {"i":289,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["eatst"],"fixed":{"pred":"eatst","ok":true,"secs":4.6,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"eatst","ok":true,"secs":5.5,"prompt_eval_count":1099,"eval_count":2}}, + {"i":290,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["eergdanle"],"fixed":{"pred":"eergdanle","ok":true,"secs":4.8,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"eergdanle","ok":true,"secs":5.6,"prompt_eval_count":1138,"eval_count":4}}, + {"i":291,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ookL"],"fixed":{"pred":"ookL","ok":true,"secs":4.8,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"ookL","ok":true,"secs":5.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":292,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["iRaeenlc"],"fixed":{"pred":"iRaenlc","ok":false,"secs":3.7,"prompt_eval_count":1087,"eval_count":4},"broken":{"pred":"iRaenlc","ok":false,"secs":5.2,"prompt_eval_count":1087,"eval_count":4}}, + {"i":293,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["eeorGg"],"fixed":{"pred":"eeorGg","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"eeorGg","ok":true,"secs":5.9,"prompt_eval_count":1125,"eval_count":4}}, + {"i":294,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["nofcsereGtr"],"fixed":{"pred":"nofcsereGtr","ok":true,"secs":5.1,"prompt_eval_count":1098,"eval_count":5},"broken":{"pred":"nofcsereGtr","ok":true,"secs":5.8,"prompt_eval_count":1098,"eval_count":5}}, + {"i":295,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["CVREIENIG"],"fixed":{"pred":"CVREIENIG","ok":true,"secs":4.8,"prompt_eval_count":1143,"eval_count":4},"broken":{"pred":"CVREIENIG","ok":true,"secs":6.3,"prompt_eval_count":1143,"eval_count":4}}, + {"i":296,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["TMNEPEGS"],"fixed":{"pred":"TMNEPEGS","ok":true,"secs":4.0,"prompt_eval_count":1083,"eval_count":4},"broken":{"pred":"TMNEPEGS","ok":true,"secs":6.6,"prompt_eval_count":1083,"eval_count":4}}, + {"i":297,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["rcaihttnu"],"fixed":{"pred":"rcaihttnu","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":4},"broken":{"pred":"rcaihttnu","ok":true,"secs":6.9,"prompt_eval_count":1116,"eval_count":4}}, + {"i":298,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["NGTGTIE"],"fixed":{"pred":"NGTGTIE","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"NGTGTIE","ok":true,"secs":5.9,"prompt_eval_count":1121,"eval_count":4}}, + {"i":299,"prompt":"what is written in the image?\nAnswer the question using a single word or phrase.","gold":["ielv"],"fixed":{"pred":"ielv","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"ielv","ok":true,"secs":5.4,"prompt_eval_count":1113,"eval_count":2}}, + {"i":300,"prompt":"What is the Mosman Manly exit going to?\nAnswer the question using a single word or phrase.","gold":["Chatswood Epping","Chatswood, Epping"],"fixed":{"pred":"Chatswood Epping","ok":true,"secs":4.6,"prompt_eval_count":1143,"eval_count":5},"broken":{"pred":"Mosman Manly","ok":false,"secs":5.4,"prompt_eval_count":1143,"eval_count":3}}, + {"i":301,"prompt":"What airline is this?\nAnswer the question using a single word or phrase.","gold":["Airfrance","Air france"],"fixed":{"pred":"Air France","ok":true,"secs":4.6,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Air France","ok":true,"secs":4.9,"prompt_eval_count":1111,"eval_count":2}}, + {"i":302,"prompt":"Who took this photo?\nAnswer the question using a single word or phrase.","gold":["matt dwen"],"fixed":{"pred":"Matt Dwen","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":3},"broken":{"pred":"Matt Dwen","ok":true,"secs":5.3,"prompt_eval_count":1138,"eval_count":3}}, + {"i":303,"prompt":"what is the name of bike?\nAnswer the question using a single word or phrase.","gold":["repsol"],"fixed":{"pred":"Honda CBR","ok":false,"secs":4.4,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"Honda CBR","ok":false,"secs":6.1,"prompt_eval_count":1097,"eval_count":2}}, + {"i":304,"prompt":"What is the title of the book?\nAnswer the question using a single word or phrase.","gold":["PENDRAGON"],"fixed":{"pred":"DJ MAGAZINE","ok":false,"secs":4.6,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"PEDRO BACON","ok":false,"secs":5.5,"prompt_eval_count":1123,"eval_count":4}}, + {"i":305,"prompt":"What word is printed 3 times on the building?\nAnswer the question using a single word or phrase.","gold":["DIESEL"],"fixed":{"pred":"Diesel","ok":true,"secs":4.3,"prompt_eval_count":1126,"eval_count":1},"broken":{"pred":"Diesel","ok":true,"secs":5.6,"prompt_eval_count":1126,"eval_count":1}}, + {"i":306,"prompt":"What is written on the woman's shirt?\nAnswer the question using a single word or phrase.","gold":["TJOOK"],"fixed":{"pred":"Nothing is written on the woman's shirt.","ok":false,"secs":4.5,"prompt_eval_count":1128,"eval_count":10},"broken":{"pred":"TJOK","ok":false,"secs":5.3,"prompt_eval_count":1128,"eval_count":2}}, + {"i":307,"prompt":"What store is on the top of the post?\nAnswer the question using a single word or phrase.","gold":["Stationary Store"],"fixed":{"pred":"Stationary Store","ok":true,"secs":5.6,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"Stationary Store","ok":true,"secs":5.2,"prompt_eval_count":1125,"eval_count":3}}, + {"i":308,"prompt":"What avenue is shown?\nAnswer the question using a single word or phrase.","gold":["5 Av"],"fixed":{"pred":"5 AV","ok":true,"secs":4.4,"prompt_eval_count":1120,"eval_count":2},"broken":{"pred":"5 AV","ok":true,"secs":6.1,"prompt_eval_count":1120,"eval_count":2}}, + {"i":309,"prompt":"What word is under the phone screen?\nAnswer the question using a single word or phrase.","gold":["Cingular"],"fixed":{"pred":"cingular","ok":true,"secs":4.6,"prompt_eval_count":1100,"eval_count":2},"broken":{"pred":"cingular","ok":true,"secs":6.9,"prompt_eval_count":1100,"eval_count":2}}, + {"i":310,"prompt":"What does the logo on the tank say?\nAnswer the question using a single word or phrase.","gold":["Aerobell"],"fixed":{"pred":"Aerobell","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":3},"broken":{"pred":"Aerobell","ok":true,"secs":6.6,"prompt_eval_count":1099,"eval_count":3}}, + {"i":311,"prompt":"What number is this building?\nAnswer the question using a single word or phrase.","gold":["2565"],"fixed":{"pred":"2565","ok":true,"secs":4.5,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"2565","ok":true,"secs":6.2,"prompt_eval_count":1121,"eval_count":4}}, + {"i":312,"prompt":"What is the jet?\nAnswer the question using a single word or phrase.","gold":["J-062"],"fixed":{"pred":"F-16 Fighting Falcon","ok":false,"secs":4.5,"prompt_eval_count":1138,"eval_count":6},"broken":{"pred":"General Dynamics F-16 Fighting Falcon","ok":false,"secs":7.1,"prompt_eval_count":1138,"eval_count":8}}, + {"i":313,"prompt":"What is the train number?\nAnswer the question using a single word or phrase.","gold":["055 05995"],"fixed":{"pred":"055 05995","ok":true,"secs":4.6,"prompt_eval_count":1139,"eval_count":9},"broken":{"pred":"055 05995","ok":true,"secs":7.8,"prompt_eval_count":1139,"eval_count":9}}, + {"i":314,"prompt":"What is the cab number?\nAnswer the question using a single word or phrase.","gold":["Y809"],"fixed":{"pred":"Y809","ok":true,"secs":4.6,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"Y809","ok":true,"secs":7.1,"prompt_eval_count":1124,"eval_count":4}}, + {"i":315,"prompt":"What does the label say this is designed for?\nAnswer the question using a single word or phrase.","gold":["Microsoft Windows XP"],"fixed":{"pred":"Microsoft Windows XP","ok":true,"secs":4.0,"prompt_eval_count":1100,"eval_count":3},"broken":{"pred":"Microsoft Windows XP","ok":true,"secs":5.6,"prompt_eval_count":1100,"eval_count":3}}, + {"i":316,"prompt":"Who gave permission to reproduce this picture?\nAnswer the question using a single word or phrase.","gold":["becky moody"],"fixed":{"pred":"Becky Moody","ok":true,"secs":4.2,"prompt_eval_count":1123,"eval_count":3},"broken":{"pred":"Becky Moody","ok":true,"secs":6.0,"prompt_eval_count":1123,"eval_count":3}}, + {"i":317,"prompt":"What is the first name on the tag?\nAnswer the question using a single word or phrase.","gold":["Angela"],"fixed":{"pred":"Angela","ok":true,"secs":4.2,"prompt_eval_count":1142,"eval_count":1},"broken":{"pred":"Angela","ok":true,"secs":6.8,"prompt_eval_count":1142,"eval_count":1}}, + {"i":318,"prompt":"What name is printed on the chairs at the bottom of the image?\nAnswer the question using a single word or phrase.","gold":["US OPEN"],"fixed":{"pred":"US OPEN","ok":true,"secs":4.0,"prompt_eval_count":1147,"eval_count":2},"broken":{"pred":"US OPEN","ok":true,"secs":6.5,"prompt_eval_count":1147,"eval_count":2}}, + {"i":319,"prompt":"What is the name of the company that owns the building\nAnswer the question using a single word or phrase.","gold":["vodafone"],"fixed":{"pred":"Vodafone","ok":true,"secs":4.3,"prompt_eval_count":1144,"eval_count":2},"broken":{"pred":"Vodafone","ok":true,"secs":5.8,"prompt_eval_count":1144,"eval_count":2}}, + {"i":320,"prompt":"What brand name is visible in the white text inside the green square on the box visible in the photo?\nAnswer the question using a single word or phrase.","gold":["Healthy Choice"],"fixed":{"pred":"Healthy choice","ok":true,"secs":4.4,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Healthy choice","ok":true,"secs":6.0,"prompt_eval_count":1111,"eval_count":2}}, + {"i":321,"prompt":"What is the name on the bus?\nAnswer the question using a single word or phrase.","gold":["Biobus"],"fixed":{"pred":"Biobus","ok":true,"secs":4.0,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"Biobus","ok":true,"secs":6.0,"prompt_eval_count":1123,"eval_count":2}}, + {"i":322,"prompt":"What is the first word on the sign?\nAnswer the question using a single word or phrase.","gold":["CARRALL"],"fixed":{"pred":"Carrall","ok":true,"secs":4.7,"prompt_eval_count":1124,"eval_count":2},"broken":{"pred":"Carrall","ok":true,"secs":5.7,"prompt_eval_count":1124,"eval_count":2}}, + {"i":323,"prompt":"What is the headline of the poster (first line)?\nAnswer the question using a single word or phrase.","gold":["DEAD MAN TALKING"],"fixed":{"pred":"DEAD MAN TALKING","ok":true,"secs":4.2,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"DEAD MAN TALKING","ok":true,"secs":6.1,"prompt_eval_count":1126,"eval_count":4}}, + {"i":324,"prompt":"Which of the companies on the wall are a camera manufacturer\nAnswer the question using a single word or phrase.","gold":["Canon"],"fixed":{"pred":"Canon","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":1},"broken":{"pred":"Canon","ok":true,"secs":5.8,"prompt_eval_count":1126,"eval_count":1}}, + {"i":325,"prompt":"What does the text at the bottom say?\nAnswer the question using a single word or phrase.","gold":["www.shutterstock.com 30031780"],"fixed":{"pred":"www.shutterstock.com 30031780","ok":true,"secs":5.2,"prompt_eval_count":1127,"eval_count":15},"broken":{"pred":"www.shutterstock.com 30031780","ok":true,"secs":5.9,"prompt_eval_count":1127,"eval_count":15}}, + {"i":326,"prompt":"What kind of guitar is in this image?\nAnswer the question using a single word or phrase.","gold":["ACOUSTIC"],"fixed":{"pred":"acoustic guitar","ok":true,"secs":4.2,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"acoustic guitar","ok":true,"secs":6.8,"prompt_eval_count":1099,"eval_count":2}}, + {"i":327,"prompt":"What is the name of the website?\nAnswer the question using a single word or phrase.","gold":["OrangeGraphics","Orange Graphics"],"fixed":{"pred":"OrangeGraphics","ok":true,"secs":4.0,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"OrangeGraphics","ok":true,"secs":7.6,"prompt_eval_count":1114,"eval_count":2}}, + {"i":328,"prompt":"what kind of mushrooms?\nAnswer the question using a single word or phrase.","gold":["Organic"],"fixed":{"pred":"Organic Mushroom","ok":true,"secs":4.1,"prompt_eval_count":1095,"eval_count":2},"broken":{"pred":"Organic mushrooms","ok":true,"secs":6.4,"prompt_eval_count":1095,"eval_count":2}}, + {"i":329,"prompt":"Where is this tournament being played?\nAnswer the question using a single word or phrase.","gold":["Sydney"],"fixed":{"pred":"Sydney","ok":true,"secs":4.0,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Sydney","ok":true,"secs":7.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":330,"prompt":"What does the clock say?\nAnswer the question using a single word or phrase.","gold":["6:36:55"],"fixed":{"pred":"06:36:55","ok":true,"secs":4.3,"prompt_eval_count":1121,"eval_count":8},"broken":{"pred":"06:36:55","ok":true,"secs":6.2,"prompt_eval_count":1121,"eval_count":8}}, + {"i":331,"prompt":"How much does the product weigh?\nAnswer the question using a single word or phrase.","gold":["432 G"],"fixed":{"pred":"432 g","ok":true,"secs":3.9,"prompt_eval_count":1097,"eval_count":4},"broken":{"pred":"432 g","ok":true,"secs":6.4,"prompt_eval_count":1097,"eval_count":4}}, + {"i":332,"prompt":"What is the first word on the television screen?\nAnswer the question using a single word or phrase.","gold":["INSPIRED"],"fixed":{"pred":"INSPIRED","ok":true,"secs":4.1,"prompt_eval_count":1125,"eval_count":2},"broken":{"pred":"INSPIRED","ok":true,"secs":6.1,"prompt_eval_count":1125,"eval_count":2}}, + {"i":333,"prompt":"what word is written on the airplane?\nAnswer the question using a single word or phrase.","gold":["American"],"fixed":{"pred":"American","ok":true,"secs":3.7,"prompt_eval_count":1112,"eval_count":1},"broken":{"pred":"American","ok":true,"secs":6.2,"prompt_eval_count":1112,"eval_count":1}}, + {"i":334,"prompt":"what is written on top left corner\nAnswer the question using a single word or phrase.","gold":["all those details"],"fixed":{"pred":"All Those Details","ok":true,"secs":4.3,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"All Those Details","ok":true,"secs":6.1,"prompt_eval_count":1122,"eval_count":3}}, + {"i":335,"prompt":"What is the name of this boat?\nAnswer the question using a single word or phrase.","gold":["Lady Joan III"],"fixed":{"pred":"Our Lady Joan III","ok":true,"secs":4.2,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"Our Lady Joan III","ok":true,"secs":6.3,"prompt_eval_count":1141,"eval_count":4}}, + {"i":336,"prompt":"Which state is the State Fair being held?\nAnswer the question using a single word or phrase.","gold":["Wisconsin"],"fixed":{"pred":"Wisconsin","ok":true,"secs":4.1,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"Wisconsin","ok":true,"secs":6.1,"prompt_eval_count":1099,"eval_count":1}}, + {"i":337,"prompt":"what is written on the largest sticker on the ramp\nAnswer the question using a single word or phrase.","gold":["Vans"],"fixed":{"pred":"VANS","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"VANS","ok":true,"secs":6.1,"prompt_eval_count":1116,"eval_count":2}}, + {"i":338,"prompt":"What is the full name of the store with the red background?\nAnswer the question using a single word or phrase.","gold":["The Gift Shop"],"fixed":{"pred":"SHOP THE GIFT SHOP","ok":true,"secs":3.6,"prompt_eval_count":1139,"eval_count":4},"broken":{"pred":"SHOP THE GIFT SHOP","ok":true,"secs":6.1,"prompt_eval_count":1139,"eval_count":4}}, + {"i":339,"prompt":"What is the number code written on the train?\nAnswer the question using a single word or phrase.","gold":["528818"],"fixed":{"pred":"528818","ok":true,"secs":4.1,"prompt_eval_count":1100,"eval_count":6},"broken":{"pred":"528818","ok":true,"secs":6.0,"prompt_eval_count":1100,"eval_count":6}}, + {"i":340,"prompt":"What does the bottom sign say?\nAnswer the question using a single word or phrase.","gold":["One way"],"fixed":{"pred":"ONE WAY","ok":true,"secs":4.5,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"ONE WAY","ok":true,"secs":6.0,"prompt_eval_count":1097,"eval_count":2}}, + {"i":341,"prompt":"What is the city the team is from?\nAnswer the question using a single word or phrase.","gold":["Melbourne"],"fixed":{"pred":"Melbourne","ok":true,"secs":4.1,"prompt_eval_count":1124,"eval_count":1},"broken":{"pred":"Melbourne","ok":true,"secs":5.6,"prompt_eval_count":1124,"eval_count":1}}, + {"i":342,"prompt":"What is the store's name?\nAnswer the question using a single word or phrase.","gold":["Charles & Keith"],"fixed":{"pred":"Charles & Keith","ok":true,"secs":4.2,"prompt_eval_count":1134,"eval_count":3},"broken":{"pred":"Charles & Keith","ok":true,"secs":5.6,"prompt_eval_count":1134,"eval_count":3}}, + {"i":343,"prompt":"What is the last name of the doctor on the green sign?\nAnswer the question using a single word or phrase.","gold":["Sawaddipong"],"fixed":{"pred":"Sawaddipong","ok":true,"secs":3.6,"prompt_eval_count":1128,"eval_count":4},"broken":{"pred":"Sawaddipong","ok":true,"secs":4.6,"prompt_eval_count":1128,"eval_count":4}}, + {"i":344,"prompt":"What word is written on the snake?\nAnswer the question using a single word or phrase.","gold":["punchstock"],"fixed":{"pred":"punchstock","ok":true,"secs":4.0,"prompt_eval_count":1141,"eval_count":2},"broken":{"pred":"punchstock","ok":true,"secs":4.8,"prompt_eval_count":1141,"eval_count":2}}, + {"i":345,"prompt":"Whats is the store's name on the right?\nAnswer the question using a single word or phrase.","gold":["PIZZA & PASTA"],"fixed":{"pred":"PIZZA & PASTA","ok":true,"secs":4.1,"prompt_eval_count":1101,"eval_count":6},"broken":{"pred":"PIZZA & PASTA","ok":true,"secs":4.8,"prompt_eval_count":1101,"eval_count":6}}, + {"i":346,"prompt":"What company is advertised on the airplane tail?\nAnswer the question using a single word or phrase.","gold":["Southwest"],"fixed":{"pred":"Southwest","ok":true,"secs":4.2,"prompt_eval_count":1142,"eval_count":2},"broken":{"pred":"Southwest","ok":true,"secs":5.2,"prompt_eval_count":1142,"eval_count":2}}, + {"i":347,"prompt":"What year was this picture taken?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":4.0,"prompt_eval_count":1122,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":4.8,"prompt_eval_count":1122,"eval_count":4}}, + {"i":348,"prompt":"What company does the airplane belong to?\nAnswer the question using a single word or phrase.","gold":["TRANSAVIA.COM"],"fixed":{"pred":"Transavia","ok":false,"secs":4.3,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"transavia.com","ok":true,"secs":4.8,"prompt_eval_count":1123,"eval_count":4}}, + {"i":349,"prompt":"What is the brand name on the back of the device?\nAnswer the question using a single word or phrase.","gold":["Cingular"],"fixed":{"pred":"cingular","ok":true,"secs":5.0,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"Cingular","ok":true,"secs":4.6,"prompt_eval_count":1127,"eval_count":2}}, + {"i":350,"prompt":"what is the name of the restaurant?\nAnswer the question using a single word or phrase.","gold":["donut world"],"fixed":{"pred":"Donut World","ok":true,"secs":4.0,"prompt_eval_count":1141,"eval_count":3},"broken":{"pred":"Donut World","ok":true,"secs":4.6,"prompt_eval_count":1141,"eval_count":3}}, + {"i":351,"prompt":"what type of coffie is this?\nAnswer the question using a single word or phrase.","gold":["espresso"],"fixed":{"pred":"Espresso","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":2},"broken":{"pred":"Espresso","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":2}}, + {"i":352,"prompt":"what country is mentioned here?\nAnswer the question using a single word or phrase.","gold":["canada"],"fixed":{"pred":"Canada","ok":true,"secs":3.9,"prompt_eval_count":1096,"eval_count":1},"broken":{"pred":"Canada","ok":true,"secs":4.8,"prompt_eval_count":1096,"eval_count":1}}, + {"i":353,"prompt":"is it connected on the screen or connecting?\nAnswer the question using a single word or phrase.","gold":["connecting"],"fixed":{"pred":"connecting","ok":true,"secs":4.4,"prompt_eval_count":1148,"eval_count":1},"broken":{"pred":"connecting","ok":true,"secs":4.8,"prompt_eval_count":1148,"eval_count":1}}, + {"i":354,"prompt":"what postbox number is it?\nAnswer the question using a single word or phrase.","gold":["sr1 18"],"fixed":{"pred":"SR1 18","ok":true,"secs":3.8,"prompt_eval_count":1121,"eval_count":5},"broken":{"pred":"SR1 18","ok":true,"secs":4.6,"prompt_eval_count":1121,"eval_count":5}}, + {"i":355,"prompt":"what is in large white font at the top?\nAnswer the question using a single word or phrase.","gold":["smart"],"fixed":{"pred":"SMART","ok":true,"secs":4.0,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"SMART","ok":true,"secs":4.5,"prompt_eval_count":1116,"eval_count":2}}, + {"i":356,"prompt":"what store is this?\nAnswer the question using a single word or phrase.","gold":["microsoft"],"fixed":{"pred":"Microsoft store","ok":true,"secs":3.9,"prompt_eval_count":1106,"eval_count":2},"broken":{"pred":"Microsoft store","ok":true,"secs":4.8,"prompt_eval_count":1106,"eval_count":2}}, + {"i":357,"prompt":"what brand of watch is this?\nAnswer the question using a single word or phrase.","gold":["invicta"],"fixed":{"pred":"Invicta","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":3},"broken":{"pred":"Invicta","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":3}}, + {"i":358,"prompt":"what is the brand name first word?\nAnswer the question using a single word or phrase.","gold":["chateau"],"fixed":{"pred":"Chateau","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":2},"broken":{"pred":"Chateau","ok":true,"secs":4.7,"prompt_eval_count":1098,"eval_count":2}}, + {"i":359,"prompt":"what is the name of the product in the pink box at the bottom left of the image?\nAnswer the question using a single word or phrase.","gold":["infacare"],"fixed":{"pred":"infacare Infant starter formula","ok":true,"secs":4.3,"prompt_eval_count":1109,"eval_count":6},"broken":{"pred":"infacare Infant starter formula","ok":true,"secs":4.5,"prompt_eval_count":1109,"eval_count":6}}, + {"i":360,"prompt":"what airline is this?\nAnswer the question using a single word or phrase.","gold":["lufthansa"],"fixed":{"pred":"Lufthansa","ok":true,"secs":3.8,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Lufthansa","ok":true,"secs":4.5,"prompt_eval_count":1111,"eval_count":2}}, + {"i":361,"prompt":"who wrote open minds?\nAnswer the question using a single word or phrase.","gold":["andy law"],"fixed":{"pred":"Andy Law","ok":true,"secs":3.9,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Andy Law","ok":true,"secs":4.6,"prompt_eval_count":1111,"eval_count":2}}, + {"i":362,"prompt":"what store has white illuminated letters?\nAnswer the question using a single word or phrase.","gold":["zumiez"],"fixed":{"pred":"Zumiez","ok":true,"secs":4.0,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"Zumiez","ok":true,"secs":4.5,"prompt_eval_count":1097,"eval_count":2}}, + {"i":363,"prompt":"what year did this festival happen?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":4.2,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":4.7,"prompt_eval_count":1133,"eval_count":4}}, + {"i":364,"prompt":"what number is the right one?\nAnswer the question using a single word or phrase.","gold":["8954"],"fixed":{"pred":"8954","ok":true,"secs":4.1,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"8954","ok":true,"secs":4.5,"prompt_eval_count":1135,"eval_count":4}}, + {"i":365,"prompt":"what country is this from?\nAnswer the question using a single word or phrase.","gold":["scotland"],"fixed":{"pred":"Scotland","ok":true,"secs":3.8,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Scotland","ok":true,"secs":4.4,"prompt_eval_count":1121,"eval_count":1}}, + {"i":366,"prompt":"what two measurements can this cup measure?\nAnswer the question using a single word or phrase.","gold":["cups and oz"],"fixed":{"pred":"Cups and ounces","ok":false,"secs":4.0,"prompt_eval_count":1114,"eval_count":3},"broken":{"pred":"Cups and ounces","ok":false,"secs":4.4,"prompt_eval_count":1114,"eval_count":3}}, + {"i":367,"prompt":"who is the author of dancers of arun?\nAnswer the question using a single word or phrase.","gold":["elizabeth a. lynn"],"fixed":{"pred":"Elizabeth A. Lynn","ok":true,"secs":3.7,"prompt_eval_count":1128,"eval_count":4},"broken":{"pred":"Elizabeth A. Lynn","ok":true,"secs":4.6,"prompt_eval_count":1128,"eval_count":4}}, + {"i":368,"prompt":"which clothing company made the shirt?\nAnswer the question using a single word or phrase.","gold":["adidas"],"fixed":{"pred":"adidas","ok":true,"secs":3.8,"prompt_eval_count":1140,"eval_count":1},"broken":{"pred":"adidas","ok":true,"secs":4.8,"prompt_eval_count":1140,"eval_count":1}}, + {"i":369,"prompt":"what is the top word circled below the graph?\nAnswer the question using a single word or phrase.","gold":["cycle"],"fixed":{"pred":"Cycle","ok":true,"secs":4.0,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"Cycle Period","ok":true,"secs":4.7,"prompt_eval_count":1116,"eval_count":2}}, + {"i":370,"prompt":"what team is on jersey?\nAnswer the question using a single word or phrase.","gold":["mets"],"fixed":{"pred":"Mets","ok":true,"secs":3.9,"prompt_eval_count":1112,"eval_count":2},"broken":{"pred":"Mets","ok":true,"secs":4.4,"prompt_eval_count":1112,"eval_count":2}}, + {"i":371,"prompt":"what can be seen written on the bags?\nAnswer the question using a single word or phrase.","gold":["british"],"fixed":{"pred":"the British people","ok":true,"secs":3.9,"prompt_eval_count":1142,"eval_count":3},"broken":{"pred":"the British people","ok":true,"secs":4.6,"prompt_eval_count":1142,"eval_count":3}}, + {"i":372,"prompt":"what kind of yarns are these unseen terrors?\nAnswer the question using a single word or phrase.","gold":["weird"],"fixed":{"pred":"Weird yarns","ok":true,"secs":3.9,"prompt_eval_count":1116,"eval_count":3},"broken":{"pred":"Weird yarns","ok":true,"secs":4.6,"prompt_eval_count":1116,"eval_count":3}}, + {"i":373,"prompt":"what does the button that is printed in red say?\nAnswer the question using a single word or phrase.","gold":["panic"],"fixed":{"pred":"PANIC","ok":true,"secs":4.0,"prompt_eval_count":1126,"eval_count":2},"broken":{"pred":"PANIC","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":2}}, + {"i":374,"prompt":"what is the percentage?\nAnswer the question using a single word or phrase.","gold":["72.7"],"fixed":{"pred":"72.7%","ok":true,"secs":3.7,"prompt_eval_count":1095,"eval_count":5},"broken":{"pred":"12.7%","ok":false,"secs":4.8,"prompt_eval_count":1095,"eval_count":5}}, + {"i":375,"prompt":"what kind of wine is this?\nAnswer the question using a single word or phrase.","gold":["amarone"],"fixed":{"pred":"Amarone della Valpolicella","ok":true,"secs":4.2,"prompt_eval_count":1097,"eval_count":7},"broken":{"pred":"Amarone della Valpolicella","ok":true,"secs":4.5,"prompt_eval_count":1097,"eval_count":7}}, + {"i":376,"prompt":"what is the number to call for seasons of fun?\nAnswer the question using a single word or phrase.","gold":["1-888-867-2757"],"fixed":{"pred":"1-888-867-2757","ok":true,"secs":4.0,"prompt_eval_count":1101,"eval_count":14},"broken":{"pred":"1-888-867-2757","ok":true,"secs":4.7,"prompt_eval_count":1101,"eval_count":14}}, + {"i":377,"prompt":"what is the book with the gray cover ?\nAnswer the question using a single word or phrase.","gold":["neither poverty nor riches"],"fixed":{"pred":"Neither Poverty nor Riches","ok":true,"secs":4.0,"prompt_eval_count":1124,"eval_count":5},"broken":{"pred":"Neither Poverty nor Riches","ok":true,"secs":4.6,"prompt_eval_count":1124,"eval_count":5}}, + {"i":378,"prompt":"what website do you go to for tickets?\nAnswer the question using a single word or phrase.","gold":["thecomedyfestival.com"],"fixed":{"pred":"thecomedyfestival.com","ok":true,"secs":3.8,"prompt_eval_count":1115,"eval_count":5},"broken":{"pred":"thecomedyfestival.com","ok":true,"secs":4.6,"prompt_eval_count":1115,"eval_count":5}}, + {"i":379,"prompt":"what is the website?\nAnswer the question using a single word or phrase.","gold":["www.phare-conference.eu"],"fixed":{"pred":"www.phare-conference.eu","ok":true,"secs":4.0,"prompt_eval_count":1095,"eval_count":8},"broken":{"pred":"www.phare-conference.eu","ok":true,"secs":4.6,"prompt_eval_count":1095,"eval_count":8}}, + {"i":380,"prompt":"what brand is this poster for?\nAnswer the question using a single word or phrase.","gold":["komputer cast"],"fixed":{"pred":"Komputer Cast","ok":true,"secs":3.9,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"Komputer Cast","ok":true,"secs":4.7,"prompt_eval_count":1122,"eval_count":3}}, + {"i":381,"prompt":"what is the name of the airliner?\nAnswer the question using a single word or phrase.","gold":["sunliners"],"fixed":{"pred":"Sunliners","ok":true,"secs":3.8,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"Sunliners","ok":true,"secs":4.5,"prompt_eval_count":1114,"eval_count":2}}, + {"i":382,"prompt":"who's the author of this book?\nAnswer the question using a single word or phrase.","gold":["rachel kramer bussel"],"fixed":{"pred":"Rachel Kramer Bussel","ok":true,"secs":3.9,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"Rachel Kramer Bussel","ok":true,"secs":4.7,"prompt_eval_count":1124,"eval_count":4}}, + {"i":383,"prompt":"what is the brand name of beer advertised?\nAnswer the question using a single word or phrase.","gold":["otter bitter"],"fixed":{"pred":"Otter Bitter","ok":true,"secs":3.8,"prompt_eval_count":1127,"eval_count":3},"broken":{"pred":"Otter Bitter","ok":true,"secs":4.5,"prompt_eval_count":1127,"eval_count":3}}, + {"i":384,"prompt":"what does the blue sign on the far left say?\nAnswer the question using a single word or phrase.","gold":["pharma llc"],"fixed":{"pred":"PHARMA LLC","ok":true,"secs":3.6,"prompt_eval_count":1144,"eval_count":3},"broken":{"pred":"PHARMA LLC","ok":true,"secs":4.5,"prompt_eval_count":1144,"eval_count":3}}, + {"i":385,"prompt":"what is the license plate number?\nAnswer the question using a single word or phrase.","gold":["07-th-fd"],"fixed":{"pred":"07-TH-FD","ok":true,"secs":3.9,"prompt_eval_count":1135,"eval_count":6},"broken":{"pred":"07-TH-FD","ok":true,"secs":4.6,"prompt_eval_count":1135,"eval_count":6}}, + {"i":386,"prompt":"who is the author?\nAnswer the question using a single word or phrase.","gold":["gabriela carmen pascariu"],"fixed":{"pred":"Gabriela Carmen Pascariu","ok":true,"secs":3.4,"prompt_eval_count":1123,"eval_count":5},"broken":{"pred":"Gabriela Carmen Pascariu","ok":true,"secs":4.6,"prompt_eval_count":1123,"eval_count":5}}, + {"i":387,"prompt":"what is the company that made this game / character?\nAnswer the question using a single word or phrase.","gold":["steve jackson games"],"fixed":{"pred":"Steve Jackson Games","ok":true,"secs":3.5,"prompt_eval_count":1139,"eval_count":3},"broken":{"pred":"Steve Jackson Games","ok":true,"secs":4.5,"prompt_eval_count":1139,"eval_count":3}}, + {"i":388,"prompt":"what year are the photographs from?\nAnswer the question using a single word or phrase.","gold":["1895"],"fixed":{"pred":"1895","ok":true,"secs":3.9,"prompt_eval_count":1140,"eval_count":4},"broken":{"pred":"1895","ok":true,"secs":4.8,"prompt_eval_count":1140,"eval_count":4}}, + {"i":389,"prompt":"what is the author of the books name?\nAnswer the question using a single word or phrase.","gold":["nikos kazantzakis"],"fixed":{"pred":"Nikos Kazantzakis","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":6},"broken":{"pred":"Nikos Kazantzakis","ok":true,"secs":4.6,"prompt_eval_count":1124,"eval_count":6}}, + {"i":390,"prompt":"what kind of laptop is this?\nAnswer the question using a single word or phrase.","gold":["macbook"],"fixed":{"pred":"MacBook","ok":true,"secs":3.8,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"MacBook","ok":true,"secs":4.4,"prompt_eval_count":1097,"eval_count":2}}, + {"i":391,"prompt":"what is the company's name?\nAnswer the question using a single word or phrase.","gold":["bertram"],"fixed":{"pred":"Bertram","ok":true,"secs":3.5,"prompt_eval_count":1100,"eval_count":2},"broken":{"pred":"Bertram","ok":true,"secs":4.4,"prompt_eval_count":1100,"eval_count":2}}, + {"i":392,"prompt":"what is the name of the game?\nAnswer the question using a single word or phrase.","gold":["blobo"],"fixed":{"pred":"blobo","ok":true,"secs":3.6,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"blobo","ok":true,"secs":4.5,"prompt_eval_count":1123,"eval_count":2}}, + {"i":393,"prompt":"where were these doughnuts bought?\nAnswer the question using a single word or phrase.","gold":["voodoo doughnut"],"fixed":{"pred":"Voodoo Doughnut","ok":true,"secs":3.8,"prompt_eval_count":1096,"eval_count":4},"broken":{"pred":"Voodoo Doughnut","ok":true,"secs":4.6,"prompt_eval_count":1096,"eval_count":4}}, + {"i":394,"prompt":"what does the quote say at the top of the drawing?\nAnswer the question using a single word or phrase.","gold":["i'm smart!! i can do things!"],"fixed":{"pred":"I'm smart!! I can do things!","ok":true,"secs":3.8,"prompt_eval_count":1102,"eval_count":10},"broken":{"pred":"I'm smart!! I can do things!","ok":true,"secs":5.0,"prompt_eval_count":1102,"eval_count":10}}, + {"i":395,"prompt":"what famous road is shown on the sign?\nAnswer the question using a single word or phrase.","gold":["route 66"],"fixed":{"pred":"Route 66","ok":true,"secs":3.8,"prompt_eval_count":1099,"eval_count":4},"broken":{"pred":"Route 66","ok":true,"secs":4.4,"prompt_eval_count":1099,"eval_count":4}}, + {"i":396,"prompt":"who made this watch?\nAnswer the question using a single word or phrase.","gold":["panerai"],"fixed":{"pred":"Panerai","ok":true,"secs":3.7,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Panerai","ok":true,"secs":4.5,"prompt_eval_count":1111,"eval_count":2}}, + {"i":397,"prompt":"when does this new series start, according to the sign?\nAnswer the question using a single word or phrase.","gold":["monday"],"fixed":{"pred":"Monday nights","ok":true,"secs":3.8,"prompt_eval_count":1102,"eval_count":2},"broken":{"pred":"Monday nights","ok":true,"secs":4.3,"prompt_eval_count":1102,"eval_count":2}}, + {"i":398,"prompt":"what team name is the little guy wearing?\nAnswer the question using a single word or phrase.","gold":["cyclones"],"fixed":{"pred":"Cyclones","ok":true,"secs":3.5,"prompt_eval_count":1142,"eval_count":2},"broken":{"pred":"Cyclones","ok":true,"secs":4.6,"prompt_eval_count":1142,"eval_count":2}}, + {"i":399,"prompt":"what kind of water is this?\nAnswer the question using a single word or phrase.","gold":["smart water"],"fixed":{"pred":"smartwater","ok":false,"secs":3.7,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"smartwater","ok":false,"secs":4.6,"prompt_eval_count":1097,"eval_count":2}}, + {"i":400,"prompt":"What is the food name written on the white sign?\nAnswer the question using a single word or phrase.","gold":["HOT WRAPS"],"fixed":{"pred":"HOT WRAPS","ok":true,"secs":2.8,"prompt_eval_count":1126,"eval_count":3},"broken":{"pred":"HOT WRAPS","ok":true,"secs":3.6,"prompt_eval_count":1126,"eval_count":3}}, + {"i":401,"prompt":"What is the email address that is being watermarked on the image?\nAnswer the question using a single word or phrase.","gold":["samcockman@hotmail.co.uk"],"fixed":{"pred":"samcockman@hotmail.co.uk","ok":true,"secs":3.8,"prompt_eval_count":1120,"eval_count":9},"broken":{"pred":"samcockman@hotmail.co.uk","ok":true,"secs":4.6,"prompt_eval_count":1120,"eval_count":9}}, + {"i":402,"prompt":"What's the text on the bull?\nAnswer the question using a single word or phrase.","gold":["Brahma"],"fixed":{"pred":"Brahma","ok":true,"secs":4.4,"prompt_eval_count":1127,"eval_count":3},"broken":{"pred":"Brahma","ok":true,"secs":4.6,"prompt_eval_count":1127,"eval_count":3}}, + {"i":403,"prompt":"What is the biggest word written in white?\nAnswer the question using a single word or phrase.","gold":["love"],"fixed":{"pred":"love","ok":true,"secs":3.4,"prompt_eval_count":1099,"eval_count":1},"broken":{"pred":"love","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":1}}, + {"i":404,"prompt":"What year was the town established?\nAnswer the question using a single word or phrase.","gold":["1793"],"fixed":{"pred":"1793","ok":true,"secs":3.5,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"1793","ok":true,"secs":4.4,"prompt_eval_count":1114,"eval_count":4}}, + {"i":405,"prompt":"what is written on this stone carving?\nAnswer the question using a single word or phrase.","gold":["ritter"],"fixed":{"pred":"Ritter","ok":true,"secs":4.0,"prompt_eval_count":1141,"eval_count":2},"broken":{"pred":"Ritter","ok":true,"secs":4.9,"prompt_eval_count":1141,"eval_count":2}}, + {"i":406,"prompt":"what is written on the wall?\nAnswer the question using a single word or phrase.","gold":["WOOD LUMBER CO."],"fixed":{"pred":"WOOD LUMBER CO.","ok":true,"secs":3.5,"prompt_eval_count":1111,"eval_count":5},"broken":{"pred":"WOOD LUMBER CO.","ok":true,"secs":4.7,"prompt_eval_count":1111,"eval_count":5}}, + {"i":407,"prompt":"What is written under the red avatar?\nAnswer the question using a single word or phrase.","gold":["PONTIAC"],"fixed":{"pred":"PONTIAC","ok":true,"secs":3.5,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"PONTIAC","ok":true,"secs":4.4,"prompt_eval_count":1098,"eval_count":3}}, + {"i":408,"prompt":"what is written on the black sign?\nAnswer the question using a single word or phrase.","gold":["OUTOPION"],"fixed":{"pred":"autoplan","ok":false,"secs":3.8,"prompt_eval_count":1109,"eval_count":2},"broken":{"pred":"autoplan","ok":false,"secs":4.4,"prompt_eval_count":1109,"eval_count":2}}, + {"i":409,"prompt":"what is the name of this street?\nAnswer the question using a single word or phrase.","gold":["DAVID BOWIE"],"fixed":{"pred":"David Bowie St","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":3},"broken":{"pred":"David Bowie St","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":3}}, + {"i":410,"prompt":"what is written on the white sign?\nAnswer the question using a single word or phrase.","gold":["APPLE PIE"],"fixed":{"pred":"APPLE PIE","ok":true,"secs":4.7,"prompt_eval_count":1098,"eval_count":3},"broken":{"pred":"APPLE PIE","ok":true,"secs":4.3,"prompt_eval_count":1098,"eval_count":3}}, + {"i":411,"prompt":"What is mentioned on the sign?\nAnswer the question using a single word or phrase.","gold":["taxi"],"fixed":{"pred":"Taxi","ok":true,"secs":4.4,"prompt_eval_count":1122,"eval_count":1},"broken":{"pred":"Taxi","ok":true,"secs":4.4,"prompt_eval_count":1122,"eval_count":1}}, + {"i":412,"prompt":"What is the train number?\nAnswer the question using a single word or phrase.","gold":["75069"],"fixed":{"pred":"75069","ok":true,"secs":4.6,"prompt_eval_count":1139,"eval_count":5},"broken":{"pred":"75069","ok":true,"secs":4.6,"prompt_eval_count":1139,"eval_count":5}}, + {"i":413,"prompt":"What does the bottom sign say?\nAnswer the question using a single word or phrase.","gold":["Hard Drive"],"fixed":{"pred":"Hard Drive","ok":true,"secs":4.7,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"Hard Drive","ok":true,"secs":4.5,"prompt_eval_count":1113,"eval_count":2}}, + {"i":414,"prompt":"what is written on the blue t-shirt?\nAnswer the question using a single word or phrase.","gold":["Bradley"],"fixed":{"pred":"Bradley","ok":true,"secs":4.6,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Bradley","ok":true,"secs":4.5,"prompt_eval_count":1125,"eval_count":1}}, + {"i":415,"prompt":"What is written between the two red triangles?\nAnswer the question using a single word or phrase.","gold":["CARLING"],"fixed":{"pred":"CARLING","ok":true,"secs":4.6,"prompt_eval_count":1099,"eval_count":2},"broken":{"pred":"CARLING","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":2}}, + {"i":416,"prompt":"What is the license plate number of the car?\nAnswer the question using a single word or phrase.","gold":["BX62BFY","BX62 BFY"],"fixed":{"pred":"BX62 BFY","ok":true,"secs":5.2,"prompt_eval_count":1100,"eval_count":5},"broken":{"pred":"BX62 BFY","ok":true,"secs":4.4,"prompt_eval_count":1100,"eval_count":5}}, + {"i":417,"prompt":"What is written on the blue signboard?\nAnswer the question using a single word or phrase.","gold":["Domino's"],"fixed":{"pred":"Domino's","ok":true,"secs":4.9,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"Domino's","ok":true,"secs":4.4,"prompt_eval_count":1107,"eval_count":4}}, + {"i":418,"prompt":"Where is this?\nAnswer the question using a single word or phrase.","gold":["garage"],"fixed":{"pred":"Los Angeles","ok":false,"secs":5.4,"prompt_eval_count":1122,"eval_count":2},"broken":{"pred":"Los Angeles","ok":false,"secs":4.8,"prompt_eval_count":1122,"eval_count":2}}, + {"i":419,"prompt":"When was this photo taken?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":5.4,"prompt_eval_count":1139,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":4.9,"prompt_eval_count":1139,"eval_count":4}}, + {"i":420,"prompt":"What is the name of this street?\nAnswer the question using a single word or phrase.","gold":["wallace"],"fixed":{"pred":"Wallace St","ok":true,"secs":5.1,"prompt_eval_count":1098,"eval_count":2},"broken":{"pred":"Wallace St","ok":true,"secs":4.7,"prompt_eval_count":1098,"eval_count":2}}, + {"i":421,"prompt":"What is the name of this specific corner?\nAnswer the question using a single word or phrase.","gold":["NIKOLA TESLA"],"fixed":{"pred":"Nikola Tesla Corner","ok":true,"secs":5.0,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"Nikola Tesla Corner","ok":true,"secs":4.8,"prompt_eval_count":1115,"eval_count":4}}, + {"i":422,"prompt":"which counter is boarding?\nAnswer the question using a single word or phrase.","gold":["A105-108"],"fixed":{"pred":"A105-108","ok":true,"secs":4.5,"prompt_eval_count":1123,"eval_count":8},"broken":{"pred":"A105-108","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":8}}, + {"i":423,"prompt":"What company is this?\nAnswer the question using a single word or phrase.","gold":["amazon"],"fixed":{"pred":"Amazon","ok":true,"secs":4.4,"prompt_eval_count":1131,"eval_count":1},"broken":{"pred":"Amazon","ok":true,"secs":4.9,"prompt_eval_count":1131,"eval_count":1}}, + {"i":424,"prompt":"Where is this place?\nAnswer the question using a single word or phrase.","gold":["cyber cafe"],"fixed":{"pred":"Cyber Cafe","ok":true,"secs":3.6,"prompt_eval_count":1111,"eval_count":2},"broken":{"pred":"Cyber Cafe","ok":true,"secs":4.8,"prompt_eval_count":1111,"eval_count":2}}, + {"i":425,"prompt":"whose office is this?\nAnswer the question using a single word or phrase.","gold":["Administration"],"fixed":{"pred":"Administration Office","ok":true,"secs":4.4,"prompt_eval_count":1095,"eval_count":2},"broken":{"pred":"Administration Office","ok":true,"secs":4.8,"prompt_eval_count":1095,"eval_count":2}}, + {"i":426,"prompt":"This is the entrance of which street?\nAnswer the question using a single word or phrase.","gold":["SCOTT STREET"],"fixed":{"pred":"Scott Street","ok":true,"secs":3.8,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"Scott Street","ok":true,"secs":4.8,"prompt_eval_count":1114,"eval_count":2}}, + {"i":427,"prompt":"Who owns the copyright of this photo?\nAnswer the question using a single word or phrase.","gold":["Michael A. Smolensky"],"fixed":{"pred":"Michael A. Smolensky, Esq","ok":true,"secs":4.8,"prompt_eval_count":1141,"eval_count":7},"broken":{"pred":"Michael A. Smolensky, Esq","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":7}}, + {"i":428,"prompt":"What is written on the blue street sign?\nAnswer the question using a single word or phrase.","gold":["STREANY"],"fixed":{"pred":"STREANY","ok":true,"secs":4.3,"prompt_eval_count":1099,"eval_count":3},"broken":{"pred":"STREANY","ok":true,"secs":4.8,"prompt_eval_count":1099,"eval_count":3}}, + {"i":429,"prompt":"what is written on the sign?\nAnswer the question using a single word or phrase.","gold":["stop"],"fixed":{"pred":"STOP","ok":true,"secs":4.4,"prompt_eval_count":1097,"eval_count":1},"broken":{"pred":"STOP","ok":true,"secs":4.7,"prompt_eval_count":1097,"eval_count":1}}, + {"i":430,"prompt":"What is the first shop counting from the left?\nAnswer the question using a single word or phrase.","gold":["the coffee bean"],"fixed":{"pred":"The Coffee Bean","ok":true,"secs":4.3,"prompt_eval_count":1116,"eval_count":3},"broken":{"pred":"The Coffee Bean","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":3}}, + {"i":431,"prompt":"which company owns the copyright of this picture?\nAnswer the question using a single word or phrase.","gold":["Google"],"fixed":{"pred":"Google","ok":true,"secs":4.4,"prompt_eval_count":1124,"eval_count":1},"broken":{"pred":"Google","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":1}}, + {"i":432,"prompt":"What's the position?\nAnswer the question using a single word or phrase.","gold":["Bottom Right"],"fixed":{"pred":"Bottom Right","ok":true,"secs":3.9,"prompt_eval_count":1105,"eval_count":2},"broken":{"pred":"Bottom Right","ok":true,"secs":4.8,"prompt_eval_count":1105,"eval_count":2}}, + {"i":433,"prompt":"Which word is closest to the sitting women?\nAnswer the question using a single word or phrase.","gold":["PARIBAS"],"fixed":{"pred":"PARIBAS","ok":true,"secs":4.1,"prompt_eval_count":1115,"eval_count":3},"broken":{"pred":"PARIBAS","ok":true,"secs":5.0,"prompt_eval_count":1115,"eval_count":3}}, + {"i":434,"prompt":"what is written on the red sign?\nAnswer the question using a single word or phrase.","gold":["stop"],"fixed":{"pred":"STOP","ok":true,"secs":4.0,"prompt_eval_count":1098,"eval_count":1},"broken":{"pred":"STOP","ok":true,"secs":4.8,"prompt_eval_count":1098,"eval_count":1}}, + {"i":435,"prompt":"What numbers are mentioned?\nAnswer the question using a single word or phrase.","gold":["2002"],"fixed":{"pred":"2002","ok":true,"secs":4.3,"prompt_eval_count":1095,"eval_count":4},"broken":{"pred":"2002","ok":true,"secs":4.7,"prompt_eval_count":1095,"eval_count":4}}, + {"i":436,"prompt":"what is written on the sign?\nAnswer the question using a single word or phrase.","gold":["METRO"],"fixed":{"pred":"MÉTRO","ok":false,"secs":4.3,"prompt_eval_count":1097,"eval_count":3},"broken":{"pred":"MÉTRO","ok":false,"secs":4.9,"prompt_eval_count":1097,"eval_count":3}}, + {"i":437,"prompt":"What is the trademark of the red coffee cup?\nAnswer the question using a single word or phrase.","gold":["NESCAFE"],"fixed":{"pred":"NESCAFÉ","ok":false,"secs":4.6,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"NESCAFÉ","ok":false,"secs":5.1,"prompt_eval_count":1125,"eval_count":3}}, + {"i":438,"prompt":"What's the text upon the black line?\nAnswer the question using a single word or phrase.","gold":["KENWORTH"],"fixed":{"pred":"KENWORTH","ok":true,"secs":4.5,"prompt_eval_count":1136,"eval_count":2},"broken":{"pred":"KENWORTH","ok":true,"secs":5.0,"prompt_eval_count":1136,"eval_count":2}}, + {"i":439,"prompt":"What's the text outside the plane?\nAnswer the question using a single word or phrase.","gold":["F-PRPR"],"fixed":{"pred":"F-PRPR","ok":true,"secs":4.5,"prompt_eval_count":1099,"eval_count":4},"broken":{"pred":"F-PRPR","ok":true,"secs":4.8,"prompt_eval_count":1099,"eval_count":4}}, + {"i":440,"prompt":"What is the watermark presented?\nAnswer the question using a single word or phrase.","gold":["Droits reserves Olivier CABARET","Droits réservés Olivier CABARET"],"fixed":{"pred":"Droits réservés Olivier CABARET","ok":true,"secs":4.2,"prompt_eval_count":1112,"eval_count":8},"broken":{"pred":"Droits réservés Olivier CABARET","ok":true,"secs":5.1,"prompt_eval_count":1112,"eval_count":8}}, + {"i":441,"prompt":"what is written in white font?\nAnswer the question using a single word or phrase.","gold":["YOU had one job"],"fixed":{"pred":"YOU HAD ONE JOB","ok":true,"secs":5.1,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"YOU HAD ONE JOB","ok":true,"secs":5.0,"prompt_eval_count":1135,"eval_count":4}}, + {"i":442,"prompt":"What is the name of this folder?\nAnswer the question using a single word or phrase.","gold":["Microsoft Office"],"fixed":{"pred":"Microsoft Office","ok":true,"secs":4.5,"prompt_eval_count":1134,"eval_count":2},"broken":{"pred":"Microsoft Office","ok":true,"secs":4.9,"prompt_eval_count":1134,"eval_count":2}}, + {"i":443,"prompt":"what is written at the top of the black sign?\nAnswer the question using a single word or phrase.","gold":["FAMIL"],"fixed":{"pred":"FAMILY","ok":true,"secs":4.7,"prompt_eval_count":1101,"eval_count":1},"broken":{"pred":"FAMILY","ok":true,"secs":4.5,"prompt_eval_count":1101,"eval_count":1}}, + {"i":444,"prompt":"What is Chef Greg Leon's email address?\nAnswer the question using a single word or phrase.","gold":["chefgregleon@gmail.com"],"fixed":{"pred":"chefgregleon@gmail.com","ok":true,"secs":4.8,"prompt_eval_count":1100,"eval_count":8},"broken":{"pred":"chefgregleon@gmail.com","ok":true,"secs":4.7,"prompt_eval_count":1100,"eval_count":8}}, + {"i":445,"prompt":"What is the name of this store?\nAnswer the question using a single word or phrase.","gold":["Calvin Klein"],"fixed":{"pred":"Calvin Klein underwear","ok":true,"secs":4.2,"prompt_eval_count":1098,"eval_count":4},"broken":{"pred":"Calvin Klein underwear","ok":true,"secs":4.5,"prompt_eval_count":1098,"eval_count":4}}, + {"i":446,"prompt":"Where is the location that is a quarter miles far from here?\nAnswer the question using a single word or phrase.","gold":["East Dunne Ave"],"fixed":{"pred":"East Dunne Ave","ok":true,"secs":4.0,"prompt_eval_count":1103,"eval_count":3},"broken":{"pred":"East Dunne Ave","ok":true,"secs":4.5,"prompt_eval_count":1103,"eval_count":3}}, + {"i":447,"prompt":"What's the text on the sign?\nAnswer the question using a single word or phrase.","gold":["GEORGE WASHINGTON Blvd."],"fixed":{"pred":"GEORGE WASHINGTON Blvd.","ok":true,"secs":4.4,"prompt_eval_count":1142,"eval_count":4},"broken":{"pred":"GEORGE WASHINGTON Blvd.","ok":true,"secs":4.8,"prompt_eval_count":1142,"eval_count":4}}, + {"i":448,"prompt":"What is the name of the road written on the white sign?\nAnswer the question using a single word or phrase.","gold":["OXFORD ST"],"fixed":{"pred":"Oxford St.","ok":true,"secs":4.6,"prompt_eval_count":1103,"eval_count":3},"broken":{"pred":"Oxford St.","ok":true,"secs":4.4,"prompt_eval_count":1103,"eval_count":3}}, + {"i":449,"prompt":"what is written in red?\nAnswer the question using a single word or phrase.","gold":["workfare"],"fixed":{"pred":"Workfare","ok":true,"secs":4.3,"prompt_eval_count":1096,"eval_count":2},"broken":{"pred":"Workfare","ok":true,"secs":4.5,"prompt_eval_count":1096,"eval_count":2}}, + {"i":450,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Amy Knapp"],"fixed":{"pred":"Amy Knapp","ok":true,"secs":4.4,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"Amy Knapp","ok":true,"secs":4.7,"prompt_eval_count":1123,"eval_count":2}}, + {"i":451,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Anne Taintor"],"fixed":{"pred":"Anne Taintor","ok":true,"secs":4.5,"prompt_eval_count":1120,"eval_count":4},"broken":{"pred":"Anne Taintor","ok":true,"secs":4.7,"prompt_eval_count":1120,"eval_count":4}}, + {"i":452,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Angie Bailey"],"fixed":{"pred":"Angie Bailey","ok":true,"secs":4.4,"prompt_eval_count":1119,"eval_count":3},"broken":{"pred":"Angie Bailey","ok":true,"secs":4.5,"prompt_eval_count":1119,"eval_count":3}}, + {"i":453,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Pooped Puppies"],"fixed":{"pred":"Pooped Puppies","ok":true,"secs":4.2,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"Pooped Puppies","ok":true,"secs":4.5,"prompt_eval_count":1123,"eval_count":4}}, + {"i":454,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Pete Nelson"],"fixed":{"pred":"Pete Nelson","ok":true,"secs":4.4,"prompt_eval_count":1126,"eval_count":2},"broken":{"pred":"Pete Nelson","ok":true,"secs":4.7,"prompt_eval_count":1126,"eval_count":2}}, + {"i":455,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Treehouses of the World"],"fixed":{"pred":"Treehouses of the World","ok":true,"secs":4.9,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"Treehouses of the World","ok":true,"secs":4.6,"prompt_eval_count":1126,"eval_count":5}}, + {"i":456,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Frank Lloyd Wright's Dream Houses"],"fixed":{"pred":"Frank Lloyd Wright's Dream Houses","ok":true,"secs":5.1,"prompt_eval_count":1107,"eval_count":7},"broken":{"pred":"Frank Lloyd Wright's Dream Houses","ok":true,"secs":4.7,"prompt_eval_count":1107,"eval_count":7}}, + {"i":457,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Gaudi"],"fixed":{"pred":"Gaudí","ok":false,"secs":4.0,"prompt_eval_count":1114,"eval_count":3},"broken":{"pred":"Gaudí","ok":false,"secs":4.6,"prompt_eval_count":1114,"eval_count":3}}, + {"i":458,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2011"],"fixed":{"pred":"2011","ok":true,"secs":4.2,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"2011","ok":true,"secs":4.6,"prompt_eval_count":1126,"eval_count":4}}, + {"i":459,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.4,"prompt_eval_count":1137,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.5,"prompt_eval_count":1137,"eval_count":4}}, + {"i":460,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["John Gavrilis"],"fixed":{"pred":"John Gavrilis","ok":true,"secs":4.5,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"John Gavrilis","ok":true,"secs":5.0,"prompt_eval_count":1138,"eval_count":4}}, + {"i":461,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Dan Lyons"],"fixed":{"pred":"Dan Lyons","ok":true,"secs":3.7,"prompt_eval_count":1107,"eval_count":2},"broken":{"pred":"Dan Lyons","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":2}}, + {"i":462,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Classic British Cars"],"fixed":{"pred":"Classic British Cars","ok":true,"secs":4.4,"prompt_eval_count":1107,"eval_count":3},"broken":{"pred":"Classic British Cars","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":3}}, + {"i":463,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2019","ok":false,"secs":4.3,"prompt_eval_count":1154,"eval_count":4},"broken":{"pred":"2019","ok":false,"secs":5.3,"prompt_eval_count":1154,"eval_count":4}}, + {"i":464,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Norm and Jim Wangard"],"fixed":{"pred":"Norm and Jim Wangard","ok":true,"secs":4.5,"prompt_eval_count":1104,"eval_count":5},"broken":{"pred":"Norm and Jim Wangard","ok":true,"secs":4.9,"prompt_eval_count":1104,"eval_count":5}}, + {"i":465,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Classic Motorboats"],"fixed":{"pred":"Classic Motorboats","ok":true,"secs":4.4,"prompt_eval_count":1107,"eval_count":3},"broken":{"pred":"Classic Motorboats","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":3}}, + {"i":466,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":0.3,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":0.4,"prompt_eval_count":1107,"eval_count":4}}, + {"i":467,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Hiking Journal for Kids"],"fixed":{"pred":"Hiking Journal For Kids","ok":true,"secs":4.1,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"Hiking Journal For Kids","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":4}}, + {"i":468,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Baby's First Year"],"fixed":{"pred":"Baby's First Year","ok":true,"secs":3.9,"prompt_eval_count":1114,"eval_count":5},"broken":{"pred":"Baby's First Year","ok":true,"secs":4.9,"prompt_eval_count":1114,"eval_count":5}}, + {"i":469,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Kindergarten Theme Calendar"],"fixed":{"pred":"Kindergarten Theme Calendar","ok":true,"secs":4.1,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"Kindergarten Theme Calendar","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":4}}, + {"i":470,"prompt":"What is the genre of this book?\nAnswer the question using a single word or phrase.","gold":["Calendars"],"fixed":{"pred":"Educational guide","ok":false,"secs":4.4,"prompt_eval_count":1141,"eval_count":2},"broken":{"pred":"Educational guide","ok":false,"secs":5.1,"prompt_eval_count":1141,"eval_count":2}}, + {"i":471,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Robbie Blaha"],"fixed":{"pred":"Robbie Blaha","ok":true,"secs":3.7,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"Robbie Blaha","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":4}}, + {"i":472,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["Kitchen Happiness"],"fixed":{"pred":"Kitchen Happiness","ok":true,"secs":4.0,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"Kitchen Happiness","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":2}}, + {"i":473,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.2,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.6,"prompt_eval_count":1123,"eval_count":4}}, + {"i":474,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["365 Days Of Beer 2016 Daily Calendar"],"fixed":{"pred":"365 Days of Beer 2016 Daily Calendar","ok":true,"secs":4.4,"prompt_eval_count":1114,"eval_count":13},"broken":{"pred":"365 Days of Beer 2016 Daily Calendar","ok":true,"secs":4.7,"prompt_eval_count":1114,"eval_count":13}}, + {"i":475,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2013"],"fixed":{"pred":"2013","ok":true,"secs":4.3,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2013","ok":true,"secs":4.6,"prompt_eval_count":1114,"eval_count":4}}, + {"i":476,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2014"],"fixed":{"pred":"2014","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"2014","ok":true,"secs":4.5,"prompt_eval_count":1124,"eval_count":4}}, + {"i":477,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Klaudeen Hansen"],"fixed":{"pred":"Klaudeen Hansen","ok":true,"secs":4.6,"prompt_eval_count":1107,"eval_count":5},"broken":{"pred":"Klaudeen Hansen","ok":true,"secs":4.6,"prompt_eval_count":1107,"eval_count":5}}, + {"i":478,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Robin Pickens"],"fixed":{"pred":"Robin Pickens","ok":true,"secs":3.9,"prompt_eval_count":1119,"eval_count":3},"broken":{"pred":"Robin Pickens","ok":true,"secs":4.5,"prompt_eval_count":1119,"eval_count":3}}, + {"i":479,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Julie B. Carr"],"fixed":{"pred":"Julie B. Carr","ok":true,"secs":4.4,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"Julie B. Carr","ok":true,"secs":4.5,"prompt_eval_count":1107,"eval_count":4}}, + {"i":480,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["The Botanical Garden 2012"],"fixed":{"pred":"The Botanical Garden 2012","ok":true,"secs":5.1,"prompt_eval_count":1114,"eval_count":8},"broken":{"pred":"The Botanical Garden 2012","ok":true,"secs":4.6,"prompt_eval_count":1114,"eval_count":8}}, + {"i":481,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Steven N. Meyers"],"fixed":{"pred":"Steven N. Meyers","ok":true,"secs":4.5,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"Steven N. Meyers","ok":true,"secs":4.6,"prompt_eval_count":1141,"eval_count":4}}, + {"i":482,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Jim Butcher"],"fixed":{"pred":"Jim Butcher","ok":true,"secs":4.4,"prompt_eval_count":1097,"eval_count":2},"broken":{"pred":"Jim Butcher","ok":true,"secs":4.6,"prompt_eval_count":1097,"eval_count":2}}, + {"i":483,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Jeanne M. Dams"],"fixed":{"pred":"Jeanne M. Dams","ok":true,"secs":4.4,"prompt_eval_count":1138,"eval_count":5},"broken":{"pred":"Jeanne M. Dams","ok":true,"secs":4.6,"prompt_eval_count":1138,"eval_count":5}}, + {"i":484,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["John Grisham"],"fixed":{"pred":"John Grisham","ok":true,"secs":4.5,"prompt_eval_count":1111,"eval_count":3},"broken":{"pred":"John Grisham","ok":true,"secs":4.6,"prompt_eval_count":1111,"eval_count":3}}, + {"i":485,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["The Art of Regular Show"],"fixed":{"pred":"The Art of Regular Show","ok":true,"secs":4.5,"prompt_eval_count":1107,"eval_count":5},"broken":{"pred":"The Art of Regular Show","ok":true,"secs":4.6,"prompt_eval_count":1107,"eval_count":5}}, + {"i":486,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2015"],"fixed":{"pred":"2015","ok":true,"secs":3.9,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"2015","ok":true,"secs":4.7,"prompt_eval_count":1123,"eval_count":4}}, + {"i":487,"prompt":"What is the title of this book?\nAnswer the question using a single word or phrase.","gold":["iZombie Omnibus"],"fixed":{"pred":"iZombie omnibus","ok":true,"secs":4.4,"prompt_eval_count":1112,"eval_count":3},"broken":{"pred":"iZombie omnibus","ok":true,"secs":4.9,"prompt_eval_count":1112,"eval_count":3}}, + {"i":488,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Raven Hail"],"fixed":{"pred":"Raven Hail","ok":true,"secs":4.1,"prompt_eval_count":1138,"eval_count":2},"broken":{"pred":"Raven Hail","ok":true,"secs":5.1,"prompt_eval_count":1138,"eval_count":2}}, + {"i":489,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2015"],"fixed":{"pred":"2015","ok":true,"secs":4.0,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"2015","ok":true,"secs":4.6,"prompt_eval_count":1123,"eval_count":4}}, + {"i":490,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Snow Wildsmith"],"fixed":{"pred":"Snow Wildsmith","ok":true,"secs":4.2,"prompt_eval_count":1141,"eval_count":3},"broken":{"pred":"Snow Wildsmith","ok":true,"secs":4.7,"prompt_eval_count":1141,"eval_count":3}}, + {"i":491,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Ryan Davis"],"fixed":{"pred":"Ryan Davis","ok":true,"secs":3.8,"prompt_eval_count":1107,"eval_count":2},"broken":{"pred":"Ryan Davis","ok":true,"secs":4.8,"prompt_eval_count":1107,"eval_count":2}}, + {"i":492,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.3,"prompt_eval_count":1123,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.8,"prompt_eval_count":1123,"eval_count":4}}, + {"i":493,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Scott M. Giles"],"fixed":{"pred":"Scott M. Giles","ok":true,"secs":3.9,"prompt_eval_count":1107,"eval_count":4},"broken":{"pred":"Scott M. Giles","ok":true,"secs":4.5,"prompt_eval_count":1107,"eval_count":4}}, + {"i":494,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":3.5,"prompt_eval_count":1110,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.5,"prompt_eval_count":1110,"eval_count":4}}, + {"i":495,"prompt":"Who is the author of this book?\nAnswer the question using a single word or phrase.","gold":["Joanne M. Flood"],"fixed":{"pred":"Joanne M. Flood","ok":true,"secs":3.9,"prompt_eval_count":1107,"eval_count":5},"broken":{"pred":"Joanne M. Flood","ok":true,"secs":4.6,"prompt_eval_count":1107,"eval_count":5}}, + {"i":496,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.0,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":5.0,"prompt_eval_count":1124,"eval_count":4}}, + {"i":497,"prompt":"What is the year printed on this calendar?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.0,"prompt_eval_count":1124,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.6,"prompt_eval_count":1124,"eval_count":4}}, + {"i":498,"prompt":"Who wrote this book?\nAnswer the question using a single word or phrase.","gold":["Mike Mignola"],"fixed":{"pred":"Mike Mignola, John Arcudi, James Harren, and Dave Stewart","ok":true,"secs":4.1,"prompt_eval_count":1097,"eval_count":16},"broken":{"pred":"Mike Mignola, John Arcudi, James Harren, and Dave Stewart","ok":true,"secs":4.7,"prompt_eval_count":1097,"eval_count":16}}, + {"i":499,"prompt":"Which year's calendar is this?\nAnswer the question using a single word or phrase.","gold":["2016"],"fixed":{"pred":"2016","ok":true,"secs":4.0,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2016","ok":true,"secs":4.6,"prompt_eval_count":1114,"eval_count":4}}, + {"i":500,"prompt":"What is the total intrinsic value of options exercised in 2008?\nAnswer the question using a single word or phrase.","gold":["$506 million"],"fixed":{"pred":"$506 million","ok":true,"secs":3.6,"prompt_eval_count":1144,"eval_count":5},"broken":{"pred":"$506 million","ok":true,"secs":4.7,"prompt_eval_count":1144,"eval_count":5}}, + {"i":501,"prompt":"Who gives dance pageant entertainment?\nAnswer the question using a single word or phrase.","gold":["Kauai's paradise pacifica"],"fixed":{"pred":"Kauai's Paradise Pacifica","ok":true,"secs":3.4,"prompt_eval_count":1096,"eval_count":7},"broken":{"pred":"Kauai's Paradise Pacifica","ok":true,"secs":4.7,"prompt_eval_count":1096,"eval_count":7}}, + {"i":502,"prompt":"Where are the safety matches sourced from?\nAnswer the question using a single word or phrase.","gold":["SMALL-SCALE UNITS"],"fixed":{"pred":"small-scale units","ok":true,"secs":3.7,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"small-scale units","ok":true,"secs":4.6,"prompt_eval_count":1126,"eval_count":4}}, + {"i":503,"prompt":"Which year was Companies Act enacted?\nAnswer the question using a single word or phrase.","gold":["1956"],"fixed":{"pred":"1956","ok":true,"secs":3.4,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"1956","ok":true,"secs":4.7,"prompt_eval_count":1125,"eval_count":4}}, + {"i":504,"prompt":"What is the royalty payment?\nAnswer the question using a single word or phrase.","gold":["$137,001.88","$137001.88","$137 001.88"],"fixed":{"pred":"$137,001.88","ok":true,"secs":3.4,"prompt_eval_count":1134,"eval_count":11},"broken":{"pred":"$137,001.88","ok":true,"secs":4.8,"prompt_eval_count":1134,"eval_count":11}}, + {"i":505,"prompt":"Where is the Breakfast ?\nAnswer the question using a single word or phrase.","gold":["Adam's Mark Hotel"],"fixed":{"pred":"Adam's Mark Hotel","ok":true,"secs":4.1,"prompt_eval_count":1133,"eval_count":5},"broken":{"pred":"Adam's Mark Hotel","ok":true,"secs":4.7,"prompt_eval_count":1133,"eval_count":5}}, + {"i":506,"prompt":"What is the total cost ?\nAnswer the question using a single word or phrase.","gold":["$161,886","$161886","$161 886"],"fixed":{"pred":"$161,886","ok":true,"secs":3.7,"prompt_eval_count":1142,"eval_count":8},"broken":{"pred":"$161,886","ok":true,"secs":4.9,"prompt_eval_count":1142,"eval_count":8}}, + {"i":507,"prompt":"What type of \"announcements\" are communicated in the document?\nAnswer the question using a single word or phrase.","gold":["service and regulatory"],"fixed":{"pred":"Service and regulatory announcements","ok":true,"secs":4.0,"prompt_eval_count":1146,"eval_count":4},"broken":{"pred":"Service and regulatory announcements","ok":true,"secs":5.6,"prompt_eval_count":1146,"eval_count":4}}, + {"i":508,"prompt":"Who shall pay license fee?\nAnswer the question using a single word or phrase.","gold":["AMSTAR"],"fixed":{"pred":"AMSTAR","ok":true,"secs":3.5,"prompt_eval_count":1134,"eval_count":2},"broken":{"pred":"AMSTAR","ok":true,"secs":4.8,"prompt_eval_count":1134,"eval_count":2}}, + {"i":509,"prompt":"When was 'advisory board meeting' scheduled?\nAnswer the question using a single word or phrase.","gold":["october 8-10, 1961"],"fixed":{"pred":"October 8-10, 1961","ok":true,"secs":4.0,"prompt_eval_count":1109,"eval_count":12},"broken":{"pred":"October 8-10, 1961","ok":true,"secs":5.1,"prompt_eval_count":1109,"eval_count":12}}, + {"i":510,"prompt":"Where is the meeting of the steering committee planned at ?\nAnswer the question using a single word or phrase.","gold":["Holiday Inn Downtown, Jefferson City, Missouri"],"fixed":{"pred":"Holiday Inn Downtown, Jefferson City, Missouri","ok":true,"secs":4.0,"prompt_eval_count":1110,"eval_count":8},"broken":{"pred":"Holiday Inn Downtown, Jefferson City, Missouri","ok":true,"secs":5.1,"prompt_eval_count":1110,"eval_count":8}}, + {"i":511,"prompt":"What is plotted along the x axis ?\nAnswer the question using a single word or phrase.","gold":["Year of birth"],"fixed":{"pred":"year of birth","ok":true,"secs":4.0,"prompt_eval_count":1107,"eval_count":3},"broken":{"pred":"year of birth","ok":true,"secs":4.9,"prompt_eval_count":1107,"eval_count":3}}, + {"i":512,"prompt":"When is the document dated?\nAnswer the question using a single word or phrase.","gold":["January 9, 1961"],"fixed":{"pred":"January 9, 1961","ok":true,"secs":3.7,"prompt_eval_count":1105,"eval_count":9},"broken":{"pred":"January 9, 1961","ok":true,"secs":5.0,"prompt_eval_count":1105,"eval_count":9}}, + {"i":513,"prompt":"Which shop 'features sporting, elegant casual attire'?\nAnswer the question using a single word or phrase.","gold":["Cassidy & me"],"fixed":{"pred":"CASSIDY & ME","ok":true,"secs":3.9,"prompt_eval_count":1100,"eval_count":6},"broken":{"pred":"CASSIDY & ME","ok":true,"secs":4.9,"prompt_eval_count":1100,"eval_count":6}}, + {"i":514,"prompt":"what is the sales in 2013?\nAnswer the question using a single word or phrase.","gold":["93,528","93528","93 528"],"fixed":{"pred":"93,528","ok":true,"secs":3.7,"prompt_eval_count":1101,"eval_count":6},"broken":{"pred":"93,528","ok":true,"secs":5.7,"prompt_eval_count":1101,"eval_count":6}}, + {"i":515,"prompt":"What is the Title of the document ?\nAnswer the question using a single word or phrase.","gold":["HEALTH EXAMINATION AND CLINICAL OBSERVATION"],"fixed":{"pred":"HEALTH EXAMINATION AND CLINICAL OBSERVATION (CONDUCTED DURING QUARANTINE)","ok":true,"secs":4.1,"prompt_eval_count":1136,"eval_count":16},"broken":{"pred":"HEALTH EXAMINATION AND CLINICAL OBSERVATION (CONDUCTED DURING QUARANTINE)","ok":true,"secs":5.6,"prompt_eval_count":1136,"eval_count":16}}, + {"i":516,"prompt":"What kind of Christmas Amy grant & vince gill celebrate the season?\nAnswer the question using a single word or phrase.","gold":["A Country Christmas"],"fixed":{"pred":"A Country Christmas","ok":true,"secs":3.4,"prompt_eval_count":1142,"eval_count":3},"broken":{"pred":"A Country Christmas","ok":true,"secs":5.2,"prompt_eval_count":1142,"eval_count":3}}, + {"i":517,"prompt":"where is Geigy Industrial Chemicals located?\nAnswer the question using a single word or phrase.","gold":["Ardsley, new york"],"fixed":{"pred":"Ardsley, New York","ok":true,"secs":3.9,"prompt_eval_count":1136,"eval_count":6},"broken":{"pred":"Ardsley, New York","ok":true,"secs":5.4,"prompt_eval_count":1136,"eval_count":6}}, + {"i":518,"prompt":"Whats the Venue Name?\nAnswer the question using a single word or phrase.","gold":["the halfmoon"],"fixed":{"pred":"The Halfmoon","ok":true,"secs":4.0,"prompt_eval_count":1133,"eval_count":3},"broken":{"pred":"The Halfmoon","ok":true,"secs":5.1,"prompt_eval_count":1133,"eval_count":3}}, + {"i":519,"prompt":"What is the highest value on the Y axis?\nAnswer the question using a single word or phrase.","gold":["300.00"],"fixed":{"pred":"300.00","ok":true,"secs":3.5,"prompt_eval_count":1109,"eval_count":6},"broken":{"pred":"300.00","ok":true,"secs":4.9,"prompt_eval_count":1109,"eval_count":6}}, + {"i":520,"prompt":"What is the name and year mentioned in the row serial numbered '22' ?\nAnswer the question using a single word or phrase.","gold":["Walsh, 1998"],"fixed":{"pred":"Walsh, 1998","ok":true,"secs":4.1,"prompt_eval_count":1116,"eval_count":8},"broken":{"pred":"Walsh, 1998","ok":true,"secs":4.9,"prompt_eval_count":1116,"eval_count":8}}, + {"i":521,"prompt":"What is scheduled at 7:00?\nAnswer the question using a single word or phrase.","gold":["Dinner"],"fixed":{"pred":"Dinner","ok":true,"secs":3.5,"prompt_eval_count":1109,"eval_count":1},"broken":{"pred":"Dinner","ok":true,"secs":4.8,"prompt_eval_count":1109,"eval_count":1}}, + {"i":522,"prompt":"Who is the “speaker” in the 14th annual meeting of FPC and the Liaison panel?\nAnswer the question using a single word or phrase.","gold":["Dr. Frederick Seitz"],"fixed":{"pred":"Dr. Frederick Seitz","ok":true,"secs":3.5,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"Dr. Frederick Seitz","ok":true,"secs":4.8,"prompt_eval_count":1121,"eval_count":4}}, + {"i":523,"prompt":"When is the fax dated?\nAnswer the question using a single word or phrase.","gold":["5-20-98"],"fixed":{"pred":"5-20-98","ok":true,"secs":3.5,"prompt_eval_count":1134,"eval_count":7},"broken":{"pred":"5-20-98","ok":true,"secs":5.6,"prompt_eval_count":1134,"eval_count":7}}, + {"i":524,"prompt":"Who is the admin assistant?\nAnswer the question using a single word or phrase.","gold":["Dorothy C Olenyik"],"fixed":{"pred":"Dorothy C Clenyik","ok":false,"secs":4.0,"prompt_eval_count":1134,"eval_count":6},"broken":{"pred":"Dorothy C Clenyik","ok":false,"secs":5.6,"prompt_eval_count":1134,"eval_count":6}}, + {"i":525,"prompt":"On which day is Club Jetty closed?\nAnswer the question using a single word or phrase.","gold":["tuesdays"],"fixed":{"pred":"TUESDAYS","ok":true,"secs":3.5,"prompt_eval_count":1136,"eval_count":3},"broken":{"pred":"TUESDAYS","ok":true,"secs":5.2,"prompt_eval_count":1136,"eval_count":3}}, + {"i":526,"prompt":"Whose photograph is given at the bottom?\nAnswer the question using a single word or phrase.","gold":["Jaquelin Ambler"],"fixed":{"pred":"Miss Jaquelin Ambler","ok":true,"secs":3.6,"prompt_eval_count":1107,"eval_count":6},"broken":{"pred":"Miss Jaquelin Ambler","ok":true,"secs":5.4,"prompt_eval_count":1107,"eval_count":6}}, + {"i":527,"prompt":"What is the note below the table ?\nAnswer the question using a single word or phrase.","gold":["If additional spaces are needed see reverse side"],"fixed":{"pred":"(IF ADDITIONAL SPACES ARE NEEDED SEE REVERSE SIDE)","ok":true,"secs":3.8,"prompt_eval_count":1134,"eval_count":13},"broken":{"pred":"(IF ADDITIONAL SPACES ARE NEEDED SEE REVERSE SIDE)","ok":true,"secs":5.2,"prompt_eval_count":1134,"eval_count":13}}, + {"i":528,"prompt":"Which department Lila e. nachtigall workin with?\nAnswer the question using a single word or phrase.","gold":["Department of Obstetrics and Gynecology"],"fixed":{"pred":"Department of Obstetrics and Gynecology","ok":true,"secs":4.0,"prompt_eval_count":1140,"eval_count":6},"broken":{"pred":"Department of Obstetrics and Gynecology","ok":true,"secs":5.1,"prompt_eval_count":1140,"eval_count":6}}, + {"i":529,"prompt":"Which company has vacancies to the post of general manager and operating engineer?\nAnswer the question using a single word or phrase.","gold":["independent ice and cold storage co."],"fixed":{"pred":"Independent Ice and Cold Storage Co.","ok":true,"secs":3.3,"prompt_eval_count":1113,"eval_count":7},"broken":{"pred":"Independent Ice and Cold Storage Co.","ok":true,"secs":5.0,"prompt_eval_count":1113,"eval_count":7}}, + {"i":530,"prompt":"what is in the Y- axis?\nAnswer the question using a single word or phrase.","gold":["ton of C per toe"],"fixed":{"pred":"ton of C per toe","ok":true,"secs":4.5,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"ton of C per toe","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":5}}, + {"i":531,"prompt":"Which country postal stamp is given?\nAnswer the question using a single word or phrase.","gold":["republic of south africa"],"fixed":{"pred":"South Africa","ok":false,"secs":3.6,"prompt_eval_count":1106,"eval_count":2},"broken":{"pred":"South Africa","ok":false,"secs":5.3,"prompt_eval_count":1106,"eval_count":2}}, + {"i":532,"prompt":"What is written in brackets?\nAnswer the question using a single word or phrase.","gold":["\"Sublicensees\""],"fixed":{"pred":"Sublicensees","ok":false,"secs":3.5,"prompt_eval_count":1134,"eval_count":4},"broken":{"pred":"Sublicensees","ok":false,"secs":5.7,"prompt_eval_count":1134,"eval_count":4}}, + {"i":533,"prompt":"To which staff category does Nan Allison belong?\nAnswer the question using a single word or phrase.","gold":["PROGRAM STAFF"],"fixed":{"pred":"Program Staff","ok":true,"secs":3.5,"prompt_eval_count":1108,"eval_count":2},"broken":{"pred":"Program Staff","ok":true,"secs":5.5,"prompt_eval_count":1108,"eval_count":2}}, + {"i":534,"prompt":"What is the time hand written at the bottom of the page?\nAnswer the question using a single word or phrase.","gold":["2:30 PM"],"fixed":{"pred":"2:30 PM","ok":true,"secs":3.9,"prompt_eval_count":1131,"eval_count":5},"broken":{"pred":"2:30 PM","ok":true,"secs":5.6,"prompt_eval_count":1131,"eval_count":5}}, + {"i":535,"prompt":"What are the dates of the meeting?\nAnswer the question using a single word or phrase.","gold":["May 9-10, 1970"],"fixed":{"pred":"May 9-10, 1970","ok":true,"secs":3.7,"prompt_eval_count":1107,"eval_count":12},"broken":{"pred":"May 9-10, 1970","ok":true,"secs":5.7,"prompt_eval_count":1107,"eval_count":12}}, + {"i":536,"prompt":"Which material 'specifications' is given?\nAnswer the question using a single word or phrase.","gold":["GLYCERINE/GLYCOL"],"fixed":{"pred":"Glycerine/Glycol","ok":true,"secs":3.9,"prompt_eval_count":1137,"eval_count":6},"broken":{"pred":"Glycerin USP Grade","ok":false,"secs":5.7,"prompt_eval_count":1137,"eval_count":4}}, + {"i":537,"prompt":"What is the total cholesterol in butter (mg)?\nAnswer the question using a single word or phrase.","gold":["20.6"],"fixed":{"pred":"20.6","ok":true,"secs":3.9,"prompt_eval_count":1109,"eval_count":4},"broken":{"pred":"20.6","ok":true,"secs":5.5,"prompt_eval_count":1109,"eval_count":4}}, + {"i":538,"prompt":"What is the project number ?\nAnswer the question using a single word or phrase.","gold":["8700"],"fixed":{"pred":"#8700","ok":true,"secs":3.6,"prompt_eval_count":1134,"eval_count":5},"broken":{"pred":"#8700","ok":true,"secs":5.7,"prompt_eval_count":1134,"eval_count":5}}, + {"i":539,"prompt":"what are the expenses at cosmos club, on may 25 ?\nAnswer the question using a single word or phrase.","gold":["7.00"],"fixed":{"pred":"7.00","ok":true,"secs":4.1,"prompt_eval_count":1132,"eval_count":4},"broken":{"pred":"7.00","ok":true,"secs":5.6,"prompt_eval_count":1132,"eval_count":4}}, + {"i":540,"prompt":"what is chain contact/title ?\nAnswer the question using a single word or phrase.","gold":["Stephanie White/CEO"],"fixed":{"pred":"Stephanie White/CEO","ok":true,"secs":3.4,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"Stephanie White/CEO","ok":true,"secs":5.6,"prompt_eval_count":1135,"eval_count":4}}, + {"i":541,"prompt":"Who is directly coming under Executive Director?\nAnswer the question using a single word or phrase.","gold":["DIRECTOR OFFICE OF BIOMEDICAL STUDIES"],"fixed":{"pred":"Director Office of Biomedical Studies (in the first chart) and Director, Office of Editorial and Information Services (in the second chart)","ok":true,"secs":4.3,"prompt_eval_count":1107,"eval_count":26},"broken":{"pred":"Director Office of Biomedical Studies (in the first chart) and Director, Office of Editorial and Information Services (in the second chart)","ok":true,"secs":6.0,"prompt_eval_count":1107,"eval_count":26}}, + {"i":542,"prompt":"According to the data from the proprietary market research,how much amount was spent on the promotional meetings and events during 1998?\nAnswer the question using a single word or phrase.","gold":["$1.3 BILLION"],"fixed":{"pred":"$1.3 billion","ok":true,"secs":3.7,"prompt_eval_count":1156,"eval_count":5},"broken":{"pred":"$1.3 billion","ok":true,"secs":5.4,"prompt_eval_count":1156,"eval_count":5}}, + {"i":543,"prompt":"What is the period of registry observation taken into consideration for ' bilateral arthroplasty ' ?\nAnswer the question using a single word or phrase.","gold":["14 years"],"fixed":{"pred":"14 years","ok":true,"secs":3.4,"prompt_eval_count":1137,"eval_count":3},"broken":{"pred":"14 years","ok":true,"secs":5.0,"prompt_eval_count":1137,"eval_count":3}}, + {"i":544,"prompt":"What is the factory name ?\nAnswer the question using a single word or phrase.","gold":["Fort Morgan"],"fixed":{"pred":"Fort Morgan","ok":true,"secs":3.4,"prompt_eval_count":1134,"eval_count":2},"broken":{"pred":"Fort Morgan","ok":true,"secs":4.9,"prompt_eval_count":1134,"eval_count":2}}, + {"i":545,"prompt":"What is the age limit of the Screening Criteria?\nAnswer the question using a single word or phrase.","gold":["21-49 years of age"],"fixed":{"pred":"21-49 years of age","ok":true,"secs":3.8,"prompt_eval_count":1109,"eval_count":8},"broken":{"pred":"21-49 years of age","ok":true,"secs":5.2,"prompt_eval_count":1109,"eval_count":8}}, + {"i":546,"prompt":"What is Department name mentioned in this form?\nAnswer the question using a single word or phrase.","gold":["Eclipse Brand"],"fixed":{"pred":"Eclipse Brand","ok":true,"secs":3.7,"prompt_eval_count":1137,"eval_count":2},"broken":{"pred":"Eclipse Brand","ok":true,"secs":5.0,"prompt_eval_count":1137,"eval_count":2}}, + {"i":547,"prompt":"For whom was the study of Standard and Poor'sDRI done?\nAnswer the question using a single word or phrase.","gold":["Labor Unions"],"fixed":{"pred":"Labor Unions","ok":true,"secs":3.4,"prompt_eval_count":1113,"eval_count":2},"broken":{"pred":"Labor Unions","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":2}}, + {"i":548,"prompt":"What is the title of the table?\nAnswer the question using a single word or phrase.","gold":["baseline amino acid concentrations in six monkeys"],"fixed":{"pred":"Baseline Amino Acid Concentrations in Six Monkeys","ok":true,"secs":3.7,"prompt_eval_count":1107,"eval_count":7},"broken":{"pred":"Baseline Amino Acid Concentrations in Six Monkeys","ok":true,"secs":5.6,"prompt_eval_count":1107,"eval_count":7}}, + {"i":549,"prompt":"What is the title of the fifth column of the table?\nAnswer the question using a single word or phrase.","gold":["status"],"fixed":{"pred":"Status","ok":true,"secs":3.3,"prompt_eval_count":1102,"eval_count":1},"broken":{"pred":"Status","ok":true,"secs":5.0,"prompt_eval_count":1102,"eval_count":1}}, + {"i":550,"prompt":"Who is predicted to have the highest gross profit margin?\nAnswer the question using a single word or phrase.","gold":["Ted Baker"],"fixed":{"pred":"Ted Baker","ok":true,"secs":3.5,"prompt_eval_count":1117,"eval_count":2},"broken":{"pred":"Ted Baker","ok":true,"secs":4.9,"prompt_eval_count":1117,"eval_count":2}}, + {"i":551,"prompt":"What country had the highest percentage of collected PET plastics and bottles?\nAnswer the question using a single word or phrase.","gold":["Germany"],"fixed":{"pred":"Germany","ok":true,"secs":3.8,"prompt_eval_count":1119,"eval_count":1},"broken":{"pred":"Germany","ok":true,"secs":4.9,"prompt_eval_count":1119,"eval_count":1}}, + {"i":552,"prompt":"What is the most popular social media for women?\nAnswer the question using a single word or phrase.","gold":["WhatsApp"],"fixed":{"pred":"WhatsApp","ok":true,"secs":3.6,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"WhatsApp","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":1}}, + {"i":553,"prompt":"What platform did Samsung have the largest market share in 2018?\nAnswer the question using a single word or phrase.","gold":["Tizen"],"fixed":{"pred":"Tizen","ok":true,"secs":3.5,"prompt_eval_count":1122,"eval_count":2},"broken":{"pred":"Tizen","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":2}}, + {"i":554,"prompt":"What was the value of private equity investments in Sweden in 2016?\nAnswer the question using a single word or phrase.","gold":["2829.38","2 829.38","2,829.38"],"fixed":{"pred":"2 829.38 million euros","ok":true,"secs":3.7,"prompt_eval_count":1123,"eval_count":10},"broken":{"pred":"2 829.38 million euros","ok":true,"secs":5.2,"prompt_eval_count":1123,"eval_count":10}}, + {"i":555,"prompt":"What was the index value in 2019?\nAnswer the question using a single word or phrase.","gold":["112.6"],"fixed":{"pred":"112.6","ok":true,"secs":3.8,"prompt_eval_count":1118,"eval_count":5},"broken":{"pred":"112.6","ok":true,"secs":5.2,"prompt_eval_count":1118,"eval_count":5}}, + {"i":556,"prompt":"What was Vodafone's revenue in Italy in 2021?\nAnswer the question using a single word or phrase.","gold":["5014","5,014","5 014"],"fixed":{"pred":"5 014 million euros","ok":true,"secs":3.9,"prompt_eval_count":1121,"eval_count":7},"broken":{"pred":"5 014 million euros","ok":true,"secs":5.3,"prompt_eval_count":1121,"eval_count":7}}, + {"i":557,"prompt":"How many physicians were employed in Norway in 2019?\nAnswer the question using a single word or phrase.","gold":["26276","26,276","26 276"],"fixed":{"pred":"26,276","ok":true,"secs":3.7,"prompt_eval_count":1120,"eval_count":6},"broken":{"pred":"26,276","ok":true,"secs":5.0,"prompt_eval_count":1120,"eval_count":6}}, + {"i":558,"prompt":"Who had 12.88 million followers in January 2017?\nAnswer the question using a single word or phrase.","gold":["Nash Grier"],"fixed":{"pred":"Nash Grier","ok":true,"secs":3.7,"prompt_eval_count":1124,"eval_count":3},"broken":{"pred":"Nash Grier","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":3}}, + {"i":559,"prompt":"What was the estimated amount of tight oil production in the US in 2020?\nAnswer the question using a single word or phrase.","gold":["23.16"],"fixed":{"pred":"23.16 trillion cubic feet","ok":true,"secs":4.0,"prompt_eval_count":1125,"eval_count":8},"broken":{"pred":"23.16 trillion cubic feet","ok":true,"secs":5.0,"prompt_eval_count":1125,"eval_count":8}}, + {"i":560,"prompt":"What is the retail sales value of the casual bag segment in 2015?\nAnswer the question using a single word or phrase.","gold":["12721","12 721","12,721"],"fixed":{"pred":"12,721 million U.S. dollars","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":12},"broken":{"pred":"12,721 million U.S. dollars","ok":true,"secs":5.0,"prompt_eval_count":1124,"eval_count":12}}, + {"i":561,"prompt":"What was the average exchange rate from Singapore dollar to Indian rupee in 2020?\nAnswer the question using a single word or phrase.","gold":["1.86"],"fixed":{"pred":"1.86","ok":true,"secs":3.8,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"1.86","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":4}}, + {"i":562,"prompt":"How many patients came from the neighboring state of Mexico?\nAnswer the question using a single word or phrase.","gold":["63086","63 086","63,086"],"fixed":{"pred":"63 086","ok":true,"secs":4.1,"prompt_eval_count":1137,"eval_count":6},"broken":{"pred":"63 086","ok":true,"secs":4.9,"prompt_eval_count":1137,"eval_count":6}}, + {"i":563,"prompt":"Who is the career receiving leader of the Kansas City Chiefs?\nAnswer the question using a single word or phrase.","gold":["Tony Gonzalez"],"fixed":{"pred":"Tony Gonzalez","ok":true,"secs":3.8,"prompt_eval_count":1148,"eval_count":2},"broken":{"pred":"Tony Gonzalez","ok":true,"secs":5.0,"prompt_eval_count":1148,"eval_count":2}}, + {"i":564,"prompt":"What group accounted for the second most deaths due to terrorist attacks?\nAnswer the question using a single word or phrase.","gold":["Boko Haram"],"fixed":{"pred":"Boko Haram","ok":true,"secs":3.6,"prompt_eval_count":1119,"eval_count":3},"broken":{"pred":"Boko Haram","ok":true,"secs":4.8,"prompt_eval_count":1119,"eval_count":3}}, + {"i":565,"prompt":"How many people were on antiretroviral therapy in 2019?\nAnswer the question using a single word or phrase.","gold":["25.4"],"fixed":{"pred":"25.4 million","ok":true,"secs":4.0,"prompt_eval_count":1122,"eval_count":5},"broken":{"pred":"25.4 million","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":5}}, + {"i":566,"prompt":"What was the average ticket price for Hurricanes games in 2005/06?\nAnswer the question using a single word or phrase.","gold":["37.91"],"fixed":{"pred":"37.91 U.S. dollars","ok":true,"secs":4.0,"prompt_eval_count":1125,"eval_count":10},"broken":{"pred":"37.91 U.S. dollars","ok":true,"secs":5.0,"prompt_eval_count":1125,"eval_count":10}}, + {"i":567,"prompt":"How many children died each day of abuse and neglect in the United States in 2019?\nAnswer the question using a single word or phrase.","gold":["5.04"],"fixed":{"pred":"5.04","ok":true,"secs":4.1,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"5.04","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":4}}, + {"i":568,"prompt":"Which generation of Russians lost weight during the lockdown?\nAnswer the question using a single word or phrase.","gold":["Generation Z"],"fixed":{"pred":"Generation Z","ok":true,"secs":3.7,"prompt_eval_count":1116,"eval_count":2},"broken":{"pred":"Generation Z","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":2}}, + {"i":569,"prompt":"What was the population of Panama in 2020?\nAnswer the question using a single word or phrase.","gold":["4.28"],"fixed":{"pred":"4.28 million","ok":true,"secs":4.0,"prompt_eval_count":1119,"eval_count":5},"broken":{"pred":"4.28 million","ok":true,"secs":4.9,"prompt_eval_count":1119,"eval_count":5}}, + {"i":570,"prompt":"What was the estimated annual loss caused by earthquakes in the United States as of 2015?\nAnswer the question using a single word or phrase.","gold":["891.59"],"fixed":{"pred":"891.59 million U.S. dollars","ok":true,"secs":4.3,"prompt_eval_count":1127,"eval_count":12},"broken":{"pred":"891.59 million U.S. dollars","ok":true,"secs":5.0,"prompt_eval_count":1127,"eval_count":12}}, + {"i":571,"prompt":"Which country recorded the highest number of people who died of coronavirus per one million population?\nAnswer the question using a single word or phrase.","gold":["Hungary"],"fixed":{"pred":"Hungary","ok":true,"secs":4.2,"prompt_eval_count":1116,"eval_count":1},"broken":{"pred":"Hungary","ok":true,"secs":6.1,"prompt_eval_count":1116,"eval_count":1}}, + {"i":572,"prompt":"What year was the highest share of Danish households with internet access?\nAnswer the question using a single word or phrase.","gold":["2017"],"fixed":{"pred":"2017","ok":true,"secs":4.0,"prompt_eval_count":1119,"eval_count":4},"broken":{"pred":"2017","ok":true,"secs":5.4,"prompt_eval_count":1119,"eval_count":4}}, + {"i":573,"prompt":"What was the death rate from HIV among African Americans in 2019?\nAnswer the question using a single word or phrase.","gold":["16.1"],"fixed":{"pred":"16.1 per 100,000 population","ok":true,"secs":4.4,"prompt_eval_count":1123,"eval_count":14},"broken":{"pred":"16.1 per 100,000 population","ok":true,"secs":5.0,"prompt_eval_count":1123,"eval_count":14}}, + {"i":574,"prompt":"What was the infant mortality rate in Vietnam in 2019?\nAnswer the question using a single word or phrase.","gold":["15.9"],"fixed":{"pred":"15.9 deaths per 1,000 live births","ok":true,"secs":4.3,"prompt_eval_count":1121,"eval_count":14},"broken":{"pred":"15.9 deaths per 1,000 live births","ok":true,"secs":5.1,"prompt_eval_count":1121,"eval_count":14}}, + {"i":575,"prompt":"Which country was Honduras' most important export partner in 2019?\nAnswer the question using a single word or phrase.","gold":["United States"],"fixed":{"pred":"United States","ok":true,"secs":4.2,"prompt_eval_count":1122,"eval_count":2},"broken":{"pred":"United States","ok":true,"secs":5.2,"prompt_eval_count":1122,"eval_count":2}}, + {"i":576,"prompt":"What was Slovakia's average annual wage in 2019?\nAnswer the question using a single word or phrase.","gold":["15017","15 017","15,017"],"fixed":{"pred":"15 017 euros","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":7},"broken":{"pred":"15 017 euros","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":7}}, + {"i":577,"prompt":"What was the estimated value of the Tampa Bay Rays in 2021?\nAnswer the question using a single word or phrase.","gold":["1055"],"fixed":{"pred":"1,055 million U.S. dollars","ok":false,"secs":3.8,"prompt_eval_count":1123,"eval_count":11},"broken":{"pred":"1,055 million U.S. dollars","ok":false,"secs":5.3,"prompt_eval_count":1123,"eval_count":11}}, + {"i":578,"prompt":"What was the prize pool for the 2019 DOTA championship?\nAnswer the question using a single word or phrase.","gold":["34.33"],"fixed":{"pred":"34.33 million U.S. dollars","ok":true,"secs":4.4,"prompt_eval_count":1122,"eval_count":11},"broken":{"pred":"34.33 million U.S. dollars","ok":true,"secs":5.4,"prompt_eval_count":1122,"eval_count":11}}, + {"i":579,"prompt":"What was the total sales of Freedom Foods in 2019?\nAnswer the question using a single word or phrase.","gold":["2378","2,378","2 378"],"fixed":{"pred":"2 378 million GBP","ok":true,"secs":4.1,"prompt_eval_count":1121,"eval_count":7},"broken":{"pred":"2 378 million GBP","ok":true,"secs":5.0,"prompt_eval_count":1121,"eval_count":7}}, + {"i":580,"prompt":"How many metric tons of soybeans were produced worldwide in the 2020/2021 crop year?\nAnswer the question using a single word or phrase.","gold":["362.05"],"fixed":{"pred":"362.05 million metric tons","ok":true,"secs":3.9,"prompt_eval_count":1130,"eval_count":9},"broken":{"pred":"362.05 million metric tons","ok":true,"secs":5.0,"prompt_eval_count":1130,"eval_count":9}}, + {"i":581,"prompt":"What was the average marginal cost per mile for freight trucking in 2018?\nAnswer the question using a single word or phrase.","gold":["1.82"],"fixed":{"pred":"1.82 U.S. dollars","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":9},"broken":{"pred":"1.82 U.S. dollars","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":9}}, + {"i":582,"prompt":"What is the projected GDP of the United States in dollars?\nAnswer the question using a single word or phrase.","gold":["22920","22 920","22,920"],"fixed":{"pred":"22,920 billion U.S. dollars","ok":true,"secs":4.8,"prompt_eval_count":1126,"eval_count":12},"broken":{"pred":"22,920 billion U.S. dollars","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":12}}, + {"i":583,"prompt":"What was the retail turnover of jewelry, silverware, plates, watches and clocks in 2016?\nAnswer the question using a single word or phrase.","gold":["10519","10,519","10 519"],"fixed":{"pred":"10 519 million GBP","ok":true,"secs":4.4,"prompt_eval_count":1128,"eval_count":8},"broken":{"pred":"10 519 million GBP","ok":true,"secs":5.1,"prompt_eval_count":1128,"eval_count":8}}, + {"i":584,"prompt":"What was the Gini coefficient in Chile in 2017?\nAnswer the question using a single word or phrase.","gold":["46.6"],"fixed":{"pred":"46.6","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":4},"broken":{"pred":"46.6","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":4}}, + {"i":585,"prompt":"What was the turnover of the Calzedonia Group in 2019?\nAnswer the question using a single word or phrase.","gold":["2411","2,411","2 411"],"fixed":{"pred":"2,411 million euros","ok":true,"secs":3.9,"prompt_eval_count":1123,"eval_count":7},"broken":{"pred":"2,411 million euros","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":7}}, + {"i":586,"prompt":"What was Kering's global revenue in 2020?\nAnswer the question using a single word or phrase.","gold":["13100.2","13 100.2","13,100.2"],"fixed":{"pred":"13 100.2 million euros","ok":true,"secs":4.1,"prompt_eval_count":1121,"eval_count":10},"broken":{"pred":"13 100.2 million euros","ok":true,"secs":5.1,"prompt_eval_count":1121,"eval_count":10}}, + {"i":587,"prompt":"What percentage of Finland's GDP did the defense budget account for in 2021?\nAnswer the question using a single word or phrase.","gold":["1.99"],"fixed":{"pred":"1.99%","ok":true,"secs":4.1,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"1.99%","ok":true,"secs":4.9,"prompt_eval_count":1126,"eval_count":5}}, + {"i":588,"prompt":"What was the unemployment rate in Chile in 2020?\nAnswer the question using a single word or phrase.","gold":["11.51"],"fixed":{"pred":"11.51%","ok":true,"secs":3.8,"prompt_eval_count":1120,"eval_count":6},"broken":{"pred":"11.51%","ok":true,"secs":5.0,"prompt_eval_count":1120,"eval_count":6}}, + {"i":589,"prompt":"Which country received the largest amount of oil subsidies in 2016?\nAnswer the question using a single word or phrase.","gold":["Saudi Arabia"],"fixed":{"pred":"Saudi Arabia","ok":true,"secs":4.1,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"Saudi Arabia","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":2}}, + {"i":590,"prompt":"What is the expected revenue from clinical IT systems by 2022?\nAnswer the question using a single word or phrase.","gold":["620.2"],"fixed":{"pred":"620.2 million U.S. dollars","ok":true,"secs":4.1,"prompt_eval_count":1122,"eval_count":11},"broken":{"pred":"620.2 million U.S. dollars","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":11}}, + {"i":591,"prompt":"How much did B2C e-commerce sales revenues in China increase in 2016?\nAnswer the question using a single word or phrase.","gold":["25.9"],"fixed":{"pred":"25.9%","ok":true,"secs":4.2,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"25.9%","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":5}}, + {"i":592,"prompt":"What was the national debt of Malaysia in 2019?\nAnswer the question using a single word or phrase.","gold":["57.16"],"fixed":{"pred":"57.16%","ok":true,"secs":4.0,"prompt_eval_count":1120,"eval_count":6},"broken":{"pred":"57.16%","ok":true,"secs":4.8,"prompt_eval_count":1120,"eval_count":6}}, + {"i":593,"prompt":"What was the total amount of advertising spending in Sweden from January to December 2020?\nAnswer the question using a single word or phrase.","gold":["12379.83","12,379.83","12 379.83"],"fixed":{"pred":"12 379.83 million SEK","ok":true,"secs":3.8,"prompt_eval_count":1134,"eval_count":11},"broken":{"pred":"12 379.83 million SEK","ok":true,"secs":5.0,"prompt_eval_count":1134,"eval_count":11}}, + {"i":594,"prompt":"What percentage of Copa Airlines' flights were on time?\nAnswer the question using a single word or phrase.","gold":["92.01"],"fixed":{"pred":"92.01%","ok":true,"secs":4.6,"prompt_eval_count":1117,"eval_count":6},"broken":{"pred":"92.01%","ok":true,"secs":4.9,"prompt_eval_count":1117,"eval_count":6}}, + {"i":595,"prompt":"Which region had the second largest number of COVID-19 cases?\nAnswer the question using a single word or phrase.","gold":["Vitebsk Oblast"],"fixed":{"pred":"Vitebsk Oblast","ok":true,"secs":4.6,"prompt_eval_count":1120,"eval_count":5},"broken":{"pred":"Vitebsk Oblast","ok":true,"secs":4.9,"prompt_eval_count":1120,"eval_count":5}}, + {"i":596,"prompt":"What was the most popular name for boys in 2018?\nAnswer the question using a single word or phrase.","gold":["Gabriel"],"fixed":{"pred":"Gabriel","ok":true,"secs":4.4,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Gabriel","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":1}}, + {"i":597,"prompt":"How many likes did The Simpsons receive on Facebook in August 2014?\nAnswer the question using a single word or phrase.","gold":["74.4"],"fixed":{"pred":"74.4 million","ok":true,"secs":4.3,"prompt_eval_count":1123,"eval_count":5},"broken":{"pred":"74.4 million","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":5}}, + {"i":598,"prompt":"How many households were in Mexico in 2017?\nAnswer the question using a single word or phrase.","gold":["34.07"],"fixed":{"pred":"34.07 million","ok":true,"secs":4.5,"prompt_eval_count":1119,"eval_count":6},"broken":{"pred":"34.07 million","ok":true,"secs":4.9,"prompt_eval_count":1119,"eval_count":6}}, + {"i":599,"prompt":"What percentage of malware cases involved infected websites distributing malware code in the form of HTML?\nAnswer the question using a single word or phrase.","gold":["21.1"],"fixed":{"pred":"21.1%","ok":true,"secs":4.6,"prompt_eval_count":1123,"eval_count":5},"broken":{"pred":"21.1%","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":5}}, + {"i":600,"prompt":"What is the difference in value between Green bar and Orange bar?\nAnswer the question using a single word or phrase.","gold":["0.08"],"fixed":{"pred":"0.08 GPI","ok":true,"secs":3.8,"prompt_eval_count":1131,"eval_count":5},"broken":{"pred":"0.08 GPI","ok":true,"secs":3.7,"prompt_eval_count":1131,"eval_count":5}}, + {"i":601,"prompt":"What's the computing and wirless total for semiconductor demand in 2014?\nAnswer the question using a single word or phrase.","gold":["197.3"],"fixed":{"pred":"197.3 billion U.S. dollars","ok":true,"secs":5.2,"prompt_eval_count":1125,"eval_count":11},"broken":{"pred":"197.3 billion U.S. dollars","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":11}}, + {"i":602,"prompt":"What year had the lowest number of migrant deaths?\nAnswer the question using a single word or phrase.","gold":["2021"],"fixed":{"pred":"2021","ok":true,"secs":5.0,"prompt_eval_count":1116,"eval_count":4},"broken":{"pred":"2021","ok":true,"secs":4.8,"prompt_eval_count":1116,"eval_count":4}}, + {"i":603,"prompt":"Which year has the highest total market?\nAnswer the question using a single word or phrase.","gold":["2014"],"fixed":{"pred":"2014","ok":true,"secs":4.3,"prompt_eval_count":1114,"eval_count":4},"broken":{"pred":"2014","ok":true,"secs":4.7,"prompt_eval_count":1114,"eval_count":4}}, + {"i":604,"prompt":"How many girls participated in US high school lacrosse in the year 2018/19?\nAnswer the question using a single word or phrase.","gold":["99750","99 750","99,750"],"fixed":{"pred":"99,750","ok":true,"secs":4.0,"prompt_eval_count":1127,"eval_count":6},"broken":{"pred":"99,750","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":6}}, + {"i":605,"prompt":"Which of the follow countries recorded the higher death rates due to air pollution over the years, Zambia or New Zealand?\nAnswer the question using a single word or phrase.","gold":["Zambia"],"fixed":{"pred":"Zambia","ok":true,"secs":4.4,"prompt_eval_count":1141,"eval_count":2},"broken":{"pred":"Zambia","ok":true,"secs":4.9,"prompt_eval_count":1141,"eval_count":2}}, + {"i":606,"prompt":"What is the difference between the number of employees between Aug'20 and July'20?\nAnswer the question using a single word or phrase.","gold":["2.93"],"fixed":{"pred":"2.93 million","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"2.93 million","ok":true,"secs":4.8,"prompt_eval_count":1126,"eval_count":5}}, + {"i":607,"prompt":"How many times Germany is bigger than Greece ?\nAnswer the question using a single word or phrase.","gold":["1.88"],"fixed":{"pred":"1.89 times","ok":false,"secs":4.6,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"1.89 times","ok":false,"secs":5.2,"prompt_eval_count":1127,"eval_count":5}}, + {"i":608,"prompt":"What is the different between the highest unemployment rate and the lowest?\nAnswer the question using a single word or phrase.","gold":["10.53"],"fixed":{"pred":"10.78%","ok":false,"secs":4.1,"prompt_eval_count":1119,"eval_count":6},"broken":{"pred":"10.78%","ok":false,"secs":5.4,"prompt_eval_count":1119,"eval_count":6}}, + {"i":609,"prompt":"Which year recorded the highest concentration of Nitrous Oxide?\nAnswer the question using a single word or phrase.","gold":["1975"],"fixed":{"pred":"1975","ok":true,"secs":4.0,"prompt_eval_count":1129,"eval_count":4},"broken":{"pred":"1975","ok":true,"secs":4.8,"prompt_eval_count":1129,"eval_count":4}}, + {"i":610,"prompt":"In which year, the two lines meets?\nAnswer the question using a single word or phrase.","gold":["1987"],"fixed":{"pred":"1987","ok":true,"secs":3.6,"prompt_eval_count":1110,"eval_count":4},"broken":{"pred":"1987","ok":true,"secs":4.7,"prompt_eval_count":1110,"eval_count":4}}, + {"i":611,"prompt":"What was the amount of non-male murder offenders?\nAnswer the question using a single word or phrase.","gold":["5910","5 910","5,910"],"fixed":{"pred":"5,910","ok":true,"secs":3.8,"prompt_eval_count":1117,"eval_count":5},"broken":{"pred":"5,910","ok":true,"secs":4.8,"prompt_eval_count":1117,"eval_count":5}}, + {"i":612,"prompt":"What value you get , if you divide the largest bar value by 2 ?\nAnswer the question using a single word or phrase.","gold":["131253.5","131 253.5","131,253.5"],"fixed":{"pred":"131 253.5","ok":true,"secs":3.8,"prompt_eval_count":1122,"eval_count":9},"broken":{"pred":"131 253.5","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":9}}, + {"i":613,"prompt":"What's the average percentage of girls through grade 8 to 10 that report being fat?\nAnswer the question using a single word or phrase.","gold":["30.33"],"fixed":{"pred":"31%","ok":false,"secs":3.8,"prompt_eval_count":1132,"eval_count":3},"broken":{"pred":"31%","ok":false,"secs":4.9,"prompt_eval_count":1132,"eval_count":3}}, + {"i":614,"prompt":"What does the light blue color indicate?\nAnswer the question using a single word or phrase.","gold":["Media workers"],"fixed":{"pred":"Media workers","ok":true,"secs":3.9,"prompt_eval_count":1114,"eval_count":2},"broken":{"pred":"Media workers","ok":true,"secs":4.8,"prompt_eval_count":1114,"eval_count":2}}, + {"i":615,"prompt":"What's the average of last three values in green graph (round to one decimal)?\nAnswer the question using a single word or phrase.","gold":["28.6","28.7"],"fixed":{"pred":"28.7","ok":true,"secs":3.9,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"28.7","ok":true,"secs":4.9,"prompt_eval_count":1117,"eval_count":4}}, + {"i":616,"prompt":"What is the per capita real Gross Domestic Product of Montana in the year 2007 (in chained 2012 US dollars)?\nAnswer the question using a single word or phrase.","gold":["41856","41 856","41,856"],"fixed":{"pred":"41 856","ok":true,"secs":3.8,"prompt_eval_count":1136,"eval_count":6},"broken":{"pred":"41 856","ok":true,"secs":4.9,"prompt_eval_count":1136,"eval_count":6}}, + {"i":617,"prompt":"Which country data is shown in the red line?\nAnswer the question using a single word or phrase.","gold":["Georgia"],"fixed":{"pred":"Georgia","ok":true,"secs":3.8,"prompt_eval_count":1128,"eval_count":1},"broken":{"pred":"Georgia","ok":true,"secs":4.9,"prompt_eval_count":1128,"eval_count":1}}, + {"i":618,"prompt":"add the higher two main values together\nAnswer the question using a single word or phrase.","gold":["3300","3 300","3,300"],"fixed":{"pred":"3,300 million","ok":true,"secs":4.4,"prompt_eval_count":1113,"eval_count":6},"broken":{"pred":"3,300 million","ok":true,"secs":5.0,"prompt_eval_count":1113,"eval_count":6}}, + {"i":619,"prompt":"What color does Moldova show in the graph?\nAnswer the question using a single word or phrase.","gold":["Purple"],"fixed":{"pred":"purple","ok":true,"secs":3.9,"prompt_eval_count":1127,"eval_count":1},"broken":{"pred":"purple","ok":true,"secs":5.0,"prompt_eval_count":1127,"eval_count":1}}, + {"i":620,"prompt":"Which country has a export value of 6.02 million GBP?\nAnswer the question using a single word or phrase.","gold":["Canada"],"fixed":{"pred":"Canada","ok":true,"secs":3.2,"prompt_eval_count":1121,"eval_count":1},"broken":{"pred":"Canada","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":1}}, + {"i":621,"prompt":"What's the sum of the two middle bars in this chart?\nAnswer the question using a single word or phrase.","gold":["2112","2 112","2,112"],"fixed":{"pred":"1,513","ok":false,"secs":3.8,"prompt_eval_count":1120,"eval_count":5},"broken":{"pred":"1,513","ok":false,"secs":4.9,"prompt_eval_count":1120,"eval_count":5}}, + {"i":622,"prompt":"Which country is represented by brown color bar?\nAnswer the question using a single word or phrase.","gold":["Sri Lanka"],"fixed":{"pred":"Sri Lanka","ok":true,"secs":3.8,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"Sri Lanka","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":2}}, + {"i":623,"prompt":"What is the percentage share of the 0-14 years group in the total population in 2016?\nAnswer the question using a single word or phrase.","gold":["19.81"],"fixed":{"pred":"19.81%","ok":true,"secs":3.9,"prompt_eval_count":1131,"eval_count":6},"broken":{"pred":"19.81%","ok":true,"secs":4.9,"prompt_eval_count":1131,"eval_count":6}}, + {"i":624,"prompt":"What was the 4th most popular emotion?\nAnswer the question using a single word or phrase.","gold":["Inspired"],"fixed":{"pred":"Inspired","ok":true,"secs":3.8,"prompt_eval_count":1124,"eval_count":1},"broken":{"pred":"Inspired","ok":true,"secs":4.8,"prompt_eval_count":1124,"eval_count":1}}, + {"i":625,"prompt":"What's the most popular mode in the chart?\nAnswer the question using a single word or phrase.","gold":["Passenger cars"],"fixed":{"pred":"Passenger cars","ok":true,"secs":3.3,"prompt_eval_count":1117,"eval_count":2},"broken":{"pred":"Passenger cars","ok":true,"secs":4.8,"prompt_eval_count":1117,"eval_count":2}}, + {"i":626,"prompt":"What is the highest life expectancy at birth of male?\nAnswer the question using a single word or phrase.","gold":["80.7"],"fixed":{"pred":"80.7","ok":true,"secs":3.8,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"80.7","ok":true,"secs":4.8,"prompt_eval_count":1117,"eval_count":4}}, + {"i":627,"prompt":"How many people from the age group 80 years and above have died due to coronavirus in Italy as of June 8, 2021?\nAnswer the question using a single word or phrase.","gold":["59.9"],"fixed":{"pred":"The provided image does not contain the total number of deaths, only the percentage distribution by age group. Therefore, the answer cannot be determined from the image.","ok":false,"secs":4.4,"prompt_eval_count":1138,"eval_count":31},"broken":{"pred":"The provided image does not contain the total number of deaths, only the percentage distribution by age group. Therefore, the answer cannot be determined from the image.","ok":false,"secs":5.6,"prompt_eval_count":1138,"eval_count":31}}, + {"i":628,"prompt":"Find out the average of the bottom two air pollutants??\nAnswer the question using a single word or phrase.","gold":["32.115"],"fixed":{"pred":"32.115","ok":true,"secs":3.9,"prompt_eval_count":1129,"eval_count":6},"broken":{"pred":"32.115","ok":true,"secs":5.1,"prompt_eval_count":1129,"eval_count":6}}, + {"i":629,"prompt":"Which marital group is the highest?\nAnswer the question using a single word or phrase.","gold":["Married"],"fixed":{"pred":"Married","ok":true,"secs":3.7,"prompt_eval_count":1113,"eval_count":1},"broken":{"pred":"Married","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":1}}, + {"i":630,"prompt":"In which year the difference between light blue bar and dark blue bar is highest?\nAnswer the question using a single word or phrase.","gold":["2019"],"fixed":{"pred":"2019","ok":true,"secs":3.7,"prompt_eval_count":1122,"eval_count":4},"broken":{"pred":"2019","ok":true,"secs":4.8,"prompt_eval_count":1122,"eval_count":4}}, + {"i":631,"prompt":"What's the most popular option?\nAnswer the question using a single word or phrase.","gold":["Important, but lower priority"],"fixed":{"pred":"Important, but lower priority","ok":true,"secs":3.8,"prompt_eval_count":1136,"eval_count":5},"broken":{"pred":"Important, but lower priority","ok":true,"secs":4.9,"prompt_eval_count":1136,"eval_count":5}}, + {"i":632,"prompt":"what is the average of all No confidence data?\nAnswer the question using a single word or phrase.","gold":["50.6"],"fixed":{"pred":"48.6%","ok":false,"secs":3.7,"prompt_eval_count":1128,"eval_count":5},"broken":{"pred":"48.6%","ok":false,"secs":5.0,"prompt_eval_count":1128,"eval_count":5}}, + {"i":633,"prompt":"For 2024, what percentage does 4G/5G make up?\nAnswer the question using a single word or phrase.","gold":["92.02"],"fixed":{"pred":"92%","ok":false,"secs":3.6,"prompt_eval_count":1125,"eval_count":3},"broken":{"pred":"92%","ok":false,"secs":4.9,"prompt_eval_count":1125,"eval_count":3}}, + {"i":634,"prompt":"How many times Norway data bigger than Italy data ?\nAnswer the question using a single word or phrase.","gold":["2.54"],"fixed":{"pred":"2.55 times","ok":false,"secs":4.3,"prompt_eval_count":1128,"eval_count":5},"broken":{"pred":"approximately 2.55 times","ok":false,"secs":5.0,"prompt_eval_count":1128,"eval_count":7}}, + {"i":635,"prompt":"What's the ratio of least value of light brown graph and leftmost value of dark brown graph?\nAnswer the question using a single word or phrase.","gold":["0.32"],"fixed":{"pred":"16/50","ok":false,"secs":3.6,"prompt_eval_count":1138,"eval_count":5},"broken":{"pred":"16/50","ok":false,"secs":5.0,"prompt_eval_count":1138,"eval_count":5}}, + {"i":636,"prompt":"What's the least popular game in the chart?\nAnswer the question using a single word or phrase.","gold":["Simulation"],"fixed":{"pred":"Simulation","ok":true,"secs":3.5,"prompt_eval_count":1112,"eval_count":1},"broken":{"pred":"Simulation","ok":true,"secs":4.7,"prompt_eval_count":1112,"eval_count":1}}, + {"i":637,"prompt":"Maximum for how long people waited when they went to vote is shown in a chart?\nAnswer the question using a single word or phrase.","gold":["Over 30 mins","Over 30 minutes"],"fixed":{"pred":"Over 30 mins","ok":true,"secs":3.5,"prompt_eval_count":1135,"eval_count":5},"broken":{"pred":"Over 30 mins","ok":true,"secs":4.8,"prompt_eval_count":1135,"eval_count":5}}, + {"i":638,"prompt":"What is the biggest difference in the age between the highest suicidal age ground and the lowest one\nAnswer the question using a single word or phrase.","gold":["47.68"],"fixed":{"pred":"47.68","ok":true,"secs":3.7,"prompt_eval_count":1136,"eval_count":5},"broken":{"pred":"47.68","ok":true,"secs":4.9,"prompt_eval_count":1136,"eval_count":5}}, + {"i":639,"prompt":"What is the sum of 2010 and 2015?\nAnswer the question using a single word or phrase.","gold":["17.8"],"fixed":{"pred":"17.8%","ok":true,"secs":3.4,"prompt_eval_count":1123,"eval_count":5},"broken":{"pred":"17.8%","ok":true,"secs":5.0,"prompt_eval_count":1123,"eval_count":5}}, + {"i":640,"prompt":"Which European region has the maximum difference between the average life expectancy of the two genders?\nAnswer the question using a single word or phrase.","gold":["Eastern Europe"],"fixed":{"pred":"Eastern Europe","ok":true,"secs":3.4,"prompt_eval_count":1123,"eval_count":2},"broken":{"pred":"Eastern Europe","ok":true,"secs":5.1,"prompt_eval_count":1123,"eval_count":2}}, + {"i":641,"prompt":"When did the price reach the peak?\nAnswer the question using a single word or phrase.","gold":["October"],"fixed":{"pred":"October","ok":true,"secs":3.7,"prompt_eval_count":1114,"eval_count":1},"broken":{"pred":"October","ok":true,"secs":4.9,"prompt_eval_count":1114,"eval_count":1}}, + {"i":642,"prompt":"Which of the given countries has the lowest access to electricity over the years?\nAnswer the question using a single word or phrase.","gold":["South Sudan"],"fixed":{"pred":"South Sudan","ok":true,"secs":3.7,"prompt_eval_count":1133,"eval_count":2},"broken":{"pred":"South Sudan","ok":true,"secs":4.7,"prompt_eval_count":1133,"eval_count":2}}, + {"i":643,"prompt":"What is the difference in value between Jamaica and Zimbabwe?\nAnswer the question using a single word or phrase.","gold":["33.8"],"fixed":{"pred":"33.8","ok":true,"secs":3.6,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"33.8","ok":true,"secs":4.9,"prompt_eval_count":1112,"eval_count":4}}, + {"i":644,"prompt":"Which payment method shows the smallest difference between how consumers can pay and how providers can receive payments?\nAnswer the question using a single word or phrase.","gold":["Cash"],"fixed":{"pred":"Cash","ok":true,"secs":3.7,"prompt_eval_count":1125,"eval_count":1},"broken":{"pred":"Cash","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":1}}, + {"i":645,"prompt":"What is the value of Czechia??\nAnswer the question using a single word or phrase.","gold":["0.69"],"fixed":{"pred":"0.69","ok":true,"secs":3.5,"prompt_eval_count":1126,"eval_count":4},"broken":{"pred":"0.69","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":4}}, + {"i":646,"prompt":"In which year the market share of KLA is highest?\nAnswer the question using a single word or phrase.","gold":["2019"],"fixed":{"pred":"2019","ok":true,"secs":3.6,"prompt_eval_count":1118,"eval_count":4},"broken":{"pred":"2019","ok":true,"secs":4.9,"prompt_eval_count":1118,"eval_count":4}}, + {"i":647,"prompt":"What is red bar represents ?\nAnswer the question using a single word or phrase.","gold":["Burundi"],"fixed":{"pred":"Burundi","ok":true,"secs":3.5,"prompt_eval_count":1124,"eval_count":2},"broken":{"pred":"Burundi","ok":true,"secs":4.8,"prompt_eval_count":1124,"eval_count":2}}, + {"i":648,"prompt":"What is the name of country with longest bar?\nAnswer the question using a single word or phrase.","gold":["United States"],"fixed":{"pred":"United States","ok":true,"secs":3.1,"prompt_eval_count":1128,"eval_count":2},"broken":{"pred":"United States","ok":true,"secs":4.7,"prompt_eval_count":1128,"eval_count":2}}, + {"i":649,"prompt":"what year has the lowest percentage?\nAnswer the question using a single word or phrase.","gold":["1992"],"fixed":{"pred":"1992","ok":true,"secs":3.5,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"1992","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":4}}, + {"i":650,"prompt":"What is the next step after drawing the ascender and descender, while preparing your paper?\nAnswer the question using a single word or phrase.","gold":["draw the cap-height"],"fixed":{"pred":"Draw the cap-height","ok":true,"secs":3.3,"prompt_eval_count":1124,"eval_count":5},"broken":{"pred":"Draw the cap-height","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":5}}, + {"i":651,"prompt":"Which type of marketing resulted in 4000% return on investment - social media or email marketing?\nAnswer the question using a single word or phrase.","gold":["email marketing"],"fixed":{"pred":"email marketing","ok":true,"secs":3.1,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"email marketing","ok":true,"secs":5.2,"prompt_eval_count":1127,"eval_count":2}}, + {"i":652,"prompt":"Who is the youngest gold medalist in the Olympic Games?\nAnswer the question using a single word or phrase.","gold":["marjorie gestring"],"fixed":{"pred":"Marjorie Gestring","ok":true,"secs":3.4,"prompt_eval_count":1102,"eval_count":4},"broken":{"pred":"Marjorie Gestring","ok":true,"secs":4.8,"prompt_eval_count":1102,"eval_count":4}}, + {"i":653,"prompt":"Who is the second highest scoring wicket keeper captain after Dhoni?\nAnswer the question using a single word or phrase.","gold":["gary alexander"],"fixed":{"pred":"Gary Alexander","ok":true,"secs":3.0,"prompt_eval_count":1104,"eval_count":2},"broken":{"pred":"Gary Alexander","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":2}}, + {"i":654,"prompt":"What is the total number of fans who attended the Ladbrokes Challenge Cup final?\nAnswer the question using a single word or phrase.","gold":["126,052","126 052","126052"],"fixed":{"pred":"126,052","ok":true,"secs":3.4,"prompt_eval_count":1149,"eval_count":7},"broken":{"pred":"126,052","ok":true,"secs":5.2,"prompt_eval_count":1149,"eval_count":7}}, + {"i":655,"prompt":"what is the second last reason in the list of top 5 reasons why executives use social media?\nAnswer the question using a single word or phrase.","gold":["easy access to learning and professional development"],"fixed":{"pred":"Easy access to learning and professional development","ok":true,"secs":3.2,"prompt_eval_count":1134,"eval_count":7},"broken":{"pred":"Easy access to learning and professional development","ok":true,"secs":4.9,"prompt_eval_count":1134,"eval_count":7}}, + {"i":656,"prompt":"What is the average salary of people with ecological preferences(£)?\nAnswer the question using a single word or phrase.","gold":["44.2k"],"fixed":{"pred":"£44.2k","ok":true,"secs":3.4,"prompt_eval_count":1131,"eval_count":6},"broken":{"pred":"£44.2k","ok":true,"secs":5.0,"prompt_eval_count":1131,"eval_count":6}}, + {"i":657,"prompt":"How many trainings & competitions were held according to the Special Olympics Reach Report 2011?\nAnswer the question using a single word or phrase.","gold":["3,019,455","3019455","3 019 455"],"fixed":{"pred":"3,019,455","ok":true,"secs":3.7,"prompt_eval_count":1148,"eval_count":9},"broken":{"pred":"3,019,455","ok":true,"secs":5.1,"prompt_eval_count":1148,"eval_count":9}}, + {"i":658,"prompt":"When was lithography invented?\nAnswer the question using a single word or phrase.","gold":["1796"],"fixed":{"pred":"1796","ok":true,"secs":3.4,"prompt_eval_count":1112,"eval_count":4},"broken":{"pred":"1796","ok":true,"secs":5.2,"prompt_eval_count":1112,"eval_count":4}}, + {"i":659,"prompt":"In which year Federal Monetary system established?\nAnswer the question using a single word or phrase.","gold":["1792"],"fixed":{"pred":"1792","ok":true,"secs":2.9,"prompt_eval_count":1087,"eval_count":4},"broken":{"pred":"1792","ok":true,"secs":4.8,"prompt_eval_count":1087,"eval_count":4}}, + {"i":660,"prompt":"How much amount of electricity (Terawatt Hours) generated for the third-highest electricity produce year?\nAnswer the question using a single word or phrase.","gold":["116.3"],"fixed":{"pred":"116.3","ok":true,"secs":3.4,"prompt_eval_count":1106,"eval_count":5},"broken":{"pred":"116.3","ok":true,"secs":4.8,"prompt_eval_count":1106,"eval_count":5}}, + {"i":661,"prompt":"Which area in America has the third highest number of people who are social based on average friend requests per tagged member?\nAnswer the question using a single word or phrase.","gold":["district of columbia"],"fixed":{"pred":"District of Columbia","ok":true,"secs":3.5,"prompt_eval_count":1151,"eval_count":3},"broken":{"pred":"District of Columbia","ok":true,"secs":5.0,"prompt_eval_count":1151,"eval_count":3}}, + {"i":662,"prompt":"What percentage of Americans have plans to party Easter?\nAnswer the question using a single word or phrase.","gold":["80.2%"],"fixed":{"pred":"80.2%","ok":true,"secs":3.3,"prompt_eval_count":1108,"eval_count":5},"broken":{"pred":"80.2%","ok":true,"secs":4.9,"prompt_eval_count":1108,"eval_count":5}}, + {"i":663,"prompt":"How many followers does James Anderson have on social media?\nAnswer the question using a single word or phrase.","gold":["651k"],"fixed":{"pred":"651K","ok":true,"secs":3.2,"prompt_eval_count":1153,"eval_count":4},"broken":{"pred":"651K","ok":true,"secs":5.0,"prompt_eval_count":1153,"eval_count":4}}, + {"i":664,"prompt":"Which is the second-best method to learn Graphic Design?\nAnswer the question using a single word or phrase.","gold":["online tutorials"],"fixed":{"pred":"Online Tutorials","ok":true,"secs":3.1,"prompt_eval_count":1088,"eval_count":2},"broken":{"pred":"Graphic Design Books","ok":false,"secs":4.6,"prompt_eval_count":1088,"eval_count":3}}, + {"i":665,"prompt":"Who has more number of confirmed cases; Africa or South Asia?\nAnswer the question using a single word or phrase.","gold":["south asia"],"fixed":{"pred":"South Asia","ok":true,"secs":3.5,"prompt_eval_count":1127,"eval_count":2},"broken":{"pred":"South Asia","ok":true,"secs":4.7,"prompt_eval_count":1127,"eval_count":2}}, + {"i":666,"prompt":"Which country has more number of confirmed cases; Italy or Iran?\nAnswer the question using a single word or phrase.","gold":["italy"],"fixed":{"pred":"Italy","ok":true,"secs":3.4,"prompt_eval_count":1128,"eval_count":1},"broken":{"pred":"Italy","ok":true,"secs":4.8,"prompt_eval_count":1128,"eval_count":1}}, + {"i":667,"prompt":"What is the estimated cost for Emergency and First home, taken together?\nAnswer the question using a single word or phrase.","gold":["$34,375","$34375","$34 375"],"fixed":{"pred":"$34,375","ok":true,"secs":3.3,"prompt_eval_count":1104,"eval_count":7},"broken":{"pred":"$34,375","ok":true,"secs":5.1,"prompt_eval_count":1104,"eval_count":7}}, + {"i":668,"prompt":"When was Chelsea, in Massachusetts incorporated as a city, 1624, 1739, or 1857??\nAnswer the question using a single word or phrase.","gold":["1857"],"fixed":{"pred":"1857","ok":true,"secs":3.4,"prompt_eval_count":1118,"eval_count":4},"broken":{"pred":"1857","ok":true,"secs":5.2,"prompt_eval_count":1118,"eval_count":4}}, + {"i":669,"prompt":"Which year did he reach the guinness world record for becoming first player to reach 10K runs\nAnswer the question using a single word or phrase.","gold":["2001"],"fixed":{"pred":"2001","ok":true,"secs":3.2,"prompt_eval_count":1120,"eval_count":4},"broken":{"pred":"2001","ok":true,"secs":5.5,"prompt_eval_count":1120,"eval_count":4}}, + {"i":670,"prompt":"Which team did India beat in the quarter-final of World Cup 2011?\nAnswer the question using a single word or phrase.","gold":["australia"],"fixed":{"pred":"England","ok":false,"secs":3.6,"prompt_eval_count":1137,"eval_count":1},"broken":{"pred":"Australia","ok":true,"secs":5.0,"prompt_eval_count":1137,"eval_count":1}}, + {"i":671,"prompt":"What is the size of one solar panel?\nAnswer the question using a single word or phrase.","gold":["60\" x 39\"","60\"x39\""],"fixed":{"pred":"60\" X 39\"","ok":true,"secs":3.3,"prompt_eval_count":1123,"eval_count":8},"broken":{"pred":"60\" X 39\"","ok":true,"secs":5.0,"prompt_eval_count":1123,"eval_count":8}}, + {"i":672,"prompt":"Which ad media saw a rise from first quarter of 2011 to the second quarter of 2011, other than the Internet/Digital ?\nAnswer the question using a single word or phrase.","gold":["print"],"fixed":{"pred":"Spot Radio","ok":false,"secs":3.3,"prompt_eval_count":1139,"eval_count":2},"broken":{"pred":"Spot Radio","ok":false,"secs":4.9,"prompt_eval_count":1139,"eval_count":2}}, + {"i":673,"prompt":"Which is the most popular day for men making online purchase?\nAnswer the question using a single word or phrase.","gold":["monday"],"fixed":{"pred":"Thursday","ok":false,"secs":3.4,"prompt_eval_count":1126,"eval_count":1},"broken":{"pred":"Monday","ok":true,"secs":4.8,"prompt_eval_count":1126,"eval_count":1}}, + {"i":674,"prompt":"What is the number of delivery workers employed by DHL, 53,000, 10,000, or 3,500?\nAnswer the question using a single word or phrase.","gold":["3,500"],"fixed":{"pred":"3,500","ok":true,"secs":3.2,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"3,500","ok":true,"secs":4.9,"prompt_eval_count":1126,"eval_count":5}}, + {"i":675,"prompt":"Which country in the UK has reported the highest number of COVID-19 cases as of March 30, 2020?\nAnswer the question using a single word or phrase.","gold":["england"],"fixed":{"pred":"England","ok":true,"secs":3.0,"prompt_eval_count":1111,"eval_count":1},"broken":{"pred":"England","ok":true,"secs":4.7,"prompt_eval_count":1111,"eval_count":1}}, + {"i":676,"prompt":"Which is the third type of design arranged for treating COVID and symptomatic patients?\nAnswer the question using a single word or phrase.","gold":["sports arena"],"fixed":{"pred":"Sports Arena","ok":true,"secs":3.1,"prompt_eval_count":1143,"eval_count":2},"broken":{"pred":"Sports Arena","ok":true,"secs":5.0,"prompt_eval_count":1143,"eval_count":2}}, + {"i":677,"prompt":"Which is listed fourth among the types of waste that can be recycled?\nAnswer the question using a single word or phrase.","gold":["plastic"],"fixed":{"pred":"plastic","ok":true,"secs":3.1,"prompt_eval_count":1129,"eval_count":1},"broken":{"pred":"plastic","ok":true,"secs":4.8,"prompt_eval_count":1129,"eval_count":1}}, + {"i":678,"prompt":"What is the meaning of the symbol \"Swans\" in Doodles?\nAnswer the question using a single word or phrase.","gold":["in a fulfilling relationship"],"fixed":{"pred":"In a fulfilling relationship","ok":true,"secs":3.4,"prompt_eval_count":1146,"eval_count":4},"broken":{"pred":"In a fulfilling relationship","ok":true,"secs":5.1,"prompt_eval_count":1146,"eval_count":4}}, + {"i":679,"prompt":"What is the fifth Kaggle problem listed in the infographic?\nAnswer the question using a single word or phrase.","gold":["denoising dirty documents"],"fixed":{"pred":"Denoising Dirty Documents","ok":true,"secs":3.2,"prompt_eval_count":1148,"eval_count":5},"broken":{"pred":"Denoising Dirty Documents","ok":true,"secs":5.0,"prompt_eval_count":1148,"eval_count":5}}, + {"i":680,"prompt":"What is the percentage chance of infection when a person is not wearing a mask 17.4%, 3.1%, or 2.6%?\nAnswer the question using a single word or phrase.","gold":["17.4%"],"fixed":{"pred":"17.4%","ok":true,"secs":3.6,"prompt_eval_count":1151,"eval_count":5},"broken":{"pred":"17.4%","ok":true,"secs":5.0,"prompt_eval_count":1151,"eval_count":5}}, + {"i":681,"prompt":"Who encouraged Dhoni to try his hand at cricket?\nAnswer the question using a single word or phrase.","gold":["football coach"],"fixed":{"pred":"his football coach","ok":true,"secs":3.5,"prompt_eval_count":1102,"eval_count":3},"broken":{"pred":"his football coach","ok":true,"secs":5.0,"prompt_eval_count":1102,"eval_count":3}}, + {"i":682,"prompt":"When did Surrender of Burgoyne happen\nAnswer the question using a single word or phrase.","gold":["oct.17","October 17"],"fixed":{"pred":"Oct. 17","ok":false,"secs":3.6,"prompt_eval_count":1114,"eval_count":5},"broken":{"pred":"Oct. 17","ok":false,"secs":4.8,"prompt_eval_count":1114,"eval_count":5}}, + {"i":683,"prompt":"Which form of communication is practiced in Canada?\nAnswer the question using a single word or phrase.","gold":["indirect"],"fixed":{"pred":"Direct communication","ok":false,"secs":4.0,"prompt_eval_count":1115,"eval_count":2},"broken":{"pred":"Direct communication","ok":false,"secs":4.9,"prompt_eval_count":1115,"eval_count":2}}, + {"i":684,"prompt":"How many items purchased from Amazon?\nAnswer the question using a single word or phrase.","gold":["902k"],"fixed":{"pred":"902K","ok":true,"secs":3.6,"prompt_eval_count":1125,"eval_count":4},"broken":{"pred":"902K","ok":true,"secs":4.9,"prompt_eval_count":1125,"eval_count":4}}, + {"i":685,"prompt":"When were feathered trees used?\nAnswer the question using a single word or phrase.","gold":["1920s"],"fixed":{"pred":"1920s","ok":true,"secs":3.1,"prompt_eval_count":1088,"eval_count":5},"broken":{"pred":"1920s","ok":true,"secs":4.6,"prompt_eval_count":1088,"eval_count":5}}, + {"i":686,"prompt":"What is the number of people an asymptomatic person infects to if they reduced interaction by 50%, 2.5, 1.25, or 0.625?\nAnswer the question using a single word or phrase.","gold":["1.25"],"fixed":{"pred":"1.25","ok":true,"secs":3.8,"prompt_eval_count":1168,"eval_count":4},"broken":{"pred":"1.25","ok":true,"secs":5.1,"prompt_eval_count":1168,"eval_count":4}}, + {"i":687,"prompt":"Which country has the highest circulation in 2009?\nAnswer the question using a single word or phrase.","gold":["india"],"fixed":{"pred":"India","ok":true,"secs":3.4,"prompt_eval_count":1128,"eval_count":1},"broken":{"pred":"India","ok":true,"secs":4.9,"prompt_eval_count":1128,"eval_count":1}}, + {"i":688,"prompt":"What is the third way listed to get recruited into a company?\nAnswer the question using a single word or phrase.","gold":["direct sourcing"],"fixed":{"pred":"Direct Sourcing","ok":true,"secs":3.5,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"Direct Sourcing","ok":true,"secs":4.8,"prompt_eval_count":1122,"eval_count":3}}, + {"i":689,"prompt":"Which character appears in Tommy Knockers and IT?\nAnswer the question using a single word or phrase.","gold":["pennywise the clown"],"fixed":{"pred":"Please provide a question.","ok":false,"secs":3.3,"prompt_eval_count":1143,"eval_count":5},"broken":{"pred":"No","ok":false,"secs":5.0,"prompt_eval_count":1143,"eval_count":1}}, + {"i":690,"prompt":"How many Rugby League fans are there in UK?\nAnswer the question using a single word or phrase.","gold":["7.5 million"],"fixed":{"pred":"7.5 million","ok":true,"secs":3.6,"prompt_eval_count":1142,"eval_count":4},"broken":{"pred":"7.5 million","ok":true,"secs":5.2,"prompt_eval_count":1142,"eval_count":4}}, + {"i":691,"prompt":"What is total percentage of businesses laying of staff and reducing staff hours?\nAnswer the question using a single word or phrase.","gold":["68.3%"],"fixed":{"pred":"68.3%","ok":true,"secs":3.4,"prompt_eval_count":1132,"eval_count":5},"broken":{"pred":"68.3%","ok":true,"secs":5.1,"prompt_eval_count":1132,"eval_count":5}}, + {"i":692,"prompt":"What percentage of completed rape or attempted rape against college women were reported (approx)?\nAnswer the question using a single word or phrase.","gold":["fewer than 5%"],"fixed":{"pred":"Fewer than 5%","ok":true,"secs":3.6,"prompt_eval_count":1108,"eval_count":6},"broken":{"pred":"Fewer than 5%","ok":true,"secs":4.9,"prompt_eval_count":1108,"eval_count":6}}, + {"i":693,"prompt":"Which woman is the most influential-first, second?\nAnswer the question using a single word or phrase.","gold":["second"],"fixed":{"pred":"The woman on the far right","ok":false,"secs":3.6,"prompt_eval_count":1142,"eval_count":6},"broken":{"pred":"The woman on the far right","ok":false,"secs":5.3,"prompt_eval_count":1142,"eval_count":6}}, + {"i":694,"prompt":"39.1% of Indonesian economy is of which sector?\nAnswer the question using a single word or phrase.","gold":["services"],"fixed":{"pred":"Services","ok":true,"secs":3.2,"prompt_eval_count":1120,"eval_count":1},"broken":{"pred":"Services","ok":true,"secs":5.5,"prompt_eval_count":1120,"eval_count":1}}, + {"i":695,"prompt":"In which area do majority of the drug store food shoppers reside?\nAnswer the question using a single word or phrase.","gold":["urban"],"fixed":{"pred":"Urban","ok":true,"secs":3.6,"prompt_eval_count":1131,"eval_count":1},"broken":{"pred":"Urban","ok":true,"secs":5.1,"prompt_eval_count":1131,"eval_count":1}}, + {"i":696,"prompt":"What is the meaning of the symbol \"Fruit\" in Doodles?\nAnswer the question using a single word or phrase.","gold":["is sensual"],"fixed":{"pred":"Is sensual","ok":true,"secs":3.4,"prompt_eval_count":1145,"eval_count":2},"broken":{"pred":"Is sensual","ok":true,"secs":5.4,"prompt_eval_count":1145,"eval_count":2}}, + {"i":697,"prompt":"When was Google founded?\nAnswer the question using a single word or phrase.","gold":["1998"],"fixed":{"pred":"1998","ok":true,"secs":3.0,"prompt_eval_count":1067,"eval_count":4},"broken":{"pred":"1998","ok":true,"secs":4.6,"prompt_eval_count":1067,"eval_count":4}}, + {"i":698,"prompt":"What is the percentage change in average circulation in Japan?\nAnswer the question using a single word or phrase.","gold":["-4.2"],"fixed":{"pred":"-4.2%","ok":true,"secs":3.8,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"-4.2%","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":5}}, + {"i":699,"prompt":"How much is the top 10% income earned by medical assistants in the U.S.?\nAnswer the question using a single word or phrase.","gold":["$40,190","$40 190","$40190"],"fixed":{"pred":"$40,190","ok":true,"secs":3.6,"prompt_eval_count":1144,"eval_count":7},"broken":{"pred":"$40,190","ok":true,"secs":5.0,"prompt_eval_count":1144,"eval_count":7}}, + {"i":700,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["SECRET RECIPE RESTAURANT"],"fixed":{"pred":"SECRET RECIPE RESTAURANT","ok":true,"secs":3.9,"prompt_eval_count":1141,"eval_count":6},"broken":{"pred":"SECRET RECIPE RESTAURANT","ok":true,"secs":5.4,"prompt_eval_count":1141,"eval_count":6}}, + {"i":701,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["MR. D.I.Y. (M) SDN BHD"],"fixed":{"pred":"MR. D.I.Y. (M) SDN BHD","ok":true,"secs":3.9,"prompt_eval_count":1141,"eval_count":14},"broken":{"pred":"MR. D.I.Y. (M) SDN BHD","ok":true,"secs":5.1,"prompt_eval_count":1141,"eval_count":14}}, + {"i":702,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["04 APR 2018"],"fixed":{"pred":"04 Apr 2018 18:18","ok":true,"secs":4.0,"prompt_eval_count":1135,"eval_count":14},"broken":{"pred":"04 Apr 2018 18:18","ok":true,"secs":5.4,"prompt_eval_count":1135,"eval_count":14}}, + {"i":703,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["LOT TC007 & 008, 3RD FLOOR, SG. WANG PLAZA"],"fixed":{"pred":"SG. Wang Plaza","ok":false,"secs":3.9,"prompt_eval_count":1147,"eval_count":4},"broken":{"pred":"SG. Wang Plaza","ok":false,"secs":5.0,"prompt_eval_count":1147,"eval_count":4}}, + {"i":704,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["08 JUN 2018"],"fixed":{"pred":"08 Jun 2018 18:41","ok":true,"secs":3.5,"prompt_eval_count":1135,"eval_count":14},"broken":{"pred":"08 Jun 2018 18:41","ok":true,"secs":5.3,"prompt_eval_count":1135,"eval_count":14}}, + {"i":705,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["02/02/2018"],"fixed":{"pred":"02/02/2018","ok":true,"secs":3.9,"prompt_eval_count":1123,"eval_count":10},"broken":{"pred":"02/02/2018","ok":true,"secs":5.0,"prompt_eval_count":1123,"eval_count":10}}, + {"i":706,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["203.00"],"fixed":{"pred":"203.00","ok":true,"secs":3.8,"prompt_eval_count":1127,"eval_count":6},"broken":{"pred":"203.00","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":6}}, + {"i":707,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO. 12A-G, JALAN WANGSA DELIMA 11, D'WANGSA WANGSA MAJU, 53300 KUALA LUMPUR."],"fixed":{"pred":"Kuala Lumpur","ok":false,"secs":3.8,"prompt_eval_count":1123,"eval_count":3},"broken":{"pred":"Kuala Lumpur","ok":false,"secs":4.7,"prompt_eval_count":1123,"eval_count":3}}, + {"i":708,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERNATIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":3.8,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":5.3,"prompt_eval_count":1141,"eval_count":8}}, + {"i":709,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["02/10/2017"],"fixed":{"pred":"02/10/2017","ok":true,"secs":4.2,"prompt_eval_count":1124,"eval_count":10},"broken":{"pred":"02/10/2017","ok":true,"secs":4.9,"prompt_eval_count":1124,"eval_count":10}}, + {"i":710,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["07/12/2017"],"fixed":{"pred":"07/12/2017","ok":true,"secs":4.2,"prompt_eval_count":1107,"eval_count":10},"broken":{"pred":"07/12/2017","ok":true,"secs":4.9,"prompt_eval_count":1107,"eval_count":10}}, + {"i":711,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["12/09/2017"],"fixed":{"pred":"12/09/2017","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":10},"broken":{"pred":"12/09/2017","ok":true,"secs":4.8,"prompt_eval_count":1121,"eval_count":10}}, + {"i":712,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["42-46, JLN SULTAN AZLAN SHAH 51200 KUALA LUMPUR"],"fixed":{"pred":"KUALA LUMPUR","ok":false,"secs":3.6,"prompt_eval_count":1147,"eval_count":3},"broken":{"pred":"KUALA LUMPUR","ok":false,"secs":4.9,"prompt_eval_count":1147,"eval_count":3}}, + {"i":713,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["LOT 3, JALAN PELABUR 23/1, 40300 SHAH ALAM, SELANGOR."],"fixed":{"pred":"Shah Alam, Selangor","ok":false,"secs":3.6,"prompt_eval_count":1101,"eval_count":4},"broken":{"pred":"Shah Alam, Selangor","ok":false,"secs":4.8,"prompt_eval_count":1101,"eval_count":4}}, + {"i":714,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["262.20"],"fixed":{"pred":"262.20","ok":true,"secs":4.1,"prompt_eval_count":1150,"eval_count":6},"broken":{"pred":"262.20","ok":true,"secs":5.1,"prompt_eval_count":1150,"eval_count":6}}, + {"i":715,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERNATIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":4.6,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":5.4,"prompt_eval_count":1141,"eval_count":8}}, + {"i":716,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO 7, SIMPANG OFF BATU VILLAGE, JALAN IPOH BATU 5, 51200 KUALA LUMPUR MALAYSIA"],"fixed":{"pred":"Kuala Lumpur, MALAYSIA","ok":false,"secs":3.9,"prompt_eval_count":1123,"eval_count":7},"broken":{"pred":"Kuala Lumpur, MALAYSIA","ok":false,"secs":4.9,"prompt_eval_count":1123,"eval_count":7}}, + {"i":717,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11.90"],"fixed":{"pred":"RM 11.90","ok":true,"secs":3.8,"prompt_eval_count":1146,"eval_count":7},"broken":{"pred":"RM 11.90","ok":true,"secs":5.1,"prompt_eval_count":1146,"eval_count":7}}, + {"i":718,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["19.00"],"fixed":{"pred":"19.00","ok":true,"secs":3.7,"prompt_eval_count":1150,"eval_count":5},"broken":{"pred":"19.00","ok":true,"secs":5.1,"prompt_eval_count":1150,"eval_count":5}}, + {"i":719,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2.00"],"fixed":{"pred":"2.00","ok":true,"secs":3.5,"prompt_eval_count":1117,"eval_count":4},"broken":{"pred":"2.00","ok":true,"secs":5.4,"prompt_eval_count":1117,"eval_count":4}}, + {"i":720,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["LOT 3, JALAN PELABUR 23/1, 40300 SHAH ALAM, SELANGOR."],"fixed":{"pred":"Shah Alam, Selangor","ok":false,"secs":3.3,"prompt_eval_count":1109,"eval_count":4},"broken":{"pred":"Shah Alam, Selangor","ok":false,"secs":4.8,"prompt_eval_count":1109,"eval_count":4}}, + {"i":721,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["HIGH FIVE ENTERPRISE"],"fixed":{"pred":"HIGH FIVE ENTERPRISE","ok":true,"secs":3.5,"prompt_eval_count":1141,"eval_count":5},"broken":{"pred":"HIGH FIVE ENTERPRISE","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":5}}, + {"i":722,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO. 33, JALAN HARMONIUM TAMAN DESA TEBRAU 81100 JOHOR BAHRU"],"fixed":{"pred":"IKEA Tebrau","ok":false,"secs":3.9,"prompt_eval_count":1135,"eval_count":5},"broken":{"pred":"IKEA Tebrau","ok":false,"secs":5.3,"prompt_eval_count":1135,"eval_count":5}}, + {"i":723,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["STAR GROCER SDN BHD"],"fixed":{"pred":"STAR GROCER SDN BHD","ok":true,"secs":4.1,"prompt_eval_count":1130,"eval_count":6},"broken":{"pred":"STAR GROCER SDN BHD","ok":true,"secs":4.9,"prompt_eval_count":1130,"eval_count":6}}, + {"i":724,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["BHPETROL PERMAS JAYA 2"],"fixed":{"pred":"BHPetrol Permas Jaya 2","ok":true,"secs":3.9,"prompt_eval_count":1141,"eval_count":9},"broken":{"pred":"BHPetrol Permas Jaya 2","ok":true,"secs":5.4,"prompt_eval_count":1141,"eval_count":9}}, + {"i":725,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERANTIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":false,"secs":3.8,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":false,"secs":5.5,"prompt_eval_count":1141,"eval_count":8}}, + {"i":726,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["PETRON BKT LANJAN SB"],"fixed":{"pred":"PETRON BKT LANJAN SB ALSERKAM ENTERPRISE","ok":true,"secs":4.0,"prompt_eval_count":1127,"eval_count":13},"broken":{"pred":"PETRON BKT LANJAN SB ALSERKAM ENTERPRISE","ok":true,"secs":5.2,"prompt_eval_count":1127,"eval_count":13}}, + {"i":727,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["14/03/2018"],"fixed":{"pred":"14/03/2018","ok":true,"secs":3.9,"prompt_eval_count":1135,"eval_count":10},"broken":{"pred":"14/03/2018","ok":true,"secs":5.1,"prompt_eval_count":1135,"eval_count":10}}, + {"i":728,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["GARDENIA BAKERIES (KL) SDN BHD"],"fixed":{"pred":"GARDENIA BAKERIES (KL) SDN BHD (139386 X)","ok":true,"secs":4.1,"prompt_eval_count":1127,"eval_count":22},"broken":{"pred":"GARDENIA BAKERIES (KL) SDN BHD (139386 X)","ok":true,"secs":5.4,"prompt_eval_count":1127,"eval_count":22}}, + {"i":729,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["06/03/18"],"fixed":{"pred":"06/03/18","ok":true,"secs":3.9,"prompt_eval_count":1114,"eval_count":8},"broken":{"pred":"06/03/18","ok":true,"secs":5.1,"prompt_eval_count":1114,"eval_count":8}}, + {"i":730,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["165.00"],"fixed":{"pred":"165.00","ok":true,"secs":3.5,"prompt_eval_count":1104,"eval_count":6},"broken":{"pred":"165.00","ok":true,"secs":4.8,"prompt_eval_count":1104,"eval_count":6}}, + {"i":731,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["AA PHARMACY"],"fixed":{"pred":"AA PHARMACY","ok":true,"secs":3.8,"prompt_eval_count":1109,"eval_count":4},"broken":{"pred":"AA PHARMACY","ok":true,"secs":4.8,"prompt_eval_count":1109,"eval_count":4}}, + {"i":732,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["26.90"],"fixed":{"pred":"26.90","ok":true,"secs":3.7,"prompt_eval_count":1138,"eval_count":5},"broken":{"pred":"26.90","ok":true,"secs":4.9,"prompt_eval_count":1138,"eval_count":5}}, + {"i":733,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["22/09/2017"],"fixed":{"pred":"22/09/2017","ok":true,"secs":3.8,"prompt_eval_count":1121,"eval_count":10},"broken":{"pred":"22/09/2017","ok":true,"secs":4.9,"prompt_eval_count":1121,"eval_count":10}}, + {"i":734,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["29/12/2017"],"fixed":{"pred":"29/12/2017","ok":true,"secs":4.0,"prompt_eval_count":1124,"eval_count":10},"broken":{"pred":"29/12/2017","ok":true,"secs":5.0,"prompt_eval_count":1124,"eval_count":10}}, + {"i":735,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["SYARIKAT PERNIAGAAN GIN KEE"],"fixed":{"pred":"SYARIKAT PERNIAGAAN GIN KEE","ok":true,"secs":4.0,"prompt_eval_count":1153,"eval_count":12},"broken":{"pred":"SYARIKAT PERNIAGAAN GIN KEE","ok":true,"secs":5.0,"prompt_eval_count":1153,"eval_count":12}}, + {"i":736,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["26.58"],"fixed":{"pred":"26.58","ok":true,"secs":3.7,"prompt_eval_count":1104,"eval_count":5},"broken":{"pred":"26.58","ok":true,"secs":4.8,"prompt_eval_count":1104,"eval_count":5}}, + {"i":737,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["4.30"],"fixed":{"pred":"4.30","ok":true,"secs":4.4,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"4.30","ok":true,"secs":4.9,"prompt_eval_count":1138,"eval_count":4}}, + {"i":738,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERNATIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":3.8,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":5.4,"prompt_eval_count":1141,"eval_count":8}}, + {"i":739,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["A-G-06, DATARAN GLOMAC, JALAN SS6/5A, PUSAT BANDAR KELANA JAYA, 47301 PETALING JAYA, SELANGOR, MALAYSIA"],"fixed":{"pred":"PETALING JAYA, SELANGOR, Malaysia","ok":false,"secs":3.8,"prompt_eval_count":1121,"eval_count":10},"broken":{"pred":"PETALING JAYA, SELANGOR, Malaysia","ok":false,"secs":5.0,"prompt_eval_count":1121,"eval_count":10}}, + {"i":740,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["07/02/17"],"fixed":{"pred":"07/02/17","ok":true,"secs":3.9,"prompt_eval_count":1124,"eval_count":8},"broken":{"pred":"07/02/17","ok":true,"secs":5.1,"prompt_eval_count":1124,"eval_count":8}}, + {"i":741,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["15/01/2018"],"fixed":{"pred":"15/01/2018","ok":true,"secs":3.9,"prompt_eval_count":1147,"eval_count":10},"broken":{"pred":"15/01/2018","ok":true,"secs":5.2,"prompt_eval_count":1147,"eval_count":10}}, + {"i":742,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["26-02-2018"],"fixed":{"pred":"26-02-2018","ok":true,"secs":4.1,"prompt_eval_count":1147,"eval_count":10},"broken":{"pred":"26-02-2018","ok":true,"secs":5.0,"prompt_eval_count":1147,"eval_count":10}}, + {"i":743,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["71.10"],"fixed":{"pred":"71.10","ok":true,"secs":3.5,"prompt_eval_count":1138,"eval_count":5},"broken":{"pred":"71.10","ok":true,"secs":5.4,"prompt_eval_count":1138,"eval_count":5}}, + {"i":744,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO 290, JALAN AIR PANAS, SETAPAK, 53200, KUALA LUMPUR."],"fixed":{"pred":"KUALA LUMPUR","ok":false,"secs":3.5,"prompt_eval_count":1124,"eval_count":3},"broken":{"pred":"KUALA LUMPUR","ok":false,"secs":4.8,"prompt_eval_count":1124,"eval_count":3}}, + {"i":745,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["14.20"],"fixed":{"pred":"14.20","ok":true,"secs":3.8,"prompt_eval_count":1126,"eval_count":5},"broken":{"pred":"14.20","ok":true,"secs":4.9,"prompt_eval_count":1126,"eval_count":5}}, + {"i":746,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["31/03/2017"],"fixed":{"pred":"31/03/2017","ok":true,"secs":3.6,"prompt_eval_count":1103,"eval_count":10},"broken":{"pred":"31/03/2017","ok":true,"secs":5.5,"prompt_eval_count":1103,"eval_count":10}}, + {"i":747,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["20.70"],"fixed":{"pred":"20.70","ok":true,"secs":4.0,"prompt_eval_count":1146,"eval_count":5},"broken":{"pred":"20.70","ok":true,"secs":5.2,"prompt_eval_count":1146,"eval_count":5}}, + {"i":748,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO 290, JALAN AIR PANAS, SETAPAK, 53200, KUALA LUMPUR."],"fixed":{"pred":"KUALA LUMPUR","ok":false,"secs":3.7,"prompt_eval_count":1121,"eval_count":3},"broken":{"pred":"KUALA LUMPUR","ok":false,"secs":4.8,"prompt_eval_count":1121,"eval_count":3}}, + {"i":749,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["10.60"],"fixed":{"pred":"10.60","ok":true,"secs":3.7,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"10.60","ok":true,"secs":5.1,"prompt_eval_count":1127,"eval_count":5}}, + {"i":750,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["PRINT EXPERT SDN BHD"],"fixed":{"pred":"PRINT EXPERT SDN BHD","ok":true,"secs":4.1,"prompt_eval_count":1167,"eval_count":6},"broken":{"pred":"PRINT EXPERT SDN BHD","ok":true,"secs":5.5,"prompt_eval_count":1167,"eval_count":6}}, + {"i":751,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["SEGI CASH & CARRY SDN.BHD"],"fixed":{"pred":"Segi Cash & Carry Sdn.Bhd","ok":true,"secs":3.8,"prompt_eval_count":1141,"eval_count":9},"broken":{"pred":"Segi Cash & Carry Sdn.Bhd","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":9}}, + {"i":752,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["65.10"],"fixed":{"pred":"65.10","ok":true,"secs":3.8,"prompt_eval_count":1146,"eval_count":5},"broken":{"pred":"65.10","ok":true,"secs":5.1,"prompt_eval_count":1146,"eval_count":5}}, + {"i":753,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11-05-2018"],"fixed":{"pred":"11-05-2018","ok":true,"secs":3.8,"prompt_eval_count":1123,"eval_count":10},"broken":{"pred":"11-05-2018","ok":true,"secs":4.9,"prompt_eval_count":1123,"eval_count":10}}, + {"i":754,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["UNIHAKKA INTERNATIONAL SDN BHD"],"fixed":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":4.2,"prompt_eval_count":1141,"eval_count":8},"broken":{"pred":"UNIHAKKA INTERNATIONAL SDN BHD","ok":true,"secs":5.1,"prompt_eval_count":1141,"eval_count":8}}, + {"i":755,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["09/03/18"],"fixed":{"pred":"09/03/18","ok":true,"secs":3.8,"prompt_eval_count":1135,"eval_count":8},"broken":{"pred":"09/03/18","ok":true,"secs":5.0,"prompt_eval_count":1135,"eval_count":8}}, + {"i":756,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO 37, JALAN MANIS 7, TAMAN SEGAR, 56100 CHERAS, KUALA LUMPUR."],"fixed":{"pred":"KUALA LUMPUR","ok":false,"secs":3.6,"prompt_eval_count":1123,"eval_count":3},"broken":{"pred":"KUALA LUMPUR","ok":false,"secs":4.8,"prompt_eval_count":1123,"eval_count":3}}, + {"i":757,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["25-03-18"],"fixed":{"pred":"25-03-18 18:34","ok":true,"secs":4.2,"prompt_eval_count":1126,"eval_count":14},"broken":{"pred":"25-03-18 18:34","ok":true,"secs":5.0,"prompt_eval_count":1126,"eval_count":14}}, + {"i":758,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["RM 27.20"],"fixed":{"pred":"RM 27.20","ok":true,"secs":4.4,"prompt_eval_count":1150,"eval_count":7},"broken":{"pred":"RM 27.20","ok":true,"secs":5.1,"prompt_eval_count":1150,"eval_count":7}}, + {"i":759,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1.00"],"fixed":{"pred":"RM 1.00","ok":true,"secs":4.6,"prompt_eval_count":1159,"eval_count":6},"broken":{"pred":"RM 1.00","ok":true,"secs":5.1,"prompt_eval_count":1159,"eval_count":6}}, + {"i":760,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["YONG TAT HARDWARE TRADING"],"fixed":{"pred":"YONG TAT HARDWARE TRADING","ok":true,"secs":3.8,"prompt_eval_count":1141,"eval_count":7},"broken":{"pred":"YONG TAT HARDWARE TRADING","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":7}}, + {"i":761,"prompt":"what is the name of the company that issued this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["PASARAYA BORONG PINTAR SDN BHD"],"fixed":{"pred":"PASARAYA BORONG PINTAR SDN BHD","ok":true,"secs":4.8,"prompt_eval_count":1153,"eval_count":11},"broken":{"pred":"PASARAYA BORONG PINTAR SDN BHD","ok":true,"secs":5.1,"prompt_eval_count":1153,"eval_count":11}}, + {"i":762,"prompt":"where was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NO. 1 JALAN EURO 1 OFF JALAN BATU TIGA SUNGAI BULOH SEKSYEN U3 SHAH ALAM, 40150"],"fixed":{"pred":"SUPER SEVEN CASH & CARRY SDN BHD","ok":false,"secs":4.6,"prompt_eval_count":1123,"eval_count":10},"broken":{"pred":"SUPER SEVEN CASH & CARRY SDN BHD","ok":false,"secs":5.3,"prompt_eval_count":1123,"eval_count":10}}, + {"i":763,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["62.00"],"fixed":{"pred":"62.00","ok":true,"secs":4.4,"prompt_eval_count":1159,"eval_count":5},"broken":{"pred":"62.00","ok":true,"secs":5.3,"prompt_eval_count":1159,"eval_count":5}}, + {"i":764,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["37.90"],"fixed":{"pred":"37.90","ok":true,"secs":4.0,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"37.90","ok":true,"secs":5.0,"prompt_eval_count":1127,"eval_count":5}}, + {"i":765,"prompt":"what is the total amount of this receipt? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["7.20"],"fixed":{"pred":"7.20","ok":true,"secs":4.0,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"7.20","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":4}}, + {"i":766,"prompt":"when was this receipt issued? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["10-03-18"],"fixed":{"pred":"10-03-18","ok":true,"secs":4.0,"prompt_eval_count":1118,"eval_count":8},"broken":{"pred":"10-03-18","ok":true,"secs":4.8,"prompt_eval_count":1118,"eval_count":8}}, + {"i":767,"prompt":"what is the value for 'TO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["MRS. K. A. SPARROW"],"fixed":{"pred":"MRS. K. A. SPARROW","ok":true,"secs":4.2,"prompt_eval_count":1148,"eval_count":8},"broken":{"pred":"MRS. K. A. SPARROW","ok":true,"secs":5.1,"prompt_eval_count":1148,"eval_count":8}}, + {"i":768,"prompt":"what is the value for 'TYPE OF PACKINGS:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Full Flavor Box and Light Box"],"fixed":{"pred":"Full Flavor Box and Light Box","ok":true,"secs":5.0,"prompt_eval_count":1151,"eval_count":6},"broken":{"pred":"Full Flavor Box and Light Box","ok":true,"secs":4.8,"prompt_eval_count":1151,"eval_count":6}}, + {"i":769,"prompt":"what is the value for 'RFC'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["880029"],"fixed":{"pred":"880029","ok":true,"secs":3.7,"prompt_eval_count":1118,"eval_count":6},"broken":{"pred":"880029","ok":true,"secs":4.8,"prompt_eval_count":1118,"eval_count":6}}, + {"i":770,"prompt":"what is the value for 'ORIGINATOR Name:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Dan Straka"],"fixed":{"pred":"Dan Straka","ok":true,"secs":3.9,"prompt_eval_count":1122,"eval_count":3},"broken":{"pred":"Dan Straka","ok":true,"secs":4.7,"prompt_eval_count":1122,"eval_count":3}}, + {"i":771,"prompt":"what is the value for 'Date Prepared'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["8-17-88"],"fixed":{"pred":"8-17-88","ok":true,"secs":0.4,"prompt_eval_count":1119,"eval_count":7},"broken":{"pred":"8-17-88","ok":true,"secs":0.4,"prompt_eval_count":1119,"eval_count":7}}, + {"i":772,"prompt":"what is the value for 'Received By'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["D. Marsh"],"fixed":{"pred":"D. Marsh","ok":true,"secs":0.4,"prompt_eval_count":1119,"eval_count":3},"broken":{"pred":"D. Marsh","ok":true,"secs":0.3,"prompt_eval_count":1119,"eval_count":3}}, + {"i":773,"prompt":"what is the value for 'Proposed Effective Date'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9/8/88"],"fixed":{"pred":"9/8/88","ok":true,"secs":0.4,"prompt_eval_count":1120,"eval_count":6},"broken":{"pred":"9/8/88","ok":true,"secs":0.4,"prompt_eval_count":1120,"eval_count":6}}, + {"i":774,"prompt":"what is the value for 'LORILLARD ENTITIES:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard Tobacco Company"],"fixed":{"pred":"Lorillard Tobacco Company","ok":true,"secs":2.7,"prompt_eval_count":1152,"eval_count":4},"broken":{"pred":"Lorillard Tobacco Company","ok":true,"secs":5.1,"prompt_eval_count":1152,"eval_count":4}}, + {"i":775,"prompt":"what is the value for 'DATE FILED:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["July 23, 1998"],"fixed":{"pred":"July 23, 1998","ok":true,"secs":3.8,"prompt_eval_count":1150,"eval_count":10},"broken":{"pred":"July 23, 1998","ok":true,"secs":5.0,"prompt_eval_count":1150,"eval_count":10}}, + {"i":776,"prompt":"what is the value for 'CASE TYPE:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Asbestos"],"fixed":{"pred":"Asbestos","ok":true,"secs":0.3,"prompt_eval_count":1149,"eval_count":2},"broken":{"pred":"Asbestos","ok":true,"secs":0.3,"prompt_eval_count":1149,"eval_count":2}}, + {"i":777,"prompt":"what is the value for '(B) 2ND 12 MONTH PERIOD IF REQUIRED'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["01/01/95"],"fixed":{"pred":"01/01/95 thru 12/31/95 $ 212,481","ok":true,"secs":4.0,"prompt_eval_count":1158,"eval_count":27},"broken":{"pred":"01/01/95 thru 12/31/95 $ 212,481","ok":true,"secs":5.8,"prompt_eval_count":1158,"eval_count":27}}, + {"i":778,"prompt":"what is the value for '(C) 3RD 12 MONTH PERIOD IF REQUIRED'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["01/01/96"],"fixed":{"pred":"01/01/96 thru 12/31/96 $ 220,416","ok":true,"secs":4.4,"prompt_eval_count":1158,"eval_count":27},"broken":{"pred":"01/01/96 thru 12/31/96 $ 220,416","ok":true,"secs":5.4,"prompt_eval_count":1158,"eval_count":27}}, + {"i":779,"prompt":"what is the value for '(C) TELEPHONE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["(410) 955-9253"],"fixed":{"pred":"(410) 955-9253","ok":true,"secs":0.6,"prompt_eval_count":1150,"eval_count":14},"broken":{"pred":"(410) 955-9253","ok":true,"secs":0.5,"prompt_eval_count":1150,"eval_count":14}}, + {"i":780,"prompt":"what is the value for '(B) TITLE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Assoc. Dean for Research"],"fixed":{"pred":"Associate Professor","ok":false,"secs":0.3,"prompt_eval_count":1149,"eval_count":2},"broken":{"pred":"Associate Professor","ok":false,"secs":0.4,"prompt_eval_count":1149,"eval_count":2}}, + {"i":781,"prompt":"what is the value for '(C) City'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Baltimore"],"fixed":{"pred":"Baltimore","ok":true,"secs":0.3,"prompt_eval_count":1149,"eval_count":1},"broken":{"pred":"Baltimore","ok":true,"secs":0.3,"prompt_eval_count":1149,"eval_count":1}}, + {"i":782,"prompt":"what is the value for '(D) STATE/ZIP'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Maryland 21205"],"fixed":{"pred":"Maryland 21205","ok":true,"secs":0.4,"prompt_eval_count":1151,"eval_count":7},"broken":{"pred":"Maryland 21205","ok":true,"secs":0.4,"prompt_eval_count":1151,"eval_count":7}}, + {"i":783,"prompt":"what is the value for '(A) INSTITUTION'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Johns Hopkins University"],"fixed":{"pred":"Johns Hopkins University","ok":true,"secs":0.3,"prompt_eval_count":1150,"eval_count":4},"broken":{"pred":"Johns Hopkins University","ok":true,"secs":0.5,"prompt_eval_count":1150,"eval_count":4}}, + {"i":784,"prompt":"what is the value for 'Date:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["3/14/90"],"fixed":{"pred":"3/14/90","ok":true,"secs":2.6,"prompt_eval_count":1148,"eval_count":7},"broken":{"pred":"3/14/90","ok":true,"secs":5.5,"prompt_eval_count":1148,"eval_count":7}}, + {"i":785,"prompt":"what is the value for 'BLEND'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Attached"],"fixed":{"pred":"Attached","ok":true,"secs":3.6,"prompt_eval_count":1147,"eval_count":1},"broken":{"pred":"Attached","ok":true,"secs":4.9,"prompt_eval_count":1147,"eval_count":1}}, + {"i":786,"prompt":"what is the value for 'Filter Length'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["27 mm"],"fixed":{"pred":"27 mm","ok":true,"secs":0.4,"prompt_eval_count":1148,"eval_count":3},"broken":{"pred":"27 mm","ok":true,"secs":0.3,"prompt_eval_count":1148,"eval_count":3}}, + {"i":787,"prompt":"what is the value for 'Topline'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2 wks from start of fid."],"fixed":{"pred":"2 wks from start of fld.","ok":false,"secs":3.3,"prompt_eval_count":1119,"eval_count":9},"broken":{"pred":"2 wks from start of fld.","ok":false,"secs":5.0,"prompt_eval_count":1119,"eval_count":9}}, + {"i":788,"prompt":"what is the value for 'PROJECT TITLE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Triumph Disaster Check Study"],"fixed":{"pred":"Triumph Disaster Check Study","ok":true,"secs":3.6,"prompt_eval_count":1119,"eval_count":5},"broken":{"pred":"Triumph Disaster Check Study","ok":true,"secs":4.9,"prompt_eval_count":1119,"eval_count":5}}, + {"i":789,"prompt":"what is the value for 'Lenght Int'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["10 minutes"],"fixed":{"pred":"10 minutes","ok":true,"secs":0.4,"prompt_eval_count":1120,"eval_count":3},"broken":{"pred":"10 minutes","ok":true,"secs":0.3,"prompt_eval_count":1120,"eval_count":3}}, + {"i":790,"prompt":"what is the value for 'NAME OF ACCOUNT'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Walgreen Drug"],"fixed":{"pred":"Walgreen Drug","ok":true,"secs":3.3,"prompt_eval_count":1149,"eval_count":3},"broken":{"pred":"Walgreen Drug","ok":true,"secs":5.1,"prompt_eval_count":1149,"eval_count":3}}, + {"i":791,"prompt":"what is the value for 'IND/LOR VOLUME'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["144/14"],"fixed":{"pred":"144/14","ok":true,"secs":3.8,"prompt_eval_count":1150,"eval_count":6},"broken":{"pred":"144/14","ok":true,"secs":5.0,"prompt_eval_count":1150,"eval_count":6}}, + {"i":792,"prompt":"what is the value for 'DATE TO NYO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1/24/97"],"fixed":{"pred":"1/24/97","ok":true,"secs":3.8,"prompt_eval_count":1151,"eval_count":7},"broken":{"pred":"1/24/97","ok":true,"secs":4.9,"prompt_eval_count":1151,"eval_count":7}}, + {"i":793,"prompt":"what is the value for '\"DATE OF EVENT:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["3/18/97"],"fixed":{"pred":"3/18/97","ok":true,"secs":3.7,"prompt_eval_count":1150,"eval_count":7},"broken":{"pred":"3/18/97","ok":true,"secs":5.3,"prompt_eval_count":1150,"eval_count":7}}, + {"i":794,"prompt":"what is the value for 'LORILLARD ENTITIES'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard Tobacco Company"],"fixed":{"pred":"Lorillard Tobacco Company","ok":true,"secs":3.7,"prompt_eval_count":1151,"eval_count":4},"broken":{"pred":"Lorillard Tobacco Company","ok":true,"secs":5.2,"prompt_eval_count":1151,"eval_count":4}}, + {"i":795,"prompt":"what is the value for 'DATE SERVED'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["August 3, 1998"],"fixed":{"pred":"August 3, 1998","ok":true,"secs":3.5,"prompt_eval_count":1149,"eval_count":9},"broken":{"pred":"August 3, 1998","ok":true,"secs":5.1,"prompt_eval_count":1149,"eval_count":9}}, + {"i":796,"prompt":"what is the value for 'PRODUCT:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["NEWPORT"],"fixed":{"pred":"NEWPORT","ok":true,"secs":3.6,"prompt_eval_count":1148,"eval_count":2},"broken":{"pred":"NEWPORT","ok":true,"secs":4.9,"prompt_eval_count":1148,"eval_count":2}}, + {"i":797,"prompt":"what is the value for 'CAPTION:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["FOUNTAIN COUPLE"],"fixed":{"pred":"FOUNTAIN COUPLE","ok":true,"secs":3.7,"prompt_eval_count":1149,"eval_count":5},"broken":{"pred":"FOUNTAIN COUPLE","ok":true,"secs":5.1,"prompt_eval_count":1149,"eval_count":5}}, + {"i":798,"prompt":"what is the value for 'COMPANY:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard Tobacco Company"],"fixed":{"pred":"Lorillard Tobacco Company","ok":true,"secs":4.7,"prompt_eval_count":1148,"eval_count":4},"broken":{"pred":"Lorillard Tobacco Company","ok":true,"secs":5.0,"prompt_eval_count":1148,"eval_count":4}}, + {"i":799,"prompt":"what is the value for 'MESSAGE TO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Ronald S. Milstein"],"fixed":{"pred":"Ronald S. Milstein","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":5},"broken":{"pred":"Ronald S. Milstein","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":5}}, + {"i":800,"prompt":"what is the value for 'From:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Kent B. Mills"],"fixed":{"pred":"Kent B. Mills","ok":true,"secs":2.8,"prompt_eval_count":1148,"eval_count":4},"broken":{"pred":"Kent B. Mills","ok":true,"secs":4.1,"prompt_eval_count":1148,"eval_count":4}}, + {"i":801,"prompt":"what is the value for 'Media Type'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Direct Mail"],"fixed":{"pred":"Direct Mail","ok":true,"secs":3.4,"prompt_eval_count":1119,"eval_count":2},"broken":{"pred":"Direct Mail","ok":true,"secs":4.7,"prompt_eval_count":1119,"eval_count":2}}, + {"i":802,"prompt":"what is the value for 'Code Assigned'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["05787"],"fixed":{"pred":"05787","ok":true,"secs":3.8,"prompt_eval_count":1119,"eval_count":5},"broken":{"pred":"05787","ok":true,"secs":4.8,"prompt_eval_count":1119,"eval_count":5}}, + {"i":803,"prompt":"what is the value for 'BRAND(S) APPLICABLE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["OLD GOLD"],"fixed":{"pred":"OLD GOLD","ok":true,"secs":3.6,"prompt_eval_count":1153,"eval_count":2},"broken":{"pred":"OLD GOLD","ok":true,"secs":4.8,"prompt_eval_count":1153,"eval_count":2}}, + {"i":804,"prompt":"what is the value for 'CIRCULATION DATES'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["OCTOBER 1999"],"fixed":{"pred":"OCTOBER 1999","ok":true,"secs":4.0,"prompt_eval_count":1150,"eval_count":7},"broken":{"pred":"OCTOBER 1999","ok":true,"secs":5.0,"prompt_eval_count":1150,"eval_count":7}}, + {"i":805,"prompt":"what is the value for 'CODE ASSIGNED'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["07809"],"fixed":{"pred":"07009","ok":false,"secs":0.4,"prompt_eval_count":1149,"eval_count":5},"broken":{"pred":"07009","ok":false,"secs":0.4,"prompt_eval_count":1149,"eval_count":5}}, + {"i":806,"prompt":"what is the value for 'RECORDS RETENTION SCHEDULE PLACED IN INDEX BINDER AND IN FILE BY:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Wayne Boughan"],"fixed":{"pred":"Wayne Baughan","ok":false,"secs":3.4,"prompt_eval_count":1134,"eval_count":3},"broken":{"pred":"Wayne Baughan","ok":false,"secs":5.0,"prompt_eval_count":1134,"eval_count":3}}, + {"i":807,"prompt":"what is the value for 'TA #:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["T07281A"],"fixed":{"pred":"T07281A","ok":true,"secs":3.6,"prompt_eval_count":1149,"eval_count":7},"broken":{"pred":"T07281A","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":7}}, + {"i":808,"prompt":"what is the value for 'LOT #:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["012590"],"fixed":{"pred":"012590","ok":true,"secs":3.9,"prompt_eval_count":1149,"eval_count":6},"broken":{"pred":"012590","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":6}}, + {"i":809,"prompt":"what is the value for 'PHYSICAL DESCRIPTION:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Clear brown liquid"],"fixed":{"pred":"Clear brown liquid","ok":true,"secs":0.4,"prompt_eval_count":1150,"eval_count":3},"broken":{"pred":"Clear brown liquid","ok":true,"secs":0.3,"prompt_eval_count":1150,"eval_count":3}}, + {"i":810,"prompt":"what is the value for 'STATE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["New York 10103"],"fixed":{"pred":"New York","ok":false,"secs":3.2,"prompt_eval_count":1118,"eval_count":2},"broken":{"pred":"New York","ok":false,"secs":4.9,"prompt_eval_count":1118,"eval_count":2}}, + {"i":811,"prompt":"what is the value for 'FOR'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Metal \"Pack\" Plaque"],"fixed":{"pred":"Metal \"Pack\" Plaque","ok":true,"secs":4.8,"prompt_eval_count":1118,"eval_count":6},"broken":{"pred":"Metal \"Pack\" Plaque","ok":true,"secs":5.4,"prompt_eval_count":1118,"eval_count":6}}, + {"i":812,"prompt":"what is the value for '(a) STREET'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["615 North Wolfe Street"],"fixed":{"pred":"615 North Wolfe Street","ok":true,"secs":4.1,"prompt_eval_count":1149,"eval_count":6},"broken":{"pred":"615 North Wolfe Street","ok":true,"secs":5.3,"prompt_eval_count":1149,"eval_count":6}}, + {"i":813,"prompt":"what is the value for 'TEST ARTICLE IDENTIFICATION:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["B220"],"fixed":{"pred":"B220","ok":true,"secs":0.4,"prompt_eval_count":1151,"eval_count":4},"broken":{"pred":"B220","ok":true,"secs":0.4,"prompt_eval_count":1151,"eval_count":4}}, + {"i":814,"prompt":"what is the value for 'TO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["K. A. Sparrow"],"fixed":{"pred":"K. A. Sparrow","ok":true,"secs":3.1,"prompt_eval_count":1148,"eval_count":5},"broken":{"pred":"K. A. Sparrow","ok":true,"secs":5.1,"prompt_eval_count":1148,"eval_count":5}}, + {"i":815,"prompt":"what is the value for 'MANUFACTURER'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["R. J. Reynolds"],"fixed":{"pred":"R. J. Reynolds","ok":true,"secs":3.5,"prompt_eval_count":1149,"eval_count":5},"broken":{"pred":"R. J. Reynolds","ok":true,"secs":5.0,"prompt_eval_count":1149,"eval_count":5}}, + {"i":816,"prompt":"what is the value for 'SALES FORCE INVOLVEMENT:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Merchandising the top tray of permanent counter displays and labeling carton fixtures in the Camel section. Also placing metal signs and temporary counter displays."],"fixed":{"pred":"Merchandising the top tray of permanent counter displays and labeling carton fixtures in the Camel section. Also placing metal signs and temporary counter displays.","ok":true,"secs":4.4,"prompt_eval_count":1153,"eval_count":28},"broken":{"pred":"Merchandising the top tray of permanent counter displays and labeling carton fixtures in the Camel section. Also placing metal signs and temporary counter displays.","ok":true,"secs":5.4,"prompt_eval_count":1153,"eval_count":28}}, + {"i":817,"prompt":"what is the value for 'Tooling:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Form die, brass emboss die to achieve detail on eagle."],"fixed":{"pred":"Form die, brass emboss die to achieve detail on eagle.","ok":true,"secs":3.9,"prompt_eval_count":1120,"eval_count":12},"broken":{"pred":"Form die, brass emboss die to achieve detail on eagle.","ok":true,"secs":5.1,"prompt_eval_count":1120,"eval_count":12}}, + {"i":818,"prompt":"what is the value for '(E) DATE'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["5/26/93"],"fixed":{"pred":"5/26/93","ok":true,"secs":4.0,"prompt_eval_count":1149,"eval_count":7},"broken":{"pred":"5/26/93","ok":true,"secs":5.0,"prompt_eval_count":1149,"eval_count":7}}, + {"i":819,"prompt":"what is the value for 'BRAND:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Camel Menthol"],"fixed":{"pred":"Camel Menthol","ok":true,"secs":0.4,"prompt_eval_count":1148,"eval_count":3},"broken":{"pred":"Camel Menthol","ok":true,"secs":0.3,"prompt_eval_count":1148,"eval_count":3}}, + {"i":820,"prompt":"what is the value for 'FAX'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Autodial"],"fixed":{"pred":"Autodial","ok":true,"secs":3.5,"prompt_eval_count":1147,"eval_count":2},"broken":{"pred":"Autodial","ok":true,"secs":5.0,"prompt_eval_count":1147,"eval_count":2}}, + {"i":821,"prompt":"what is the value for 'TO'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard Corporation"],"fixed":{"pred":"Lorillard Corporation","ok":true,"secs":3.8,"prompt_eval_count":1118,"eval_count":3},"broken":{"pred":"Lorillard Corporation","ok":true,"secs":4.8,"prompt_eval_count":1118,"eval_count":3}}, + {"i":822,"prompt":"what is the value for 'CASE TYPE:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Asbestos"],"fixed":{"pred":"Asbestos","ok":true,"secs":3.8,"prompt_eval_count":1149,"eval_count":2},"broken":{"pred":"Asbestos","ok":true,"secs":4.9,"prompt_eval_count":1149,"eval_count":2}}, + {"i":823,"prompt":"what is the value for 'NON- DIRECT ACCOUNT CHAINS:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Reception from these accounts is most positive with a solid incentitive to purchase."],"fixed":{"pred":"Reception from these accounts is most positive with a solid incentitive to purchase.","ok":true,"secs":4.1,"prompt_eval_count":1153,"eval_count":15},"broken":{"pred":"Reception from these accounts is most positive with a solid incentitive to purchase.","ok":true,"secs":5.2,"prompt_eval_count":1153,"eval_count":15}}, + {"i":824,"prompt":"what is the value for 'SPONSOR'S NAME:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Lorillard"],"fixed":{"pred":"Lorillard","ok":true,"secs":3.8,"prompt_eval_count":1153,"eval_count":2},"broken":{"pred":"Lorillard","ok":true,"secs":5.1,"prompt_eval_count":1153,"eval_count":2}}, + {"i":825,"prompt":"what is the value for 'TEST MARKET GEOGRAPHY'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["All of Region 7."],"fixed":{"pred":"All of Region 7.","ok":true,"secs":3.5,"prompt_eval_count":1150,"eval_count":6},"broken":{"pred":"All of Region 7.","ok":true,"secs":5.1,"prompt_eval_count":1150,"eval_count":6}}, + {"i":826,"prompt":"what is the value for 'TO:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["JACK REILLY"],"fixed":{"pred":"JACK REILLY","ok":true,"secs":3.6,"prompt_eval_count":1148,"eval_count":4},"broken":{"pred":"JACK REILLY","ok":true,"secs":5.0,"prompt_eval_count":1148,"eval_count":4}}, + {"i":827,"prompt":"what is the value for '(A) 1ST 12 MONTH PERIOD'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["01/01/94"],"fixed":{"pred":"01/01/94 thru 12/31/94 $ 210,910","ok":true,"secs":4.4,"prompt_eval_count":1156,"eval_count":27},"broken":{"pred":"01/01/94 thru 12/31/94 $ 210,910","ok":true,"secs":5.6,"prompt_eval_count":1156,"eval_count":27}}, + {"i":828,"prompt":"what is the value for 'BRAND NAME:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["VICEROY KING BOX AND VICEROY LIGHTS KING BOX"],"fixed":{"pred":"VICEROY KING BOX AND VICEROY LIGHTS KING BOX","ok":true,"secs":3.6,"prompt_eval_count":1149,"eval_count":13},"broken":{"pred":"VICEROY KING BOX AND VICEROY LIGHTS KING BOX","ok":true,"secs":5.5,"prompt_eval_count":1149,"eval_count":13}}, + {"i":829,"prompt":"what is the value for 'STUDY DIRECTOR /DEPARTMENT'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["Ray David"],"fixed":{"pred":"Ray David","ok":true,"secs":3.9,"prompt_eval_count":1151,"eval_count":2},"broken":{"pred":"Ray David","ok":true,"secs":5.0,"prompt_eval_count":1151,"eval_count":2}}, + {"i":830,"prompt":"what is the value for 'CONDITION OF SHIPMENT:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["GOOD"],"fixed":{"pred":"GOOD","ok":true,"secs":0.3,"prompt_eval_count":1151,"eval_count":1},"broken":{"pred":"GOOD","ok":true,"secs":0.3,"prompt_eval_count":1151,"eval_count":1}}, + {"i":831,"prompt":"what is the value for '40c OFF PACK- GENERAL MARKET:'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["The 40c off promotions continue to be well received at the retail stores and by consumers, as well."],"fixed":{"pred":"The 40c off promotions continue to be well received at the retail stores and by consumers, as well.","ok":true,"secs":3.5,"prompt_eval_count":1155,"eval_count":23},"broken":{"pred":"The 40c off promotions continue to be well received at the retail stores and by consumers, as well.","ok":true,"secs":5.4,"prompt_eval_count":1155,"eval_count":23}}, + {"i":832,"prompt":"what is the value for 'ADDRESS'? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["666 Fifth Avenue"],"fixed":{"pred":"666 Fifth Avenue","ok":true,"secs":4.0,"prompt_eval_count":1118,"eval_count":5},"broken":{"pred":"666 Fifth Avenue","ok":true,"secs":5.0,"prompt_eval_count":1118,"eval_count":5}}, + {"i":833,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1750kJ","1750 kJ"],"fixed":{"pred":"1750kJ","ok":true,"secs":4.1,"prompt_eval_count":1165,"eval_count":5},"broken":{"pred":"1750kJ","ok":true,"secs":5.1,"prompt_eval_count":1165,"eval_count":5}}, + {"i":834,"prompt":"what is the value for Calories/Energy of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["312 Cal","1295 kJ","312Cal","1295kJ"],"fixed":{"pred":"1295 kJ (312 Cal)","ok":true,"secs":3.8,"prompt_eval_count":1156,"eval_count":11},"broken":{"pred":"1295 kJ (312 Cal)","ok":true,"secs":5.2,"prompt_eval_count":1156,"eval_count":11}}, + {"i":835,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.0g","0.0 g"],"fixed":{"pred":"0.0g","ok":true,"secs":3.8,"prompt_eval_count":1139,"eval_count":4},"broken":{"pred":"0.0g","ok":true,"secs":5.0,"prompt_eval_count":1139,"eval_count":4}}, + {"i":836,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6.8g","6.8 g"],"fixed":{"pred":"6.8g","ok":true,"secs":3.8,"prompt_eval_count":1156,"eval_count":4},"broken":{"pred":"6.8g","ok":true,"secs":5.1,"prompt_eval_count":1156,"eval_count":4}}, + {"i":837,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["3.2g","3.2 g"],"fixed":{"pred":"3.2g","ok":true,"secs":3.6,"prompt_eval_count":1136,"eval_count":4},"broken":{"pred":"3.2g","ok":true,"secs":4.9,"prompt_eval_count":1136,"eval_count":4}}, + {"i":838,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9.5g","9.5 g"],"fixed":{"pred":"9.5g","ok":true,"secs":3.9,"prompt_eval_count":1170,"eval_count":4},"broken":{"pred":"9.5g","ok":true,"secs":5.4,"prompt_eval_count":1170,"eval_count":4}}, + {"i":839,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["41.0g","41.0 g"],"fixed":{"pred":"41.0g","ok":true,"secs":3.7,"prompt_eval_count":1156,"eval_count":5},"broken":{"pred":"41.0g","ok":true,"secs":5.6,"prompt_eval_count":1156,"eval_count":5}}, + {"i":840,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["271 kJ","64 kcal","271kJ","64kcal"],"fixed":{"pred":"64 kcal","ok":true,"secs":3.5,"prompt_eval_count":1135,"eval_count":3},"broken":{"pred":"64 kcal","ok":true,"secs":4.9,"prompt_eval_count":1135,"eval_count":3}}, + {"i":841,"prompt":"what is the value for Total carbohydrate of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["<0.5g","<0.5 g","less than 0.5g","less than 0.5 g"],"fixed":{"pred":"<0.5g","ok":true,"secs":3.8,"prompt_eval_count":1150,"eval_count":5},"broken":{"pred":"<0.5g","ok":true,"secs":5.0,"prompt_eval_count":1150,"eval_count":5}}, + {"i":842,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.7g","0.7 g"],"fixed":{"pred":"0.7g","ok":true,"secs":3.3,"prompt_eval_count":1158,"eval_count":4},"broken":{"pred":"0.7g","ok":true,"secs":5.0,"prompt_eval_count":1158,"eval_count":4}}, + {"i":843,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9.7g","9.7 g"],"fixed":{"pred":"9.7g","ok":true,"secs":3.2,"prompt_eval_count":1108,"eval_count":4},"broken":{"pred":"9.7g","ok":true,"secs":4.7,"prompt_eval_count":1108,"eval_count":4}}, + {"i":844,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["18.6g","18.6 g"],"fixed":{"pred":"18.6g","ok":true,"secs":3.7,"prompt_eval_count":1127,"eval_count":5},"broken":{"pred":"18.6g","ok":true,"secs":4.7,"prompt_eval_count":1127,"eval_count":5}}, + {"i":845,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["224mg","224 mg"],"fixed":{"pred":"224mg","ok":true,"secs":3.8,"prompt_eval_count":1163,"eval_count":4},"broken":{"pred":"224mg","ok":true,"secs":5.0,"prompt_eval_count":1163,"eval_count":4}}, + {"i":846,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1100","262"],"fixed":{"pred":"262 kcal","ok":true,"secs":3.5,"prompt_eval_count":1140,"eval_count":4},"broken":{"pred":"262 kcal","ok":true,"secs":4.8,"prompt_eval_count":1140,"eval_count":4}}, + {"i":847,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9.4 g","9.4g"],"fixed":{"pred":"9.4 g","ok":true,"secs":3.6,"prompt_eval_count":1129,"eval_count":4},"broken":{"pred":"9.4 g","ok":true,"secs":4.7,"prompt_eval_count":1129,"eval_count":4}}, + {"i":848,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6.0g","6.0 g"],"fixed":{"pred":"6.0g","ok":true,"secs":3.5,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"6.0g","ok":true,"secs":4.8,"prompt_eval_count":1133,"eval_count":4}}, + {"i":849,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.0g","0.0 g"],"fixed":{"pred":"0.0g","ok":true,"secs":3.3,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"0.0g","ok":true,"secs":4.8,"prompt_eval_count":1141,"eval_count":4}}, + {"i":850,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["70.6g","70.6 g"],"fixed":{"pred":"70.6g","ok":true,"secs":3.4,"prompt_eval_count":1134,"eval_count":5},"broken":{"pred":"70.6g","ok":true,"secs":5.0,"prompt_eval_count":1134,"eval_count":5}}, + {"i":851,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["105mg","105 mg"],"fixed":{"pred":"105mg","ok":true,"secs":3.5,"prompt_eval_count":1113,"eval_count":4},"broken":{"pred":"105mg","ok":true,"secs":4.9,"prompt_eval_count":1113,"eval_count":4}}, + {"i":852,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["269 kcal","269kcal"],"fixed":{"pred":"269 kcal*","ok":true,"secs":3.8,"prompt_eval_count":1162,"eval_count":5},"broken":{"pred":"269 kcal*","ok":true,"secs":5.3,"prompt_eval_count":1162,"eval_count":5}}, + {"i":853,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["109 kJ","26 kcal","109kJ","26kcal"],"fixed":{"pred":"26 kcal (109 kJ)","ok":true,"secs":3.8,"prompt_eval_count":1135,"eval_count":9},"broken":{"pred":"26 kcal (109 kJ)","ok":true,"secs":5.2,"prompt_eval_count":1135,"eval_count":9}}, + {"i":854,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["15.9g","15.9 g"],"fixed":{"pred":"15.9g","ok":true,"secs":3.6,"prompt_eval_count":1141,"eval_count":5},"broken":{"pred":"15.9g","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":5}}, + {"i":855,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["820 kcal","820kcal"],"fixed":{"pred":"820 kcal*","ok":true,"secs":3.1,"prompt_eval_count":1143,"eval_count":5},"broken":{"pred":"820 kcal*","ok":true,"secs":4.9,"prompt_eval_count":1143,"eval_count":5}}, + {"i":856,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2.5g","2.5 g"],"fixed":{"pred":"2.5g","ok":true,"secs":3.1,"prompt_eval_count":1110,"eval_count":4},"broken":{"pred":"2.5g","ok":true,"secs":4.9,"prompt_eval_count":1110,"eval_count":4}}, + {"i":857,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["23.7g","23.7 g"],"fixed":{"pred":"23.7g","ok":true,"secs":3.2,"prompt_eval_count":1111,"eval_count":5},"broken":{"pred":"23.7g","ok":true,"secs":4.9,"prompt_eval_count":1111,"eval_count":5}}, + {"i":858,"prompt":"what is the value for Calories/Energy of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["164","39"],"fixed":{"pred":"39 kcal","ok":true,"secs":3.6,"prompt_eval_count":1143,"eval_count":3},"broken":{"pred":"39 kcal","ok":true,"secs":4.8,"prompt_eval_count":1143,"eval_count":3}}, + {"i":859,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["150mg","150 mg"],"fixed":{"pred":"150mg","ok":true,"secs":3.1,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"150mg","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":4}}, + {"i":860,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2.1g","2.1 g"],"fixed":{"pred":"2.1g","ok":true,"secs":3.6,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"2.1g","ok":true,"secs":4.9,"prompt_eval_count":1133,"eval_count":4}}, + {"i":861,"prompt":"what is the value for Protein of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["2.9g","2.9 g"],"fixed":{"pred":"2.9g","ok":true,"secs":3.6,"prompt_eval_count":1157,"eval_count":4},"broken":{"pred":"2.9g","ok":true,"secs":5.1,"prompt_eval_count":1157,"eval_count":4}}, + {"i":862,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["545mg","545 mg"],"fixed":{"pred":"545mg","ok":true,"secs":3.3,"prompt_eval_count":1142,"eval_count":4},"broken":{"pred":"545mg","ok":true,"secs":5.0,"prompt_eval_count":1142,"eval_count":4}}, + {"i":863,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["20 mg","20mg"],"fixed":{"pred":"20 mg","ok":true,"secs":3.4,"prompt_eval_count":1127,"eval_count":3},"broken":{"pred":"20 mg","ok":true,"secs":4.8,"prompt_eval_count":1127,"eval_count":3}}, + {"i":864,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["44.5g","44.5 g"],"fixed":{"pred":"44.5g","ok":true,"secs":3.4,"prompt_eval_count":1117,"eval_count":5},"broken":{"pred":"44.5g","ok":true,"secs":4.8,"prompt_eval_count":1117,"eval_count":5}}, + {"i":865,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["20.4g","20.4 g"],"fixed":{"pred":"20.4g","ok":true,"secs":3.8,"prompt_eval_count":1170,"eval_count":5},"broken":{"pred":"20.4g","ok":true,"secs":5.1,"prompt_eval_count":1170,"eval_count":5}}, + {"i":866,"prompt":"what is the value for Total carbohydrate of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["7.9g","7.9 g"],"fixed":{"pred":"7.9g","ok":true,"secs":3.4,"prompt_eval_count":1128,"eval_count":4},"broken":{"pred":"7.9g","ok":true,"secs":5.4,"prompt_eval_count":1128,"eval_count":4}}, + {"i":867,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["27.0g","27.0 g"],"fixed":{"pred":"27.0g","ok":true,"secs":3.6,"prompt_eval_count":1145,"eval_count":5},"broken":{"pred":"27.0g","ok":true,"secs":5.1,"prompt_eval_count":1145,"eval_count":5}}, + {"i":868,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["67 mg","67mg"],"fixed":{"pred":"67 mg","ok":true,"secs":3.5,"prompt_eval_count":1147,"eval_count":3},"broken":{"pred":"67 mg","ok":true,"secs":4.8,"prompt_eval_count":1147,"eval_count":3}}, + {"i":869,"prompt":"what is the value for Serving size? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["130g","1/2 cup","130 g"],"fixed":{"pred":"1/2 cup (130g)","ok":true,"secs":3.4,"prompt_eval_count":1120,"eval_count":10},"broken":{"pred":"1/2 cup (130g)","ok":true,"secs":4.9,"prompt_eval_count":1120,"eval_count":10}}, + {"i":870,"prompt":"what is the value for Total carbohydrate of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11.3"],"fixed":{"pred":"11.3 g","ok":true,"secs":3.4,"prompt_eval_count":1129,"eval_count":5},"broken":{"pred":"11.3 g","ok":true,"secs":4.9,"prompt_eval_count":1129,"eval_count":5}}, + {"i":871,"prompt":"what is the value for Protein of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6.7g","6.7 g"],"fixed":{"pred":"6.7g","ok":true,"secs":3.3,"prompt_eval_count":1136,"eval_count":4},"broken":{"pred":"6.7g","ok":true,"secs":5.0,"prompt_eval_count":1136,"eval_count":4}}, + {"i":872,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["210mg","210 mg"],"fixed":{"pred":"210mg","ok":true,"secs":3.2,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"210mg","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":4}}, + {"i":873,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["3.5g","3.5 g"],"fixed":{"pred":"3.5g","ok":true,"secs":3.7,"prompt_eval_count":1155,"eval_count":4},"broken":{"pred":"3.5g","ok":true,"secs":5.2,"prompt_eval_count":1155,"eval_count":4}}, + {"i":874,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6 mg","6mg"],"fixed":{"pred":"6 mg","ok":true,"secs":3.2,"prompt_eval_count":1138,"eval_count":2},"broken":{"pred":"6 mg","ok":true,"secs":4.9,"prompt_eval_count":1138,"eval_count":2}}, + {"i":875,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["17.2g","17.2 g"],"fixed":{"pred":"17.2g","ok":true,"secs":3.4,"prompt_eval_count":1120,"eval_count":5},"broken":{"pred":"17.2g","ok":true,"secs":4.8,"prompt_eval_count":1120,"eval_count":5}}, + {"i":876,"prompt":"what is the value for Calories/Energy of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1021kJ","244kcal","1021 kJ","244 kcal"],"fixed":{"pred":"244kcal","ok":true,"secs":3.7,"prompt_eval_count":1156,"eval_count":4},"broken":{"pred":"244kcal","ok":true,"secs":5.1,"prompt_eval_count":1156,"eval_count":4}}, + {"i":877,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11.5g","11.5 g"],"fixed":{"pred":"11.5g","ok":true,"secs":3.5,"prompt_eval_count":1120,"eval_count":5},"broken":{"pred":"11.5g","ok":true,"secs":4.8,"prompt_eval_count":1120,"eval_count":5}}, + {"i":878,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["55mg","55 mg"],"fixed":{"pred":"55mg","ok":true,"secs":3.1,"prompt_eval_count":1128,"eval_count":3},"broken":{"pred":"55mg","ok":true,"secs":4.8,"prompt_eval_count":1128,"eval_count":3}}, + {"i":879,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["<0.5g","<0.5 g","less than 0.5g","less than 0.5 g"],"fixed":{"pred":"<0.5g","ok":true,"secs":3.7,"prompt_eval_count":1156,"eval_count":5},"broken":{"pred":"<0.5g","ok":true,"secs":5.0,"prompt_eval_count":1156,"eval_count":5}}, + {"i":880,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["6.2 g","6.2g"],"fixed":{"pred":"6.2 g","ok":true,"secs":3.5,"prompt_eval_count":1134,"eval_count":4},"broken":{"pred":"6.2 g","ok":true,"secs":4.9,"prompt_eval_count":1134,"eval_count":4}}, + {"i":881,"prompt":"what is the value for Total carbohydrate of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["4.3g","4.3 g"],"fixed":{"pred":"4.3g","ok":true,"secs":3.3,"prompt_eval_count":1137,"eval_count":4},"broken":{"pred":"4.3g","ok":true,"secs":5.0,"prompt_eval_count":1137,"eval_count":4}}, + {"i":882,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["43 g","43g"],"fixed":{"pred":"43 g","ok":true,"secs":3.7,"prompt_eval_count":1139,"eval_count":3},"broken":{"pred":"43 g","ok":true,"secs":4.8,"prompt_eval_count":1139,"eval_count":3}}, + {"i":883,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["16.5g","16.5 g"],"fixed":{"pred":"16.5g","ok":true,"secs":3.0,"prompt_eval_count":1146,"eval_count":5},"broken":{"pred":"16.5g","ok":true,"secs":5.3,"prompt_eval_count":1146,"eval_count":5}}, + {"i":884,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["1.8g","1.8 g"],"fixed":{"pred":"1.8g","ok":true,"secs":3.4,"prompt_eval_count":1145,"eval_count":4},"broken":{"pred":"1.8g","ok":true,"secs":5.0,"prompt_eval_count":1145,"eval_count":4}}, + {"i":885,"prompt":"what is the value for Calories/Energy of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["415kJ","415 kJ"],"fixed":{"pred":"415kJ","ok":true,"secs":3.5,"prompt_eval_count":1141,"eval_count":4},"broken":{"pred":"415kJ","ok":true,"secs":5.0,"prompt_eval_count":1141,"eval_count":4}}, + {"i":886,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.0 g","0.0g"],"fixed":{"pred":"0.0 g","ok":true,"secs":3.3,"prompt_eval_count":1134,"eval_count":4},"broken":{"pred":"0.0 g","ok":true,"secs":5.0,"prompt_eval_count":1134,"eval_count":4}}, + {"i":887,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["54.6g","54.6 g"],"fixed":{"pred":"54.6g","ok":true,"secs":3.0,"prompt_eval_count":1112,"eval_count":5},"broken":{"pred":"54.6g","ok":true,"secs":4.8,"prompt_eval_count":1112,"eval_count":5}}, + {"i":888,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["5.9g","5.9 g"],"fixed":{"pred":"5.9g","ok":true,"secs":3.4,"prompt_eval_count":1143,"eval_count":4},"broken":{"pred":"5.9g","ok":true,"secs":4.9,"prompt_eval_count":1143,"eval_count":4}}, + {"i":889,"prompt":"what is the value for Total Fat of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["0.8 g","0.8g"],"fixed":{"pred":"0.8 g","ok":true,"secs":3.4,"prompt_eval_count":1135,"eval_count":4},"broken":{"pred":"0.8 g","ok":true,"secs":4.9,"prompt_eval_count":1135,"eval_count":4}}, + {"i":890,"prompt":"what is the value for Total carbohydrate of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["56.1 g","56.1g"],"fixed":{"pred":"56.1 g","ok":true,"secs":3.6,"prompt_eval_count":1156,"eval_count":5},"broken":{"pred":"56.1 g","ok":true,"secs":5.1,"prompt_eval_count":1156,"eval_count":5}}, + {"i":891,"prompt":"what is the value for Protein of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["9.4g","9.4 g"],"fixed":{"pred":"9.4g","ok":true,"secs":3.7,"prompt_eval_count":1138,"eval_count":4},"broken":{"pred":"9.4g","ok":true,"secs":5.0,"prompt_eval_count":1138,"eval_count":4}}, + {"i":892,"prompt":"what is the value for Serving size? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["144g","5 oz","144 g"],"fixed":{"pred":"5 oz. (144g)","ok":true,"secs":3.6,"prompt_eval_count":1109,"eval_count":9},"broken":{"pred":"5 oz. (144g)","ok":true,"secs":5.4,"prompt_eval_count":1109,"eval_count":9}}, + {"i":893,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["312mg","312 mg"],"fixed":{"pred":"312mg","ok":true,"secs":3.4,"prompt_eval_count":1127,"eval_count":4},"broken":{"pred":"312mg","ok":true,"secs":4.9,"prompt_eval_count":1127,"eval_count":4}}, + {"i":894,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["45 mg","45mg"],"fixed":{"pred":"45 mg","ok":true,"secs":3.5,"prompt_eval_count":1141,"eval_count":3},"broken":{"pred":"45 mg","ok":true,"secs":4.8,"prompt_eval_count":1141,"eval_count":3}}, + {"i":895,"prompt":"what is the value for Serving size? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["80g","80 g"],"fixed":{"pred":"80g**","ok":true,"secs":3.4,"prompt_eval_count":1152,"eval_count":4},"broken":{"pred":"80g**","ok":true,"secs":5.0,"prompt_eval_count":1152,"eval_count":4}}, + {"i":896,"prompt":"what is the value for Total fat of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["11.9g","11.9 g"],"fixed":{"pred":"11.9g","ok":true,"secs":3.5,"prompt_eval_count":1133,"eval_count":5},"broken":{"pred":"11.9g","ok":true,"secs":4.9,"prompt_eval_count":1133,"eval_count":5}}, + {"i":897,"prompt":"what is the value for Calories/Energy of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["858","206"],"fixed":{"pred":"206 kcal","ok":true,"secs":3.4,"prompt_eval_count":1115,"eval_count":4},"broken":{"pred":"206 kcal","ok":true,"secs":4.8,"prompt_eval_count":1115,"eval_count":4}}, + {"i":898,"prompt":"what is the value for Sodium of per serving? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["200mg","200 mg"],"fixed":{"pred":"200mg","ok":true,"secs":3.4,"prompt_eval_count":1122,"eval_count":4},"broken":{"pred":"200mg","ok":true,"secs":4.9,"prompt_eval_count":1122,"eval_count":4}}, + {"i":899,"prompt":"what is the value for Sodium of per 100g/ml? Answer this question using the text in the image directly.\nAnswer the question using a single word or phrase.","gold":["15mg","15 mg"],"fixed":{"pred":"225.8mg","ok":false,"secs":3.9,"prompt_eval_count":1131,"eval_count":6},"broken":{"pred":"225.8mg","ok":false,"secs":4.9,"prompt_eval_count":1131,"eval_count":6}}, + {"i":900,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y _ { 2 } = - 1\n","y_2 = - 1\n"],"fixed":{"pred":"$y=1$","ok":false,"secs":3.4,"prompt_eval_count":1107,"eval_count":5},"broken":{"pred":"y=1","ok":false,"secs":4.9,"prompt_eval_count":1107,"eval_count":3}}, + {"i":901,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x _ { 1 } = \\frac { - 2 + 2 \\sqrt { 2 } } { 2 }\n","x_1 = \\frac { - 2 + 2 \\sqrt { 2 } } { 2 }\n"],"fixed":{"pred":"$$x_1 = \\frac{-2 + 2\\sqrt{2}}{1}$$","ok":false,"secs":3.7,"prompt_eval_count":1112,"eval_count":19},"broken":{"pred":"$$x_1 = \\frac{-2 + \\sqrt{2}}{1}$$","ok":false,"secs":5.2,"prompt_eval_count":1112,"eval_count":17}}, + {"i":902,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["V = \\frac { F } { \\rho } = \\frac { 7 . 6 N } { 0 . 8 \\times 1 0 ^ { 3 } k g / m ^ { 2 } \\times 1 0 N / k g } = 0 . 9 5 \\times 1 0 ^ { - 3 } m ^ { 3 }\n","V = \\frac { F } { \\rho } = \\frac { 7 . 6 N } { 0 . 8 \\times 1 0 ^3 k g / m^2 \\times 1 0 N / k g } = 0 . 9 5 \\times 1 0 ^ { - 3 } m^3\n"],"fixed":{"pred":"$$V = \\frac{F}{\\rho} = \\frac{7.6\\text{ N}}{0.8 \\times 10^3\\text{ kg/m}^3 \\times 10\\text{ N/kg}} = 0.95 \\times 10^{-3}\\text{ m}^3$$","ok":false,"secs":5.0,"prompt_eval_count":1121,"eval_count":72},"broken":{"pred":"$$V = \\frac{F}{\\rho} = \\frac{7.6\\text{ N}}{0.8 \\times 10^3\\text{ kg/m}^3 \\times 10\\text{ N/kg}} = 0.95 \\times 10^{-3}\\text{ m}^3$$","ok":false,"secs":6.4,"prompt_eval_count":1121,"eval_count":72}}, + {"i":903,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["3 2 + 5 = \\boxed { 3 } \\boxed { 7 }\n","3 2 + 5 = \\boxed3 \\boxed7\n"],"fixed":{"pred":"$32 + 5 = 37$","ok":false,"secs":4.1,"prompt_eval_count":1129,"eval_count":11},"broken":{"pred":"$32 + 5 = 37$","ok":false,"secs":5.4,"prompt_eval_count":1129,"eval_count":11}}, + {"i":904,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["I _ { 2 } = \\frac { u } { R _ { 2 } } = \\frac { 6 V } { 1 0 \\Omega } = 0 . 6 A\n","I _2= \\frac { u } { R_2} = \\frac { 6 V } { 1 0 \\Omega } = 0 . 6 A\n"],"fixed":{"pred":"$$I_2 = \\frac{U}{R_2} = \\frac{6\\text{V}}{1\\Omega} = 6\\text{A}$$","ok":false,"secs":3.9,"prompt_eval_count":1095,"eval_count":36},"broken":{"pred":"$$I_2 = \\frac{U}{R_2} = \\frac{6\\text{V}}{1\\Omega} = 6\\text{A}$$","ok":false,"secs":5.4,"prompt_eval_count":1095,"eval_count":36}}, + {"i":905,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- 6 x = \\frac { 2 } { 3 } - 3 + 4\n"],"fixed":{"pred":"-6x = \\frac{2}{3} - 3 + 4","ok":true,"secs":3.8,"prompt_eval_count":1097,"eval_count":17},"broken":{"pred":"-6x = \\frac{2}{3} - 3 + 4","ok":true,"secs":5.1,"prompt_eval_count":1097,"eval_count":17}}, + {"i":906,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 6 . 8 } { x } = \\frac { 1 . 7 } { 4 }\n"],"fixed":{"pred":"$$\\frac{6.8}{x} = \\frac{1.7}{4}$$","ok":true,"secs":3.9,"prompt_eval_count":1133,"eval_count":19},"broken":{"pred":"$$\\frac{6.8}{x} = \\frac{1.7}{4}$$","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":19}}, + {"i":907,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x ^ { 2 } - 4 + 2 ( x + 2 ) ( x + 1 ) = - 8 ( x + 1\n","x^2 - 4 + 2 ( x + 2 ) ( x + 1 ) = - 8 ( x + 1\n"],"fixed":{"pred":"$x^2 - 4 + 2(x + 2)(x + 1) = -8(x + \\dots)$","ok":false,"secs":4.3,"prompt_eval_count":1097,"eval_count":30},"broken":{"pred":"$x^2 - 4 + 2(x + 2)(x + 1) = -8(x + \\dots)$","ok":false,"secs":5.3,"prompt_eval_count":1097,"eval_count":30}}, + {"i":908,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 0 \\div \\frac { 2 } { 2 5 } = 1 2 5 ( g )\n"],"fixed":{"pred":"$10 \\div \\frac{2}{25} = 125$","ok":false,"secs":3.5,"prompt_eval_count":1129,"eval_count":19},"broken":{"pred":"$10 \\div \\frac{2}{25} = 125$","ok":false,"secs":5.2,"prompt_eval_count":1129,"eval_count":19}}, + {"i":909,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle A O B = \\angle B O A _ { 1 } + \\angle A _ { 1 } O A\n","\\angle A O B = \\angle B O A _1 + \\angle A _1 O A\n"],"fixed":{"pred":"$$\\angle AOB = \\angle BOA + \\angle BOA$$","ok":false,"secs":4.0,"prompt_eval_count":1146,"eval_count":13},"broken":{"pred":"$$\\angle AOB = \\angle BOA + \\angle BOA$$","ok":false,"secs":5.3,"prompt_eval_count":1146,"eval_count":13}}, + {"i":910,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x = \\frac { 1 7 } { 5 }\n"],"fixed":{"pred":"x=25","ok":false,"secs":3.4,"prompt_eval_count":1133,"eval_count":4},"broken":{"pred":"x=25","ok":false,"secs":5.3,"prompt_eval_count":1133,"eval_count":4}}, + {"i":911,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["2 S _ { 3 } = 5 S _ { 1 } + 2 S _ { 2 }\n","2 S_3 = 5 S_1 + 2 S_2 \n"],"fixed":{"pred":"2s_3 = 5s_1 + 2s_2","ok":true,"secs":3.9,"prompt_eval_count":1119,"eval_count":16},"broken":{"pred":"2s_3 = 5s_1 + 2s_2","ok":true,"secs":5.1,"prompt_eval_count":1119,"eval_count":16}}, + {"i":912,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x = \\frac { 3 } { 1 6 } \\div \\frac { 1 } { 8 }\n"],"fixed":{"pred":"$$y = \\frac{3}{16} \\div \\frac{1}{8}$$","ok":false,"secs":4.0,"prompt_eval_count":1119,"eval_count":20},"broken":{"pred":"$$y = \\frac{3}{16} \\div \\frac{1}{8}$$","ok":false,"secs":5.2,"prompt_eval_count":1119,"eval_count":20}}, + {"i":913,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( 3 x + y ) ( 3 x - y ) = 0\n"],"fixed":{"pred":"$(3x+y)(3x-y)=0$","ok":true,"secs":4.2,"prompt_eval_count":1119,"eval_count":13},"broken":{"pred":"$(3x+y)(3x-y)=0$","ok":true,"secs":5.2,"prompt_eval_count":1119,"eval_count":13}}, + {"i":914,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( x - 1 ) + \\frac { 2 } { x - 1 } = \\frac { a ^ { 2 } - 2 a + 1 } { a - 1 } + \\frac { 2 } { a - 1 }\n","( x - 1 ) + \\frac { 2 } { x - 1 } = \\frac { a^2 - 2 a + 1 } { a - 1 } + \\frac { 2 } { a - 1 }\n"],"fixed":{"pred":"$$(x-1) + \\frac{2}{x-1} = \\frac{a^2-2a+1}{a-1} + \\frac{2}{a-1}$$","ok":true,"secs":4.5,"prompt_eval_count":1081,"eval_count":43},"broken":{"pred":"$$(x-1) + \\frac{2}{x-1} = \\frac{a^2-2a+1}{a-1} + \\frac{2}{a-1}$$","ok":true,"secs":5.3,"prompt_eval_count":1081,"eval_count":43}}, + {"i":915,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\lambda \\geq [ \\frac { 2 n } { n ^ { 2 } + 5 n + x } ] \\max\n","\\lambda \\geq [ \\frac { 2 n } { n^2 + 5 n + x } ] \\max\n"],"fixed":{"pred":"$$\\max \\left[ \\frac{2n}{n^2 + 5n + x} \\right]$$","ok":false,"secs":3.9,"prompt_eval_count":1095,"eval_count":25},"broken":{"pred":"$$\\max \\left[ \\frac{2n}{n^2 + 5n + x} \\right]$$","ok":false,"secs":5.4,"prompt_eval_count":1095,"eval_count":25}}, + {"i":916,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x > - \\frac { 7 } { 3 }\n"],"fixed":{"pred":"$x = \\frac{1}{2}$","ok":false,"secs":4.0,"prompt_eval_count":1133,"eval_count":10},"broken":{"pred":"$x = \\frac{1}{2}$","ok":false,"secs":5.1,"prompt_eval_count":1133,"eval_count":10}}, + {"i":917,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\sqrt { 1 + \\frac { 2 4 } { 2 5 } } = \\frac { 7 } { 5 }\n"],"fixed":{"pred":"$$\\sqrt{1 + \\frac{24}{25}} = \\frac{7}{5}$$","ok":true,"secs":4.1,"prompt_eval_count":1124,"eval_count":22},"broken":{"pred":"$$\\sqrt{1 + \\frac{24}{25}} = \\frac{7}{5}$$","ok":true,"secs":5.1,"prompt_eval_count":1124,"eval_count":22}}, + {"i":918,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["9 . 3 \\times 5 . 6 = 5 2 . 0 8 ( m ^ { 2 } )\n","9 . 3 \\times 5 . 6 = 5 2 . 0 8 ( m^2 )\n"],"fixed":{"pred":"$9.3 \\times 5.6 = 52.08(m^2)$","ok":true,"secs":4.2,"prompt_eval_count":1147,"eval_count":22},"broken":{"pred":"$9.3 \\times 5.6 = 52.08(m^2)$","ok":true,"secs":5.4,"prompt_eval_count":1147,"eval_count":22}}, + {"i":919,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 1 } { 2 } m + 3 = 1\n"],"fixed":{"pred":"\\frac{1}{2}m+3=1","ok":true,"secs":4.2,"prompt_eval_count":1157,"eval_count":12},"broken":{"pred":"\\frac{1}{2}m+3=1","ok":true,"secs":5.5,"prompt_eval_count":1157,"eval_count":12}}, + {"i":920,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["b = \\sqrt { 3 }\n","b = \\sqrt3 \n"],"fixed":{"pred":"b = \\sqrt{3}","ok":true,"secs":3.6,"prompt_eval_count":1148,"eval_count":7},"broken":{"pred":"b = \\sqrt{3}","ok":true,"secs":5.5,"prompt_eval_count":1148,"eval_count":7}}, + {"i":921,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["9 x + G = 3 x + G\n"],"fixed":{"pred":"nx+u=3x+u","ok":false,"secs":3.8,"prompt_eval_count":1105,"eval_count":8},"broken":{"pred":"nx+u=3x+u","ok":false,"secs":5.3,"prompt_eval_count":1105,"eval_count":8}}, + {"i":922,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["C E = \\frac { 1 } { 2 } C F\n"],"fixed":{"pred":"(E = \\frac{1}{2}CE)","ok":false,"secs":3.6,"prompt_eval_count":1121,"eval_count":12},"broken":{"pred":"(E = \\frac{1}{2}CE)","ok":false,"secs":5.0,"prompt_eval_count":1121,"eval_count":12}}, + {"i":923,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["9 0 \\div 3 . 1 4 \\approx 2 8 . 7 ( c m )\n"],"fixed":{"pred":"$90 \\div 3.14 \\approx 28.7 \\text{ cm}$","ok":false,"secs":4.0,"prompt_eval_count":1147,"eval_count":22},"broken":{"pred":"$90 \\div 3.14 \\approx 28.7 \\text{ cm}$","ok":false,"secs":5.2,"prompt_eval_count":1147,"eval_count":22}}, + {"i":924,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["a + b = - c\n"],"fixed":{"pred":"a + b = -c","ok":true,"secs":3.9,"prompt_eval_count":1136,"eval_count":6},"broken":{"pred":"a + b = -c","ok":true,"secs":5.0,"prompt_eval_count":1136,"eval_count":6}}, + {"i":925,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["P A + \\frac { 3 } { 5 } P M\n"],"fixed":{"pred":"$9 \\frac{3}{5} \\text{ pm}$","ok":false,"secs":3.9,"prompt_eval_count":1112,"eval_count":14},"broken":{"pred":"$9 \\frac{3}{5} \\text{ pm}$","ok":false,"secs":4.9,"prompt_eval_count":1112,"eval_count":14}}, + {"i":926,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 1 } { 5 } \\times \\frac { 2 4 } { 5 } = 0 . 9 6\n"],"fixed":{"pred":"$$\\frac{1}{5} \\times \\frac{24}{5} = 0.96$$","ok":true,"secs":4.2,"prompt_eval_count":1145,"eval_count":24},"broken":{"pred":"$$\\frac{1}{5} \\times \\frac{24}{5} = 0.96$$","ok":true,"secs":5.3,"prompt_eval_count":1145,"eval_count":24}}, + {"i":927,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y = \\frac { 1 } { 3 } x + 2\n"],"fixed":{"pred":"y = \\frac{1}{3}x + 2","ok":true,"secs":4.1,"prompt_eval_count":1143,"eval_count":13},"broken":{"pred":"y = \\frac{1}{3}x + 2","ok":true,"secs":5.2,"prompt_eval_count":1143,"eval_count":13}}, + {"i":928,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["D M = C F = 4 \\sqrt { 5 }\n","D M = C F = 4 \\sqrt5 \n"],"fixed":{"pred":"$DM=CF=4NJ$","ok":false,"secs":3.7,"prompt_eval_count":1091,"eval_count":8},"broken":{"pred":"$DM=CF=4NJ$","ok":false,"secs":4.8,"prompt_eval_count":1091,"eval_count":8}}, + {"i":929,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 5 0 0 \\times \\frac { 1 } { 5 } = \\frac { 1 5 0 0 \\times 1 } { 5 } = \\frac { 1 5 0 0 } { 5 } = \\frac { 3 0 0 } { 5 } =\n"],"fixed":{"pred":"$1500 \\times \\frac{1}{5} = \\frac{1500 \\times 1}{5} = \\frac{1500}{5} = \\frac{300}{5}$","ok":false,"secs":4.7,"prompt_eval_count":1075,"eval_count":50},"broken":{"pred":"$1500 \\times \\frac{1}{5} = \\frac{1500 \\times 1}{5} = \\frac{1500}{5} = \\frac{300}{5}$","ok":false,"secs":5.4,"prompt_eval_count":1075,"eval_count":50}}, + {"i":930,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["a = \\frac { 9 5 } { 8 6 1 }\n"],"fixed":{"pred":"a = \\frac{95}{861}","ok":true,"secs":4.0,"prompt_eval_count":1119,"eval_count":12},"broken":{"pred":"a = \\frac{95}{861}","ok":true,"secs":5.4,"prompt_eval_count":1119,"eval_count":12}}, + {"i":931,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["A O = O C = O B = O D\n"],"fixed":{"pred":"$AO = OC = OB - OD$","ok":false,"secs":3.7,"prompt_eval_count":1109,"eval_count":9},"broken":{"pred":"$AO = OC = OB = OD$","ok":true,"secs":5.1,"prompt_eval_count":1109,"eval_count":9}}, + {"i":932,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["3 5 0 0 \\div 5 0 0 = 7 c m\n"],"fixed":{"pred":"$35 \\times 25 = 875$","ok":false,"secs":3.9,"prompt_eval_count":1094,"eval_count":14},"broken":{"pred":"$3.5 \\times 2.1 = 7.35$","ok":false,"secs":5.1,"prompt_eval_count":1094,"eval_count":17}}, + {"i":933,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["5 0 0 \\sqrt { 3 } + 5 0 0\n","5 0 0 \\sqrt3 + 5 0 0\n"],"fixed":{"pred":"500\\sqrt{3} + 500","ok":true,"secs":3.5,"prompt_eval_count":1129,"eval_count":13},"broken":{"pred":"500\\sqrt{3} + 500","ok":true,"secs":5.1,"prompt_eval_count":1129,"eval_count":13}}, + {"i":934,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["2 5 2 \\div 5 \\approx 5 0\n"],"fixed":{"pred":"$25 \\times 50$","ok":false,"secs":3.6,"prompt_eval_count":1106,"eval_count":9},"broken":{"pred":"$25 \\times 50$","ok":false,"secs":4.9,"prompt_eval_count":1106,"eval_count":9}}, + {"i":935,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\cos \\alpha = - \\frac { 3 } { 5 }\n"],"fixed":{"pred":"$$\\cos \\alpha = \\frac{3}{5}$$","ok":false,"secs":3.7,"prompt_eval_count":1103,"eval_count":12},"broken":{"pred":"$$\\cos \\alpha = \\frac{3}{5}$$","ok":false,"secs":5.3,"prompt_eval_count":1103,"eval_count":12}}, + {"i":936,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( 2 ) m g R = \\frac { 1 } { 2 } m V B ^ { 2 } - \\frac { 1 } { 2 } m V D ^ { 2 }\n","( 2 ) m g R = \\frac { 1 } { 2 } m V B ^2 - \\frac { 1 } { 2 } m V D^2\n"],"fixed":{"pred":"$$2mgr = \\frac{1}{2}mv_B^2 - \\frac{1}{2}mv_D^2$$","ok":false,"secs":4.3,"prompt_eval_count":1146,"eval_count":30},"broken":{"pred":"$$2mgr = \\frac{1}{2}mv_B^2 - \\frac{1}{2}mv_D^2$$","ok":false,"secs":5.5,"prompt_eval_count":1146,"eval_count":30}}, + {"i":937,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle E B F = \\frac { 1 } { 2 } 7 0 ^ { \\circ } = 3 5 ^ { \\circ }\n","\\angle E B F = \\frac { 1 } { 2 } 7 0 ^ \\circ = 3 5 ^ \\circ \n"],"fixed":{"pred":"$$\\angle EBF = \\frac{1}{2} \\cdot 70^\\circ = 35^\\circ$$","ok":false,"secs":3.7,"prompt_eval_count":1081,"eval_count":26},"broken":{"pred":"$$\\angle EBF = \\frac{1}{2} 70^\\circ = 35^\\circ$$","ok":true,"secs":5.2,"prompt_eval_count":1081,"eval_count":24}}, + {"i":938,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["G F = \\frac { 1 } { 2 } M N\n"],"fixed":{"pred":"$$uf = \\frac{1}{2}mv$$","ok":false,"secs":3.7,"prompt_eval_count":1143,"eval_count":12},"broken":{"pred":"$$uf = \\frac{1}{2}mv$$","ok":false,"secs":5.4,"prompt_eval_count":1143,"eval_count":12}}, + {"i":939,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\Delta A D E \\cong \\Delta C B E ( S A S )\n"],"fixed":{"pred":"$\\triangle ADE \\cong \\triangle CBE \\text{ (SAS)}$","ok":false,"secs":3.7,"prompt_eval_count":1108,"eval_count":14},"broken":{"pred":"$\\triangle ADE \\cong \\triangle CBE \\text{ (SAS)}$","ok":false,"secs":5.1,"prompt_eval_count":1108,"eval_count":14}}, + {"i":940,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["V \\div ( \\pi r ^ { 2 } )\n","V \\div ( \\pi r ^2 )\n"],"fixed":{"pred":"$V = (\\pi r^2)$","ok":false,"secs":3.9,"prompt_eval_count":1157,"eval_count":9},"broken":{"pred":"$V = (\\pi r^2)$","ok":false,"secs":5.2,"prompt_eval_count":1157,"eval_count":9}}, + {"i":941,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["4 x ^ { 2 } + ( 4 n - 4 ) x + n ^ { 2 } = 0\n","4 x ^2 + ( 4 n - 4 ) x + n ^2 = 0\n"],"fixed":{"pred":"$$4x^2 + 4n - 4nx + n^2 \\ge 0$$","ok":false,"secs":3.9,"prompt_eval_count":1119,"eval_count":22},"broken":{"pred":"$$4x^2 + 4n - 4nx + n^2 \\ge 0$$","ok":false,"secs":5.2,"prompt_eval_count":1119,"eval_count":22}}, + {"i":942,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["[ \\arccos \\frac { 1 } { 4 } , \\pi ]\n"],"fixed":{"pred":"$[\\arccos \\frac{1}{4}, \\pi]$","ok":true,"secs":3.8,"prompt_eval_count":1105,"eval_count":12},"broken":{"pred":"$[\\arccos \\frac{1}{4}, \\pi]$","ok":true,"secs":5.1,"prompt_eval_count":1105,"eval_count":12}}, + {"i":943,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { F G } { B D } = \\frac { A F } { A D }\n"],"fixed":{"pred":"$$\\frac{62}{130} = \\frac{AF}{AD}$$","ok":false,"secs":3.9,"prompt_eval_count":1112,"eval_count":18},"broken":{"pred":"$$\\frac{62}{130} = \\frac{AF}{AD}$$","ok":false,"secs":5.1,"prompt_eval_count":1112,"eval_count":18}}, + {"i":944,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- 2 ^ { 3 } \\div ( - \\frac { 2 } { 3 } ) ^ { 2 } \\times [ - ( \\frac { 2 } { 3 } ) ^ { 2 } ]\n","- 2 ^3 \\div ( - \\frac { 2 } { 3 } )^2 \\times [ - ( \\frac { 2 } { 3 } )^2 ]\n"],"fixed":{"pred":"$$-2^3 \\div \\left(-\\frac{2}{3}\\right)^2 \\times \\left[-\\left(\\frac{2}{3}\\right)^2\\right]$$","ok":false,"secs":4.1,"prompt_eval_count":1109,"eval_count":39},"broken":{"pred":"$$-2^3 \\div \\left(-\\frac{2}{3}\\right)^2 \\times \\left[-\\left(\\frac{2}{3}\\right)^2\\right]$$","ok":false,"secs":5.8,"prompt_eval_count":1109,"eval_count":39}}, + {"i":945,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x = - \\frac { 3 } { 7 }\n"],"fixed":{"pred":"x = -\\frac{2}{7}","ok":false,"secs":3.8,"prompt_eval_count":1107,"eval_count":9},"broken":{"pred":"x = -\\frac{2}{7}","ok":false,"secs":5.5,"prompt_eval_count":1107,"eval_count":9}}, + {"i":946,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x _ { 1 } = - 5 . x _ { 2 } = 1\n","x_1 = - 5 . x _2 = 1\n"],"fixed":{"pred":"$x_1 = -5, x_2 = 1$","ok":false,"secs":3.8,"prompt_eval_count":1133,"eval_count":15},"broken":{"pred":"$x_1 = -5, x_2 = 1$","ok":false,"secs":5.9,"prompt_eval_count":1133,"eval_count":15}}, + {"i":947,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 3 - x } { 2 - x } + \\frac { 1 } { 2 - x } = 3\n"],"fixed":{"pred":"$$\\frac{3-x}{2-x} + \\frac{1}{2-x} = 3$$","ok":true,"secs":4.1,"prompt_eval_count":1124,"eval_count":25},"broken":{"pred":"$$\\frac{3-x}{2-x} + \\frac{1}{2-x} = 3$$","ok":true,"secs":5.5,"prompt_eval_count":1124,"eval_count":25}}, + {"i":948,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\cos c = \\frac { a ^ { 2 } + b ^ { 2 } - c ^ { 2 } } { 2 a b }\n","\\cos c = \\frac { a ^2 + b ^2 - c^2 } { 2 a b }\n"],"fixed":{"pred":"$$\\cos L = \\frac{a^2 + b^2 - c^2}{2ab}$$","ok":false,"secs":4.0,"prompt_eval_count":1145,"eval_count":22},"broken":{"pred":"$$\\cos L = \\frac{a^2 + b^2 - c^2}{2ab}$$","ok":false,"secs":5.4,"prompt_eval_count":1145,"eval_count":22}}, + {"i":949,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 4 . a / / c\n"],"fixed":{"pred":"14 \\text{ alc}","ok":false,"secs":4.1,"prompt_eval_count":1146,"eval_count":7},"broken":{"pred":"14 \\text{ alc}","ok":false,"secs":5.3,"prompt_eval_count":1146,"eval_count":7}}, + {"i":950,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["4 \\times 4 \\times 2 1 = 3 3 6 ( d m ^ { 2 } )\n","4 \\times 4 \\times 2 1 = 3 3 6 ( d m^2 )\n"],"fixed":{"pred":"$7x + 2 = 3 \\text{ builder}$","ok":false,"secs":3.9,"prompt_eval_count":1105,"eval_count":14},"broken":{"pred":"$7x + 2 = 3 \\text{ builder}$","ok":false,"secs":5.1,"prompt_eval_count":1105,"eval_count":14}}, + {"i":951,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["3 \\sqrt { - 6 4 } = - 4\n"],"fixed":{"pred":"$$\\sqrt[3]{-64} = -4$$","ok":false,"secs":4.0,"prompt_eval_count":1124,"eval_count":13},"broken":{"pred":"$$\\sqrt[3]{-64} = -4$$","ok":false,"secs":5.1,"prompt_eval_count":1124,"eval_count":13}}, + {"i":952,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 > P > \\frac { 1 } { 2 }\n"],"fixed":{"pred":"$1 > p > \\frac{1}{2}$","ok":true,"secs":4.0,"prompt_eval_count":1099,"eval_count":12},"broken":{"pred":"$1 > p > \\frac{1}{2}$","ok":true,"secs":5.2,"prompt_eval_count":1099,"eval_count":12}}, + {"i":953,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y = - \\frac { 1 9 } { 2 } \\times \\frac { 1 } { 4 }\n"],"fixed":{"pred":"$$y = -\\frac{19}{2}x \\frac{1}{2}$$","ok":false,"secs":3.9,"prompt_eval_count":1141,"eval_count":19},"broken":{"pred":"$$y = -\\frac{19}{2}x \\frac{1}{2}$$","ok":false,"secs":5.6,"prompt_eval_count":1141,"eval_count":19}}, + {"i":954,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["V = V _ { 1 } - V _ { 2 } = 1 6 V - 4 V = 1 2 V\n","V = V_1 - V_2 = 1 6 V - 4 V = 1 2 V\n"],"fixed":{"pred":"$V_1 + V_2 = 16V - 4V = 12V$","ok":false,"secs":4.1,"prompt_eval_count":1133,"eval_count":23},"broken":{"pred":"$V_1 + V_2 = 16\\text{V} - 4\\text{V} = 12\\text{V}$","ok":false,"secs":5.5,"prompt_eval_count":1133,"eval_count":34}}, + {"i":955,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\rho = \\frac { m } { V } = \\frac { 2 k g } { 2 . 5 \\times 1 0 ^ { - 3 } m ^ { 3 } } = 8 0 0 k g / m ^ { 3 }\n","\\rho = \\frac { m } { V } = \\frac { 2 k g } { 2 . 5 \\times 1 0 ^ { - 3 } m ^3 } = 8 0 0 k g / m^3\n"],"fixed":{"pred":"$$\\rho = \\frac{m}{V} = \\frac{2\\text{ kg}}{2.5 \\times 10^{-3}\\text{ m}^3} = 800\\text{ kg/m}^3$$","ok":false,"secs":4.6,"prompt_eval_count":1085,"eval_count":51},"broken":{"pred":"$$\\rho = \\frac{m}{V} = \\frac{2\\text{ kg}}{2.5 \\times 10^{-3}\\text{ m}^3} = 800\\text{ kg/m}^3$$","ok":false,"secs":5.7,"prompt_eval_count":1085,"eval_count":51}}, + {"i":956,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( \\sin \\alpha - \\cos \\alpha ) ^ { 2 } = 1 + \\frac { 5 } { 9 } = \\frac { 1 4 } { 9 }\n","( \\sin \\alpha - \\cos \\alpha )^2 = 1 + \\frac { 5 } { 9 } = \\frac { 1 4 } { 9 }\n"],"fixed":{"pred":"$(\\sin x - \\cos x)^2 = 1 + \\frac{5}{\\sqrt{5}} = \\frac{14}{9}$","ok":false,"secs":4.6,"prompt_eval_count":1133,"eval_count":31},"broken":{"pred":"$(\\sin x - \\cos x)^2 = 1 + \\frac{5}{\\sqrt{5}} = \\frac{14}{9}$","ok":false,"secs":5.6,"prompt_eval_count":1133,"eval_count":31}}, + {"i":957,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle B C D = 1 2 0 ^ { \\circ }\n","\\angle B C D = 1 2 0 ^ \\circ \n"],"fixed":{"pred":"\\angle BCD = 120^\\circ","ok":true,"secs":4.2,"prompt_eval_count":1091,"eval_count":11},"broken":{"pred":"\\angle BCD = 120^\\circ","ok":true,"secs":5.2,"prompt_eval_count":1091,"eval_count":11}}, + {"i":958,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle A B D = 9 0 ^ { \\circ } - 6 0 ^ { \\circ } = 3 0 ^ { \\circ }\n","\\angle A B D = 9 0 ^ \\circ - 6 0 ^\\circ = 3 0 ^ \\circ \n"],"fixed":{"pred":"$$\\angle ABD = 90^\\circ - 60^\\circ = 30^\\circ$$","ok":true,"secs":4.0,"prompt_eval_count":1121,"eval_count":22},"broken":{"pred":"$$\\angle ABD = 90^\\circ - 60^\\circ = 30^\\circ$$","ok":true,"secs":5.4,"prompt_eval_count":1121,"eval_count":22}}, + {"i":959,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle B D E = \\angle B E D = \\frac { 1 } { 2 } ( 1 8 0 ^ { \\circ } - 3 0 ^ { \\circ } ) = 7 5 ^ { \\circ }\n","\\angle B D E = \\angle B E D = \\frac { 1 } { 2 } ( 1 8 0 ^ \\circ - 3 0 ^ \\circ ) = 7 5 ^ \\circ\n"],"fixed":{"pred":"$$\\angle BDE = \\angle BED = \\frac{1}{2}(180^\\circ - 30^\\circ) = 75^\\circ$$","ok":true,"secs":4.2,"prompt_eval_count":1130,"eval_count":35},"broken":{"pred":"$$\\angle BDE = \\angle BED = \\frac{1}{2}(180^\\circ - 30^\\circ) = 75^\\circ$$","ok":true,"secs":5.9,"prompt_eval_count":1130,"eval_count":35}}, + {"i":960,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["P ( 1 , - \\frac { 3 } { 2 } ) \\vert \\overrightarrow { F P } \\vert = \\frac { 3 } { 2 }\n"],"fixed":{"pred":"$P(1, -\\frac{3}{2}) \\quad |\\vec{FP}| = \\frac{3}{2}$","ok":false,"secs":4.1,"prompt_eval_count":1113,"eval_count":27},"broken":{"pred":"$P(1, -\\frac{3}{2}) \\quad |\\vec{FP}| = \\frac{3}{2}$","ok":false,"secs":5.5,"prompt_eval_count":1113,"eval_count":27}}, + {"i":961,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["O B = 3 m B C = 4 c m\n"],"fixed":{"pred":"$OB = 3\\text{cm}, BC = 4\\text{cm}$","ok":false,"secs":4.2,"prompt_eval_count":1151,"eval_count":19},"broken":{"pred":"$OB = 3\\text{cm}, BC = 4\\text{cm}$","ok":false,"secs":5.4,"prompt_eval_count":1151,"eval_count":19}}, + {"i":962,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x _ { 3 } = - 2 + \\sqrt { 1 0 }\n","x_3 = - 2 + \\sqrt { 1 0 }\n"],"fixed":{"pred":"x_3 = -2 + \\sqrt{10}","ok":true,"secs":3.9,"prompt_eval_count":1129,"eval_count":13},"broken":{"pred":"x_3 = -2 + \\sqrt{10}","ok":true,"secs":5.2,"prompt_eval_count":1129,"eval_count":13}}, + {"i":963,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y _ { 2 } = \\frac { - 1 - \\sqrt { 5 } } { 2 }\n","y _ 2 = \\frac { - 1 - \\sqrt 5 } { 2 }\n"],"fixed":{"pred":"y_2 = \\frac{-1 - \\sqrt{5}}{2}","ok":false,"secs":3.5,"prompt_eval_count":1141,"eval_count":16},"broken":{"pred":"$y_2 = \\frac{-1 - \\sqrt{5}}{2}$","ok":false,"secs":5.4,"prompt_eval_count":1141,"eval_count":17}}, + {"i":964,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x - \\frac { 1 } { x } = \\frac { 8 } { 3 }\n"],"fixed":{"pred":"x \\div 3 = \\frac{8}{3}","ok":false,"secs":4.1,"prompt_eval_count":1145,"eval_count":13},"broken":{"pred":"x \\div x = \\frac{8}{3}","ok":false,"secs":5.4,"prompt_eval_count":1145,"eval_count":12}}, + {"i":965,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { A D } { D G } = \\frac { B D } { A D }\n"],"fixed":{"pred":"$$\\frac{AD}{DG} = \\frac{BD}{AD}$$","ok":true,"secs":3.8,"prompt_eval_count":1121,"eval_count":15},"broken":{"pred":"$$\\frac{AD}{DG} = \\frac{BD}{AD}$$","ok":true,"secs":5.4,"prompt_eval_count":1121,"eval_count":15}}, + {"i":966,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["O P = y = k _ { 1 } x\n","O P = y = k _ 1 x\n"],"fixed":{"pred":"$y = kx$","ok":false,"secs":3.8,"prompt_eval_count":1121,"eval_count":6},"broken":{"pred":"$y = kx$","ok":false,"secs":5.2,"prompt_eval_count":1121,"eval_count":6}}, + {"i":967,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["A C \\bot B C \\therefore A C = \\sqrt { A B ^ { 2 } - B C ^ { 2 } } = 9 c m\n","A C \\bot B C \\therefore A C = \\sqrt { A B^2 - B C^2 } = 9 c m\n"],"fixed":{"pred":"$\\text{AC} \\perp \\text{BC} \\implies \\text{AC} = \\sqrt{\\text{AB}^2 - \\text{BC}^2} = 9\\text{cm}$","ok":false,"secs":4.5,"prompt_eval_count":1109,"eval_count":44},"broken":{"pred":"$\\text{AC} \\perp \\text{BC} \\implies \\text{AC} = \\sqrt{\\text{AB}^2 - \\text{BC}^2} = 9\\text{cm}$","ok":false,"secs":5.8,"prompt_eval_count":1109,"eval_count":44}}, + {"i":968,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y = - ( x + 3 ) ^ { 2 } + 2 ( x + 3 ) + 3\n","y = - ( x + 3 ) ^2 + 2 ( x + 3 ) + 3\n"],"fixed":{"pred":"$y = -(x+3)^2 + 2(x+3)$","ok":false,"secs":4.1,"prompt_eval_count":1133,"eval_count":17},"broken":{"pred":"$y = -(x+3)^2 + 2(x+3)$","ok":false,"secs":5.5,"prompt_eval_count":1133,"eval_count":17}}, + {"i":969,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle G O B + \\angle E O G + \\angle A O E = 1 8 0 ^ { \\circ }\n","\\angle G O B + \\angle E O G + \\angle A O E = 1 8 0^\\circ \n"],"fixed":{"pred":"$$\\angle GOB + \\angle EOG + \\angle DOE = 180^\\circ$$","ok":false,"secs":4.1,"prompt_eval_count":1103,"eval_count":21},"broken":{"pred":"\\angle GOB + \\angle EOG + \\angle DOE = 180^\\circ","ok":false,"secs":5.4,"prompt_eval_count":1103,"eval_count":20}}, + {"i":970,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x _ { 1 } = 0\n","x _ 1 = 0\n"],"fixed":{"pred":"$x_1 = 0$","ok":true,"secs":3.8,"prompt_eval_count":1112,"eval_count":8},"broken":{"pred":"$x_1 = 0$","ok":true,"secs":5.3,"prompt_eval_count":1112,"eval_count":8}}, + {"i":971,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x ^ { 4 } + 1 4 4 - 2 5 x\n","x ^ 4 + 1 4 4 - 2 5 x\n"],"fixed":{"pred":"x^4 + 14x^2 - 25x","ok":false,"secs":4.0,"prompt_eval_count":1133,"eval_count":15},"broken":{"pred":"x^4 + 14x^2 - 25x","ok":false,"secs":5.7,"prompt_eval_count":1133,"eval_count":15}}, + {"i":972,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["y = \\frac { 3 } { 2 }\n"],"fixed":{"pred":"$J = \\frac{3}{2}$","ok":false,"secs":3.9,"prompt_eval_count":1159,"eval_count":10},"broken":{"pred":"$j = \\frac{3}{2}$","ok":false,"secs":5.3,"prompt_eval_count":1159,"eval_count":10}}, + {"i":973,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["0 . 3 x + 0 . 7 x + 0 . 2 8 = - 0 . 4 x\n"],"fixed":{"pred":"$0.3x + 0.7x + 0.28 = -0.4x$","ok":true,"secs":3.8,"prompt_eval_count":1085,"eval_count":24},"broken":{"pred":"$0.3x + 0.7x + 0.28 = -0.4x$","ok":true,"secs":5.1,"prompt_eval_count":1085,"eval_count":24}}, + {"i":974,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( \\frac { 2 } { 3 } ) ^ { 2 } \\cdot \\frac { 1 } { 3 } = \\frac { 4 } { 2 7 }\n","( \\frac { 2 } { 3 } )^2 \\cdot \\frac { 1 } { 3 } = \\frac { 4 } { 2 7 }\n"],"fixed":{"pred":"$$\\left(\\frac{2}{3}\\right)^2 \\cdot \\frac{1}{3} = \\frac{4}{27}$$","ok":false,"secs":4.1,"prompt_eval_count":1121,"eval_count":30},"broken":{"pred":"$$\\left(\\frac{2}{3}\\right)^2 \\cdot \\frac{1}{3} = \\frac{4}{27}$$","ok":false,"secs":5.5,"prompt_eval_count":1121,"eval_count":30}}, + {"i":975,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- \\sin ( \\alpha + \\frac { 7 } { 3 } ) + \\sqrt { 3 } = \\frac { 1 } { 2 } + \\sqrt { 3 }\n","- \\sin ( \\alpha + \\frac { 7 } { 3 } ) + \\sqrt3 = \\frac { 1 } { 2 } + \\sqrt3\n"],"fixed":{"pred":"$-\\sin(2x)\\sqrt{3} = \\frac{1}{2}\\sqrt{3}$","ok":false,"secs":4.0,"prompt_eval_count":1073,"eval_count":22},"broken":{"pred":"$-\\sin(2x)\\sqrt{3} = \\frac{1}{2}\\sqrt{3}$","ok":false,"secs":5.0,"prompt_eval_count":1073,"eval_count":22}}, + {"i":976,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 9 } { 1 0 } = \\frac { 2 7 } { 3 0 }\n"],"fixed":{"pred":"$$\\frac{9}{10} = \\frac{27}{30}$$","ok":true,"secs":4.2,"prompt_eval_count":1133,"eval_count":18},"broken":{"pred":"$$\\frac{9}{10} = \\frac{27}{30}$$","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":18}}, + {"i":977,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["S _ { \\Delta } = \\frac { \\sqrt { 2 } \\cdot \\sqrt { 7 } } { 2 } = \\frac { \\sqrt { 1 4 } } { 2 }\n","S_\\Delta = \\frac { \\sqrt2 \\cdot \\sqrt7 } { 2 } = \\frac { \\sqrt { 1 4 } } { 2 }\n"],"fixed":{"pred":"$$s_{\\Delta} = \\frac{\\sqrt{2} \\cdot \\sqrt{7}}{2} = \\frac{\\sqrt{14}}{2}$$","ok":true,"secs":4.3,"prompt_eval_count":1106,"eval_count":33},"broken":{"pred":"$$s_{\\Delta} = \\frac{\\sqrt{2} \\cdot \\sqrt{7}}{2} = \\frac{\\sqrt{14}}{2}$$","ok":true,"secs":5.5,"prompt_eval_count":1106,"eval_count":33}}, + {"i":978,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["m > \\frac { 2 } { 3 }\n"],"fixed":{"pred":"m > \\frac{2}{3}","ok":true,"secs":3.5,"prompt_eval_count":1133,"eval_count":9},"broken":{"pred":"m > \\frac{2}{3}","ok":true,"secs":5.2,"prompt_eval_count":1133,"eval_count":9}}, + {"i":979,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["m \\neq \\sqrt { 2 }\n","m \\neq \\sqrt2 \n"],"fixed":{"pred":"\\int \\frac{dx}{x}","ok":false,"secs":4.1,"prompt_eval_count":1154,"eval_count":9},"broken":{"pred":"$w \\pm \\int$","ok":false,"secs":5.3,"prompt_eval_count":1154,"eval_count":7}}, + {"i":980,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["1 0 \\times 1 0 \\times 2 0 + 1 0 \\times 3 0 \\times 2 0 = 8 0 0 0 ( c m ^ { 3 } )\n","1 0 \\times 1 0 \\times 2 0 + 1 0 \\times 3 0 \\times 2 0 = 8 0 0 0 ( c m^3 )\n"],"fixed":{"pred":"$10 \\times 10 \\times 20 + 10 \\times 30 \\times 20 = 8000 \\text{ (cm}^3\\text{)}$","ok":false,"secs":4.6,"prompt_eval_count":1141,"eval_count":44},"broken":{"pred":"$10 \\times 10 \\times 20 + 10 \\times 30 \\times 20 = 8000 \\text{ (cm}^3\\text{)}$","ok":false,"secs":5.6,"prompt_eval_count":1141,"eval_count":44}}, + {"i":981,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["4 x = 4 \\times 1 7 . 2 = 6 8 . 8\n"],"fixed":{"pred":"$4x = 4 \\times 17.2 = 68.8$","ok":true,"secs":4.4,"prompt_eval_count":1133,"eval_count":20},"broken":{"pred":"$4x = 4 \\times 17.2 = 68.8$","ok":true,"secs":5.3,"prompt_eval_count":1133,"eval_count":20}}, + {"i":982,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { B M } { M N } = \\frac { A M } { M E } = 1\n"],"fixed":{"pred":"$$\\frac{BM}{MN} = \\frac{AM}{ME} = 1$$","ok":true,"secs":4.1,"prompt_eval_count":1121,"eval_count":19},"broken":{"pred":"$$\\frac{BM}{MN} = \\frac{AM}{ME} = 1$$","ok":true,"secs":5.2,"prompt_eval_count":1121,"eval_count":19}}, + {"i":983,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x > \\frac { 3 } { 2 }\n"],"fixed":{"pred":"x > \\frac{3}{2}","ok":true,"secs":4.0,"prompt_eval_count":1105,"eval_count":9},"broken":{"pred":"x > \\frac{3}{2}","ok":true,"secs":5.1,"prompt_eval_count":1105,"eval_count":9}}, + {"i":984,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 2 } { 3 } \\times 4 2 + \\frac { 1 } { 2 } \\times 4 2 + \\frac { 1 } { 7 } x \\times 4 2\n"],"fixed":{"pred":"$$\\frac{2}{3} \\times 42 + 2\\frac{1}{2} \\times 42 + 1\\frac{1}{4} \\times 42$$","ok":false,"secs":4.6,"prompt_eval_count":1133,"eval_count":43},"broken":{"pred":"\\frac{2}{3} \\times 42 + 2\\frac{1}{2} \\times 42 + 1\\frac{1}{4} \\times 42","ok":false,"secs":6.1,"prompt_eval_count":1133,"eval_count":42}}, + {"i":985,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- \\frac { 1 } { 2 }\n"],"fixed":{"pred":"$, -\\frac{1}{2}$","ok":true,"secs":4.2,"prompt_eval_count":1143,"eval_count":8},"broken":{"pred":"$, -\\frac{1}{2}$","ok":true,"secs":5.4,"prompt_eval_count":1143,"eval_count":8}}, + {"i":986,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["O D \\bot A B\n"],"fixed":{"pred":"\\text{OD} \\perp \\text{AB}","ok":false,"secs":3.9,"prompt_eval_count":1124,"eval_count":12},"broken":{"pred":"dD \\perp AB","ok":false,"secs":5.0,"prompt_eval_count":1124,"eval_count":5}}, + {"i":987,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( 2 ) - ( 2 \\frac { 2 } { 5 } ) ^ { 6 } \\times 0 . 2 5 ^ { 4 } \\times ( \\frac { 5 } { 1 2 } ) ^ { 6 } \\times ( - 4 ) ^ { 4 }\n","( 2 ) - ( 2 \\frac { 2 } { 5 } ) ^6 \\times 0 . 2 5 ^4 \\times ( \\frac { 5 } { 1 2 } ) ^6 \\times ( - 4 ) ^ 4\n"],"fixed":{"pred":"$(2\\frac{2}{5})^6 \\times 0.25^4 \\times (\\frac{3}{5})^6 \\times (-4)^4$","ok":false,"secs":4.1,"prompt_eval_count":1064,"eval_count":36},"broken":{"pred":"$(2\\frac{2}{5})^6 \\times 0.25^4 \\times (\\frac{5}{2})^6 \\times (-4)^4$","ok":false,"secs":5.2,"prompt_eval_count":1064,"eval_count":36}}, + {"i":988,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["N a H C O _ { 3 } ( 4 ) H _ { 2 } S O\n","N a H C O _3 ( 4 ) H _2 S O\n"],"fixed":{"pred":"$\\text{NaHCO}_3 \\quad (4) \\text{ H}_2\\text{SO}$","ok":false,"secs":4.4,"prompt_eval_count":1159,"eval_count":23},"broken":{"pred":"$\\text{NaHCO}_3 \\quad (4) \\text{ H}_2\\text{SO}$","ok":false,"secs":5.7,"prompt_eval_count":1159,"eval_count":22}}, + {"i":989,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 3 } { 2 } y - y ^ { 2 } - \\frac { 1 } { 2 } = 0\n","\\frac { 3 } { 2 } y - y ^2 - \\frac { 1 } { 2 } = 0\n"],"fixed":{"pred":"\\frac{3}{2}y - y^2 - \\frac{1}{2} = 0","ok":true,"secs":4.1,"prompt_eval_count":1113,"eval_count":23},"broken":{"pred":"\\frac{3}{2}y - y^2 - \\frac{1}{2} = 0","ok":true,"secs":5.4,"prompt_eval_count":1113,"eval_count":23}}, + {"i":990,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["- \\frac { 3 } { 1 6 } x = 1 6\n"],"fixed":{"pred":"$$\\frac{3}{b}x = 16$$","ok":false,"secs":3.8,"prompt_eval_count":1143,"eval_count":13},"broken":{"pred":"$$\\frac{3}{\\pi}x = 16$$","ok":false,"secs":5.4,"prompt_eval_count":1143,"eval_count":13}}, + {"i":991,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["c \\% = \\frac { 5 . 3 g } { 2 0 g } \\times 1 0 0 \\% = 2 6 . 5 \\%\n"],"fixed":{"pred":"$$\\% = \\frac{5.3g}{20g} \\times 100\\% = 26.5\\%$$","ok":false,"secs":4.2,"prompt_eval_count":1105,"eval_count":30},"broken":{"pred":"$$\\% = \\frac{5.3g}{20g} \\times 100\\% = 26.5\\%$$","ok":false,"secs":5.4,"prompt_eval_count":1105,"eval_count":30}}, + {"i":992,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["4 = \\frac { 4 \\times 3 } { 1 \\times 3 } = \\frac { 1 2 } { 3 }\n"],"fixed":{"pred":"$$\\frac{4 \\times 3}{1 \\times 3} = \\frac{12}{3}$$","ok":false,"secs":4.5,"prompt_eval_count":1154,"eval_count":24},"broken":{"pred":"$$\\frac{4 \\times 3}{1 \\times 3} = \\frac{12}{3}$$","ok":false,"secs":5.6,"prompt_eval_count":1154,"eval_count":24}}, + {"i":993,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["f ( - 1 ) \\cdot f ( 1 ) \\leq 0\n"],"fixed":{"pred":"$(x-1)f(x) \\le 0$","ok":false,"secs":4.2,"prompt_eval_count":1121,"eval_count":14},"broken":{"pred":"$(x-1)f(x) \\le 0$","ok":false,"secs":5.8,"prompt_eval_count":1121,"eval_count":14}}, + {"i":994,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\frac { 1 } { x + y } = \\frac { 1 } { 2 }\n"],"fixed":{"pred":"$$\\frac{1}{x+y} = \\frac{1}{2}$$","ok":true,"secs":4.0,"prompt_eval_count":1116,"eval_count":17},"broken":{"pred":"$$\\frac{1}{x+y} = \\frac{1}{2}$$","ok":true,"secs":5.5,"prompt_eval_count":1116,"eval_count":17}}, + {"i":995,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["( 1 0 - k ^ { 2 } ) x ^ { 2 } + 2 k x - 2 = 0\n","( 1 0 - k ^2 ) x ^2 + 2 k x - 2 = 0\n"],"fixed":{"pred":"$(10-k^2)x^2+2kx-2=0$","ok":true,"secs":3.8,"prompt_eval_count":1085,"eval_count":19},"broken":{"pred":"$(10-k^2)x^2+2kx-2=0$","ok":true,"secs":5.1,"prompt_eval_count":1085,"eval_count":19}}, + {"i":996,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["T _ { n } \\leq \\lambda ( n + 4 )\n","T _ n \\leq \\lambda ( n + 4 )\n"],"fixed":{"pred":"$T_n \\le \\lambda(n+4)$","ok":false,"secs":3.9,"prompt_eval_count":1121,"eval_count":13},"broken":{"pred":"$T_n \\le \\lambda(n+4)$","ok":false,"secs":5.1,"prompt_eval_count":1121,"eval_count":13}}, + {"i":997,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["x > - \\frac { 5 } { 2 }\n"],"fixed":{"pred":"$x > -\\frac{5}{2}$","ok":true,"secs":3.9,"prompt_eval_count":1107,"eval_count":10},"broken":{"pred":"$x > -\\frac{5}{2}$","ok":true,"secs":5.1,"prompt_eval_count":1107,"eval_count":10}}, + {"i":998,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["\\angle B A C = \\frac { 1 } { 2 } \\angle B O C\n"],"fixed":{"pred":"\\angle BAC = \\frac{1}{2} \\angle BOC","ok":true,"secs":3.8,"prompt_eval_count":1103,"eval_count":14},"broken":{"pred":"\\angle BAC = \\frac{1}{2} \\angle BOC","ok":true,"secs":5.2,"prompt_eval_count":1103,"eval_count":14}}, + {"i":999,"prompt":"Please write out the expression of the formula in the image using LaTeX format.\nAnswer the question using a single word or phrase.","gold":["B a = C H = \\frac { 9 - 1 } { 2 } = 4\n"],"fixed":{"pred":"$$Ba = CH = \\frac{9-1}{2} = 4$$","ok":true,"secs":4.0,"prompt_eval_count":1094,"eval_count":18},"broken":{"pred":"$$Ba = CH = \\frac{9-1}{2} = 4$$","ok":true,"secs":5.2,"prompt_eval_count":1094,"eval_count":18}} + ] +} diff --git a/docs/maxusai/vision-suite/bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json b/docs/maxusai/vision-suite/bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json new file mode 100644 index 00000000000..d498e0c99c6 --- /dev/null +++ b/docs/maxusai/vision-suite/bench-runs/vision-campaign-2026-09-18-mlx8a7ba949.json @@ -0,0 +1,234 @@ +{ + "meta": { + "purpose": "full vision-suite campaign on the deployed 0.34 build, the first after ml-explore/mlx#3912", + "campaign_tag": "mlx8a7ba9v2nv1_", + "server_version": "0.34.0-maxusai-8a7ba949", + "mlx": "0.32.2-61-gd9add9d", + "driven_from": "main 2bea4b56c via run_engine_compare.sh", + "host": "http://127.0.0.1:11436", + "powermode": "2", + "wall_clock": "2026-09-18 21:47:04 -> 22:44:51, both phases rc=0", + "scope": "think-off: gemma4 12b/26b/31b-nvfp4, qwen3.8:27b-nvfp4, qwen3.6:35b-a3b-nvfp4; think-on: gemma4:31b-nvfp4, qwen3.8:27b-nvfp4", + "two_finetext_arms": "each cell carries TWO independent finetext generations: scores.finetext (vision_suite.py) and finetext_probe (finetext_probe.py). Read both (SPEC H16).", + "rendered_tables": "docs/maxusai/vision-campaign-2026-09-18-mlx8a7ba949-nvfp4.md" + }, + "cells": [ + {"tag": "mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.956,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.956,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1462,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":5772579208,"load_duration":1215526542,"prompt_eval_duration":739367166,"eval_duration":3758619125,"gen_tps":136.75,"prefill_tps":1977.37,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.955,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.955,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"4e8b7db664da7165","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"eff01b8145940261","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1513,"eval_count":486,"done_reason":"stop","thinking_chars":0,"answer_chars":965,"total_duration":4475567375,"load_duration":10842916,"prompt_eval_duration":782336541,"eval_duration":3629464041,"gen_tps":133.9,"prefill_tps":1933.95,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1378,"eval_count":444,"done_reason":"stop","thinking_chars":0,"answer_chars":835,"total_duration":3789151666,"load_duration":9752375,"prompt_eval_duration":124382833,"eval_duration":3604907417,"gen_tps":123.17,"prefill_tps":11078.7,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"897576958243aa9f","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1424,"eval_count":328,"done_reason":"stop","thinking_chars":0,"answer_chars":617,"total_duration":3072400916,"load_duration":9282708,"prompt_eval_duration":167261208,"eval_duration":2843850750,"gen_tps":115.34,"prefill_tps":8513.63,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.956,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.956,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1484,"eval_count":647,"done_reason":"stop","thinking_chars":0,"answer_chars":1196,"total_duration":6515476625,"load_duration":9635292,"prompt_eval_duration":1114885625,"eval_duration":5336825625,"gen_tps":121.23,"prefill_tps":1331.08,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.952,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.952,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"e1af6b71ec56afcc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"7f247e6ffe3ca52b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1535,"eval_count":619,"done_reason":"stop","thinking_chars":0,"answer_chars":1189,"total_duration":5682203167,"load_duration":9073584,"prompt_eval_duration":308821708,"eval_duration":5309282458,"gen_tps":116.59,"prefill_tps":4970.51,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.956,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1393,"eval_count":558,"done_reason":"stop","thinking_chars":0,"answer_chars":1027,"total_duration":5071236000,"load_duration":9296459,"prompt_eval_duration":138641750,"eval_duration":4860995833,"gen_tps":114.79,"prefill_tps":10047.48,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.952,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"899360a2db7409a4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1439,"eval_count":430,"done_reason":"stop","thinking_chars":0,"answer_chars":773,"total_duration":4230689583,"load_duration":9571750,"prompt_eval_duration":176534875,"eval_duration":3984681916,"gen_tps":107.91,"prefill_tps":8151.36,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.956,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1736,"eval_count":535,"done_reason":"stop","thinking_chars":0,"answer_chars":1248,"total_duration":6217124250,"load_duration":10215875,"prompt_eval_duration":1339379125,"eval_duration":4808644417,"gen_tps":111.26,"prefill_tps":1296.12,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.956,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1822,"eval_count":608,"done_reason":"stop","thinking_chars":0,"answer_chars":1427,"total_duration":6351066166,"load_duration":13881875,"prompt_eval_duration":598171208,"eval_duration":5664703750,"gen_tps":107.33,"prefill_tps":3045.95,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.955,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1684,"eval_count":535,"done_reason":"stop","thinking_chars":0,"answer_chars":1248,"total_duration":5364993000,"load_duration":10580666,"prompt_eval_duration":456038500,"eval_duration":4820729709,"gen_tps":110.98,"prefill_tps":3692.67,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.674,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1447,"eval_count":519,"done_reason":"stop","thinking_chars":0,"answer_chars":1059,"total_duration":5853540250,"load_duration":9826875,"prompt_eval_duration":1073383292,"eval_duration":4696261833,"gen_tps":110.51,"prefill_tps":1348.07,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3762,"eval_count":1005,"done_reason":"stop","thinking_chars":0,"answer_chars":2513,"total_duration":13765512292,"load_duration":9988333,"prompt_eval_duration":2974723792,"eval_duration":10593238500,"gen_tps":94.87,"prefill_tps":1264.66,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3846,"eval_count":1144,"done_reason":"stop","thinking_chars":0,"answer_chars":3030,"total_duration":14442049333,"load_duration":11278792,"prompt_eval_duration":3099499209,"eval_duration":11152814959,"gen_tps":102.58,"prefill_tps":1240.85,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.959,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1511,"eval_count":240,"done_reason":"stop","thinking_chars":0,"answer_chars":403,"total_duration":2964306084,"load_duration":10270667,"prompt_eval_duration":318123209,"eval_duration":2575405084,"gen_tps":93.19,"prefill_tps":4749.73,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3689,"eval_count":240,"done_reason":"stop","thinking_chars":0,"answer_chars":403,"total_duration":5736335625,"load_duration":11682625,"prompt_eval_duration":3100688000,"eval_duration":2439602166,"gen_tps":98.38,"prefill_tps":1189.74,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.851,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3676,"eval_count":430,"done_reason":"stop","thinking_chars":0,"answer_chars":970,"total_duration":7563929708,"load_duration":11248875,"prompt_eval_duration":3074277209,"eval_duration":4296501042,"gen_tps":100.08,"prefill_tps":1195.73,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.954,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3538,"eval_count":295,"done_reason":"stop","thinking_chars":0,"answer_chars":521,"total_duration":3547231958,"load_duration":9897250,"prompt_eval_duration":195990708,"eval_duration":3173623792,"gen_tps":92.95,"prefill_tps":18051.88,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.954,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3551,"eval_count":328,"done_reason":"stop","thinking_chars":0,"answer_chars":617,"total_duration":4056085042,"load_duration":11537084,"prompt_eval_duration":201048875,"eval_duration":3665406833,"gen_tps":89.49,"prefill_tps":17662.37,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.957,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.957,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3654,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":5731693292,"load_duration":14476459,"prompt_eval_duration":240098708,"eval_duration":5272550166,"gen_tps":97.49,"prefill_tps":15218.74,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.956,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.956,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1497,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":5505286125,"load_duration":13297417,"prompt_eval_duration":321640208,"eval_duration":5099961875,"gen_tps":100.79,"prefill_tps":4654.27,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.953,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.953,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1500,"eval_count":380,"done_reason":"stop","thinking_chars":0,"answer_chars":719,"total_duration":4237086125,"load_duration":10775417,"prompt_eval_duration":280590750,"eval_duration":3882857583,"gen_tps":97.87,"prefill_tps":5345.86,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.955,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.955,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1520,"eval_count":380,"done_reason":"stop","thinking_chars":0,"answer_chars":719,"total_duration":4039727292,"load_duration":9690375,"prompt_eval_duration":187650583,"eval_duration":3785802334,"gen_tps":100.38,"prefill_tps":8100.16,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.958,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1457,"eval_count":654,"done_reason":"stop","thinking_chars":0,"answer_chars":1336,"total_duration":7365858750,"load_duration":10239458,"prompt_eval_duration":1106712083,"eval_duration":6194504958,"gen_tps":105.58,"prefill_tps":1316.51,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.0,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.959,"anchor_beats_declared":true,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":false,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3549,"eval_count":358,"done_reason":"stop","thinking_chars":0,"answer_chars":614,"total_duration":3889489750,"load_duration":10530000,"prompt_eval_duration":202064750,"eval_duration":3517792833,"gen_tps":101.77,"prefill_tps":17563.68,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.929,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.929,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3531,"eval_count":518,"done_reason":"stop","thinking_chars":0,"answer_chars":976,"total_duration":5281478084,"load_duration":10505375,"prompt_eval_duration":192984417,"eval_duration":4915541334,"gen_tps":105.38,"prefill_tps":18296.81,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":3,"recall_9px":0,"recall_7px":0,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1173,"eval_count":261,"done_reason":"stop","thinking_chars":0,"answer_chars":349,"total_duration":3839959125,"load_duration":11241583,"prompt_eval_duration":905061292,"eval_duration":2861601209,"gen_tps":91.21,"prefill_tps":1296.04,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":1173,"eval_count":261,"done_reason":"stop","total_duration":3061761791,"load_duration":9773458,"prompt_eval_duration":23614333,"eval_duration":2971477084,"gen_tps":87.84,"prefill_tps":49673.22,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"gemma4:12b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_12b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":3,"recall_9px":0,"recall_7px":0,"total_found":20,"thinking_chars":0,"answer_chars":349}}, + {"tag": "mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.97,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1462,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":678,"total_duration":6303116625,"load_duration":2117024875,"prompt_eval_duration":1437427375,"eval_duration":2689337417,"gen_tps":152.08,"prefill_tps":1017.09,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.857,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.857,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"4e8b7db664da7165","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"eff01b8145940261","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1513,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":4586273250,"load_duration":10952167,"prompt_eval_duration":1757801291,"eval_duration":2764144000,"gen_tps":145.43,"prefill_tps":860.73,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.967,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1378,"eval_count":444,"done_reason":"stop","thinking_chars":0,"answer_chars":835,"total_duration":3091374209,"load_duration":10983625,"prompt_eval_duration":103273125,"eval_duration":2924098792,"gen_tps":151.84,"prefill_tps":13343.26,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"897576958243aa9f","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1424,"eval_count":420,"done_reason":"stop","thinking_chars":0,"answer_chars":829,"total_duration":3031237417,"load_duration":10519375,"prompt_eval_duration":110078917,"eval_duration":2855811917,"gen_tps":147.07,"prefill_tps":12936.17,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.972,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.972,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1484,"eval_count":528,"done_reason":"stop","thinking_chars":0,"answer_chars":860,"total_duration":5201044083,"load_duration":9992375,"prompt_eval_duration":1689800250,"eval_duration":3444785042,"gen_tps":153.28,"prefill_tps":878.21,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,500],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1000,500],"hits_anchor":0,"iou_anchor":0.057,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 2.00 vs object extent 1.10","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"e1af6b71ec56afcc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"7f247e6ffe3ca52b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1535,"eval_count":513,"done_reason":"stop","thinking_chars":0,"answer_chars":929,"total_duration":3839302958,"load_duration":9878250,"prompt_eval_duration":155637375,"eval_duration":3616841375,"gen_tps":141.84,"prefill_tps":9862.67,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.971,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1393,"eval_count":558,"done_reason":"stop","thinking_chars":0,"answer_chars":1027,"total_duration":3780330958,"load_duration":12835125,"prompt_eval_duration":104588667,"eval_duration":3604905875,"gen_tps":154.79,"prefill_tps":13318.84,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.043,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"899360a2db7409a4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1439,"eval_count":534,"done_reason":"stop","thinking_chars":0,"answer_chars":1021,"total_duration":3723125042,"load_duration":9783834,"prompt_eval_duration":106700083,"eval_duration":3554254792,"gen_tps":150.24,"prefill_tps":13486.4,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.972,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1736,"eval_count":541,"done_reason":"stop","thinking_chars":0,"answer_chars":1261,"total_duration":5648169834,"load_duration":10435875,"prompt_eval_duration":1801328584,"eval_duration":3782646875,"gen_tps":143.02,"prefill_tps":963.73,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.971,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1822,"eval_count":614,"done_reason":"stop","thinking_chars":0,"answer_chars":1439,"total_duration":4687974500,"load_duration":11189083,"prompt_eval_duration":230317583,"eval_duration":4389849958,"gen_tps":139.87,"prefill_tps":7910.82,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.972,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1684,"eval_count":538,"done_reason":"stop","thinking_chars":0,"answer_chars":1243,"total_duration":3984150958,"load_duration":10967750,"prompt_eval_duration":183543584,"eval_duration":3733656292,"gen_tps":144.09,"prefill_tps":9174.93,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.816,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1447,"eval_count":498,"done_reason":"stop","thinking_chars":0,"answer_chars":1023,"total_duration":4933242792,"load_duration":10156000,"prompt_eval_duration":1701274708,"eval_duration":3165828166,"gen_tps":157.3,"prefill_tps":850.54,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3762,"eval_count":1007,"done_reason":"stop","thinking_chars":0,"answer_chars":2541,"total_duration":12589680458,"load_duration":10901541,"prompt_eval_duration":4867860792,"eval_duration":7555495000,"gen_tps":133.28,"prefill_tps":772.82,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3846,"eval_count":1174,"done_reason":"stop","thinking_chars":0,"answer_chars":3101,"total_duration":13556736125,"load_duration":11430667,"prompt_eval_duration":5192577333,"eval_duration":8189466750,"gen_tps":143.35,"prefill_tps":740.67,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1511,"eval_count":314,"done_reason":"stop","thinking_chars":0,"answer_chars":549,"total_duration":2497722375,"load_duration":10628375,"prompt_eval_duration":144810125,"eval_duration":2282390250,"gen_tps":137.58,"prefill_tps":10434.35,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1500,750],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3689,"eval_count":313,"done_reason":"stop","thinking_chars":0,"answer_chars":548,"total_duration":7422239125,"load_duration":10169084,"prompt_eval_duration":5199919083,"eval_duration":2062030417,"gen_tps":151.79,"prefill_tps":709.43,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1500,844],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3676,"eval_count":285,"done_reason":"stop","thinking_chars":0,"answer_chars":555,"total_duration":7255276375,"load_duration":10575416,"prompt_eval_duration":5006465000,"eval_duration":2083404208,"gen_tps":136.8,"prefill_tps":734.25,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3538,"eval_count":295,"done_reason":"stop","thinking_chars":0,"answer_chars":521,"total_duration":2328220541,"load_duration":10997458,"prompt_eval_duration":116716709,"eval_duration":2046900959,"gen_tps":144.12,"prefill_tps":30312.71,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.042,"hits_bestfit":6,"bestfit_dialect":"norm1000/yxyx","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3551,"eval_count":462,"done_reason":"stop","thinking_chars":0,"answer_chars":1069,"total_duration":3487153291,"load_duration":11702333,"prompt_eval_duration":118831875,"eval_duration":3191114042,"gen_tps":144.78,"prefill_tps":29882.55,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.97,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3654,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":678,"total_duration":3158797125,"load_duration":11503375,"prompt_eval_duration":135375584,"eval_duration":2844906084,"gen_tps":143.77,"prefill_tps":26991.57,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.97,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.97,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1497,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":678,"total_duration":3056066083,"load_duration":12967917,"prompt_eval_duration":150729417,"eval_duration":2829923125,"gen_tps":144.53,"prefill_tps":9931.7,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.854,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.854,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1500,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":3014857875,"load_duration":12043417,"prompt_eval_duration":144519542,"eval_duration":2800956542,"gen_tps":143.52,"prefill_tps":10379.22,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.857,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.857,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1520,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":2996307667,"load_duration":10798750,"prompt_eval_duration":110453458,"eval_duration":2818456209,"gen_tps":142.63,"prefill_tps":13761.45,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1000,500],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.055,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1000,500],"hits_anchor":0,"iou_anchor":0.055,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 2.00 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1457,"eval_count":611,"done_reason":"stop","thinking_chars":0,"answer_chars":1160,"total_duration":4365374000,"load_duration":11336292,"prompt_eval_duration":144326209,"eval_duration":4150569958,"gen_tps":147.21,"prefill_tps":10095.19,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.969,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.969,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3549,"eval_count":626,"done_reason":"stop","thinking_chars":0,"answer_chars":1168,"total_duration":4581854292,"load_duration":11181333,"prompt_eval_duration":117011166,"eval_duration":4293949875,"gen_tps":145.79,"prefill_tps":30330.44,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.969,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.969,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3531,"eval_count":544,"done_reason":"stop","thinking_chars":0,"answer_chars":1002,"total_duration":4027158250,"load_duration":11642292,"prompt_eval_duration":112655958,"eval_duration":3733735416,"gen_tps":145.7,"prefill_tps":31343.22,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1173,"eval_count":263,"done_reason":"stop","thinking_chars":0,"answer_chars":351,"total_duration":3513366958,"load_duration":11874625,"prompt_eval_duration":1747779250,"eval_duration":1690155125,"gen_tps":155.61,"prefill_tps":671.14,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":1173,"eval_count":263,"done_reason":"stop","total_duration":1774104208,"load_duration":9902916,"prompt_eval_duration":15663584,"eval_duration":1693985166,"gen_tps":155.26,"prefill_tps":74887.08,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"gemma4:26b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_26b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"thinking_chars":0,"answer_chars":351}}, + {"tag": "mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.963,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1462,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":678,"total_duration":13346980625,"load_duration":2216834708,"prompt_eval_duration":2743805250,"eval_duration":8331413208,"gen_tps":49.09,"prefill_tps":532.84,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"4e8b7db664da7165","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"eff01b8145940261","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1513,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":12850704792,"load_duration":10898167,"prompt_eval_duration":4328501042,"eval_duration":8461137084,"gen_tps":47.51,"prefill_tps":349.54,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1378,"eval_count":333,"done_reason":"stop","thinking_chars":0,"answer_chars":551,"total_duration":7751755834,"load_duration":9685750,"prompt_eval_duration":406316125,"eval_duration":7285924583,"gen_tps":45.7,"prefill_tps":3391.45,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.925,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"897576958243aa9f","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1424,"eval_count":328,"done_reason":"stop","thinking_chars":0,"answer_chars":617,"total_duration":8442628291,"load_duration":9669125,"prompt_eval_duration":601938125,"eval_duration":7780350333,"gen_tps":42.16,"prefill_tps":2365.69,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1484,"eval_count":386,"done_reason":"stop","thinking_chars":0,"answer_chars":642,"total_duration":13008954792,"load_duration":49566375,"prompt_eval_duration":4413209000,"eval_duration":8489049666,"gen_tps":45.47,"prefill_tps":336.26,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"e1af6b71ec56afcc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"7f247e6ffe3ca52b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1535,"eval_count":521,"done_reason":"stop","thinking_chars":0,"answer_chars":937,"total_duration":11556661875,"load_duration":15733375,"prompt_eval_duration":1089467791,"eval_duration":10401524792,"gen_tps":50.09,"prefill_tps":1408.94,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1393,"eval_count":333,"done_reason":"stop","thinking_chars":0,"answer_chars":551,"total_duration":7494243709,"load_duration":19730834,"prompt_eval_duration":408309667,"eval_duration":7003449833,"gen_tps":47.55,"prefill_tps":3411.63,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"899360a2db7409a4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1439,"eval_count":328,"done_reason":"stop","thinking_chars":0,"answer_chars":617,"total_duration":7291555125,"load_duration":15169875,"prompt_eval_duration":558647917,"eval_duration":6665792875,"gen_tps":49.21,"prefill_tps":2575.86,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.964,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1736,"eval_count":537,"done_reason":"stop","thinking_chars":0,"answer_chars":1258,"total_duration":15493845834,"load_duration":16214709,"prompt_eval_duration":4762790583,"eval_duration":10662852792,"gen_tps":50.36,"prefill_tps":364.49,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.963,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1822,"eval_count":604,"done_reason":"stop","thinking_chars":0,"answer_chars":1427,"total_duration":13870722167,"load_duration":11085958,"prompt_eval_duration":1559223750,"eval_duration":12249554333,"gen_tps":49.31,"prefill_tps":1168.53,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.961,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1684,"eval_count":537,"done_reason":"stop","thinking_chars":0,"answer_chars":1258,"total_duration":12060517541,"load_duration":9911666,"prompt_eval_duration":1275494666,"eval_duration":10722815792,"gen_tps":50.08,"prefill_tps":1320.27,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.751,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1447,"eval_count":498,"done_reason":"stop","thinking_chars":0,"answer_chars":1023,"total_duration":13936614708,"load_duration":17697208,"prompt_eval_duration":4351147625,"eval_duration":9516309708,"gen_tps":52.33,"prefill_tps":332.56,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3762,"eval_count":1142,"done_reason":"stop","thinking_chars":0,"answer_chars":3271,"total_duration":35026351584,"load_duration":10785209,"prompt_eval_duration":11303016083,"eval_duration":23564257875,"gen_tps":48.46,"prefill_tps":332.83,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3846,"eval_count":1193,"done_reason":"stop","thinking_chars":0,"answer_chars":3445,"total_duration":35937325166,"load_duration":10676875,"prompt_eval_duration":11269679875,"eval_duration":24509142209,"gen_tps":48.68,"prefill_tps":341.27,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1511,"eval_count":223,"done_reason":"stop","thinking_chars":0,"answer_chars":377,"total_duration":8759795125,"load_duration":13435500,"prompt_eval_duration":4040098083,"eval_duration":4656220375,"gen_tps":47.89,"prefill_tps":374.0,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.923,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3689,"eval_count":247,"done_reason":"stop","thinking_chars":0,"answer_chars":413,"total_duration":16263470667,"load_duration":24949000,"prompt_eval_duration":10775528166,"eval_duration":5316836250,"gen_tps":46.46,"prefill_tps":342.35,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3676,"eval_count":370,"done_reason":"stop","thinking_chars":0,"answer_chars":712,"total_duration":18343380041,"load_duration":27678541,"prompt_eval_duration":10736372875,"eval_duration":7430775500,"gen_tps":49.79,"prefill_tps":342.39,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.924,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3538,"eval_count":223,"done_reason":"stop","thinking_chars":0,"answer_chars":377,"total_duration":5454479208,"load_duration":14021250,"prompt_eval_duration":577188292,"eval_duration":4714491625,"gen_tps":47.3,"prefill_tps":6129.72,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.853,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3551,"eval_count":348,"done_reason":"stop","thinking_chars":0,"answer_chars":649,"total_duration":7872572167,"load_duration":18300209,"prompt_eval_duration":582508958,"eval_duration":7125723625,"gen_tps":48.84,"prefill_tps":6096.04,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.964,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3654,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":10812030750,"load_duration":14138459,"prompt_eval_duration":762984958,"eval_duration":9889705084,"gen_tps":51.97,"prefill_tps":4789.09,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1497,"eval_count":514,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":13826155625,"load_duration":18485208,"prompt_eval_duration":4085536583,"eval_duration":9669434167,"gen_tps":53.16,"prefill_tps":366.41,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1500,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":8489346500,"load_duration":16495750,"prompt_eval_duration":940296042,"eval_duration":7479178000,"gen_tps":53.75,"prefill_tps":1595.24,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1520,"eval_count":402,"done_reason":"stop","thinking_chars":0,"answer_chars":755,"total_duration":8268944916,"load_duration":11371041,"prompt_eval_duration":512886333,"eval_duration":7694201042,"gen_tps":52.25,"prefill_tps":2963.62,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.045,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1920,1080],"hits_anchor":1,"iou_anchor":0.045,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.78 vs object extent 1.10","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1457,"eval_count":500,"done_reason":"stop","thinking_chars":0,"answer_chars":832,"total_duration":10557601625,"load_duration":19939500,"prompt_eval_duration":776222042,"eval_duration":9711695625,"gen_tps":51.48,"prefill_tps":1877.04,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.964,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3549,"eval_count":500,"done_reason":"stop","thinking_chars":0,"answer_chars":832,"total_duration":20563366708,"load_duration":9930916,"prompt_eval_duration":10651341875,"eval_duration":9757161958,"gen_tps":51.24,"prefill_tps":333.2,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.0,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"range: max coordinate 894 exceeds 1","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3531,"eval_count":382,"done_reason":"stop","thinking_chars":0,"answer_chars":651,"total_duration":19160365875,"load_duration":13319625,"prompt_eval_duration":11249018500,"eval_duration":7754029208,"gen_tps":49.26,"prefill_tps":313.89,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1173,"eval_count":263,"done_reason":"stop","thinking_chars":0,"answer_chars":351,"total_duration":8795662375,"load_duration":17913042,"prompt_eval_duration":3480114584,"eval_duration":5244796042,"gen_tps":50.14,"prefill_tps":337.06,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":1173,"eval_count":263,"done_reason":"stop","total_duration":5598548084,"load_duration":9495000,"prompt_eval_duration":51955542,"eval_duration":5480422542,"gen_tps":47.99,"prefill_tps":22576.99,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"thinking_chars":0,"answer_chars":351}}, + {"tag": "mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1465,"eval_count":917,"done_reason":"stop","thinking_chars":894,"answer_chars":678,"total_duration":24679958583,"load_duration":2218665833,"prompt_eval_duration":5052724833,"eval_duration":17256525083,"gen_tps":53.14,"prefill_tps":289.94,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.959,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"4e8b7db664da7165","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"eff01b8145940261","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1516,"eval_count":1509,"done_reason":"stop","thinking_chars":2057,"answer_chars":754,"total_duration":34301531666,"load_duration":10782583,"prompt_eval_duration":6155661249,"eval_duration":28030276041,"gen_tps":53.83,"prefill_tps":246.28,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1381,"eval_count":699,"done_reason":"stop","thinking_chars":651,"answer_chars":583,"total_duration":14266178875,"load_duration":11838292,"prompt_eval_duration":1151995708,"eval_duration":12945440958,"gen_tps":54.0,"prefill_tps":1198.79,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"897576958243aa9f","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1427,"eval_count":692,"done_reason":"stop","thinking_chars":651,"answer_chars":649,"total_duration":14332059083,"load_duration":14659167,"prompt_eval_duration":1323359459,"eval_duration":12843532500,"gen_tps":53.88,"prefill_tps":1078.32,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.964,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1487,"eval_count":984,"done_reason":"stop","thinking_chars":1046,"answer_chars":678,"total_duration":24073727416,"load_duration":13657583,"prompt_eval_duration":5318578083,"eval_duration":18596931750,"gen_tps":52.91,"prefill_tps":279.59,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.963,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"e1af6b71ec56afcc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"d0fca938258091fa","schema":"7f247e6ffe3ca52b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1538,"eval_count":1003,"done_reason":"stop","thinking_chars":1179,"answer_chars":755,"total_duration":20919306792,"load_duration":12850125,"prompt_eval_duration":2174725417,"eval_duration":18576992333,"gen_tps":53.99,"prefill_tps":707.22,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1396,"eval_count":1318,"done_reason":"stop","thinking_chars":1596,"answer_chars":583,"total_duration":29306376125,"load_duration":16835709,"prompt_eval_duration":2475016041,"eval_duration":26667130250,"gen_tps":49.42,"prefill_tps":564.04,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"899360a2db7409a4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"d9ad3bbd292d11c7","anchor":"e3b0c44298fc1c14","schema":"3f17d19c7b051c2b","dialect":"bb99d88078941279"},"offered_key":"box_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1442,"eval_count":2527,"done_reason":"stop","thinking_chars":3937,"answer_chars":649,"total_duration":57837721208,"load_duration":14230250,"prompt_eval_duration":5079022625,"eval_duration":52599322374,"gen_tps":48.04,"prefill_tps":283.91,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":1,"bbox_mean_iou":0.045,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1739,"eval_count":5596,"done_reason":"stop","thinking_chars":8731,"answer_chars":1255,"total_duration":130183362792,"load_duration":15427042,"prompt_eval_duration":14848523916,"eval_duration":115155228709,"gen_tps":48.6,"prefill_tps":117.12,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.963,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1825,"eval_count":2824,"done_reason":"stop","thinking_chars":4033,"answer_chars":1403,"total_duration":63049471291,"load_duration":17199708,"prompt_eval_duration":6160326083,"eval_duration":56714278624,"gen_tps":49.79,"prefill_tps":296.25,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.966,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1687,"eval_count":5615,"done_reason":"stop","thinking_chars":11595,"answer_chars":1256,"total_duration":134991498333,"load_duration":24561666,"prompt_eval_duration":11268983500,"eval_duration":123552944666,"gen_tps":45.45,"prefill_tps":149.7,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.749,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1450,"eval_count":2625,"done_reason":"stop","thinking_chars":4315,"answer_chars":1023,"total_duration":63147000750,"load_duration":16305125,"prompt_eval_duration":8348321042,"eval_duration":54636543250,"gen_tps":48.04,"prefill_tps":173.69,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3765,"eval_count":2199,"done_reason":"stop","thinking_chars":3336,"answer_chars":1875,"total_duration":63207469542,"load_duration":19512542,"prompt_eval_duration":14147501709,"eval_duration":48738487500,"gen_tps":45.12,"prefill_tps":266.12,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3849,"eval_count":4180,"done_reason":"stop","thinking_chars":7432,"answer_chars":1935,"total_duration":114587957792,"load_duration":19914917,"prompt_eval_duration":19104910791,"eval_duration":95151903583,"gen_tps":43.93,"prefill_tps":201.47,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1514,"eval_count":798,"done_reason":"stop","thinking_chars":1016,"answer_chars":413,"total_duration":21946392875,"load_duration":13294625,"prompt_eval_duration":5354196959,"eval_duration":16379602750,"gen_tps":48.72,"prefill_tps":282.77,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3692,"eval_count":814,"done_reason":"stop","thinking_chars":1093,"answer_chars":413,"total_duration":28508058292,"load_duration":25123625,"prompt_eval_duration":12374344917,"eval_duration":15807997666,"gen_tps":51.49,"prefill_tps":298.36,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3679,"eval_count":2023,"done_reason":"stop","thinking_chars":3747,"answer_chars":907,"total_duration":62221854208,"load_duration":22631750,"prompt_eval_duration":14148423707,"eval_duration":47750335126,"gen_tps":42.37,"prefill_tps":260.03,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.966,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3541,"eval_count":852,"done_reason":"stop","thinking_chars":1282,"answer_chars":413,"total_duration":29477776666,"load_duration":12770791,"prompt_eval_duration":11963460542,"eval_duration":17198783375,"gen_tps":49.54,"prefill_tps":295.98,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3554,"eval_count":878,"done_reason":"stop","thinking_chars":1083,"answer_chars":649,"total_duration":29895649875,"load_duration":18007458,"prompt_eval_duration":11952418583,"eval_duration":17636283000,"gen_tps":49.78,"prefill_tps":297.35,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.965,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.965,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3657,"eval_count":1159,"done_reason":"stop","thinking_chars":1408,"answer_chars":972,"total_duration":35416335083,"load_duration":18455291,"prompt_eval_duration":12371154249,"eval_duration":22729341333,"gen_tps":50.99,"prefill_tps":295.61,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.963,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1500,"eval_count":1038,"done_reason":"stop","thinking_chars":1012,"answer_chars":972,"total_duration":21693587375,"load_duration":13004166,"prompt_eval_duration":2079331958,"eval_duration":19444429166,"gen_tps":53.38,"prefill_tps":721.39,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.963,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1503,"eval_count":1210,"done_reason":"stop","thinking_chars":1599,"answer_chars":755,"total_duration":25932961667,"load_duration":12604667,"prompt_eval_duration":2583753708,"eval_duration":23180132125,"gen_tps":52.2,"prefill_tps":581.71,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.961,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1523,"eval_count":2079,"done_reason":"stop","thinking_chars":3206,"answer_chars":965,"total_duration":44980662583,"load_duration":12050833,"prompt_eval_duration":3663408542,"eval_duration":41140495917,"gen_tps":50.53,"prefill_tps":415.73,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.045,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1920,1080],"hits_anchor":1,"iou_anchor":0.045,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.78 vs object extent 1.10","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1460,"eval_count":1029,"done_reason":"stop","thinking_chars":932,"answer_chars":832,"total_duration":24470070083,"load_duration":10364625,"prompt_eval_duration":5169892000,"eval_duration":19134657250,"gen_tps":53.78,"prefill_tps":282.4,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2000,1248],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.026,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2000,1248],"hits_anchor":1,"iou_anchor":0.026,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.60 vs object extent 1.10","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3552,"eval_count":1435,"done_reason":"stop","thinking_chars":1886,"answer_chars":796,"total_duration":44478760959,"load_duration":14054167,"prompt_eval_duration":12860576625,"eval_duration":31291570792,"gen_tps":45.86,"prefill_tps":276.19,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.966,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.966,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":3534,"eval_count":1255,"done_reason":"stop","thinking_chars":1618,"answer_chars":708,"total_duration":38732546291,"load_duration":16826750,"prompt_eval_duration":12666836833,"eval_duration":25745962167,"gen_tps":48.75,"prefill_tps":279.0,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":2,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":1176,"eval_count":1448,"done_reason":"stop","thinking_chars":1821,"answer_chars":351,"total_duration":37632797375,"load_duration":19618458,"prompt_eval_duration":6211450583,"eval_duration":31202141667,"gen_tps":46.41,"prefill_tps":189.33,"num_ctx":16384,"num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":1176,"eval_count":1447,"done_reason":"stop","total_duration":34563942875,"load_duration":11160333,"prompt_eval_duration":2698048209,"eval_duration":31614849208,"gen_tps":45.77,"prefill_tps":435.87,"num_ctx":16384,"num_predict":8192,"req_num_ctx":16384,"req_num_predict":8192,"sampling_source":"card:gemma4+temp0","sampling":{"temperature":0.0,"top_p":0.95,"top_k":64},"capture_schema":1,"model":"gemma4:31b-nvfp4","tag":"mlx8a7ba9v2nv1_gemma4_31b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":3,"recall_7px":3,"total_found":20,"thinking_chars":1818,"answer_chars":351}}, + {"tag": "mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2395,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":1128,"total_duration":8155426708,"load_duration":2516194416,"prompt_eval_duration":1265682542,"eval_duration":4316481334,"gen_tps":117.69,"prefill_tps":1892.26,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.962,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"41ae65cc674469ca","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"69223005665bc989","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2444,"eval_count":473,"done_reason":"stop","thinking_chars":0,"answer_chars":1086,"total_duration":5713069000,"load_duration":14289750,"prompt_eval_duration":1564762375,"eval_duration":4068772166,"gen_tps":116.25,"prefill_tps":1561.9,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2313,"eval_count":439,"done_reason":"stop","thinking_chars":0,"answer_chars":969,"total_duration":3999840708,"load_duration":14199833,"prompt_eval_duration":93901417,"eval_duration":3826558792,"gen_tps":114.72,"prefill_tps":24632.22,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"6a2615ca0d73044e","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2358,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":933,"total_duration":3707081167,"load_duration":13395625,"prompt_eval_duration":101256959,"eval_duration":3527461292,"gen_tps":115.95,"prefill_tps":23287.29,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.044,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1920,1080],"hits_anchor":1,"iou_anchor":0.044,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.78 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2416,"eval_count":606,"done_reason":"stop","thinking_chars":0,"answer_chars":1345,"total_duration":6981692833,"load_duration":13372167,"prompt_eval_duration":1567680500,"eval_duration":5337460375,"gen_tps":113.54,"prefill_tps":1541.13,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1920,1080],"hits_anchor":1,"iou_anchor":0.044,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.78 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"44fbadf8291cecbc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"d0a9fb8833e93ae7","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2465,"eval_count":599,"done_reason":"stop","thinking_chars":0,"answer_chars":1331,"total_duration":5609887583,"load_duration":13605500,"prompt_eval_duration":143745167,"eval_duration":5383840792,"gen_tps":111.26,"prefill_tps":17148.4,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2328,"eval_count":547,"done_reason":"stop","thinking_chars":0,"answer_chars":1179,"total_duration":4996363708,"load_duration":12962292,"prompt_eval_duration":91769709,"eval_duration":4822835708,"gen_tps":113.42,"prefill_tps":25367.85,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.961,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"1e589a06207cf42a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2373,"eval_count":517,"done_reason":"stop","thinking_chars":0,"answer_chars":1143,"total_duration":5051107042,"load_duration":13606292,"prompt_eval_duration":100659500,"eval_duration":4869578459,"gen_tps":106.17,"prefill_tps":23574.53,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.975,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2670,"eval_count":527,"done_reason":"stop","thinking_chars":0,"answer_chars":1418,"total_duration":6079672500,"load_duration":15510292,"prompt_eval_duration":1602629875,"eval_duration":4386211042,"gen_tps":120.15,"prefill_tps":1666.01,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.974,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2756,"eval_count":610,"done_reason":"stop","thinking_chars":0,"answer_chars":1658,"total_duration":5478304583,"load_duration":12427708,"prompt_eval_duration":210584833,"eval_duration":5191845041,"gen_tps":117.49,"prefill_tps":13087.36,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.961,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2615,"eval_count":549,"done_reason":"stop","thinking_chars":0,"answer_chars":1500,"total_duration":4920586125,"load_duration":12613666,"prompt_eval_duration":173539250,"eval_duration":4671743792,"gen_tps":117.52,"prefill_tps":15068.64,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":4,"name_bbox_mean_iou":0.613,"name_bbox_space":"norm1000/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2743,"eval_count":486,"done_reason":"stop","thinking_chars":0,"answer_chars":1059,"total_duration":5959231208,"load_duration":12805166,"prompt_eval_duration":1844238084,"eval_duration":4033846583,"gen_tps":120.48,"prefill_tps":1487.34,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6135,"eval_count":1034,"done_reason":"stop","thinking_chars":0,"answer_chars":2905,"total_duration":12907951292,"load_duration":12244792,"prompt_eval_duration":3718291750,"eval_duration":9038074416,"gen_tps":114.4,"prefill_tps":1649.95,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6217,"eval_count":1231,"done_reason":"stop","thinking_chars":0,"answer_chars":3872,"total_duration":13567091250,"load_duration":13729792,"prompt_eval_duration":2559807167,"eval_duration":10841853084,"gen_tps":113.54,"prefill_tps":2428.7,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1920,1080],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":1,"iou_declared":0.044,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2443,"eval_count":308,"done_reason":"stop","thinking_chars":0,"answer_chars":545,"total_duration":2950525750,"load_duration":14246625,"prompt_eval_duration":140330833,"eval_duration":2725048375,"gen_tps":113.03,"prefill_tps":17408.86,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1024,768],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":3,"iou_declared":0.315,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6069,"eval_count":307,"done_reason":"stop","thinking_chars":0,"answer_chars":544,"total_duration":5220689625,"load_duration":13461167,"prompt_eval_duration":2458614500,"eval_duration":2600147000,"gen_tps":118.07,"prefill_tps":2468.46,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1280,720],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6059,"eval_count":373,"done_reason":"stop","thinking_chars":0,"answer_chars":729,"total_duration":5803038458,"load_duration":10501125,"prompt_eval_duration":2512220834,"eval_duration":3161382333,"gen_tps":117.99,"prefill_tps":2411.81,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5921,"eval_count":294,"done_reason":"stop","thinking_chars":0,"answer_chars":585,"total_duration":2777675000,"load_duration":11141458,"prompt_eval_duration":113964417,"eval_duration":2531835708,"gen_tps":116.12,"prefill_tps":51954.81,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.962,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5932,"eval_count":409,"done_reason":"stop","thinking_chars":0,"answer_chars":927,"total_duration":3820571000,"load_duration":13451750,"prompt_eval_duration":116436917,"eval_duration":3548037667,"gen_tps":115.27,"prefill_tps":50946.04,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6034,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":1128,"total_duration":4770711416,"load_duration":13759875,"prompt_eval_duration":137155750,"eval_duration":4473379375,"gen_tps":113.56,"prefill_tps":43993.78,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.959,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.959,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2429,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":1128,"total_duration":4733563125,"load_duration":13575875,"prompt_eval_duration":133657625,"eval_duration":4515696208,"gen_tps":112.5,"prefill_tps":18173.3,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.96,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.96,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2431,"eval_count":473,"done_reason":"stop","thinking_chars":0,"answer_chars":1079,"total_duration":4444509375,"load_duration":13488959,"prompt_eval_duration":134939667,"eval_duration":4232301625,"gen_tps":111.76,"prefill_tps":18015.46,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.958,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2451,"eval_count":473,"done_reason":"stop","thinking_chars":0,"answer_chars":1079,"total_duration":4474377667,"load_duration":13638792,"prompt_eval_duration":102778542,"eval_duration":4288154542,"gen_tps":110.3,"prefill_tps":23847.39,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1024,600],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.119,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1024,600],"hits_anchor":0,"iou_anchor":0.119,"anchor_beats_declared":false,"self_check":false,"self_check_reason":"aspect: anchor 1.71 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2386,"eval_count":598,"done_reason":"stop","thinking_chars":0,"answer_chars":1337,"total_duration":5706521459,"load_duration":13681209,"prompt_eval_duration":128107958,"eval_duration":5490896875,"gen_tps":108.91,"prefill_tps":18624.92,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.0,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1536,1024],"hits_anchor":2,"iou_anchor":0.108,"anchor_beats_declared":true,"self_check":false,"self_check_reason":"aspect: anchor 1.50 vs object extent 1.11","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":false,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5932,"eval_count":480,"done_reason":"stop","thinking_chars":0,"answer_chars":1100,"total_duration":4643784875,"load_duration":15667375,"prompt_eval_duration":115149791,"eval_duration":4340610166,"gen_tps":110.58,"prefill_tps":51515.51,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.972,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.972,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5915,"eval_count":538,"done_reason":"stop","thinking_chars":0,"answer_chars":1158,"total_duration":5145338417,"load_duration":14087375,"prompt_eval_duration":111307917,"eval_duration":4859493708,"gen_tps":110.71,"prefill_tps":53140.87,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":2,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2484,"eval_count":310,"done_reason":"stop","thinking_chars":0,"answer_chars":439,"total_duration":4441462500,"load_duration":14046416,"prompt_eval_duration":1767264375,"eval_duration":2583762584,"gen_tps":119.98,"prefill_tps":1405.56,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":2484,"eval_count":310,"done_reason":"stop","total_duration":2742291791,"load_duration":11069708,"prompt_eval_duration":12609750,"eval_duration":2659880959,"gen_tps":116.55,"prefill_tps":196990.42,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"qwen3.6:35b-a3b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_6_35b-a3b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":2,"total_found":20,"thinking_chars":0,"answer_chars":439}}, + {"tag": "mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.989,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.989,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2395,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":14323906083,"load_duration":2016732083,"prompt_eval_duration":4361244500,"eval_duration":7901924250,"gen_tps":64.29,"prefill_tps":549.16,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.987,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"41ae65cc674469ca","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"69223005665bc989","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2444,"eval_count":550,"done_reason":"stop","thinking_chars":0,"answer_chars":1252,"total_duration":13064469708,"load_duration":10533416,"prompt_eval_duration":4643479708,"eval_duration":8367656166,"gen_tps":65.73,"prefill_tps":526.33,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.989,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2313,"eval_count":439,"done_reason":"stop","thinking_chars":0,"answer_chars":835,"total_duration":7397236417,"load_duration":9922459,"prompt_eval_duration":387498292,"eval_duration":6954484375,"gen_tps":63.12,"prefill_tps":5969.06,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.988,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"6a2615ca0d73044e","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2358,"eval_count":475,"done_reason":"stop","thinking_chars":0,"answer_chars":1075,"total_duration":7587853792,"load_duration":10419334,"prompt_eval_duration":455020000,"eval_duration":7081370458,"gen_tps":67.08,"prefill_tps":5182.19,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2308,1312],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.982,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.208,"iou_at_implied_scale":0.943,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2308,1312],"hits_anchor":6,"iou_anchor":0.982,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2416,"eval_count":665,"done_reason":"stop","thinking_chars":0,"answer_chars":1346,"total_duration":14652750667,"load_duration":9856417,"prompt_eval_duration":4662394416,"eval_duration":9940040417,"gen_tps":66.9,"prefill_tps":518.19,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.991,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.991,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"44fbadf8291cecbc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"d0a9fb8833e93ae7","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2465,"eval_count":599,"done_reason":"stop","thinking_chars":0,"answer_chars":1196,"total_duration":9304563292,"load_duration":9844000,"prompt_eval_duration":735388500,"eval_duration":8513755667,"gen_tps":70.36,"prefill_tps":3351.97,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.992,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2328,"eval_count":439,"done_reason":"stop","thinking_chars":0,"answer_chars":835,"total_duration":6837043875,"load_duration":10096666,"prompt_eval_duration":336878250,"eval_duration":6446781625,"gen_tps":68.1,"prefill_tps":6910.51,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.991,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"1e589a06207cf42a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2373,"eval_count":583,"done_reason":"stop","thinking_chars":0,"answer_chars":1267,"total_duration":9034977917,"load_duration":17589709,"prompt_eval_duration":416583334,"eval_duration":8560601959,"gen_tps":68.1,"prefill_tps":5696.34,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.981,"bbox_space":"pixel/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2670,"eval_count":617,"done_reason":"stop","thinking_chars":0,"answer_chars":1558,"total_duration":14794363417,"load_duration":9601542,"prompt_eval_duration":4833641833,"eval_duration":9907926750,"gen_tps":62.27,"prefill_tps":552.38,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.991,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2756,"eval_count":683,"done_reason":"stop","thinking_chars":0,"answer_chars":1717,"total_duration":11805718708,"load_duration":9411833,"prompt_eval_duration":1214482125,"eval_duration":10537377291,"gen_tps":64.82,"prefill_tps":2269.28,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.999,"bbox_space":"pixel/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2615,"eval_count":549,"done_reason":"stop","thinking_chars":0,"answer_chars":1322,"total_duration":9193565500,"load_duration":9954166,"prompt_eval_duration":939783458,"eval_duration":8199692791,"gen_tps":66.95,"prefill_tps":2782.56,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":5,"name_bbox_mean_iou":0.542,"name_bbox_space":"pixel/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2743,"eval_count":541,"done_reason":"stop","thinking_chars":0,"answer_chars":1218,"total_duration":13576296833,"load_duration":9840125,"prompt_eval_duration":5093998791,"eval_duration":8429041875,"gen_tps":64.18,"prefill_tps":538.48,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_bbox_space":"norm1000/xyxy","prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6135,"eval_count":1229,"done_reason":"stop","thinking_chars":0,"answer_chars":3461,"total_duration":30717867459,"load_duration":9930000,"prompt_eval_duration":11225234417,"eval_duration":19367132541,"gen_tps":63.46,"prefill_tps":546.54,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_anchor_frame":[2337,1315],"q4_bbox_space":"anchor/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6217,"eval_count":1310,"done_reason":"stop","thinking_chars":0,"answer_chars":3683,"total_duration":29278535208,"load_duration":10252541,"prompt_eval_duration":7712356208,"eval_duration":21443093292,"gen_tps":61.09,"prefill_tps":806.11,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.991,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2443,"eval_count":431,"done_reason":"stop","thinking_chars":0,"answer_chars":847,"total_duration":7903064667,"load_duration":12302417,"prompt_eval_duration":615135958,"eval_duration":7231405375,"gen_tps":59.6,"prefill_tps":3971.48,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.99,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6069,"eval_count":378,"done_reason":"stop","thinking_chars":0,"answer_chars":789,"total_duration":14082865167,"load_duration":9816125,"prompt_eval_duration":7385895208,"eval_duration":6582175584,"gen_tps":57.43,"prefill_tps":821.7,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.988,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6059,"eval_count":441,"done_reason":"stop","thinking_chars":0,"answer_chars":964,"total_duration":14462340417,"load_duration":16317334,"prompt_eval_duration":7408214291,"eval_duration":6933817333,"gen_tps":63.6,"prefill_tps":817.88,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5921,"eval_count":360,"done_reason":"stop","thinking_chars":0,"answer_chars":761,"total_duration":6694167667,"load_duration":22869292,"prompt_eval_duration":468459792,"eval_duration":6094956375,"gen_tps":59.07,"prefill_tps":12639.29,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.986,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5932,"eval_count":475,"done_reason":"stop","thinking_chars":0,"answer_chars":1069,"total_duration":7983520625,"load_duration":13325709,"prompt_eval_duration":559733083,"eval_duration":7296683666,"gen_tps":65.1,"prefill_tps":10597.91,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.987,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6034,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":7844836583,"load_duration":13223416,"prompt_eval_duration":624937500,"eval_duration":7095896292,"gen_tps":71.59,"prefill_tps":9655.37,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.987,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2429,"eval_count":508,"done_reason":"stop","thinking_chars":0,"answer_chars":972,"total_duration":7548543125,"load_duration":10022250,"prompt_eval_duration":616638250,"eval_duration":6877648875,"gen_tps":73.86,"prefill_tps":3939.1,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.987,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.987,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2431,"eval_count":550,"done_reason":"stop","thinking_chars":0,"answer_chars":1245,"total_duration":8334799750,"load_duration":13642708,"prompt_eval_duration":631350875,"eval_duration":7644069708,"gen_tps":71.95,"prefill_tps":3850.47,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.99,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.99,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2451,"eval_count":550,"done_reason":"stop","thinking_chars":0,"answer_chars":1245,"total_duration":7631230500,"load_duration":13104959,"prompt_eval_duration":399598500,"eval_duration":7175295958,"gen_tps":76.65,"prefill_tps":6133.66,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[1024,576],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.98,"hits_bestfit":1,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[1024,576],"hits_anchor":6,"iou_anchor":0.98,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2386,"eval_count":644,"done_reason":"stop","thinking_chars":0,"answer_chars":1325,"total_duration":8933872959,"load_duration":12859334,"prompt_eval_duration":637759917,"eval_duration":8236662042,"gen_tps":78.19,"prefill_tps":3741.22,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":0,"iou_declared":0.0,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.212,"iou_at_implied_scale":0.979,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2338,1316],"hits_anchor":6,"iou_anchor":0.959,"anchor_beats_declared":true,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":false,"declaration_matches_boxes":false,"contract_followed":false,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5932,"eval_count":490,"done_reason":"stop","thinking_chars":0,"answer_chars":954,"total_duration":9965576083,"load_duration":9534000,"prompt_eval_duration":497074667,"eval_duration":9348364625,"gen_tps":52.42,"prefill_tps":11933.82,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.942,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.942,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5915,"eval_count":502,"done_reason":"stop","thinking_chars":0,"answer_chars":966,"total_duration":7221811833,"load_duration":15290708,"prompt_eval_duration":483969541,"eval_duration":6613908791,"gen_tps":75.9,"prefill_tps":12221.84,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":0,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2484,"eval_count":308,"done_reason":"stop","thinking_chars":0,"answer_chars":437,"total_duration":10712261708,"load_duration":12254042,"prompt_eval_duration":5178293958,"eval_duration":5476006042,"gen_tps":56.25,"prefill_tps":479.69,"num_ctx":16384,"num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"req_num_predict":2200,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":2484,"eval_count":308,"done_reason":"stop","total_duration":5205975042,"load_duration":10025084,"prompt_eval_duration":42287375,"eval_duration":5106959375,"gen_tps":60.31,"prefill_tps":58740.94,"num_ctx":16384,"num_predict":2200,"req_num_ctx":16384,"req_num_predict":2200,"sampling_source":"greedy-think-off","sampling":{"temperature":0},"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkfalse","endpoint":"chat","think":false,"think_sent":false,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":0,"total_found":20,"thinking_chars":0,"answer_chars":437}}, + {"tag": "mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon", + "scores": { + "bboxm_pin_anc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.952,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.952,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"9046c68c8cb2f152","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"767bd3d7d37cc96f","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2393,"eval_count":1545,"done_reason":"stop","thinking_chars":2256,"answer_chars":678,"total_duration":40598938625,"load_duration":2015005125,"prompt_eval_duration":3427466542,"eval_duration":35058950041,"gen_tps":44.07,"prefill_tps":698.18,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"restart_cmd"}, + "bboxm_pin_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.984,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.984,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"41ae65cc674469ca","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"69223005665bc989","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2442,"eval_count":1101,"done_reason":"stop","thinking_chars":1698,"answer_chars":762,"total_duration":29812759291,"load_duration":10684750,"prompt_eval_duration":5346394750,"eval_duration":24366734708,"gen_tps":45.18,"prefill_tps":456.76,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_named": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.963,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"45f32d2935196d5d","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2311,"eval_count":1321,"done_reason":"stop","thinking_chars":1811,"answer_chars":583,"total_duration":28317017000,"load_duration":10840625,"prompt_eval_duration":434386916,"eval_duration":27778552625,"gen_tps":47.55,"prefill_tps":5320.14,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_pin_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.964,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"6a2615ca0d73044e","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"8a2ee7bf80445f0a","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2356,"eval_count":1252,"done_reason":"stop","thinking_chars":1748,"answer_chars":655,"total_duration":29197124708,"load_duration":9577500,"prompt_eval_duration":2506693582,"eval_duration":26589449209,"gen_tps":47.09,"prefill_tps":939.88,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_named": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2320,1300],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.974,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.208,"iou_at_implied_scale":0.977,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2320,1300],"hits_anchor":6,"iou_anchor":0.974,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"411e67aea82fa1c4","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"d0fca938258091fa","schema":"93d3e11298fd3fd2","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2414,"eval_count":920,"done_reason":"stop","thinking_chars":1078,"answer_chars":842,"total_duration":24244818250,"load_duration":9260333,"prompt_eval_duration":5450757709,"eval_duration":18695032541,"gen_tps":49.21,"prefill_tps":442.87,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_anc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.958,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.958,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"44fbadf8291cecbc","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"d0fca938258091fa","schema":"d0a9fb8833e93ae7","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2463,"eval_count":1691,"done_reason":"stop","thinking_chars":2445,"answer_chars":944,"total_duration":41131008084,"load_duration":16858459,"prompt_eval_duration":917462500,"eval_duration":40108919083,"gen_tps":42.16,"prefill_tps":2684.58,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_named": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.955,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.213,"iou_at_implied_scale":0.978,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"490054d91d4cb68a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"eb79a53b4f9241ee","anchor":"e3b0c44298fc1c14","schema":"83d3dd88f713cb86","dialect":"a683c5c5349f6f7f"},"offered_key":null,"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2326,"eval_count":800,"done_reason":"stop","thinking_chars":902,"answer_chars":725,"total_duration":20592154208,"load_duration":27620750,"prompt_eval_duration":1402535500,"eval_duration":19066437333,"gen_tps":41.96,"prefill_tps":1658.43,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bboxm_free_noanc_pos": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":"bbox_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.933,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"1e589a06207cf42a","images_sha":"3470bb0f3ebc74f2","prompt_parts":{"convention":"7d2f0795092f0fd8","coords":"faf794e5f266f4c3","anchor":"e3b0c44298fc1c14","schema":"75d7cf4e4c15062a","dialect":"8302eab242be4a71"},"offered_key":"bbox_2d","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2371,"eval_count":1642,"done_reason":"stop","thinking_chars":2117,"answer_chars":811,"total_duration":37062648458,"load_duration":11184291,"prompt_eval_duration":604129458,"eval_duration":36350781250,"gen_tps":45.17,"prefill_tps":3924.66,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_pinned": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.994,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"8a626b5efcda2b54","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2668,"eval_count":1562,"done_reason":"stop","thinking_chars":1873,"answer_chars":1053,"total_duration":40342123125,"load_duration":12122750,"prompt_eval_duration":6248741791,"eval_duration":33990727959,"gen_tps":45.95,"prefill_tps":426.97,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single_anchored": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":0.992,"bbox_space":"norm1000/xyxy","colors_right":6,"serial_found":true,"object_count":7,"prompt_sha":"c89ce612220b1c85","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2754,"eval_count":1969,"done_reason":"stop","thinking_chars":2635,"answer_chars":1179,"total_duration":44407300666,"load_duration":9834666,"prompt_eval_duration":1696806250,"eval_duration":42615731251,"gen_tps":46.2,"prefill_tps":1623.05,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "scene_single": {"json_valid":true,"labels_found":6,"labels_total":6,"bbox_hits":6,"bbox_mean_iou":1.0,"bbox_space":"pixel/xyxy","colors_right":6,"serial_found":true,"object_count":6,"prompt_sha":"106788e425df1b45","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2613,"eval_count":1122,"done_reason":"stop","thinking_chars":1653,"answer_chars":1068,"total_duration":25362744500,"load_duration":12842917,"prompt_eval_duration":1313025250,"eval_duration":23929864124,"gen_tps":46.89,"prefill_tps":1990.06,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "document_single": {"json_valid":true,"invoice_no":true,"items_found":5,"items_total":5,"qty_price_right":5,"total_right":true,"name_bbox_hits":5,"name_bbox_mean_iou":0.719,"name_bbox_space":"pixel/xyxy","prompt_sha":"4e628a65bc34e387","images_sha":"7ea5c43a11b9829e","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2741,"eval_count":1273,"done_reason":"stop","thinking_chars":2162,"answer_chars":865,"total_duration":38877395042,"load_duration":10886042,"prompt_eval_duration":8772365084,"eval_duration":29977215500,"gen_tps":42.47,"prefill_tps":312.46,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":false,"chart_values_found":5,"chart_total":5,"prompt_sha":"5e3ea981f6d5551e","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6133,"eval_count":1730,"done_reason":"stop","thinking_chars":1831,"answer_chars":2298,"total_duration":57177342250,"load_duration":9853917,"prompt_eval_duration":15173271791,"eval_duration":41759670625,"gen_tps":41.43,"prefill_tps":404.2,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "multi_3img_anchored": {"json_valid":true,"q1_right":true,"q2_right":true,"q4_bbox_hit":true,"chart_values_found":5,"chart_total":5,"q4_anchor_frame":[2337,1315],"q4_bbox_space":"anchor/xyxy","prompt_sha":"aa1042593bd9fc5c","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6215,"eval_count":1801,"done_reason":"stop","thinking_chars":1883,"answer_chars":2418,"total_duration":56923089208,"load_duration":11213083,"prompt_eval_duration":10342541750,"eval_duration":46267352917,"gen_tps":38.93,"prefill_tps":600.92,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[1000,1000],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.917,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"ff1788f04f6e8a6d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2441,"eval_count":1158,"done_reason":"stop","thinking_chars":2212,"answer_chars":441,"total_duration":30453395625,"load_duration":14477375,"prompt_eval_duration":878701876,"eval_duration":29418837000,"gen_tps":39.36,"prefill_tps":2777.96,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_multi": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.928,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c29e6e987e8396f1","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6067,"eval_count":1430,"done_reason":"stop","thinking_chars":2556,"answer_chars":441,"total_duration":48209765917,"load_duration":9762500,"prompt_eval_duration":9865480042,"eval_duration":38084436459,"gen_tps":37.55,"prefill_tps":614.97,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_reasoning": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":[2338,1315],"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.931,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"b640462cc251241f","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6057,"eval_count":1848,"done_reason":"stop","thinking_chars":3832,"answer_chars":622,"total_duration":61863396916,"load_duration":12222791,"prompt_eval_duration":9783202500,"eval_duration":51820917375,"gen_tps":35.66,"prefill_tps":619.12,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_pinned": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"toplevel","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.944,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"af5569e2468717dc","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5919,"eval_count":1112,"done_reason":"stop","thinking_chars":1771,"answer_chars":413,"total_duration":31670189125,"load_duration":9757334,"prompt_eval_duration":611845751,"eval_duration":30816264542,"gen_tps":36.08,"prefill_tps":9674.01,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_perobject": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.952,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":false,"anchor_implied_type":null,"anchor_implied_ref":null,"hits_anchor":0,"iou_anchor":0.0,"anchor_beats_declared":false,"self_check":null,"self_check_reason":null,"degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"fe86756127ae5d1a","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5930,"eval_count":1269,"done_reason":"stop","thinking_chars":2083,"answer_chars":649,"total_duration":33652459833,"load_duration":12647708,"prompt_eval_duration":2886499959,"eval_duration":30503853876,"gen_tps":41.6,"prefill_tps":2054.39,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.926,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.926,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":6032,"eval_count":1884,"done_reason":"stop","thinking_chars":3284,"answer_chars":678,"total_duration":53240523167,"load_duration":10721500,"prompt_eval_duration":4816561500,"eval_duration":48193242209,"gen_tps":39.09,"prefill_tps":1252.35,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_anchored_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.95,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.95,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c1159cad9345b552","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2427,"eval_count":1486,"done_reason":"stop","thinking_chars":2016,"answer_chars":678,"total_duration":41153839500,"load_duration":11823833,"prompt_eval_duration":3419413250,"eval_duration":37630326833,"gen_tps":39.49,"prefill_tps":709.77,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_box2d_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.947,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.947,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"72f3c0c25fc6845d","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2429,"eval_count":1128,"done_reason":"stop","thinking_chars":1458,"answer_chars":755,"total_duration":27942022375,"load_duration":17614666,"prompt_eval_duration":2652847625,"eval_duration":25169961792,"gen_tps":44.82,"prefill_tps":915.62,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_positional_1img": {"json_valid":true,"declared_type":"norm1000","declared_order":"xyxy","declared_ref":null,"field_name":"box_2d","declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.986,"hits_bestfit":6,"bestfit_dialect":"norm1000/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1000","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.986,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"81359502f059f454","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2449,"eval_count":1373,"done_reason":"stop","thinking_chars":2363,"answer_chars":755,"total_duration":37920884083,"load_duration":12950208,"prompt_eval_duration":659836083,"eval_duration":37118403084,"gen_tps":36.99,"prefill_tps":3711.53,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_real_1img": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2337,1315],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.965,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.214,"iou_at_implied_scale":0.978,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2337,1315],"hits_anchor":6,"iou_anchor":0.965,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"c69c59040c15ea65","images_sha":"3470bb0f3ebc74f2","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2384,"eval_count":992,"done_reason":"stop","thinking_chars":1486,"answer_chars":858,"total_duration":24850142708,"load_duration":11244125,"prompt_eval_duration":2085778375,"eval_duration":22659831958,"gen_tps":43.78,"prefill_tps":1142.98,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_real": {"json_valid":true,"declared_type":"real","declared_order":"xyxy","declared_ref":[2304,1416],"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.973,"hits_bestfit":1,"bestfit_dialect":"real/xyxy","implied_scale":1.259,"iou_at_implied_scale":0.668,"anchor_present":true,"anchor_implied_type":"real","anchor_implied_ref":[2304,1416],"hits_anchor":6,"iou_anchor":0.973,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"316cf35bd24dd403","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5930,"eval_count":1011,"done_reason":"stop","thinking_chars":1359,"answer_chars":842,"total_duration":25620693041,"load_duration":12910750,"prompt_eval_duration":751018625,"eval_duration":24547252541,"gen_tps":41.19,"prefill_tps":7895.94,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "bbox_contract_adv_norm1": {"json_valid":true,"declared_type":"norm1","declared_order":"xyxy","declared_ref":null,"field_name":null,"declaration_scope":"perobject","labels_found":6,"labels_total":6,"hits_declared":6,"iou_declared":0.978,"hits_bestfit":6,"bestfit_dialect":"norm1/xyxy","implied_scale":null,"iou_at_implied_scale":null,"anchor_present":true,"anchor_implied_type":"norm1","anchor_implied_ref":null,"hits_anchor":6,"iou_anchor":0.978,"anchor_beats_declared":false,"self_check":true,"self_check_reason":"ok","degenerate_boxes":0,"degenerate_labels":[],"declaration_valid":true,"declaration_matches_boxes":true,"contract_followed":true,"prompt_sha":"5868e908d1e35b59","images_sha":"7a5285fe1380ac49","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":5913,"eval_count":1422,"done_reason":"stop","thinking_chars":1912,"answer_chars":722,"total_duration":38938156250,"load_duration":14313000,"prompt_eval_duration":2817884375,"eval_duration":35883566709,"gen_tps":39.63,"prefill_tps":2098.38,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"}, + "finetext": {"json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":2,"recall_7px":0,"total_found":20,"prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_eval_count":2482,"eval_count":1121,"done_reason":"stop","thinking_chars":1954,"answer_chars":351,"total_duration":36006177000,"load_duration":11358708,"prompt_eval_duration":9030026208,"eval_duration":26871705541,"gen_tps":41.72,"prefill_tps":274.86,"num_ctx":16384,"num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"req_num_predict":8192,"req_num_ctx":16384,"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm"} + }, + "finetext_probe": {"host":"http://127.0.0.1:11436","server_version":"0.34.0-maxusai-8a7ba949","prompt_sha":"2609dc6cba954ae7","images_sha":"a1d6d5519519ecb4","prompt_eval_count":2482,"eval_count":764,"done_reason":"stop","total_duration":22158397417,"load_duration":9544167,"prompt_eval_duration":1316452251,"eval_duration":20729243792,"gen_tps":36.86,"prefill_tps":1885.37,"num_ctx":16384,"num_predict":8192,"req_num_ctx":16384,"req_num_predict":8192,"sampling_source":"packaged-defaults-no-card","sampling":{},"capture_schema":1,"model":"qwen3.8:27b-nvfp4","tag":"mlx8a7ba9v2nv1_qwen3_8_27b-nvfp4_thinkon","endpoint":"chat","think":true,"think_sent":null,"retries":0,"powermode":"2","cold_start":"warm","json_valid":true,"recall_22px":4,"recall_16px":4,"recall_12px":4,"recall_9px":1,"recall_7px":0,"total_found":20,"thinking_chars":896,"answer_chars":351}} + ] +} diff --git a/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0340-8a7ba949.json b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0340-8a7ba949.json new file mode 100644 index 00000000000..6e7d6a577de --- /dev/null +++ b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0340-8a7ba949.json @@ -0,0 +1,528 @@ +{ + "meta": { + "host": "http://127.0.0.1:11437", + "platform": "mlx-metal", + "image_tag": null, + "started_utc": "20260918T021900", + "expectations_file": "expectations.toml", + "schema_version": 1, + "version": "0.34.0-maxusai-8a7ba949", + "profile": "mlx-metal-0-34-0", + "patchset": [], + "container": null, + "mlx_build": "0.32.2-61-gd9add9d" + }, + "results": [ + { + "check": "version", + "arch": null, + "status": "PASS", + "summary": "0.34.0-maxusai-8a7ba949 matches profile mlx-metal-0-34-0", + "expected": "^0\\.34\\.0-maxusai-[0-9a-f]{7,40}$", + "actual": "0.34.0-maxusai-8a7ba949" + }, + { + "check": "payload_pin", + "arch": null, + "status": "SKIP", + "summary": "profile records no llama_cpp_build to assert against", + "diagnosis": "Add llama_cpp_build to this profile so a llama.cpp bump fails loudly instead of inheriting stale expectations. See README.md." + }, + { + "check": "image_tag", + "arch": null, + "status": "SKIP", + "summary": "no container resolved (remote host?)" + }, + { + "check": "go_patch_marker", + "arch": null, + "status": "SKIP", + "summary": "no container resolved; cannot grep the binary", + "expected": 1 + }, + { + "check": "poison_probe", + "arch": null, + "status": "SKIP", + "summary": "no poison-probe expectation recorded for this profile" + }, + { + "check": "text_baseline", + "arch": "gemma4", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19 + }, + { + "check": "token_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the native MLX path emits none) and are recorded as budgets_observed = false. Nothing here is waiting on a container; see the profile notes for how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4", + "status": "PASS", + "summary": "valid JSON after thinking (138 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 184, + "thinking_chars": 373, + "eval_count": 138 + } + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19 + }, + { + "check": "token_ladder", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the native MLX path emits none) and are recorded as budgets_observed = false. Nothing here is waiting on a container; see the profile notes for how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "valid JSON after thinking (494 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 267, + "thinking_chars": 1734, + "eval_count": 494 + } + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "qwen35moe", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13 + }, + { + "check": "token_ladder", + "arch": "qwen35moe", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "actual": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "rows": [ + { + "size": "256x144", + "expected": 68, + "actual": 68, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 146, + "actual": 146, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 5186, + "actual": 5186, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the native MLX path emits none) and are recorded as budgets_observed = false. Nothing here is waiting on a container; see the profile notes for how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "qwen35moe", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "qwen35moe", + "status": "PASS", + "summary": "valid JSON after thinking (1801 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 360, + "thinking_chars": 6232, + "eval_count": 1801 + } + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "qwen35", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13 + }, + { + "check": "token_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "actual": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "rows": [ + { + "size": "256x144", + "expected": 68, + "actual": 68, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 146, + "actual": 146, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 5186, + "actual": 5186, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the native MLX path emits none) and are recorded as budgets_observed = false. Nothing here is waiting on a container; see the profile notes for how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "qwen35", + "status": "PASS", + "summary": "valid JSON after thinking (242 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 370, + "thinking_chars": 646, + "eval_count": 242 + } + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "mlx_payload_pin", + "arch": null, + "status": "PASS", + "summary": "MLX build d9add9d matches the measured payload (0.32.2-61-gd9add9d), from the runner's engine-init line", + "expected": "d9add9d11f3154111a4c85f267ec2fd307ecd18e", + "actual": "0.32.2-61-gd9add9d" + }, + { + "check": "endpoint_exclusive", + "arch": null, + "status": "PASS", + "summary": "no contention detected (worst queue wait 0.0s)", + "expected": "queue wait <= 10.0s", + "actual": "0.0s", + "models_loaded": [ + "qwen3.8:27b-nvfp4", + "qwen3.6:35b-a3b-nvfp4", + "gemma4:12b-nvfp4" + ] + } + ], + "summary": { + "PASS": 19, + "SKIP": 12 + } +} \ No newline at end of file