diff --git a/docs/maxusai/adr/0049-a-verbatim-loop-ends-the-cell-as-a-verdict.md b/docs/maxusai/adr/0049-a-verbatim-loop-ends-the-cell-as-a-verdict.md new file mode 100644 index 00000000000..8ff8549f652 --- /dev/null +++ b/docs/maxusai/adr/0049-a-verbatim-loop-ends-the-cell-as-a-verdict.md @@ -0,0 +1,103 @@ +# ADR 0049: A verbatim loop ends the cell as a verdict + +Date: 2026-10-11 · Status: proposed. Opt-in (`LOOP_DETECT=1`); first used by the muse-glimmer campaign's think-on +and OCRBench arms (#463). + +## Context + +Under greedy decoding (the suite's think-off arms, and think-on at `THINK_TEMPERATURE=0`), a model that starts +repeating a block verbatim, back to back, usually keeps repeating it. The same context tends to produce the same +continuation, so the cell spends the rest of `num_predict` on the loop. Under think-on the context ladder then climbs from 16384 to 131072 and repeats +the loop at every rung, with `num_predict = num_ctx - 8192`. + +The muse-glimmer campaign's think-off pass (2026-10-11) showed every form of it: +- **MLX `30b-mxfp8`, `bboxm_pin_noanc_named`:** one box object's keys, repeated to 4350 chars. +- **MLX `30b-nvfp4`, `multi_3img`:** a 72-char fragment, repeated until it hit the length cap at every rung up to + 131072. +- **GGUF `30b`, two cases:** a run of one token. ollama's own guard aborts at 100 repeats with an HTTP 500 and no + output (`llm/llama_server.go:2483`). + +"Tokens to finish is the result; never cap to save tokens" stands, and this decision is a stated exception to it. +What a loop verdict guarantees is narrow: the stream's last 400 chars, plus two periods, are one block repeated three +times back to back. It does not guarantee the model would never have finished. The CUDA host's replay (#469 review) +found such loops on 57 length-capped cells, and on 3 of 1,505 think-on traces that went on to finish. Ending a cell +at the loop gives up those rare recoveries: the cell is labelled, not measured to its end. In exchange, a loop that +runs to the cap (57 of the 60 on that replay) costs one verdict at the rung where it starts, instead of the rest of +`num_predict` at every rung of the ladder. + +## Decision + +1. **`loop_detect.py` judges a stream as it arrives.** It is stdlib only, and the same file is shared with + ai-invoice-to-xero's `ollama_toolkit` and the structured-docs backend. It reports one of three failure modes: + - `loop detected:thinking`: the last 400 chars of the thinking trace occur 3 times in it, back to back. The last + 400 chars plus two periods must be one block repeated. + - `loop detected:content`: the same test, on the answer. + - `loop detected:token-run`: one stream emitted the same whitespace-stripped chunk 64 times in a row. That is + below the server's 100, so the client ends it first and keeps the output. + + Three copies, not two: one earlier copy is how a model restates a table. Back to back, not anywhere: three + copies with other text between them are how a thinking trace restates its draft after each check. The check runs every 64 new chars over + the last 16384 chars, so its cost is linear in the stream. +2. **`LOOP_DETECT=1` makes `client.generate` stream.** On a loop it closes the connection, which cancels the + generation server-side, and returns the partial output as a **finished** response: + - `done_reason` is `"loop"`, and `failure_mode` is `"loop detected:"`. + - The loop's fields: kind, stream, period, repeats, chars at detection, snippet, parameters. + - `eval_count` is the number of streamed chunks, and the durations are None (the server sent no final metrics). + - The server's own abort can still land first: it counts the runner's raw tokens, the client counts the chunks + that reach it after the output parser, and a parser that holds back or drops tokens delivers fewer. Mid-stream + the server can no longer send a status, so the abort arrives as a final `{"error": ...}` line. The client + raises it as `HTTP 500 (streamed): prediction aborted, token repeat limit reached`, naming the abort, and does + not retry it, as it does not retry the HTTP 500 that `stream: false` gets. It is an error, not a verdict: the + partial output is not kept. +3. **A loop the server held back is judged at the end.** A server can send a reply in its final chunk: glimmer's + parser holds back a reply that never writes its message header (the #464 path), so on the release build its + format + think-off loops stream nothing until the generation ends. The final chunk is fed to the detector too. + A loop in a reply the server stopped at `num_predict` becomes the same verdict, with the server's own metrics + kept and `server_done_reason: "length"` recorded. A reply that finished is never a loop, however repetitive. + Such a loop is labelled, not cut early: that needs the server to stream. +4. **A loop is a verdict, not an error and not a cap.** + - The suite scores the partial output as it scores a capped answer, and no table shows that score. Every + template renders `loop:` where the score would go, the way it renders `capped`: T1's quality and + throughput cells, T2's `cap_or` and latency rows, the contract matrix, geometry, reps and the bbox matrix. + - No mean pools a loop. `summarize_engine_compare.not_a_score` (capped, or a loop) is the one test, and the counts + name loops apart from caps. `cmp_scored` compares no loop, and a preflight quality run reports one as a + failure by name, outside `json_valid`. + - `was_capped` is False for `done_reason: "loop"` at any `eval_count`. A held-back loop stopped at + `num_predict` keeps `eval_count == num_predict`, and the arithmetic fallback would otherwise call it capped. + - `arm_done` treats the block as finished, so it is not re-run on resume. + - `capped_arms` does not climb for it. +5. **Every streamed block says so.** `client.loop_fields` stamps `stream: true` and the `loop_detect` parameters, + plus the loop's fields when it fires, on vision_suite and finetext_probe blocks alike. +6. **Off by default.** A campaign that turns it on mixes nothing: every block carries the stamp. + +## Validation + +- **Replay:** every saved response in the glimmer and v0.40.2 campaign trees (1817 files) was fed to the detector. + - It fired on 4. All four were cells that ran to the length cap: the two `multi_3img` and the two + `bboxm_pin_noanc_named` above. + - It fired at 800-1312 chars, against the 2182-4350 the cells generated. + - It fired on no cell that finished. +- **Thinking traces, replayed on the CUDA host (#469 review):** 1,641 traces (`client.save` writes them). + - Counting copies alone, the detector fired on 17-18 of the 1,505 think-on traces that finished + (`done_reason: "stop"`). 13 of them were `multi_3img` cells with q1, q2 and q4 all right: a draft restated + between checks, not a loop. + - Requiring the copies back to back left 3 fires on finished traces, all contiguous greedy loops that then + recovered. It kept all 57 fires on length-capped cells. +- **Thinking traces, replayed on the Metal host:** 2,176 traces saved in this checkout's worktrees. + - Finished (`stop`): 11 fires by copy count, 3 back to back. Length-capped (`length`): 57 and 57. + - Two verdicts a live think-on run had already recorded under the copy count were restated drafts: + `gemma4:31b-nvfp4` `multi_3img_anchored` (earlier copies 4118 and 7580 chars back) and `bboxm_pin_anc_pos` + (963 and 3307 chars back). The back-to-back rule calls neither. +- **Tests:** `test_loop_detect.py` (18 tests, including those two real outputs as fixtures, a restated draft that is + not a loop, and a loop that starts after one) and `test_client.py` (`TestLoopDetect`, `TestLoopFields`). + +## Consequences + +- **Think-on cells that loop end at the rung where they loop,** as `failure_mode: loop detected:thinking`. Before, + they climbed to the ceiling and ended NOT CONVERGED. A NOT CONVERGED cell under detection is therefore a + non-looping one that genuinely needs more. +- **A looped cell has no decode rate.** `metrics_block` sets `gen_tps` to None for every loop, including a held-back + one whose server metrics would allow it: that rate is a loop's, not a finishing model's. Prefill ended before the + loop, so `prefill_tps` is kept when the server reported it. +- **Streaming changes no generation.** It is the same request with `stream: true`. The drafted MLX tags' text already + depends on wall-clock draft depth (`mlx-greedy-has-baseline-spread`), and this neither fixes nor worsens that. diff --git a/docs/maxusai/spec/vision-harness-reuse.md b/docs/maxusai/spec/vision-harness-reuse.md index 194c5aa6aec..a28aa95c28d 100644 --- a/docs/maxusai/spec/vision-harness-reuse.md +++ b/docs/maxusai/spec/vision-harness-reuse.md @@ -139,7 +139,10 @@ cap (`eval == num_predict`, `done_reason "stop"`), and a thinking continuation that overshoots the recorded cap (eval 8290 against 8192, measured 2026-08-20 on qwen3.6 `bbox_contract_reasoning`). Blocks without the field keep the `eval_count >= num_predict` fallback, so historical files -never change verdict by upgrading the summarizers. +never change verdict by upgrading the summarizers. `"loop"` is not the +server's: the client writes it when it ends a cell on a loop verdict +(`LOOP_DETECT=1`, ADR 0049), and it is never capped, even at +`eval == num_predict` (a loop the server held back and stopped at the cap). The value inventory, from this repo's server code (the serving build is this fork): the runner emits `stop` or `length` (`llm/llama_server.go`, from @@ -212,6 +215,11 @@ renderings of cells that had merely hit `num_predict` 8192 and were never escalated. The owed number is the final-rung score (ADR 0012 rule 8): a campaign with `capped` cells below `CTX_MAX` has not finished running. +A cell ended on a loop verdict (`LOOP_DETECT=1`, ADR 0049) renders +`loop:` in the same cells, from `loop_verdict`, and enters no mean: +`not_a_score` (capped or looped) is the pooling test. Unlike `capped` it is a +result, not a to-do. It is finished (`arm_done`) and is not escalated. + "Every template" means every CELL, not every table: T1 guarded only its scene-derived throughput cells until later the same day, and its first prefixed-campaign render printed the capped qwen3.6 think-on multi ceiling @@ -639,7 +647,7 @@ record the machine, and the machine moves numbers that neither field explains | H7 | ADR 0012 rules 1 and 8; request examples via `emit_request.py` (payload captured from `client.py`, never re-derived). A campaign's tables re-render from its committed bundle with `summarize_engine_compare.py --bundle`, through the code a run directory goes through (`test_summarizers.py::TestBundleSource`) | | H7, H11, H13 (external benchmarks) | `extbench.py` runs render through `summarize_extbench.py`, never by hand; `extbench.py` persists the `host` / `server_version` that `client.generate()` already stamps on every response, collected as SETS so a mid-run container restart cannot let one build vouch for another's rows; `test_summarizers.py::TestExtbenchSummary` asserts the unrecorded-file, mixed-campaign and clean-footer cases, and that the paired test stays EXACT (at 3-vs-1 discordant the uncorrected chi-square reads p≈0.317 against the true 0.625). `extbench.py` also records the `prompt_cache_ram` it sent (`test_extbench.py`), and `summarize_extbench.py` names it in the footer and warns under `--timing` when arms differ, a file without it counting as the server's default (`test_summarizers.py::TestExtbenchArmsAndSlices`) | | H13 (footers) | `test_summarizers.py::TestProvenanceFooter` — clean / all-pre-H11 / mixed-recording / two-host cases against the rendered footer. `prompt_cache_stats.py ab` and `prompt_cache_probe.py compare` print the same footer through `summarize_extbench.provenance` (`test_prompt_cache_stats.py`, `test_prompt_cache_probe.py`) | -| H13 (capped rendering) | `cap_or` in `summarize_head_to_head.py` importing `was_capped` (H5); `test_summarizers.py::TestT2CappedCells` asserts a capped scene hides score and latency but keeps tok/s; `q()`/`multi_cell` in `summarize_engine_compare.py` guard every T1 quality cell — `::TestT1CappedQualityCells` | +| H13 (capped rendering) | `cap_or` in `summarize_head_to_head.py` importing `was_capped` (H5); `test_summarizers.py::TestT2CappedCells` asserts a capped scene hides score and latency but keeps tok/s; `q()`/`multi_cell` in `summarize_engine_compare.py` guard every T1 quality cell — `::TestT1CappedQualityCells`; a loop verdict renders and is excluded the same way in every template and mean — `::TestLoopVerdictReachesTheTables` | | H14 | `token_split.py`'s acceptance gate (`--write` refuses without it, and skips-by-name irreconcilable cells); `test_summarizers.py::TestT1ThinkTokColumn` asserts stamped-count-or-dash, never an estimate; `finetext_probe.py` persists as `finetext_probe` (one producer per persist name) | | H9 | `client.py` is the only module that builds a payload; `test_client.py` asserts the wire format (the test count grows with the contract — trust the suite, not this row), including the tri-state `send_think` and the `num_ctx=False` sentinel that a naive `== False` would have collapsed. `TestPromptCacheRAM` pins H4's one exception: `prompt_cache_ram: 0` on every request unless `PROMPT_CACHE_RAM` says otherwise, none from a calibrated caller (`use_env_opts=False`), and a value the server would ignore failing before the first request. CI runs the suite (`preflight-expectations.yaml`) | | H12 | `prompt_sha` / `images_sha` / `prompt_parts` on every score block, written by `client.generate()`; absence marks a pre-2026-08-20 cell | diff --git a/docs/maxusai/vision-suite/client.py b/docs/maxusai/vision-suite/client.py index 869258cdf1e..bb676bfc967 100644 --- a/docs/maxusai/vision-suite/client.py +++ b/docs/maxusai/vision-suite/client.py @@ -31,6 +31,7 @@ import urllib.error import urllib.request +import loop_detect from sampling import sampling_for DIR = os.path.dirname(os.path.abspath(__file__)) @@ -161,6 +162,107 @@ def context_error(e, num_predict, num_ctx): return RuntimeError(f"HTTP {e.code}: {body[:200]}") +def loop_detect_on(): + """LOOP_DETECT=1 streams every request through loop_detect.LoopDetector (ADR 0049). Off by default: a stream + and a single response measure the same generation, but a campaign that turns detection on must say so (it is + stamped as _stream / _loop_detect on every block), and a cancelled loop is a different record from a capped one.""" + return os.environ.get("LOOP_DETECT", "") in ("1", "true", "on", "yes") + + +class StreamError(RuntimeError): + """An error the server wrote into a stream that had already started. + + server/routes.go streamResponse cannot send a status once the first chunk + is written, so it ends the stream with one {"error": ...} line, at status + 500 unless the error carries its own. With stream:false the same failure + is an HTTP 500 (context_error), so this reads as one: "HTTP 500 + (streamed): ". It is NOT retried, as that 500 is not: the server + reached a verdict, and a retry pays for the whole generation again + (generate's transport retry would otherwise catch it as a RuntimeError).""" + + +def stream_error(err): + """The StreamError for a streamed {"error": ...} line. The server's own + repeat guard (llm/llama_server.go: more than 100 identical tokens) is + named: under LOOP_DETECT the client's 64-chunk token-run check is meant + to end such a run first, but it counts the chunks that reach the client, + after the output parser, while the server counts the runner's raw + tokens -- so a parser that holds back or drops tokens lets the server's + abort land first, and that is the server's loop verdict, not a fault.""" + msg = f"HTTP 500 (streamed): {err}" + if "token repeat limit" in str(err): + msg += (" -- the server's abort after 100 identical tokens " + "(llm/llama_server.go), before the client's 64-chunk " + "token-run check fired") + return StreamError(msg) + + +def _read_stream(resp, ep, tmo): + """Read a streamed /api/chat or /api/generate response into the shape stream:false returns, feeding a + LoopDetector as it goes. + + A loop is a VERDICT, not an error: the connection is closed (which cancels the generation server-side) and the + partial output comes back as a finished response with done_reason "loop" and failure_mode + "loop detected:", so the suite scores what was produced, as it scores a capped answer. Finished, it is + neither re-run on resume (arm_done) nor escalated (capped_arms): a greedy decode that repeats a block back to + back usually does so again at every rung, and the verdict says the cell was ended there (ADR 0049). The server never sent its final metrics, so eval_count is the number of streamed chunks (one per + token on both runners) and the durations are None. A whole generation that outlasts tmo raises the + blown-budget timeout, as a stream:false read did.""" + det = loop_detect.LoopDetector() + thinking, content, final, chunks = [], [], None, 0 + t0 = time.monotonic() + for raw in resp: + if not raw.strip(): + continue + chunk = json.loads(raw) + if chunk.get("error"): + resp.close() + raise stream_error(chunk["error"]) + if ep == "chat": + msg = chunk.get("message") or {} + th, co = msg.get("thinking") or "", msg.get("content") or "" + else: + th, co = chunk.get("thinking") or "", chunk.get("response") or "" + if th: + thinking.append(th) + if co: + content.append(co) + if chunk.get("done"): + final = chunk + # The final chunk can carry text the server held back -- glimmer's parser does for a reply with no + # message header, so nothing streamed before it. Judge it too; a loop in a reply the server cut at + # num_predict is the same verdict as one caught mid-stream, with the server's own metrics kept. + # A reply that FINISHED is not a loop, however repetitive. + loop = det.feed(thinking=th, content=co) + if loop and final.get("done_reason") == "length": + final["done_reason"] = "loop" + final["failure_mode"] = loop.failure_mode() + final["_loop"] = dict(loop.record(), loop_chunks=chunks + 1, server_done_reason="length") + break + chunks += 1 + loop = det.feed(thinking=th, content=co) + if loop: + resp.close() + final = {"done": True, "done_reason": "loop", "failure_mode": loop.failure_mode(), + "eval_count": chunks, "eval_duration": None, "prompt_eval_count": None, + "prompt_eval_duration": None, "total_duration": None, "load_duration": None, + "_loop": dict(loop.record(), loop_chunks=chunks)} + break + if time.monotonic() - t0 > tmo: + resp.close() + raise TimeoutError() + if final is None: + raise ConnectionError("stream ended without a final chunk") + th, co = "".join(thinking), "".join(content) + if ep == "chat": + final["message"] = dict(final.get("message") or {}, role="assistant", content=co, thinking=th) + else: + final["response"], final["thinking"] = co, th + final["_stream"] = True + final["_loop_detect"] = dict(det.params) + return final + + def generate(host, model, prompt, images, num_predict=None, num_ctx=None, fmt="json", extra_opts=None, endpoint_override=None, think=None, apply_sampling=True, timeout=None, @@ -244,7 +346,8 @@ def generate(host, model, prompt, images, num_predict=None, num_ctx=None, if extra_opts: opts.update(extra_opts) - payload = {"model": model, "stream": False, "options": opts} + stream = loop_detect_on() + payload = {"model": model, "stream": stream, "options": opts} # raw skips the chat template on /api/generate. token_split.py's --server # mode counts prompt_eval_count of bare text, and a template wrapper would # add tokens that belong to the template, not the text. Inert unless set @@ -328,8 +431,11 @@ def generate(host, model, prompt, images, num_predict=None, num_ctx=None, req = urllib.request.Request(url, data=body, headers={"Content-Type": "application/json"}) try: - r = json.load(urllib.request.urlopen(req, timeout=tmo)) + resp = urllib.request.urlopen(req, timeout=tmo) + r = _read_stream(resp, ep, tmo) if stream else json.load(resp) break + except StreamError: + raise # the server's verdict, as an HTTP 500 is: not retried except urllib.error.HTTPError as e: if e.code in (502, 503, 504) and attempt < len(RETRY_BACKOFF): last = f"HTTP {e.code}" @@ -451,6 +557,21 @@ def capture_stamps(r, model, tag, powermode=None, cold_start=None): return st +def loop_fields(r): + """Loop-detection capture for a block, identical for every producer (ADR 0049). A streamed request records that + detection ran and with which parameters, so a block measured with it is never mistaken for one without; a loop + adds its verdict (failure_mode "loop detected:") and the loop's own fields. A stream:false request adds + nothing, so older blocks and LOOP_DETECT-off campaigns keep their shape.""" + f = {} + if r.get("_stream"): + f["stream"] = True + f["loop_detect"] = r.get("_loop_detect") + if r.get("failure_mode"): + f["failure_mode"] = r["failure_mode"] + f.update(r.get("_loop") or {}) + return f + + def metrics_block(r): """Timing + throughput capture, identical for every producer. @@ -460,11 +581,19 @@ def metrics_block(r): publishing different quantities for the same measurement. Durations are always present (None when the server omitted them, matching the suite's historical shape); tok/s are None when underivable. + + A loop verdict (done_reason "loop", ADR 0049) has no decode rate. Cut + mid-stream it has no eval_duration anyway; held back by the server and + stopped at num_predict it does, and eval/eval_duration is then the rate + of a loop, not of a model finishing an answer. gen_tps is None for both, + explicitly, so no table or mean can quote it. Prefill ended before the + loop began, so prefill_tps is kept when the server reported it. """ m = {k: r.get(k) for k in ("total_duration", "load_duration", "prompt_eval_duration", "eval_duration")} m["gen_tps"] = (round(r["eval_count"] / (r["eval_duration"] / 1e9), 2) - if r.get("eval_count") and r.get("eval_duration") else None) + if r.get("eval_count") and r.get("eval_duration") + and r.get("done_reason") != "loop" else None) m["prefill_tps"] = (round(r["prompt_eval_count"] / (r["prompt_eval_duration"] / 1e9), 2) if r.get("prompt_eval_count") diff --git a/docs/maxusai/vision-suite/cmp_scored.py b/docs/maxusai/vision-suite/cmp_scored.py index af0fa59b78a..b591146d078 100644 --- a/docs/maxusai/vision-suite/cmp_scored.py +++ b/docs/maxusai/vision-suite/cmp_scored.py @@ -6,15 +6,15 @@ Counting every scalar field (the run directory's cmp_scores.py) is right for "are two runs identical", but a change that only moves thinking lengths then reads as hundreds of differing cells. This prints only the quality fields, with the side each move favours, for the tests that finished on both sides, then the tests that finished on one side -only. A capped block (was_capped, SPEC H5) and an errored one are not scores. Numeric moves within tol (default -0.005, the IoU band of ADR 0029) are not moves. Labels (declared type, bbox space ...) print when they change and -favour neither side. +only. A capped block (was_capped, SPEC H5), a loop verdict (loop_verdict, ADR 0049) and an errored one are not +scores; a side that looped is named. Numeric moves within tol (default 0.005, the IoU band of ADR 0029) are not moves. +Labels (declared type, bbox space ...) print when they change and favour neither side. """ import os import sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from summarize_engine_compare import load, was_capped # noqa: E402 (H5: import, never redefine) +from summarize_engine_compare import load, loop_verdict, not_a_score # noqa: E402 (H5: import, never redefine) HIGHER = {"bbox_hits", "bbox_mean_iou", "name_bbox_hits", "name_bbox_mean_iou", "hits_anchor", "hits_bestfit", "hits_declared", "iou_anchor", "iou_declared", "iou_at_implied_scale", "labels_found", "items_found", @@ -30,7 +30,7 @@ def scored(block): - return isinstance(block, dict) and not block.get("error") and not was_capped(block) + return isinstance(block, dict) and not block.get("error") and not not_a_score(block) def _num(v): @@ -46,7 +46,9 @@ def compare(a, b, tol=0.005): ba, bb = a.get(name), b.get(name) sa, sb = scored(ba), scored(bb) if sa != sb: - only.append(f"{name}: finished only in {'A' if sa else 'B'}") + other = loop_verdict(bb if sa else ba) if isinstance(bb if sa else ba, dict) else None + only.append(f"{name}: finished only in {'A' if sa else 'B'}" + + (f" ({'B' if sa else 'A'}: {other})" if other else "")) if not (sa and sb): continue moves = [] diff --git a/docs/maxusai/vision-suite/finetext_probe.py b/docs/maxusai/vision-suite/finetext_probe.py index 8e6dcc3b3f6..e063ba3f9f3 100644 --- a/docs/maxusai/vision-suite/finetext_probe.py +++ b/docs/maxusai/vision-suite/finetext_probe.py @@ -38,6 +38,7 @@ def ft_block(r, model, tag, think, num_ctx, num_predict): "eval_count": r.get("eval_count")} if r.get("done_reason"): s["done_reason"] = r["done_reason"] + s.update(client.loop_fields(r)) # ONE metrics derivation, shared with vision_suite via client.metrics_block # — this probe's first hand copy diverged at birth (round 1 vs 2, # absent-key vs None), publishing a different quantity than the suite for diff --git a/docs/maxusai/vision-suite/fixtures/loops/glimmer-mxfp8-bbox-object-repeat.txt b/docs/maxusai/vision-suite/fixtures/loops/glimmer-mxfp8-bbox-object-repeat.txt new file mode 100644 index 00000000000..1f3cee8edab --- /dev/null +++ b/docs/maxusai/vision-suite/fixtures/loops/glimmer-mxfp8-bbox-object-repeat.txt @@ -0,0 +1 @@ +{"x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", "x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", \ No newline at end of file diff --git a/docs/maxusai/vision-suite/fixtures/loops/glimmer-nvfp4-multi3img-degenerate.txt b/docs/maxusai/vision-suite/fixtures/loops/glimmer-nvfp4-multi3img-degenerate.txt new file mode 100644 index 00000000000..a7d5940beb8 --- /dev/null +++ b/docs/maxusai/vision-suite/fixtures/loops/glimmer-nvfp4-multi3img-degenerate.txt @@ -0,0 +1 @@ +Analyze each image independently and then answer cross-image questions. Be exhaustive but never invent content; if something is unreadable, say so via null values rather than guessing. All bounding boxes are ABSOLUTE PIXEL coordinates [x1, y1, x2, y2] in each image's own coordinate system. For each image produce: a " :"}{ " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " , " 1, " :"} 1, " \ No newline at end of file diff --git a/docs/maxusai/vision-suite/loop_detect.py b/docs/maxusai/vision-suite/loop_detect.py new file mode 100644 index 00000000000..f2e311fe7dc --- /dev/null +++ b/docs/maxusai/vision-suite/loop_detect.py @@ -0,0 +1,211 @@ +"""Loop detection for streamed model output: a repeating thinking trace, a repeating answer, or a run of one token. + +Stdlib only, so one file serves three codebases, kept byte-identical: MaxusAI/ollama's vision suite +(docs/maxusai/vision-suite/loop_detect.py, ADR 0049), ai-invoice-to-xero's ollama_toolkit +(packages/ollama_toolkit/loop_detect.py) and maxusai-structured-docs-backend's repair client. + +A greedy (temperature 0) decode that starts repeating a block back to back usually keeps repeating it: the same +context tends to produce the same continuation, and waiting for num_predict, or climbing a context ladder after the +length stop, spends the rest of the budget on it. Usually, not always: a replay of 1,505 finished think-on traces found +3 that repeated a block three times back to back and then finished. LoopDetector watches a stream as it arrives and +calls the loop the moment it meets the rule below, as one of three failure modes: + + loop detected:thinking the thinking trace's last WINDOW chars occur MIN_REPEATS times in it, back to back: the + last WINDOW + (MIN_REPEATS - 1) * period chars are one block repeated (period = distance) + loop detected:content the same, in the answer -- the mode ai-invoice-to-xero's detector could not see + ("Invoice own own own...", extraction-failure-modes.md) + loop detected:token-run one stream emitted the same chunk (whitespace-stripped) TOKEN_RUN times in a row -- the + whitespace runs a JSON grammar allows, which ollama's server aborts at 100 with an HTTP 500 + and no output (llm/llama_server.go) + +Why three copies, not two: a single earlier occurrence of 400 characters is how a model restates a table or quotes a +passage. ai-invoice-to-xero's thinking_is_looping fires on it; this detector does not. Its two functions are kept +below, unchanged in meaning, for callers that use them. Why back to back: three copies with other text between them +are how a trace restates its draft answer after each check, and counting copies alone called 17-18 of those 1,505 +finished traces loops. + +Cost: the block check runs every CHECK_EVERY new characters and searches only the last LOOKBACK characters, so a stream +of n characters costs O(n / CHECK_EVERY * LOOKBACK) in C-level string search -- linear in n, unlike re-scanning the +whole buffer on every chunk. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +THINKING = "thinking" +CONTENT = "content" +TOKEN_RUN = "token-run" + +LOOP_CHECK_WINDOW = 400 # chars of tail compared against the earlier stream (ai-invoice-to-xero's value) +LOOP_MIN_LENGTH = 800 # a stream shorter than this is never judged (ai-invoice-to-xero's value) +MIN_REPEATS = 3 # the tail window must occur this many times, back to back: itself plus two earlier copies +TOKEN_RUN_LEN = 64 # identical chunks in a row; below the server's 100, so the client keeps the partial output +CHECK_EVERY = 64 # chars between block checks +LOOKBACK = 16384 # chars searched; a period over (LOOKBACK - WINDOW) / (MIN_REPEATS - 1) is not seen + + +@dataclass +class Loop: + kind: str # THINKING, CONTENT or TOKEN_RUN + stream: str # THINKING or CONTENT: which stream looped (a token-run names its stream here) + at_chars: int # length of that stream when the loop was called + period: int # chars per repetition (block loops) or 0 (token-run) + repeats: int # copies seen (block) or run length (token-run) + snippet: str # one period of the loop (or the repeated chunk), truncated for logs + params: dict[str, int] = field(default_factory=dict) + + def failure_mode(self) -> str: + return f"loop detected:{self.kind}" + + def record(self) -> dict[str, Any]: + """Flat fields for a result record: everything a reader needs to see the loop without the stream.""" + return { + "loop": self.failure_mode(), + "loop_kind": self.kind, + "loop_stream": self.stream, + "loop_at_chars": self.at_chars, + "loop_period": self.period, + "loop_repeats": self.repeats, + "loop_snippet": self.snippet[:160], + "loop_params": dict(self.params), + } + + def __str__(self) -> str: + what = ( + f"period {self.period} chars x{self.repeats}" + if self.kind != TOKEN_RUN + else f"{self.repeats} identical chunks" + ) + return f"{self.failure_mode()} ({what}, at {self.at_chars} chars of {self.stream}): {self.snippet[:80]!r}" + + +class _Stream: + # buf holds only the last `keep` chars (the search region); total counts every char seen. Deltas wait in + # `pending` and are joined at each check: appending to a str attribute copies the whole buffer per chunk. + __slots__ = ("buf", "checked", "last_tok", "name", "pending", "run", "total") + + def __init__(self, name: str) -> None: + self.name, self.buf, self.total = name, "", 0 + self.pending: list[str] = [] + self.checked, self.run = 0, 0 + self.last_tok: str | None = None + + def text(self, keep: int) -> str: + if self.pending: + self.buf = (self.buf + "".join(self.pending))[-keep:] + self.pending = [] + return self.buf + + +class LoopDetector: + """Feed streamed deltas; feed() returns the first Loop found, then the same Loop on every later call.""" + + def __init__( + self, + window: int = LOOP_CHECK_WINDOW, + min_length: int = LOOP_MIN_LENGTH, + min_repeats: int = MIN_REPEATS, + token_run: int = TOKEN_RUN_LEN, + check_every: int = CHECK_EVERY, + lookback: int = LOOKBACK, + ) -> None: + if min_length < window or min_repeats < 2 or token_run < 2 or lookback < window * min_repeats: + raise ValueError( + "loop_detect: need min_length >= window, min_repeats >= 2, token_run >= 2 and " + "lookback >= window * min_repeats" + ) + self.window, self.min_length, self.min_repeats = window, min_length, min_repeats + self.token_run, self.check_every, self.lookback = token_run, check_every, lookback + self.params = { + "window": window, + "min_length": min_length, + "min_repeats": min_repeats, + "token_run": token_run, + "lookback": lookback, + } + self._streams = {THINKING: _Stream(THINKING), CONTENT: _Stream(CONTENT)} + self.loop: Loop | None = None + + def feed(self, thinking: str = "", content: str = "") -> Loop | None: + if self.loop is None and thinking: + self.loop = self._add(self._streams[THINKING], thinking) + if self.loop is None and content: + self.loop = self._add(self._streams[CONTENT], content) + return self.loop + + def _add(self, s: _Stream, delta: str) -> Loop | None: + tok = delta.strip() + if tok == s.last_tok: + s.run += 1 + else: + s.last_tok, s.run = tok, 1 + s.pending.append(delta) + s.total += len(delta) + if s.run >= self.token_run: + return Loop(TOKEN_RUN, s.name, s.total, 0, s.run, delta, self.params) + if s.total < self.min_length or s.total - s.checked < self.check_every: + return None + s.checked = s.total + return self._block(s) + + def _block(self, s: _Stream) -> Loop | None: + buf = s.text(self.lookback) + n, w = len(buf), self.window + tail = buf[n - w :] + lo = 0 + # Earlier copies, nearest first. rfind's end bound keeps each copy strictly before the previous one's end, + # so overlapping copies of a short period are all counted. + end, copies, nearest = n - 1, 1, None + while copies < self.min_repeats: + pos = buf.rfind(tail, lo, end) + if pos < 0: + return None + if nearest is None: + nearest = pos + copies += 1 + end = pos + w - 1 + assert nearest is not None # min_repeats >= 2, so the loop above found at least one copy + period = (n - w) - nearest + # Copies are not enough: a trace that restates its draft between rounds of checking has copies with other + # text between them. A loop's copies are back to back, so the tail is periodic: the last window plus + # (min_repeats - 1) periods is one block repeated. + span = w + (self.min_repeats - 1) * period + if span > n or buf[n - span : n - period] != buf[n - span + period :]: + return None + return Loop( + s.name, s.name, s.total, period, copies, buf[n - period :] if period <= w else buf[n - w :], self.params + ) + + +# ---- ai-invoice-to-xero's ollama_toolkit API, unchanged in meaning -------------------------------------------------- + + +def thinking_is_looping(thinking: str) -> bool: + """True if the last LOOP_CHECK_WINDOW chars of the trace occur verbatim earlier in it (once is enough). + + ai-invoice-to-xero's original predicate, kept for its callers. A one-copy repeat is also how a model restates a + table, so new code should use LoopDetector, which needs MIN_REPEATS copies back to back and also watches content + and token runs. + """ + if len(thinking) < LOOP_MIN_LENGTH: + return False + tail = thinking[-LOOP_CHECK_WINDOW:] + return tail in thinking[:-LOOP_CHECK_WINDOW] + + +def dedup_thinking(thinking: str) -> str: + """Trim a looping trace to its first unique pass: cut where a LOOP_CHECK_WINDOW-wide segment first recurs. + + Same result as ai-invoice-to-xero's original (which sliced the prefix for every index, O(n^2)); this keeps every + complete earlier window in a hash table, so the scan is linear. + """ + w = LOOP_CHECK_WINDOW + seen: dict[str, int] = {} + for i in range(w, len(thinking) - w + 1): + j = i - w + seen.setdefault(thinking[j : j + w], j) # windows wholly inside thinking[:i] + if thinking[i : i + w] in seen: + return thinking[:i].rstrip() + return thinking.rstrip() diff --git a/docs/maxusai/vision-suite/preflight/checks.py b/docs/maxusai/vision-suite/preflight/checks.py index 05a1d832376..e2376edb648 100644 --- a/docs/maxusai/vision-suite/preflight/checks.py +++ b/docs/maxusai/vision-suite/preflight/checks.py @@ -1380,16 +1380,21 @@ def quality_eligible(scores, tests=None): import lives here rather than at module top because release lineages carry preflight/ without the suite — and on those lineages check_quality has already SKIPped before scoring. + + A block ended on a loop verdict (LOOP_DETECT=1, ADR 0049) is neither: it + was scored on output cut at the loop. quality_looped names it. """ if SUITE_DIR not in sys.path: sys.path.insert(0, SUITE_DIR) - from summarize_engine_compare import was_capped + from summarize_engine_compare import loop_verdict, was_capped eligible, capped = {}, [] for name, blk in scores.items(): if tests is not None and name not in tests: continue if not isinstance(blk, dict) or "error" in blk: continue + if loop_verdict(blk): + continue if was_capped(blk): capped.append(name) else: @@ -1397,6 +1402,20 @@ def quality_eligible(scores, tests=None): return eligible, capped +def quality_looped(scores, tests=None): + """This run's arms that ended on a loop verdict, as " (loop:)". + + LOOP_DETECT reaches vision_suite through the inherited environment, so a + preflight can record one. It is a finding about the model, not a score: + reported as a failure by name, outside every quality ratio.""" + if SUITE_DIR not in sys.path: + sys.path.insert(0, SUITE_DIR) + from summarize_engine_compare import loop_verdict + return [f"{name} ({loop_verdict(blk)})" for name, blk in scores.items() + if (tests is None or name in tests) and isinstance(blk, dict) + and "error" not in blk and loop_verdict(blk)] + + def check_quality(host, quality, expect, arch, tag, timeout=5400): """Runs vision_suite.py and applies thresholds to the scores it already computes. The suite reports; this turns the report into a verdict.""" @@ -1465,6 +1484,9 @@ def check_quality(host, quality, expect, arch, tag, timeout=5400): # Surfaced so a run with capped arms cannot read as fully assessed. failures.append(f"arms capped, not scored: {', '.join(capped_arms_)} " f"(raise NUM_PREDICT / the ladder)") + looped = quality_looped(scores, tests=tests) + if looped: + failures.append(f"arms ended on a loop verdict, not scored: {', '.join(looped)}") if valid: metrics["json_valid"] = sum(valid) / len(valid) if metrics["json_valid"] < quality.get("min_json_valid", 1.0): diff --git a/docs/maxusai/vision-suite/preflight/test_verdicts.py b/docs/maxusai/vision-suite/preflight/test_verdicts.py index b708d0f2310..019a4da2335 100644 --- a/docs/maxusai/vision-suite/preflight/test_verdicts.py +++ b/docs/maxusai/vision-suite/preflight/test_verdicts.py @@ -986,6 +986,21 @@ def test_errored_blocks_are_neither_eligible_nor_capped(self): self.assertEqual(eligible, {}) self.assertEqual(capped, []) + def test_a_loop_verdict_leaves_the_denominator_and_is_named(self): + # LOOP_DETECT=1 reaches vision_suite through the inherited environment. A cell ended on a loop (ADR 0049) + # was scored on output cut at the loop -- held back by the server it even sits at eval == num_predict -- so + # it is neither a quality score nor capped: it is reported as the loop it is. + scores = {"scene_single": {"json_valid": False, "eval_count": 2200, "num_predict": 2200, + "done_reason": "loop", "failure_mode": "loop detected:content", + "loop_kind": "content"}, + "document_single": {"json_valid": True, "eval_count": 100, "num_predict": 2200, + "done_reason": "stop"}} + eligible, capped = checks.quality_eligible(scores) + self.assertEqual(list(eligible), ["document_single"]) + self.assertEqual(capped, []) + self.assertEqual(checks.quality_looped(scores), ["scene_single (loop:content)"]) + self.assertEqual(checks.quality_looped(scores, tests=["document_single"]), []) + diff --git a/docs/maxusai/vision-suite/summarize_contract_matrix.py b/docs/maxusai/vision-suite/summarize_contract_matrix.py index 23b24ec7608..d40a5416798 100755 --- a/docs/maxusai/vision-suite/summarize_contract_matrix.py +++ b/docs/maxusai/vision-suite/summarize_contract_matrix.py @@ -31,8 +31,8 @@ import re import sys -from summarize_engine_compare import (ctx_for, engine_for, load, resolve_tag, - was_capped) +from summarize_engine_compare import (ctx_for, engine_for, load, loop_verdict, + resolve_tag, was_capped) DIR = os.path.dirname(os.path.abspath(__file__)) ARMS = ["bbox_contract", "bbox_contract_multi", "bbox_contract_reasoning", @@ -76,6 +76,7 @@ def main(): print(f"## Contract matrix (`contract_followed`), think={think}\n") print("| Model | Engine | " + " | ".join(SHORT[a] for a in ARMS) + " | num_ctx |") print("|---" * (len(ARMS) + 3) + "|") + looped = False for model in models: tag = resolve_tag(DIR, model, think, prefix) d = load(os.path.join(DIR, f"scores_{tag}.json")) @@ -93,6 +94,12 @@ def main(): # contract to judge. It rendered ❌ for the Sep-4 cudaMallocAsync # aborts and read as two contract failures on 31b (2026-09-06). cells.append("error") + elif loop_verdict(sec): + # ADR 0049: the cell ended on a verbatim loop. Its answer was + # cut there, so contract_followed describes the cut, not the + # model. + cells.append(loop_verdict(sec)) + looped = True elif was_capped(sec): # ADR 0012 rule 8: a capped cell has no score to report. Say so # rather than rendering a False that reads as a measured failure. @@ -107,6 +114,10 @@ def main(): "`cap` = generation stopped at the `num_predict` cap rather than " "finishing, so the cell carries no score (ADR 0012 rule 8). The cap is " "a separate limit from the `num_ctx` window.") + if looped: + # Only when a cell looped, so a table without loops keeps its shape. + print("`loop:` = the cell ended on a verbatim loop (LOOP_DETECT=1, " + "ADR 0049), so its answer was cut there and carries no score.") if logpath: pm = powermodes(logpath) diff --git a/docs/maxusai/vision-suite/summarize_engine_compare.py b/docs/maxusai/vision-suite/summarize_engine_compare.py index ff2402ae1fe..e6830300139 100644 --- a/docs/maxusai/vision-suite/summarize_engine_compare.py +++ b/docs/maxusai/vision-suite/summarize_engine_compare.py @@ -169,12 +169,18 @@ def was_capped(sec): eval_count >= num_predict fallback — as does a connection-closed final, which serializes with the field omitted (llm/server.go maps DoneReasonConnectionClosed to "" and the API field is omitempty). + + done_reason "loop" is the client's loop verdict (ADR 0049) and is never + capped, even at eval == num_predict: a loop the server held back and + stopped at the cap keeps that eval_count, and the fallback re-ran it on + resume and climbed the ladder for it. A loop is a finished cell whose + result is the verdict, not a score. """ sec = sec or {} dr = sec.get("done_reason") if dr == "length": return True - if dr == "stop": + if dr in ("stop", "loop"): return False # req_num_predict is the fallback cap spelling: legacy finetext blocks # recorded the suite default in num_predict while req_num_predict held @@ -185,6 +191,38 @@ def was_capped(sec): return bool(cap and ev and ev >= cap) +def loop_verdict(sec): + """The block's loop verdict as a table cell renders it, or None. + + LOOP_DETECT=1 ends a cell on a verbatim loop (ADR 0049): done_reason + "loop", failure_mode "loop detected:". The cell is finished + (arm_done; was_capped is False) but it holds no score: the client + scored the partial output, as it scores a capped answer, and that score + measures where the loop was cut, not the model. So every template + renders "loop:" where the score would go, and no mean pools it -- + the capped discipline (ADR 0012 conv 9) applied to the third state. + Without this a looped multi_3img cell rendered "❌ q1_right, ..." and + entered every mean as a wrong answer (#469 review). + + None for every block without the verdict, including one streamed with + detection on that did not loop, so no older table changes shape. + """ + sec = sec or {} + fm = sec.get("failure_mode") + fm = fm if isinstance(fm, str) and fm.startswith("loop detected:") else None + if sec.get("done_reason") != "loop" and fm is None: + return None + kind = sec.get("loop_kind") or (fm.split(":", 1)[1] if fm else "") + return f"loop:{kind}" if kind else "loop" + + +def not_a_score(sec): + """True when a block's scores may not be rendered or pooled: capped + (was_capped) or ended on a loop verdict (loop_verdict). The one test + every mean applies (SPEC H5).""" + return was_capped(sec) or loop_verdict(sec) is not None + + def save(path, data, end=""): """Atomic JSON write: tmp + os.replace, the probes.py idiom. The scores file is the campaign's most expensive artifact; a truncate-then-dump @@ -589,7 +627,7 @@ def q(block, text): # scores and must not read as ❌ either: it is "error". if "error" in block: return "error" - return "capped" if was_capped(block) else text + return loop_verdict(block) or ("capped" if was_capped(block) else text) iou = sc.get("bbox_mean_iou") iou_cell = "—" if iou is None else (f"**{iou:.3f}**" if eng == "MLX" else f"{iou:.3f}") @@ -618,6 +656,8 @@ def multi_cell(block): # as a clean sweep. Name the three that are actually gated. if "error" in block: return "error" + if loop_verdict(block): + return loop_verdict(block) if was_capped(block): return "capped" if block.get("q1_right") and block.get("q2_right") and block.get("q4_bbox_hit"): @@ -651,7 +691,11 @@ def multi_cell(block): # char-proportional estimate (a 62-token response measured 21% control # tokens, so proportions lie). Think-off rows show a true 0. tt = sc.get("thinking_tokens") - if was_capped(sc): + if loop_verdict(sc): + # Ended on a loop (ADR 0049): eval_count is where the loop was + # cut, so no length or latency derived from it is the model's. + tok_cell = think_cell = s_cell = rh_cell = loop_verdict(sc) + elif was_capped(sc): # eval_count is the cap here, not a generated length; both it and # anything derived from it are meaningless as model results. tok_cell = f"≥{ev} ⚠" diff --git a/docs/maxusai/vision-suite/summarize_geometry.py b/docs/maxusai/vision-suite/summarize_geometry.py index 72a516ba44b..4454fe28ee9 100644 --- a/docs/maxusai/vision-suite/summarize_geometry.py +++ b/docs/maxusai/vision-suite/summarize_geometry.py @@ -27,15 +27,25 @@ import sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from summarize_engine_compare import was_capped # noqa: E402 (SPEC H5) +from summarize_engine_compare import loop_verdict, not_a_score, was_capped # noqa: E402 (SPEC H5) def pooled(got): """Blocks eligible for a pooled mean. A capped cell is an unfinished measurement (ADR 0012 conv 9): its eval_count is the harness cap and any req/h or IoU derived from it is a setting, not a result. This file - published '**N req/h** mean serial' with no capped guard at all.""" - return [s for s in got if s and not was_capped(s)] + published '**N req/h** mean serial' with no capped guard at all. A cell + ended on a loop verdict (ADR 0049) is excluded the same way: its counts + were read from output cut at the loop.""" + return [s for s in got if s and not not_a_score(s)] + + +def why_excluded(raw): + """"capped", or "capped or looped (k looped)" when any cell looped -- the + loop count is named only when there is one, so a table without loops + keeps its exact wording.""" + n_loop = sum(1 for s in raw if loop_verdict(s)) + return f"capped or looped ({n_loop} looped)" if n_loop else "capped" GEOMS = ["hd", "hd_al32", "hd_al48", "sq320", "vga", "portrait", "uhd", "uhd4k", "paste1", "paste2", "paste3", "paste4", "paste5", "paste6"] @@ -64,6 +74,8 @@ def cell(s): return " — | — | — " # SPEC H13: a capped cell renders `capped` in every template — its # anchor/bestfit counts describe a truncated generation, not the model. + if loop_verdict(s): + return " | ".join([loop_verdict(s)] * 3) if was_capped(s): return "capped | capped | capped" sz, ref = s.get("image_size"), s.get("anchor_implied_ref") @@ -110,9 +122,10 @@ def perf_table(rows, models): # included — guarding only the pooled bullet below reproduced # the named T1 defect (a capped cell published as a score while # the mean beneath it silently excluded the same row). - if was_capped(s): + if not_a_score(s): + state = loop_verdict(s) or "capped" line += (f" {s.get('prompt_eval_count','—')} | " - f"{s.get('eval_count','—')} | capped | capped |") + f"{s.get('eval_count','—')} | {state} | {state} |") continue iou = s.get("iou_anchor") line += (f" {s.get('prompt_eval_count','—')} | {s.get('eval_count','—')} | " @@ -129,7 +142,7 @@ def perf_table(rows, models): if raw and not got: # ADR 0012 conv 3: a vanished summary line is indistinguishable # from "not run". All-capped is a finding; say it. - print(f"- **{k}** — all {len(raw)} geometries capped; no pooled " + print(f"- **{k}** — all {len(raw)} geometries {why_excluded(raw)}; no pooled " f"throughput (ADR 0012 conv 9)") continue pe = [s["prompt_eval_count"] for s in got if s.get("prompt_eval_count")] @@ -158,7 +171,7 @@ def perf_table(rows, models): # ADR 0012 conv 9: the count of exclusions is a first-class # diagnostic — a 14-geometry mean over 8 must say so. if n_capped: - seg.append(f"{n_capped} of {len(raw)} geometries capped, excluded") + seg.append(f"{n_capped} of {len(raw)} geometries {why_excluded(raw)}, excluded") print(f"- **{k}** ({len(got)} geometries) — " + ", ".join(seg)) @@ -200,7 +213,7 @@ def main(): raw = [s for s in raw if s] got = pooled(raw) if len(raw) != len(got): - print(f"- **{k}** — {len(raw) - len(got)} capped geometries " + print(f"- **{k}** — {len(raw) - len(got)} {why_excluded(raw)} geometries " f"excluded from the counts below (ADR 0012 conv 9)") silent = sum(1 for s in got if s.get("self_check") and isinstance(s.get("hits_anchor"), int) diff --git a/docs/maxusai/vision-suite/summarize_head_to_head.py b/docs/maxusai/vision-suite/summarize_head_to_head.py index 5737e13aab7..bfc10787b1f 100644 --- a/docs/maxusai/vision-suite/summarize_head_to_head.py +++ b/docs/maxusai/vision-suite/summarize_head_to_head.py @@ -21,7 +21,7 @@ import sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from summarize_engine_compare import was_capped # noqa: E402 (H5: import, never redefine) +from summarize_engine_compare import loop_verdict, was_capped # noqa: E402 (H5: import, never redefine) def load(path): @@ -64,12 +64,23 @@ def cap_or(block, text): scores; rendering it through the score path printed ❌ ❌ ❌ —/— (?) for a 2026-09-04 cudaMallocAsync abort — "cannot ground" for a cell that never answered. Same rule: name the state, never a score. + + A cell ended on a loop verdict (ADR 0049) renders "loop:" with its + window: its scores were read from output cut at the loop. """ if "error" in block: return "error" + if loop_verdict(block): + return f"{loop_verdict(block)}{ctx(block)}" return f"capped{ctx(block)}" if was_capped(block) else text +def latency(sc, text): + """s/req and req/h derive from the scene's eval_count, which is the cap + for a capped cell and the cut point for a looped one: name the state.""" + return loop_verdict(sc) or ("capped" if was_capped(sc) else text) + + def main(): args = sys.argv[1:] rundir = os.path.dirname(os.path.abspath(__file__)) @@ -140,8 +151,8 @@ def main(): ("throughput", "prefill tok/s"): f"{pre:.0f}" if pre else "—", # A capped scene cell's eval_count IS the cap: s/req and req/h # derived from it quote the harness setting as model latency. - ("latency", "s/req (unique image)"): "capped" if was_capped(sc) else (f"{s_req:.1f}" if s_req else "—"), - ("latency", "req/h (serial)"): "capped" if was_capped(sc) else (f"{3600 / s_req:.0f}" if s_req else "—"), + ("latency", "s/req (unique image)"): latency(sc, f"{s_req:.1f}" if s_req else "—"), + ("latency", "req/h (serial)"): latency(sc, f"{3600 / s_req:.0f}" if s_req else "—"), }) rows = list(cells[0].keys()) diff --git a/docs/maxusai/vision-suite/summarize_matrix.py b/docs/maxusai/vision-suite/summarize_matrix.py index 20cf1144b94..79186b6331c 100644 --- a/docs/maxusai/vision-suite/summarize_matrix.py +++ b/docs/maxusai/vision-suite/summarize_matrix.py @@ -26,6 +26,11 @@ factor (the anchor bounded qwen3.6's multi_3img runaway from >122,880 tokens to 10,910 — the capped column is where that effect would show here). +**A loop verdict is excluded the same way and counted on its own.** A cell +that LOOP_DETECT=1 ended on a verbatim loop (ADR 0049) was scored on output +cut at the loop. It enters no mean, and the capped column adds "(+k loop)" +for it -- only when there is one, so a table without loops keeps its shape. + **A partial campaign renders as INCOMPLETE, with the missing cells named.** ADR 0012 rule 8: re-render after a campaign fully completes. The expected grid is the cross-product of every model, mode and geometry seen on disk × 8 arms, @@ -51,7 +56,7 @@ import sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from summarize_engine_compare import was_capped # noqa: E402 (SPEC H5) +from summarize_engine_compare import loop_verdict, not_a_score, was_capped # noqa: E402 (SPEC H5) FACTORS = ("pin", "anchor", "coords") ARM_RE = re.compile(r"^bboxm_(pin|free)_(anc|noanc)_(named|pos)$") @@ -79,7 +84,10 @@ def factors_of(arm): def load(rundir, prefix): - """[(model, mode, geometry, arm, factors, block, capped)]""" + """[(model, mode, geometry, arm, factors, block, excluded)] + + `excluded` is not_a_score: capped OR ended on a loop verdict. Every mean + below reads it; the counts tell the two apart with loop_verdict.""" out = [] for f in sorted(glob.glob(os.path.join(rundir, f"scores_{prefix}-*.json"))): m = re.match(rf"scores_{re.escape(prefix)}-(.+?)-\d+_(.+)_think(\w+)\.json", @@ -94,7 +102,7 @@ def load(rundir, prefix): for arm, blk in data.items(): fac = factors_of(arm) if fac and isinstance(blk, dict) and "error" not in blk: - out.append((model, mode, geom, arm, fac, blk, capped(blk))) + out.append((model, mode, geom, arm, fac, blk, not_a_score(blk))) return out @@ -119,11 +127,13 @@ def main(): models = sorted({r[0] for r in rows}) modes = sorted({r[1] for r in rows}) geoms = sorted({r[2] for r in rows}) - n_cap = sum(1 for r in rows if r[6]) + n_loop = sum(1 for r in rows if loop_verdict(r[5])) + n_cap = sum(1 for r in rows if r[6]) - n_loop n_salv = sum(1 for r in rows if r[5].get("json_salvaged")) print(f"\n**{a.metric}** — {len(rows)} arm-results, " f"{len(models)} models × {len(modes)} think modes × {len(geoms)} geometries") - print(f"capped (excluded from every mean): {n_cap} · json salvaged: {n_salv}\n") + print(f"capped (excluded from every mean): {n_cap} · json salvaged: {n_salv}" + + (f" · loop verdicts (excluded from every mean): {n_loop}" if n_loop else "") + "\n") # ---- completeness: a partial campaign must say so (ADR 0012 rule 8) ----- present = {(r[0], r[1], r[2], r[3]) for r in rows} @@ -176,7 +186,9 @@ def main(): means, spreads, caps, worst = [], [], [], [] for lv in levels: lvc = [r for r in cells if r[4][factor] == lv] - caps.append(f"{sum(1 for r in lvc if r[6])}/{len(lvc)}") + n_lp = sum(1 for r in lvc if loop_verdict(r[5])) + caps.append(f"{sum(1 for r in lvc if r[6]) - n_lp}/{len(lvc)}" + + (f" (+{n_lp} loop)" if n_lp else "")) vs = [val(b, a.metric) for (_, _, _, _, _, b, c) in lvc if not c and val(b, a.metric) is not None] means.append(statistics.mean(vs) if vs else None) @@ -236,7 +248,7 @@ def main(): for arm in ARMS: ac = [r for r in rows if r[0] == mdl and r[1] == mode and r[3] == arm] if not ac or any(r[6] for r in ac): - continue # capped anywhere → not trustable in this mode + continue # capped or looped anywhere → not trustable in this mode vs = [val(b, a.metric) for (_, _, _, _, _, b, _) in ac if val(b, a.metric) is not None] if not vs: @@ -250,8 +262,11 @@ def main(): cand.append((statistics.mean(vs), sp, arm, len(vs))) if not cand: if any(r[0] == mdl and r[1] == mode for r in rows): + how = ("hit the cap or looped" if any( + loop_verdict(r[5]) for r in rows if r[0] == mdl and r[1] == mode) + else "hit the cap") print(f"| {mdl.replace('_','.')[:22]} | {mode} | — | — | — | " - "every arm hit the cap at least once |") + f"every arm {how} at least once |") continue cand.sort(key=lambda t: (-t[0], t[1])) mean, sp, arm, n = cand[0] diff --git a/docs/maxusai/vision-suite/summarize_reps.py b/docs/maxusai/vision-suite/summarize_reps.py index a4eb5eb48c4..1eacbbb1e25 100755 --- a/docs/maxusai/vision-suite/summarize_reps.py +++ b/docs/maxusai/vision-suite/summarize_reps.py @@ -22,7 +22,7 @@ import sys sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -from summarize_engine_compare import was_capped # noqa: E402 (SPEC H5) +from summarize_engine_compare import loop_verdict, was_capped # noqa: E402 (SPEC H5) DIR = os.path.dirname(os.path.abspath(__file__)) @@ -189,16 +189,23 @@ def cell(runs, section, key, kind): # pooling level" clause exactly. A partial cell returns spread=None so # the within-arm spread section cannot claim reproducibility over reps # that contributed nothing. + # + # A rep that ended on a loop verdict (ADR 0049) is dropped the same way + # and counted on its own ("(k loop)"): its score was read from output cut + # at the loop, and "cannot stop" and "loops" are different findings. secs = [r[section] for r in runs if section in r and key in r.get(section, {})] - kept = [s for s in secs if not was_capped(s)] + looped = [s for s in secs if loop_verdict(s)] + capped = [s for s in secs if not loop_verdict(s) and was_capped(s)] + kept = [s for s in secs if not loop_verdict(s) and not was_capped(s)] dropped = len(secs) - len(kept) vals = [s[key] for s in kept] if secs and not vals: - return "capped", None + return " + ".join(w for w, n in (("capped", capped), ("loop", looped)) if n), None if not vals: return "—", None - suffix = f" ({dropped} capped)" if dropped else "" + why = [f"{len(n)} {w}" for w, n in (("capped", capped), ("loop", looped)) if n] + suffix = f" ({', '.join(why)})" if dropped else "" if kind == "bool": hits = sum(1 for v in vals if v) s = (f"{hits}/{len(vals)} ✅" if hits == len(vals) @@ -277,12 +284,13 @@ def main(): # the IoU spread then reaches for it by hand, which is how the wrong # 0.103 reached the campaign record's prose. worst = {"float": [], "int": []} - any_capped = False + any_capped = any_looped = False for section, key, label, kind in METRICS: if kind == "bool": continue cs, sp = cell(runs, section, key, kind) any_capped = any_capped or "capped" in cs + any_looped = any_looped or "loop" in cs if sp: worst[kind].append((sp, label)) parts = [] @@ -294,12 +302,15 @@ def main(): + ", ".join(f"{l} {s:g}" for s, l in worst[kind][:4])) if parts: print(f" {tag}: " + "; ".join(parts)) - elif any_capped: + elif any_capped and not any_looped: # "identical across all N runs" over survivors was a fabricated # reproducibility claim — the exact class the ± comment above # documents. Capped reps contributed nothing; say so instead. print(f" {tag}: no spread measurable — capped reps excluded " f"(see '(k capped)' cells)") + elif any_looped: + print(f" {tag}: no spread measurable — capped or looped reps " + f"excluded (see '(k capped)' / '(k loop)' cells)") else: print(f" {tag}: identical across all {len(runs)} runs") return 0 diff --git a/docs/maxusai/vision-suite/test_client.py b/docs/maxusai/vision-suite/test_client.py index f1a64c23e16..1987f039875 100644 --- a/docs/maxusai/vision-suite/test_client.py +++ b/docs/maxusai/vision-suite/test_client.py @@ -463,5 +463,246 @@ def test_retried_cell_records_count(self): self.assertEqual(st["retries"], 1) + + +class StreamCapture: + """Patch urlopen with a streaming server: one JSON line per chunk, then a final done chunk.""" + + def __init__(self, chunks, final=None, endpoint="chat"): + self.lines = [] + for th, co in chunks: + if endpoint == "chat": + self.lines.append({"message": {"role": "assistant", "content": co, "thinking": th}, "done": False}) + else: + self.lines.append({"response": co, "thinking": th, "done": False}) + last = {"done": True, "done_reason": "stop", "eval_count": len(chunks), "prompt_eval_count": 7} + last.update(final or {}) + if endpoint == "chat": + last["message"] = {"role": "assistant", "content": ""} + else: + last["response"] = "" + self.lines.append(last) + self.payload, self.closed, self.read_lines = None, False, 0 + + def __enter__(self): + self._orig = urllib.request.urlopen + cap = self + + class Resp: + def __iter__(self_inner): + for obj in cap.lines: + if cap.closed: + return + cap.read_lines += 1 + yield (json.dumps(obj) + "\n").encode() + + def close(self_inner): + cap.closed = True + + def fake(req, timeout=None): + cap.payload = json.loads(req.data) + return Resp() + urllib.request.urlopen = fake + return self + + def __exit__(self, *a): + urllib.request.urlopen = self._orig + + +class TestLoopDetect(unittest.TestCase): + """LOOP_DETECT=1 streams the request and cancels a verbatim loop (loop_detect.py, ADR 0049).""" + + def setUp(self): + self._env = os.environ.get("LOOP_DETECT") + os.environ["LOOP_DETECT"] = "1" + + def tearDown(self): + if self._env is None: + os.environ.pop("LOOP_DETECT", None) + else: + os.environ["LOOP_DETECT"] = self._env + + def test_off_by_default_sends_no_stream(self): + os.environ.pop("LOOP_DETECT", None) + self.assertFalse(call().payload["stream"]) + + def test_a_clean_stream_returns_the_same_shape(self): + chunks = [("thinking ", ""), ("done.", ""), ("", '{"a"'), ("", ": 1}")] + with StreamCapture(chunks) as cap: + r = client.generate("http://h", "M", "p", ["IMG"], think=True) + self.assertTrue(cap.payload["stream"]) + self.assertEqual(r["response"], '{"a": 1}') + self.assertEqual(r["thinking"], "thinking done.") + self.assertEqual(r["done_reason"], "stop") + self.assertEqual(r["eval_count"], 4) + self.assertTrue(r["_stream"]) + self.assertEqual(r["_loop_detect"]["min_repeats"], 3) + + def test_generate_endpoint_streams_too(self): + with StreamCapture([("", "hel"), ("", "lo")], endpoint="generate") as cap: + r = client.generate("http://h", "M", "p", ["IMG"], endpoint_override="generate") + self.assertTrue(cap.payload["stream"]) + self.assertEqual(r["response"], "hello") + + def test_a_content_loop_is_cancelled_and_returned_as_a_verdict(self): + obj = '{"x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", ' + chunks = [("", '{"boxes": [')] + [("", obj[i:i + 5]) for _ in range(40) for i in range(0, len(obj), 5)] + with StreamCapture(chunks) as cap: + r = client.generate("http://h", "M", "p", ["IMG"]) + self.assertTrue(cap.closed, "the stream must be closed so the server stops generating") + self.assertLess(cap.read_lines, len(cap.lines), "it must stop reading at the loop, not drain the stream") + self.assertEqual(r["done_reason"], "loop") + self.assertEqual(r["failure_mode"], "loop detected:content") + self.assertEqual(r["_loop"]["loop_period"], len(obj)) + self.assertTrue(r["response"].startswith('{"boxes": [{"x1": 70'), "the partial answer is kept for scoring") + self.assertIsNone(r["eval_duration"]) + + def test_a_token_run_is_cancelled(self): + with StreamCapture([("", '{"a": ')] + [("", " ")] * 200) as cap: + r = client.generate("http://h", "M", "p", ["IMG"]) + self.assertEqual(r["failure_mode"], "loop detected:token-run") + self.assertTrue(cap.closed) + + def test_a_thinking_loop_keeps_the_trace(self): + block = "Let me reconsider the coordinates of the second box once more carefully. " * 7 + with StreamCapture([(block[i:i + 9], "") for _ in range(4) for i in range(0, len(block), 9)]): + r = client.generate("http://h", "M", "p", ["IMG"], think=True) + self.assertEqual(r["failure_mode"], "loop detected:thinking") + self.assertEqual(r["response"], "") + self.assertTrue(r["thinking"].startswith("Let me reconsider")) + + def test_a_loop_is_finished_not_capped(self): + # A loop is a verdict, not a cap: it must neither climb the context ladder (capped_arms) nor be re-run on + # resume (arm_done / ft_done): the cell was ended at the loop, and the verdict is its result (ADR 0049). + import summarize_engine_compare as sec + import vision_suite + blk = {"done_reason": "loop", "failure_mode": "loop detected:content", "eval_count": 312, + "num_predict": 2200, "loop": "loop detected:content"} + self.assertFalse(sec.was_capped(blk)) + self.assertTrue(vision_suite.arm_done(blk)) + import tempfile + with tempfile.NamedTemporaryFile("w", suffix=".json", delete=False) as fh: + json.dump({"case_a": blk, "case_b": {"done_reason": "length", "eval_count": 2200, "num_predict": 2200}}, + fh) + self.assertEqual(sec.capped_arms(fh.name), ["case_b"]) + + +class TestBufferedLoop(unittest.TestCase): + """The server can hold a reply back and send it in the final chunk -- glimmer's parser does for a reply with no + message header (the #464 path) -- so nothing streams until the generation ends. The loop is then judged on the + final text: a length-capped reply that loops is the same verdict, a finished one that repeats is not a loop.""" + + def setUp(self): + self._env = os.environ.get("LOOP_DETECT") + os.environ["LOOP_DETECT"] = "1" + + def tearDown(self): + if self._env is None: + os.environ.pop("LOOP_DETECT", None) + else: + os.environ["LOOP_DETECT"] = self._env + + LOOPING = '{"x1": 70, "y1": 147, "x2": 210, "y2": 336, "label": "ANCHOR", "bbox_type": "norm1000", ' * 40 + + def _one_final(self, reason): + cap = StreamCapture([], final={"done_reason": reason, "eval_count": 2200, "eval_duration": 9 * 10**9}) + cap.lines[-1]["message"]["content"] = self.LOOPING + return cap + + def test_a_capped_buffered_loop_is_the_verdict(self): + with self._one_final("length"): + r = client.generate("http://h", "M", "p", ["IMG"]) + self.assertEqual(r["done_reason"], "loop") + self.assertEqual(r["failure_mode"], "loop detected:content") + self.assertEqual(r["_loop"]["server_done_reason"], "length") + self.assertEqual(r["eval_count"], 2200, "the server's own metrics are kept when it sent them") + self.assertEqual(r["response"], self.LOOPING) + + def test_a_finished_reply_that_repeats_is_not_a_loop(self): + with self._one_final("stop"): + r = client.generate("http://h", "M", "p", ["IMG"]) + self.assertEqual(r["done_reason"], "stop") + self.assertNotIn("failure_mode", r) + + +class CountingStream(StreamCapture): + """StreamCapture that counts requests, each served the whole stream afresh (so a retry is visible).""" + + def __enter__(self): + super().__enter__() + fake, self.calls = urllib.request.urlopen, 0 + + def counting(req, timeout=None): + self.calls += 1 + self.closed = False + return fake(req, timeout) + urllib.request.urlopen = counting + return self + + +class TestStreamedServerError(unittest.TestCase): + """Once a stream has started, the server can no longer send a status code: server/routes.go streamResponse writes + the error as one last {"error": ...} line instead (status 500 unless the error carries its own). With stream:false + the same failure is an HTTP 500. The streamed one must read the same, name the server's 100-token repeat abort + when that is what it is, and -- like the HTTP 500 -- not be retried: it is deterministic, and every retry pays for + the whole generation again (#469 review).""" + + def setUp(self): + self._env = os.environ.get("LOOP_DETECT") + os.environ["LOOP_DETECT"] = "1" + self._sleep = client.time.sleep + self.slept = [] + client.time.sleep = self.slept.append + + def tearDown(self): + client.time.sleep = self._sleep + if self._env is None: + os.environ.pop("LOOP_DETECT", None) + else: + os.environ["LOOP_DETECT"] = self._env + + def _fail_mid_stream(self, error): + # Distinct chunks, so the client's own detector cannot fire first: the server counts the runner's raw tokens, + # the client counts the chunks that reach it after the output parser, and those can be fewer. + cap = CountingStream([("", f"tok{i} ") for i in range(10)]) + cap.lines[-1] = {"error": error} + return cap + + def test_the_repeat_abort_names_the_server_abort_as_an_http_500(self): + cap = self._fail_mid_stream("prediction aborted, token repeat limit reached") + with cap, self.assertRaises(RuntimeError) as cm: + client.generate("http://h", "M", "p", ["IMG"]) + msg = str(cm.exception) + self.assertTrue(msg.startswith("HTTP 500 (streamed): prediction aborted, token repeat limit reached"), msg) + self.assertIn("100 identical tokens", msg) + self.assertNotIn("stream error", msg) + self.assertEqual((cap.calls, self.slept), (1, []), "a server abort is deterministic: not retried") + + def test_any_other_streamed_error_is_an_http_500_too(self): + cap = self._fail_mid_stream("mlx runner aborted: out of memory") + with cap, self.assertRaises(RuntimeError) as cm: + client.generate("http://h", "M", "p", ["IMG"]) + self.assertEqual(str(cm.exception), "HTTP 500 (streamed): mlx runner aborted: out of memory") + self.assertEqual(cap.calls, 1) + +class TestLoopFields(unittest.TestCase): + """client.loop_fields is the one derivation both producers stamp onto a block (as metrics_block is).""" + + def test_a_loop_result_stamps_its_verdict(self): + r = {"done_reason": "loop", "failure_mode": "loop detected:content", "_stream": True, + "_loop_detect": {"window": 400}, "_loop": {"loop": "loop detected:content", "loop_period": 87}} + f = client.loop_fields(r) + self.assertEqual(f["failure_mode"], "loop detected:content") + self.assertEqual(f["loop_period"], 87) + self.assertEqual(f["stream"], True) + self.assertEqual(f["loop_detect"], {"window": 400}) + + def test_a_clean_streamed_result_says_detection_ran(self): + f = client.loop_fields({"done_reason": "stop", "_stream": True, "_loop_detect": {"window": 400}}) + self.assertEqual(f, {"stream": True, "loop_detect": {"window": 400}}) + + def test_a_non_streamed_result_adds_nothing(self): + self.assertEqual(client.loop_fields({"done_reason": "stop"}), {}) + if __name__ == "__main__": unittest.main(verbosity=2) diff --git a/docs/maxusai/vision-suite/test_loop_detect.py b/docs/maxusai/vision-suite/test_loop_detect.py new file mode 100644 index 00000000000..0c54c29ed72 --- /dev/null +++ b/docs/maxusai/vision-suite/test_loop_detect.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +"""Tests for loop_detect.py. No network, stdlib only: python3 test_loop_detect.py + +The two fixtures under fixtures/loops/ are real outputs from the muse-glimmer campaign (2026-10-11, #463), both +think-off with format "json" on the release build: + glimmer-mxfp8-bbox-object-repeat.txt 30b-mxfp8, bboxm_pin_noanc_named: one box object's keys, repeated to + num_predict (2200 tokens) + glimmer-nvfp4-multi3img-degenerate.txt 30b-nvfp4, multi_3img: the prompt's words, then `, " 1, " :"}` repeated +""" +import os +import random +import time +import unittest + +import loop_detect as ld + +HERE = os.path.dirname(os.path.abspath(__file__)) + + +def fixture(name): + with open(os.path.join(HERE, "fixtures", "loops", name)) as fh: + return fh.read() + + +def prose(n, seed=0): + """Non-repeating text: a seeded word stream, so no 400-char window recurs.""" + rng = random.Random(seed) + words, size = [], 0 + while size < n: + words.append("".join(rng.choice("abcdefghijklmnopqrstuvwxyz") for _ in range(rng.randint(2, 11)))) + size += len(words[-1]) + 1 + return " ".join(words)[:n] + + +def feed_all(det, text, kind="content", chunk=7): + """Feed text in fixed chunks; return the first Loop or None.""" + for i in range(0, len(text), chunk): + piece = text[i:i + chunk] + got = det.feed(content=piece) if kind == "content" else det.feed(thinking=piece) + if got: + return got + return None + + +class TestBlockLoops(unittest.TestCase): + def test_short_text_never_triggers(self): + self.assertIsNone(feed_all(ld.LoopDetector(), "abc " * 150)) # 600 chars < min_length + + def test_long_prose_does_not_trigger(self): + for kind in ("thinking", "content"): + self.assertIsNone(feed_all(ld.LoopDetector(), prose(30000), kind=kind), kind) + + def test_a_block_restated_once_is_not_a_loop(self): + # One earlier occurrence (a table restated, a quote repeated) must not trip it: that was the one-copy + # false positive of ai-invoice-to-xero's thinking_is_looping. + block = prose(600, seed=1) + text = prose(1200, seed=2) + block + prose(300, seed=3) + block + self.assertIsNone(feed_all(ld.LoopDetector(), text)) + + def test_a_draft_restated_between_checks_is_not_a_loop(self): + # Three copies that are NOT back to back: a thinking trace that restates its draft answer after each round of + # checking. The CUDA host's replay of 1,505 finished think-on traces (#469 review) found 17-18 of these called + # loops by a copy count alone -- 13 of them multi_3img cells with q1, q2 and q4 all right. + draft = prose(500, seed=20) + text = prose(1000, seed=21) + draft + prose(300, seed=22) + draft + prose(300, seed=23) + draft + self.assertIsNone(feed_all(ld.LoopDetector(), text, kind="thinking")) + + def test_a_restated_draft_then_a_real_loop_still_fires(self): + # The periodicity test must not hide a loop that starts after earlier restatements. + draft, block = prose(500, seed=24), prose(430, seed=25) + text = prose(1000, seed=26) + draft + prose(300, seed=27) + draft + block * 4 + got = feed_all(ld.LoopDetector(), text, kind="thinking") + self.assertIsNotNone(got) + self.assertEqual((got.kind, got.period), (ld.THINKING, 430)) + + def test_a_block_repeated_is_a_loop(self): + block = prose(450, seed=4) + got = feed_all(ld.LoopDetector(), prose(900, seed=5) + block * 4, kind="thinking") + self.assertIsNotNone(got) + self.assertEqual(got.kind, ld.THINKING) + self.assertEqual(got.failure_mode(), "loop detected:thinking") + self.assertEqual(got.period, 450) + self.assertGreaterEqual(got.repeats, 3) + + def test_kind_follows_the_stream_that_loops(self): + block = prose(300, seed=6) + det = ld.LoopDetector() + self.assertIsNone(det.feed(thinking=prose(2000, seed=7))) + got = feed_all(det, block * 6, kind="content") + self.assertEqual(got.kind, ld.CONTENT) + + def test_short_period_loop(self): + got = feed_all(ld.LoopDetector(), prose(900, seed=8) + '{"a": 1, "b": 2}, ' * 200) + self.assertIsNotNone(got) + self.assertEqual(got.period, len('{"a": 1, "b": 2}, ')) + + +class TestRealLoops(unittest.TestCase): + def test_glimmer_bbox_object_repeat(self): + got = feed_all(ld.LoopDetector(), fixture("glimmer-mxfp8-bbox-object-repeat.txt")) + self.assertIsNotNone(got) + self.assertEqual(got.kind, ld.CONTENT) + self.assertLess(got.at_chars, 2000, "should fire well before the 4350 chars num_predict produced") + + def test_glimmer_multi3img_degeneration(self): + got = feed_all(ld.LoopDetector(), fixture("glimmer-nvfp4-multi3img-degenerate.txt")) + self.assertIsNotNone(got) + self.assertEqual(got.kind, ld.CONTENT) + + +class TestTokenRun(unittest.TestCase): + def test_a_run_of_one_token_is_a_loop(self): + det = ld.LoopDetector(token_run=64) + got = None + for _ in range(200): + got = det.feed(content=" ") or got + if got: + break + self.assertIsNotNone(got) + self.assertEqual(got.kind, ld.TOKEN_RUN) + self.assertEqual(got.failure_mode(), "loop detected:token-run") + self.assertEqual(got.repeats, 64) + + def test_a_short_run_is_not(self): + det = ld.LoopDetector(token_run=64) + self.assertIsNone(next((g for g in (det.feed(content=" ") for _ in range(63)) if g), None)) + self.assertIsNone(det.feed(content="x")) + + def test_token_run_is_counted_per_stream(self): + # Interleaving does not break a stream's run, and one stream's chunks never extend the other's: + # thinking alternates (never runs) while content repeats, so the run found is content's. + det = ld.LoopDetector(token_run=10) + got = None + for i in range(30): + got = got or det.feed(thinking="ab"[i % 2]) + got = got or det.feed(content="z") + self.assertEqual((got.kind, got.stream, got.repeats), (ld.TOKEN_RUN, ld.CONTENT, 10)) + + +class TestIncremental(unittest.TestCase): + def test_chunking_does_not_change_the_verdict(self): + text = prose(1500, seed=9) + prose(420, seed=10) * 5 + verdicts = set() + for chunk in (1, 3, 17, 64, 500): + got = feed_all(ld.LoopDetector(), text, chunk=chunk) + verdicts.add((got.kind, got.period) if got else None) + self.assertEqual(verdicts, {(ld.CONTENT, 420)}) + + def test_linear_enough(self): + det = ld.LoopDetector() + text = prose(300000, seed=11) + t0 = time.perf_counter() # the detector alone: generating the text is not timed + self.assertIsNone(feed_all(det, text, chunk=4)) + self.assertLess(time.perf_counter() - t0, 3.0, "a 300k-char stream must not cost quadratic time") + + def test_record_is_self_describing(self): + got = feed_all(ld.LoopDetector(), prose(900, seed=12) + prose(500, seed=13) * 4) + rec = got.record() + for key in ("loop", "loop_kind", "loop_stream", "loop_at_chars", "loop_period", "loop_repeats", + "loop_snippet", "loop_params"): + self.assertIn(key, rec) + self.assertEqual(rec["loop"], "loop detected:content") + self.assertLessEqual(len(rec["loop_snippet"]), 160) + + +class TestCompatibility(unittest.TestCase): + """The functions ai-invoice-to-xero's ollama_toolkit exports keep their names and meaning (one earlier copy).""" + + def test_thinking_is_looping(self): + self.assertFalse(ld.thinking_is_looping("x" * 799)) + block = prose(450, seed=14) + self.assertTrue(ld.thinking_is_looping(prose(500, seed=15) + block + block)) + self.assertFalse(ld.thinking_is_looping(prose(5000, seed=16))) + + def test_dedup_thinking(self): + head, block = prose(500, seed=17), prose(450, seed=18) + out = ld.dedup_thinking(head + block + block + block) + self.assertTrue(out.startswith(head)) + self.assertLess(len(out), len(head) + 2 * len(block)) + self.assertEqual(ld.dedup_thinking(" short "), " short") + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/maxusai/vision-suite/test_summarizers.py b/docs/maxusai/vision-suite/test_summarizers.py index 00eeae43c03..eb998f2b71d 100644 --- a/docs/maxusai/vision-suite/test_summarizers.py +++ b/docs/maxusai/vision-suite/test_summarizers.py @@ -651,6 +651,44 @@ def test_resume_trusts_the_verdict_through_arm_done(self): {"done_reason": "length", "eval_count": 100, "num_predict": 8192})) +class TestLoopIsNotCapped(unittest.TestCase): + """done_reason "loop" is a verdict (ADR 0049), never a cap -- including a loop the server held back and stopped + at num_predict, whose eval_count IS the cap. The arithmetic fallback read that block as capped, so arm_done + re-ran it and capped_arms climbed the ladder for it (#469 review). The row below is the shape the muse-glimmer + think-off run recorded for 30b-mxfp8 bboxm_pin_noanc_named.""" + + HELD_BACK = {"done_reason": "loop", "failure_mode": "loop detected:content", "loop_kind": "content", + "server_done_reason": "length", "eval_count": 2200, "num_predict": 2200, + "req_num_predict": 2200} + + def test_a_loop_at_the_cap_is_not_capped(self): + self.assertFalse(sec.was_capped(dict(self.HELD_BACK))) + + def test_a_loop_at_the_cap_is_done_on_resume(self): + self.assertTrue(vs.arm_done(dict(self.HELD_BACK))) + self.assertTrue(ftp.ft_done(dict(self.HELD_BACK))) + + def test_a_loop_at_the_cap_does_not_climb(self): + d = tempfile.mkdtemp() + p = os.path.join(d, "scores_x.json") + with open(p, "w") as fh: + json.dump({"a": dict(self.HELD_BACK), + "b": {"done_reason": "length", "eval_count": 2200, "num_predict": 2200}}, fh) + self.assertEqual(sec.capped_arms(p), ["b"]) + + def test_a_loop_has_no_decode_rate(self): + # The held-back loop keeps the server's metrics, so gen_tps WAS derivable -- and it is the decode rate of a + # loop, not of a model finishing. It stays None, as it is for a loop cut mid-stream (no eval_duration). + import client + r = {"done_reason": "loop", "eval_count": 2200, "eval_duration": 150_000_000_000, + "prompt_eval_count": 1000, "prompt_eval_duration": 1_000_000_000} + m = client.metrics_block(r) + self.assertIsNone(m["gen_tps"]) + self.assertEqual(m["eval_duration"], 150_000_000_000, "the server's durations are kept as recorded") + self.assertEqual(m["prefill_tps"], 1000.0, "prefill finished before the loop began; it is still a rate") + self.assertEqual(client.metrics_block(dict(r, done_reason="stop"))["gen_tps"], 14.67) + + class TestT1CampaignPrefix(unittest.TestCase): """--prefix renders a TAG_PREFIX campaign; unset, tags derive as before (H4). @@ -937,6 +975,140 @@ def test_geometry_pooled_excludes_capped(self): self.assertEqual([s["gen_tps"] for s in kept], [40.0]) +class TestLoopVerdictReachesTheTables(unittest.TestCase): + """A loop verdict (ADR 0049) renders as itself in every template and enters no mean -- the capped discipline + (ADR 0012 conv 9) applied to the third state. The client scores a loop's partial output, so without this a looped + cell published as an ordinary wrong answer and was averaged in (#469 review). A block with no loop -- including + one streamed with detection on -- renders exactly as before.""" + + MODEL = "gemma4:12b-it-q4_K_M" + LOOP = {"done_reason": "loop", "failure_mode": "loop detected:content", "loop_kind": "content", + "loop": "loop detected:content", "stream": True, "eval_count": 312, "num_predict": 2200, + "num_ctx": 16384} + CLEAN = {"bbox_mean_iou": 0.5, "eval_count": 400, "num_predict": 2200, "done_reason": "stop"} + + def looped(self, arm, **scores): + s = json.loads(json.dumps(THINKOFF)) + s[arm] = dict(s[arm], **dict(self.LOOP, **scores)) + if arm == "scene_single": + s[arm]["gen_tps"] = None # metrics_block gives a loop no decode rate + return s + + def t1_rows(self, scores): + d = tempfile.mkdtemp() + write(d, self.MODEL, "false", scores) + return [l for l in render(d, self.MODEL, "false").splitlines() if l.startswith(f"| {self.MODEL} |")] + + def test_loop_verdict_names_the_kind_and_is_none_without_one(self): + self.assertEqual(sec.loop_verdict(self.LOOP), "loop:content") + self.assertEqual(sec.loop_verdict({"failure_mode": "loop detected:token-run"}), "loop:token-run") + self.assertEqual(sec.loop_verdict({"done_reason": "loop"}), "loop") + for blk in (None, {}, dict(self.CLEAN), {"done_reason": "length", "eval_count": 2200, "num_predict": 2200}, + {"stream": True, "loop_detect": {"window": 400}, "done_reason": "stop"}): + self.assertIsNone(sec.loop_verdict(blk), blk) + + def test_t1_a_looped_scene_hides_its_scores_and_throughput(self): + grounding, throughput = self.t1_rows(self.looped("scene_single")) + self.assertIn("loop:content", grounding) + self.assertNotIn("0.900", grounding) + self.assertNotIn("6/6", grounding) + cells = [c.strip() for c in throughput.split("|")[1:-1]] + # ... | Think tok | Gen tok | Gen tok/s | Prefill tok/s | s/req | req/h + self.assertEqual(cells[-6:], ["loop:content", "loop:content", "—", "1000", "loop:content", "loop:content"]) + + def test_t1_a_looped_multi_renders_the_verdict_not_a_grounding_failure(self): + s = self.looped("multi_3img", q1_right=False, q2_right=False, q4_bbox_hit=False, + failure_mode="loop detected:thinking", loop_kind="thinking") + throughput = self.t1_rows(s)[1] + self.assertIn("| loop:thinking |", throughput) + self.assertNotIn("❌", throughput) + + def test_t1_a_streamed_block_without_a_loop_renders_exactly_as_before(self): + stamped = json.loads(json.dumps(THINKOFF)) + for blk in stamped.values(): + blk.update(stream=True, loop_detect={"window": 400, "min_repeats": 3}, done_reason="stop") + plain = json.loads(json.dumps(THINKOFF)) + for blk in plain.values(): + blk["done_reason"] = "stop" + self.assertEqual(self.t1_rows(stamped), self.t1_rows(plain)) + + def test_t2_a_looped_scene_renders_the_verdict_with_its_window(self): + rundir = tempfile.mkdtemp() + with open(os.path.join(rundir, "scores_x.json"), "w") as fh: + json.dump(self.looped("scene_single"), fh) + argv = ["summarize_head_to_head.py", "--dir", rundir, "--tags", "x"] + out = io.StringIO() + with mock_argv(argv), contextlib.redirect_stdout(out): + shh.main() + r = {l.split("|")[2].strip(): l.split("|")[3].strip() for l in out.getvalue().splitlines() + if l.startswith("|")} + self.assertEqual(r["bbox IoU"], "loop:content (16384)") + self.assertEqual(r["labels / serial"], "loop:content (16384)") + self.assertEqual(r["s/req (unique image)"], "loop:content") + self.assertEqual(r["req/h (serial)"], "loop:content") + self.assertEqual(r["gen tok/s"], "—") + + def test_contract_matrix_names_the_loop(self): + rundir = tempfile.mkdtemp() + scores = {"bbox_contract": dict(self.LOOP, contract_followed=False), + "bbox_contract_multi": {"eval_count": 400, "num_predict": 2200, "done_reason": "stop", + "contract_followed": True, "num_ctx": 8192}} + with open(os.path.join(rundir, f"scores_{sec.tag_for('m:1b', 'false')}.json"), "w") as fh: + json.dump(scores, fh) + out = io.StringIO() + with mock.patch.object(scm, "DIR", rundir), \ + mock_argv(["summarize_contract_matrix.py", "--think", "false", "m:1b"]), \ + contextlib.redirect_stdout(out): + scm.main() + row = [l for l in out.getvalue().splitlines() if l.startswith("| m:1b |")][0] + cells = [c.strip() for c in row.split("|")[1:-1]] + self.assertEqual(cells[2:4], ["loop:content", "✅"]) + self.assertIn("`loop:`", out.getvalue()) + + def test_reps_excludes_a_looped_rep_from_the_mean_and_says_so(self): + runs = [{"scene_single": dict(self.CLEAN)}, + {"scene_single": dict(self.LOOP, bbox_mean_iou=0.1)}] + s, spread = reps.cell(runs, "scene_single", "bbox_mean_iou", "float") + self.assertEqual(s, "0.500 (1 loop)") + self.assertIsNone(spread, "a partial cell claims no spread") + s, _ = reps.cell([{"scene_single": dict(self.LOOP, bbox_mean_iou=0.1)}], + "scene_single", "bbox_mean_iou", "float") + self.assertEqual(s, "loop") + + def test_geometry_pools_no_loop_and_renders_it(self): + import summarize_geometry as sg + loop = dict(self.LOOP, gen_tps=None, iou_anchor=0.1, image_size=[1280, 720]) + self.assertEqual(sg.pooled([dict(self.CLEAN, gen_tps=40.0), loop]), [dict(self.CLEAN, gen_tps=40.0)]) + self.assertEqual(sg.cell(loop), "loop:content | loop:content | loop:content") + + def test_matrix_means_exclude_loops_and_count_them(self): + import summarize_matrix as smx + d = tempfile.mkdtemp() + cell = {"bboxm_pin_anc_named": {"hits_declared": 6, "done_reason": "stop", "eval_count": 400, + "req_num_predict": 2200}, + "bboxm_free_anc_named": dict(self.LOOP, hits_declared=0, req_num_predict=2200)} + with open(os.path.join(d, "scores_mx-hd-1_m_thinkfalse.json"), "w") as fh: + json.dump(cell, fh) + out = io.StringIO() + with mock_argv(["summarize_matrix.py", "mx", "--dir", d]), contextlib.redirect_stdout(out): + smx.main() + text = out.getvalue() + self.assertIn("loop verdicts (excluded from every mean): 1", text) + pin = [l for l in text.split("### factor: pin")[1].splitlines() if l.startswith("| m |")][0] + cells = [c.strip() for c in pin.split("|")[1:-1]] + # model | think | on | off | delta | capped (on·off) | spread + self.assertEqual(cells[2:4], ["6.00", "—"], "the looped off-arm is not a 0.00 mean") + self.assertEqual(cells[5], "0/1 · 0/1 (+1 loop)") + + def test_cmp_scored_does_not_compare_a_loop(self): + import cmp_scored + a = {"t": dict(self.CLEAN, iou_declared=0.9)} + b = {"t": dict(self.LOOP, iou_declared=0.1)} + lines, fav_a, fav_b, labels = cmp_scored.compare(a, b) + self.assertEqual((fav_a, fav_b, labels), (0, 0, 0)) + self.assertEqual(lines, ["t: finished only in A (B: loop:content)"]) + + class TestCappedArms(unittest.TestCase): """The driver's escalation decision as an importable, tested function — the shell heredoc it replaces ignored done_reason (blueprint P0-1).""" diff --git a/docs/maxusai/vision-suite/vision_suite.py b/docs/maxusai/vision-suite/vision_suite.py index 78c61cc3e1a..716145991e7 100644 --- a/docs/maxusai/vision-suite/vision_suite.py +++ b/docs/maxusai/vision-suite/vision_suite.py @@ -1845,6 +1845,7 @@ def persist_scores(): # so this is copied only when the server actually said something. if r.get("done_reason"): sc["done_reason"] = r["done_reason"] + sc.update(client.loop_fields(r)) sc.update(sc_chars) # SPEC C13: a conformance rate is scoped to the geometry it was measured # at and MUST be quoted with it. Recorded on the cell for the same