Gemini 2.5 Flash structured-output repetition loop — upstream bug + client-side workaround

[tuirk]

Summary

Gemini 2.5 Flash occasionally enters an infinite token-repetition loop during structured-output extraction, truncating the response mid-JSON at max_output_tokens. This is an upstream model bug — documented by Google since Oct 2025, with ~30% hit rate on some reporters' workloads. Persists on both Flash and Pro as of April 2026, no Google fix or ETA.

This issue documents the failure signature, the client-side mitigations we've shipped, and deferred follow-ups. Keep open until either (a) Google fixes upstream, or (b) all deferred items below ship.

Observed failure signatures

Two distinct failures that hit the same code path in nlp-service/services/llm_client.py :: extract_source():

1. Repetition loop → JSON truncation:

extract_llm_parse_failed: 1 validation error for LLMExtractionResponse
  Invalid JSON: EOF while parsing a value at line 24 column 64119
  [type=json_invalid, input_value='{"entities":[..."AI","AI","AI","AI",', ...]

Config that triggers it: temperature=0.0, max_output_tokens=32768, response_mime_type=application/json, response_schema=LLMExtractionResponse, thinking_budget=512.

2. Transient connection drops (unrelated root cause, same code path):

extract_llm_call_failed: Server disconnected without sending a response.

httpx RemoteProtocolError bubbling out of google-genai with no retry logic.

Root cause

Repetition loop: Upstream sampler pathology on Gemini 2.5 Flash/Pro at temperature=0.0 with structured output. Well-documented by Google users; no Google fix:

• https://discuss.ai.google.dev/t/gemini-2-5-flash-repeats-tokens-until-max-tokens-reached-in-structured-output/107176
• https://discuss.ai.google.dev/t/gemini-2-5-pro-enters-infinite-b-repetition-loop-65k-tokens-billed-same-behavior-as-flash/115368
• https://discuss.ai.google.dev/t/truncated-response-issue-with-gemini-2-5-flash-preview/81258

Transient drops: google-genai==1.12.1 (our pinned version) has zero retry logic. HttpRetryOptions only landed in ≥1.65 — a 60-version jump we're not taking mid-debug.

Evaluated and confirmed NOT to work

• frequency_penalty / presence_penalty — NOT supported on 2.5 models; returns INVALID_ARGUMENT
• Pydantic max_length / max_items constraints — silently ignored
• Anti-repetition prompt hints alone — no reports of them working
• Lower top_p — no corroboration
• Streaming mid-body abort — streaming on 2.5 models has its own bugs (empty chunks, 400 errors)
• thinking_budget tuning — often ignored outright on 2.5 models

Applied workarounds

Shipped in two commits:

• 2234bcc — fix(api): observability overhaul for compile pipeline failures

  • Shared throwOnError helper in compile/run/route.ts replaces 8 inline status-only throws. Reads the sub-route response body (3 s race-capped), embeds first 2 000 chars in the thrown Error so failCompileProgress stores the real cause in compile_progress.error (visible in UI banner + Activity feed).
  • timed() wrapper logs [compile:<sid>] <step> <ms> on success — makes cold-start vs steady-state latency visible.
  • Top-level .catch now console.error's raw err (stack trace), not just the stringified message.
  • Each of compile/{extract,resolve,match,plan,crossref,draft,schema} now wraps its POST body in a top-level try/catch that logs the stack before returning 500. Next.js 16 production doesn't surface uncaught handler errors without explicit logging.
  • Side effect: UUIDs no longer leak into activity-feed error strings (they were baked into the old extract failed for {uuid}: 500 message).
  • api/health: bumped expected schema_version 15 → 16 (the v16 migration had been silently failing the healthcheck, leaving Docker flapping).

• 3ebb275 — feat(nlp-service): Gemini retry wrapper + truncation salvage + usage logging

  • _with_retry helper: 3 attempts, exponential backoff (0.5/1/2 s), catches httpx connect-class errors + google-genai APIError with status in {408, 429, 500, 502, 503, 504}. Re-acquires the pyrate-limiter token per attempt so retries count against the 800 RPM cap (not bypass it).
  • Applied to 7 read-only analyzers: extract, lint_scan, disambiguate, crossref, triage, generate_schema, select_pages. Skipped on draft_page, synthesize_answer, generate_digest — those commit downstream state and can't be safely re-executed after a partial response.
  • _log_usage helper: one stderr line per Gemini call with finish_reason + prompt_token_count / candidates_token_count / thoughts_token_count. Ground truth for distinguishing MAX_TOKENS truncation from other failure modes.
  • extract_source fallback chain on parse failure:
    1. json-repair salvage (new dep, json-repair>=0.59.4) — recovers the valid JSON prefix from a repetition-loop truncation for free; ~90% of loops have extractable content before the stuck tail
    2. If finish_reason == MAX_TOKENS and salvage failed → retry once with halved input; different prompt path usually escapes the loop
    3. On all-fail → chain both errors in extract_llm_parse_failed_after_fallback: first=...; second=... so compile_progress.error shows what each attempt hit
  • Anti-repetition hint added to _EXTRACTION_SYSTEM_PROMPT ("Stop as soon as the source is covered — do NOT repeat names or pad the output"). Mixed evidence it helps; costs ~10 input tokens, kept as belt-and-suspenders.

Deferred follow-ups

• Prompt reorder for implicit caching — Gemini 2.5 has implicit caching enabled (75% off cached input tokens on prefix match) but requires ≥1024 cached tokens. Our stable prefix is ~375 tokens — below the threshold. Padding it with substantive stable content (schema description, few-shot examples) would unlock the discount.
• Streaming + mid-stream repetition detection — generate_content_stream IS supported with response_schema in 1.12.1. A rolling-window detector (same ≥15-char slice repeated ≥4× in last ~100 tokens) + break the iterator could abort at ~2-4k tokens instead of 32k. ~94% output-token savings on detection. ~1 day of work.
• Env-flagged model-swap fallback to gemini-2.5-flash-preview-09-2025 after two consecutive MAX_TOKENS failures. Forum reports it's more stable for the repetition bug specifically; explicitly non-GA.
• google-genai SDK upgrade to ≥1.65 for native HttpRetryOptions (our own _with_retry becomes redundant). 60-version jump from 1.12.1 — needs its own investigation cycle.
• Batch API migration for non-interactive compiles — 50% discount on input + output tokens, 24h SLA. Session-compile is already fire-and-forget via n8n, matches the latency budget.
• Disambiguate result cache — Layer 3 entity disambiguation re-runs on every compile retry even when prior aliases were persisted. Cache by (session_id, sorted-pair-hash) to avoid re-spend.

References

• python-genai cookbook issue #1091 — retry.if_transient_error doesn't catch google-genai errors
• python-genai #782 — thinking_budget often ignored on 2.5 models
• python-genai #1984 — hallucinations/endless JSON (closed stale, no fix)
• Vertex AI Retry Strategy docs
• Gemini 2.5 implicit caching announcement
• Gemini Batch API docs
• json-repair library

[tuirk]

Scope reduction — crossref no longer calls Gemini

Commit fad98d6 replaced the LLM-backed crossref with deterministic title-matching (wikilink-injector.ts + rewritten crossref route). Effect on this issue:

• One fewer Gemini caller exposed to the repetition-loop bug. The 10 read-only LLM sites listed in the original scope (extract, disambiguate, crossref, triage, lint_scan, generate_schema, select_pages) drop to 6 — crossref is out of the LLM path entirely now.
• The other deferred follow-ups (prompt reorder for implicit caching, streaming + mid-stream abort, model-swap flag, SDK upgrade, Batch API, disambiguate cache) are unchanged.

Motivation wasn't purely the repetition loop — the trigger was a separate HeadersTimeoutError at 12m50s on an 8-source compile. But once we looked at scale (real user with 1 860 bookmarks → ~500+ pages), LLM crossref wasn't viable regardless of which upstream bug hit it: ~20 sequential Gemini calls, ~$0.06 per session, wall time 10-20 min, fragile under any timeout layer. Research across MediaWiki LinkTitles, Obsidian Automatic Linker, Logseq, Roam, LangChain KG pipelines, Karpathy's LLM-wiki write-up — every production wiki uses deterministic title-matching for this problem. Aho-Corasick / regex-alternation is the right tool.

Contradiction detection (the other half of the old crossref) dropped to 0-findings-per-session in committed runs, so it's deferred. Can be reintroduced as a targeted Gemini job on entity-overlap pairs if needed.

[tuirk]

Status check 2026-05-04 — bug migrated upstream, salvage rate below original estimate

Today's failure (single 2-source compile, session 7170feee)

Source	Chars	Outcome	Gemini calls	Output tokens	Est. cost
"Introduction to agent skills - Claude Code"	33 029	extracted ✓	1 (step=extract finish=STOP in=16303 out=2545)	2.5 K	~$0.01
"Open Standards for AI Skills - agentskills.io"	95 027	failed ✗	2 (extract + extract-fallback, both finish=MAX_TOKENS, both out=32K)	64 K wasted	~$0.16

Both attempts on source 2 hit the 32K output cap inside the repetition loop. json-repair salvage returned None for both — the loop entered before any complete entity was emitted, so there was no valid prefix to recover. The halved-input fallback (50K → 25K) did not escape the loop. Net: source 2 dropped, no entity/concept pages produced (planner had only source 1 contributing → no cross-source signal → only the unconditional source-summary pages were created).

Revising the salvage rate

The original commit (3ebb275) claimed ~90% of repetition-loop responses recover via json-repair. Today's evidence shows 0/2 recovery on a content-dense source (95K chars, technical/structural document). The 90% figure looks accurate for short repetition tails on otherwise valid output, but does not hold for documents where Gemini enters the loop early in the entity list. Suggest revising the claim to "salvages tail-truncated responses; recovery rate drops sharply when the loop starts before the first complete entity."

Upstream status — got worse

• 2026-01-14: same repetition-loop pathology migrated from Gemini 2.5 Flash to Gemini 2.5 Pro production endpoints. (forum thread) Same failure mode, billed at higher Pro pricing.
• 2026-Q1: separate Vertex Flash 2.5 transcription regression where the model emits literal [unclear] tokens until max_output_tokens. (forum thread)
• Google's own google/langextract library acknowledges the bug in their structured-extraction reference impl with no fix. No public ETA from Google for either family.

New mitigation candidates (proposed additions to deferred follow-ups)

• Surface failed source title in extract step detail. Today the UI shows "1/2 sources extracted" whether the failed source was a 200-char tweet or a 95K-char article. Should read "1/2 sources extracted — failed: <title>". UX-only, ~1 line in app/src/app/api/compile/run/route.ts.
• Lower max_output_tokens 32768 → ~12000 for the extract call. Caps the bleed when the loop hits — output cost drops by ~63% per failure. Aligned with Google's own guidance to set max_output_tokens low for character-limited applications. Tradeoff: legitimate dense extractions truncate sooner, relying on json-repair salvage. Need a baseline of typical successful out= values to set the cap right (current data: ~2.5K on a 33K-char source — 12K leaves ~5× headroom).
• Per-source circuit breaker. After N consecutive extract failures on the same source_id, mark extract-blocked in DB and skip on future retries until the user removes/re-uploads/edits. Prevents repeat burns on the same doomed source.
• Try thinking_budget=0 for the extract call specifically. Issue Gemini 2.5 Flash structured-output repetition loop — upstream bug + client-side workaround #7 already notes thinking_budget is "often ignored" on 2.5 models. But one report showed disabling thinking entirely changed output behavior dramatically (37 → 670 tokens) for a different failure mode. Worth a controlled A/B on extract calls before/after.

Reprioritization

• Streaming + mid-stream repetition detection (currently in deferred follow-ups, ~1 day of work, ~94% output-token savings on detection): this should move to prioritized. Community consensus is now stronger that this is the only structural fix until Google addresses the sampler bug. Every other workaround caps the cost of the failure rather than preventing it.

[tuirk]

Status check 2026-05-05 — streaming detector failed Stage C precision

The previous comment moved "streaming + mid-stream repetition detection" from deferred to prioritized as the only structural fix. We built it, ran a multi-stage validation pipeline against a real corpus, and the result was negative — the detector cannot ship on Gemini in its current form.

What we built

• nlp-service/services/repetition_detector.py — pure detector function (window=200, min_substr=12, min_repeats=4)
• nlp-service/tests/test_repetition_detector.py — 19 synthetic tests
• nlp-service/streaming_validation/harness.py — control (non-streaming) vs treatment (streamed + detector) on a 6-source corpus
• nlp-service/streaming_validation/debug_one_stream.py — chunk-level instrumentation
• Validation methodology: Stage A (synthetic) → Stage B (live) → Stage C (gates: recall ≥95%, precision ≥99%, cost saving ≥80%, quality preservation ±5%)

Stage A — passed

19/19 synthetic tests pass. Recall = 100% on known loop patterns ("AI","AI","AI"..., \b\b\b\b..., delimiter-only loops, entity-shaped loops, valid-prefix-then-loop hybrids). False-positive rate = 0% on real entity-JSON samples extracted from prior successful production calls.

Stage B — partial

The live run produced two intrinsic detector failures that re-tuning cannot fix:

Failure 1 — false positive on legitimate enumeration JSON (initial run):
The detector fired at chunk 2 (~220 chars) on the Vilnius corpus source. Cause: legitimate mentions: ["Vilnius", "Vilnius Cathedral", "Vilnius Old Town", ...] arrays look exactly like a loop to the detector — same substring repeats the same number of times in close succession. Mitigation attempt: added MIN_DETECTION_LENGTH=2000 gate to skip detection on the very-early stream.

Failure 2 — same root cause, larger window (after gate):
With the 2000-char gate, the Vilnius source no longer trips. But on the "Introduction to agent skills - Claude Code" source the detector fires at chunk 12 (~2400 chars) on "Claude Code" repeating four times across legitimate entities and mentions arrays. The pattern is the same: dense JSON that legitimately enumerates a small set of canonical names looks indistinguishable from a sampler loop at the detector's level.

Stage C precision = 0% (target: ≥99%). The two failure modes above don't have a knob fix. Tightening min_substr upward from 12 reduces synthetic recall (the actual "AI","AI","AI" loop has only ~6-char cycles); loosening min_repeats downward worsens precision; lengthening MIN_DETECTION_LENGTH doesn't help once we're already past 2K chars and the legit enumeration is still going.

Bonus finding — Gemini streaming has its own bug (separate from this)

The Vilnius source produces ~3,000 output tokens reliably under the non-streaming endpoint. Under generate_content_stream with the same prompt and config, the same source consistently truncates at ~1,485 output chars with finish_reason=STOP. This is unrelated to the repetition loop and unrelated to the detector — streaming + response_schema on 2.5-flash drops content before the model would naturally stop. Reproducible. Adds to the live-bug list (alongside livekit#4307 empty-final-chunks, #4706 empty-stream completions, the 400-INVALID_ARGUMENT thread).

Reprioritization

• De-prioritize streaming + mid-stream detection. Even if the precision FPs were eliminated, the Gemini-streaming truncation bug independently makes streaming unreliable for our use case. Two upstream bugs stacking; no fix for either is in our hands.
• The earlier "this is the only structural fix" call was wrong. Stage A passing gave a false sense; live data is what mattered.

New direction under investigation — provider migration

Tested DeepSeek V4 Pro side-by-side on the same corpus, including the 95K-char "Open Standards for AI Skills" source that Gemini cannot extract from. Results:

Source / mode	Outcome	Notes
Open Standards (95K → 50K truncated) on DeepSeek V4 Pro non-think × 3 runs	3/3 ok	Gemini hit 0/2 on the same source same week
Open Standards on DeepSeek V4 Pro think-high × 3 runs	3/3 ok	—
Kelly Criterion academic PDF (full pipeline)	1/1 ok, ~160s wall	15 entities, 7 concepts, 10 claims

Investigation now sized at ~3-4 days of work to add DeepSeek as a second provider behind a settings toggle (Gemini stays the default). Plan in progress; not yet committed.

Updated deferred list

• Streaming + mid-stream repetition detection — failed validation, dropped from prioritized list
• Pluggable LLM provider with DeepSeek V4 as the first alternative (default stays Gemini for OSS users with a single key) — new top item
• Prompt reorder for implicit caching (still relevant on Gemini path)
• Lower max_output_tokens 32768 → ~12000 for extract — bleed-cap mitigation, still relevant if we keep Gemini
• Surface failed source title in extract step detail — UX, independent
• Per-source circuit breaker — independent, still useful
• google-genai SDK upgrade ≥1.65 — independent
• Batch API migration — independent
• Disambiguate result cache — independent

[tuirk]

The pattern is not Gemini-specific

Cross-provider check (May 2026) confirms structured-output reliability bugs span the industry. The Gemini repetition loop documented here has direct counterparts elsewhere:

Same family — runaway output tokens until the budget cap:

• Kimi K2.5 — same repetition-loop family.
• GPT-4.1 — \n\n\n\n padding via the Responses API. Reports show >80% of output tokens being padding on affected calls. community.openai.com/t/1295284

Different mechanism, same reliability concern (silent schema breakage):

• GPT-5.4 nano — JSON schema not respected; missing required keys, type-mismatched values. langgenius/dify#31050
• GPT-5-mini — Reasoning: [] text leaking into the JSON envelope, breaking strict parsers. promptfoo#5283
• GPT-5 series — intermittent structured-output failures; nine-out-of-ten calls succeed, the tenth drops the schema entirely. agno-agi/agno#4183

Clean as of May 2026 — no documented repetition or schema-drop bug:

• Anthropic Claude family (Haiku 4.5 / Sonnet 4.6 / Opus 4.7).
• DeepSeek V4 — including on the same input bytes that loop Gemini 2.5 Flash 100% of the time.
• Mistral Medium 3 / Large 3.

Implication for the workarounds in this issue: the salvage-and-retry layer in _with_retry and extract_source is currently structured around Gemini's specific failure shape. Once the provider abstraction lands (#55 presupposes it), this should generalize into a per-provider robustness layer — each adapter declares which failure classes it's prone to (repetition loop, schema drop, padding bloat) and which salvage strategies apply. Locking the salvage code to Gemini's shape would force a parallel rewrite for every new provider.