Skip to content

fix(ai-improve): tolerate prose-wrapped JSON from the model - #4

Merged
claude[bot] merged 1 commit into
mainfrom
fix/ai-improve-json-parsing
Aug 26, 2026
Merged

fix(ai-improve): tolerate prose-wrapped JSON from the model#4
claude[bot] merged 1 commit into
mainfrom
fix/ai-improve-json-parsing

Conversation

@ssevera1

Copy link
Copy Markdown
Owner

Reproduced 3/3 in a row on Data-Science-Toolkit while verifying the pipeline: the selection call succeeded (context built, API responded), but Haiku sometimes reasons in prose ("Looking at this codebase, I need to identify...") instead of emitting bare JSON, despite SELECT_PROMPT's existing "ONLY valid JSON" instruction. ask_json() only tried json.loads() on the whole trimmed response, so any surrounding prose made it discard an otherwise-usable answer and skip the run entirely — safely (no bad file written), but silently.

What changed

  • Stronger prompt instructions in both SELECT_PROMPT and WRITE_PROMPT: explicitly forbid reasoning/commentary, spell out that a caller parses the response programmatically
  • Fallback extraction in ask_json(): if the full response isn't valid JSON on its own, scan for the first balanced {...} object anywhere in it and parse that instead of giving up
  • Print more of the raw response on a genuine parse failure (800 → 2000 chars) for easier debugging next time

Same file, byte-identical across all 11 repos before this change — opening the same fix everywhere rather than just on the repo that surfaced it.

Reproduced 3/3 on Data-Science-Toolkit: the selection call got a valid
response, but Haiku sometimes reasons in prose ("Looking at this
codebase, I need to identify...") instead of emitting bare JSON, despite
the existing "ONLY valid JSON" instruction. ask_json() only tried
json.loads() on the whole trimmed response, so any surrounding prose
made it discard an otherwise-usable answer and skip the run.

- Strengthen SELECT_PROMPT and WRITE_PROMPT: explicitly forbid
  reasoning/commentary, since a caller parses the response
  programmatically
- Add a fallback in ask_json(): if the whole response isn't valid JSON,
  scan for the first balanced {...} object anywhere in it and parse
  that instead of giving up
- Print more of the raw response on a genuine parse failure (800 -> 2000
  chars) for easier debugging next time this happens

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the two things that could bite here. First, the new prompt text uses correctly doubled {{/}} so both SELECT_PROMPT and WRITE_PROMPT still render through .format(). Second, the balanced-brace scan in ask_json() is not JSON-string-aware, but that only costs a missed recovery, never a bad parse: braces inside file_content that close early leave an unterminated string, so json.loads raises and the function returns None — exactly the pre-existing skip-the-run behaviour — and all downstream guards (safe_target, required-key check, SHRINK_FLOOR, the max_tokens stop-reason check) remain in front of any write. Change is one file, matches the stated intent, and bundles nothing unrelated.

@claude
claude Bot merged commit 7bd870b into main Aug 26, 2026
2 checks passed
@claude
claude Bot deleted the fix/ai-improve-json-parsing branch August 26, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant