Skip to content

fix(ai-improve): tolerate prose-wrapped JSON from the model - #4

Merged
claude[bot] merged 1 commit into
mainfrom
fix/ai-improve-json-parsing
Aug 26, 2026
Merged

fix(ai-improve): tolerate prose-wrapped JSON from the model#4
claude[bot] merged 1 commit into
mainfrom
fix/ai-improve-json-parsing

Conversation

@ssevera1

Copy link
Copy Markdown
Owner

Reproduced 3/3 in a row on Data-Science-Toolkit while verifying the pipeline: the selection call succeeded (context built, API responded), but Haiku sometimes reasons in prose ("Looking at this codebase, I need to identify...") instead of emitting bare JSON, despite SELECT_PROMPT's existing "ONLY valid JSON" instruction. ask_json() only tried json.loads() on the whole trimmed response, so any surrounding prose made it discard an otherwise-usable answer and skip the run entirely — safely (no bad file written), but silently.

What changed

  • Stronger prompt instructions in both SELECT_PROMPT and WRITE_PROMPT: explicitly forbid reasoning/commentary, spell out that a caller parses the response programmatically
  • Fallback extraction in ask_json(): if the full response isn't valid JSON on its own, scan for the first balanced {...} object anywhere in it and parse that instead of giving up
  • Print more of the raw response on a genuine parse failure (800 → 2000 chars) for easier debugging next time

Same file, byte-identical across all 11 repos before this change — opening the same fix everywhere rather than just on the repo that surfaced it.

Reproduced 3/3 on Data-Science-Toolkit: the selection call got a valid
response, but Haiku sometimes reasons in prose ("Looking at this
codebase, I need to identify...") instead of emitting bare JSON, despite
the existing "ONLY valid JSON" instruction. ask_json() only tried
json.loads() on the whole trimmed response, so any surrounding prose
made it discard an otherwise-usable answer and skip the run.

- Strengthen SELECT_PROMPT and WRITE_PROMPT: explicitly forbid
  reasoning/commentary, since a caller parses the response
  programmatically
- Add a fallback in ask_json(): if the whole response isn't valid JSON,
  scan for the first balanced {...} object anywhere in it and parse
  that instead of giving up
- Print more of the raw response on a genuine parse failure (800 -> 2000
  chars) for easier debugging next time this happens

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the happy path is unchanged (json.loads(raw) still runs first; the new except: pass only redirects the former failure branch), that the added {{/}} in both prompt templates escape correctly under the .format() calls at scripts/ai_improve.py:320 and :345, and that the balanced-brace fallback cannot produce an unsafe write: it is string-unaware, so an unmatched } inside a JSON string value ends the scan on a substring with an unterminated string, which fails json.loads and falls through to the same skip-the-run behaviour as before, and a stray balanced object in prose is caught by the file_path/missing key checks. All downstream guards (safe_target traversal + extension rejection, SHRINK_FLOOR, core.hooksPath=/dev/null) are untouched, and the diff is one file scoped entirely to the stated fix.

@claude
claude Bot merged commit 72fba73 into main Aug 26, 2026
3 checks passed
@claude
claude Bot deleted the fix/ai-improve-json-parsing branch August 26, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant