Skip to content

fix(ai-improve): tolerate prose-wrapped JSON from the model - #31

Merged
claude[bot] merged 1 commit into
mainfrom
fix/ai-improve-json-parsing
Aug 26, 2026
Merged

fix(ai-improve): tolerate prose-wrapped JSON from the model#31
claude[bot] merged 1 commit into
mainfrom
fix/ai-improve-json-parsing

Conversation

@ssevera1

Copy link
Copy Markdown
Owner

Reproduced 3/3 in a row on Data-Science-Toolkit while verifying the pipeline: the selection call succeeded (context built, API responded), but Haiku sometimes reasons in prose ("Looking at this codebase, I need to identify...") instead of emitting bare JSON, despite SELECT_PROMPT's existing "ONLY valid JSON" instruction. ask_json() only tried json.loads() on the whole trimmed response, so any surrounding prose made it discard an otherwise-usable answer and skip the run entirely — safely (no bad file written), but silently.

What changed

  • Stronger prompt instructions in both SELECT_PROMPT and WRITE_PROMPT: explicitly forbid reasoning/commentary, spell out that a caller parses the response programmatically
  • Fallback extraction in ask_json(): if the full response isn't valid JSON on its own, scan for the first balanced {...} object anywhere in it and parse that instead of giving up
  • Print more of the raw response on a genuine parse failure (800 → 2000 chars) for easier debugging next time

Same file, byte-identical across all 11 repos before this change — opening the same fix everywhere rather than just on the repo that surfaced it.

Reproduced 3/3 on Data-Science-Toolkit: the selection call got a valid
response, but Haiku sometimes reasons in prose ("Looking at this
codebase, I need to identify...") instead of emitting bare JSON, despite
the existing "ONLY valid JSON" instruction. ask_json() only tried
json.loads() on the whole trimmed response, so any surrounding prose
made it discard an otherwise-usable answer and skip the run.

- Strengthen SELECT_PROMPT and WRITE_PROMPT: explicitly forbid
  reasoning/commentary, since a caller parses the response
  programmatically
- Add a fallback in ask_json(): if the whole response isn't valid JSON,
  scan for the first balanced {...} object anywhere in it and parse
  that instead of giving up
- Print more of the raw response on a genuine parse failure (800 -> 2000
  chars) for easier debugging next time this happens

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the fallback JSON extractor, the prompt escaping, and the downstream call sites. The balanced-brace scan only runs after a plain json.loads failure, so the happy path is unchanged; its one weakness (brace counting ignores JSON string context) fails closed — a premature } yields an unterminated string that json.loads rejects, returning None exactly as before, so it can never hand back a silently truncated file_content. Extracting a stray object from prose is still gated downstream by safe_target(), the required-keys check, and the SHRINK_FLOOR guard, and the new prompt text correctly doubles {{/}} for the .format() templates. Scope is one file and matches the stated intent.

@claude
claude Bot merged commit 0c2099f into main Aug 26, 2026
3 checks passed
@claude
claude Bot deleted the fix/ai-improve-json-parsing branch August 26, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant