fix(ai-improve): tolerate prose-wrapped JSON from the model - #4
Conversation
Reproduced 3/3 on Data-Science-Toolkit: the selection call got a valid
response, but Haiku sometimes reasons in prose ("Looking at this
codebase, I need to identify...") instead of emitting bare JSON, despite
the existing "ONLY valid JSON" instruction. ask_json() only tried
json.loads() on the whole trimmed response, so any surrounding prose
made it discard an otherwise-usable answer and skip the run.
- Strengthen SELECT_PROMPT and WRITE_PROMPT: explicitly forbid
reasoning/commentary, since a caller parses the response
programmatically
- Add a fallback in ask_json(): if the whole response isn't valid JSON,
scan for the first balanced {...} object anywhere in it and parse
that instead of giving up
- Print more of the raw response on a genuine parse failure (800 -> 2000
chars) for easier debugging next time this happens
There was a problem hiding this comment.
Checked the happy path is unchanged (json.loads(raw) still runs first; the new except: pass only redirects the former failure branch), that the added {{/}} in both prompt templates escape correctly under the .format() calls at scripts/ai_improve.py:320 and :345, and that the balanced-brace fallback cannot produce an unsafe write: it is string-unaware, so an unmatched } inside a JSON string value ends the scan on a substring with an unterminated string, which fails json.loads and falls through to the same skip-the-run behaviour as before, and a stray balanced object in prose is caught by the file_path/missing key checks. All downstream guards (safe_target traversal + extension rejection, SHRINK_FLOOR, core.hooksPath=/dev/null) are untouched, and the diff is one file scoped entirely to the stated fix.
Reproduced 3/3 in a row on Data-Science-Toolkit while verifying the pipeline: the selection call succeeded (context built, API responded), but Haiku sometimes reasons in prose ("Looking at this codebase, I need to identify...") instead of emitting bare JSON, despite
SELECT_PROMPT's existing "ONLY valid JSON" instruction.ask_json()only triedjson.loads()on the whole trimmed response, so any surrounding prose made it discard an otherwise-usable answer and skip the run entirely — safely (no bad file written), but silently.What changed
SELECT_PROMPTandWRITE_PROMPT: explicitly forbid reasoning/commentary, spell out that a caller parses the response programmaticallyask_json(): if the full response isn't valid JSON on its own, scan for the first balanced{...}object anywhere in it and parse that instead of giving upSame file, byte-identical across all 11 repos before this change — opening the same fix everywhere rather than just on the repo that surfaced it.