Skip to content

Rank object spans ahead of array spans in JSON extraction - #32

Merged
Shashankss1205 merged 1 commit into
mainfrom
fix/issue-21
Jul 31, 2026
Merged

Rank object spans ahead of array spans in JSON extraction#32
Shashankss1205 merged 1 commit into
mainfrom
fix/issue-21

Conversation

@Shashankss1205

Copy link
Copy Markdown
Collaborator

What

extract_json ranked balanced {...}/[...] spans purely by length, so a bracketed fragment in the model's prose still hijacked the answer whenever it was longer than the real JSON:

extract_json('Verified against source lines [101, 205, 309, 412, 518, 622, 733, 848]: {"supported": false}')
# returned [101, 205, 309, 412, 518, 622, 733, 848] — now returns {'supported': False}

This resurrected the substituted-answer bug the module records as fixed: the verifier false-rejected correct, anchored claims ("reviewer reply unparseable — failing closed"), claim extraction silently dropped every claim, and three such planner replies stopped a GovernedLoop with planning_failed.

How

Prose brackets are square ([1], [lines 3-5], citation lists) while every shipped caller — the verifier's verdict, claim extraction, the planner's Subgraph — expects a top-level object, so _balanced_spans now ranks object spans longest-first ahead of array spans longest-first. The candidate order is otherwise unchanged: whole reply first, then the fence, so fenced and unfenced top-level arrays keep working, and genuinely unparseable replies still return None — the fail-closed contract is untouched.

  • grapharc/runtime/parsing.py: one-line ranking change (sorted(spans, key=lambda span: (span[0] != "{", -len(span)))) plus a _balanced_spans docstring update explaining why length alone was not sufficient.
  • tests/test_parsing.py: regression tests — the repro above for both "supported": false and "supported": true (fails on main, which returns the citation list), and an array-answer reply with a non-parsing {}-like prose fragment to pin that arrays stay reachable.
  • README.md: the "Closed" parsing note described the old "longest parse wins" ranking; it now states the object-before-array ordering.

Verification

  • pytest -q fully green (whole suite)
  • ruff check . clean
  • The new regression tests fail on current main and pass with the fix; test_verifier_accepts_a_fenced_reply, the existing hijack tests, and both top-level-array cases pass unchanged.

Fixes #21

🤖 Generated with Claude Code

extract_json ranked balanced spans by length alone, so a prose bracket
longer than the answer still hijacked it: a citation list like
[101, 205, 309, ...] outranked the real {"supported": false} verdict,
resurrecting the substituted-answer bug the module records as fixed.
Downstream, the verifier false-rejected correct claims ("reviewer reply
unparseable — failing closed"), claim extraction dropped every claim,
and three such planner replies stopped a GovernedLoop with
planning_failed.

Prose brackets are square while every shipped caller — the verifier's
verdict, claim extraction, the planner's Subgraph — expects a top-level
object, so balanced {...} spans now rank longest-first ahead of [...]
spans longest-first. The whole-reply and fenced candidates still come
first, so fenced and unfenced top-level arrays keep working, and junk
still returns None so fail-closed is unchanged.

Fixes #21

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Shashankss1205
Shashankss1205 merged commit 633129f into main Jul 31, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

runtime: a long prose bracket is substituted for the model's JSON answer

1 participant