Rank object spans ahead of array spans in JSON extraction - #32
Merged
Conversation
extract_json ranked balanced spans by length alone, so a prose bracket
longer than the answer still hijacked it: a citation list like
[101, 205, 309, ...] outranked the real {"supported": false} verdict,
resurrecting the substituted-answer bug the module records as fixed.
Downstream, the verifier false-rejected correct claims ("reviewer reply
unparseable — failing closed"), claim extraction dropped every claim,
and three such planner replies stopped a GovernedLoop with
planning_failed.
Prose brackets are square while every shipped caller — the verifier's
verdict, claim extraction, the planner's Subgraph — expects a top-level
object, so balanced {...} spans now rank longest-first ahead of [...]
spans longest-first. The whole-reply and fenced candidates still come
first, so fenced and unfenced top-level arrays keep working, and junk
still returns None so fail-closed is unchanged.
Fixes #21
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
extract_jsonranked balanced{...}/[...]spans purely by length, so a bracketed fragment in the model's prose still hijacked the answer whenever it was longer than the real JSON:This resurrected the substituted-answer bug the module records as fixed: the verifier false-rejected correct, anchored claims ("reviewer reply unparseable — failing closed"), claim extraction silently dropped every claim, and three such planner replies stopped a
GovernedLoopwithplanning_failed.How
Prose brackets are square (
[1],[lines 3-5], citation lists) while every shipped caller — the verifier's verdict, claim extraction, the planner'sSubgraph— expects a top-level object, so_balanced_spansnow ranks object spans longest-first ahead of array spans longest-first. The candidate order is otherwise unchanged: whole reply first, then the fence, so fenced and unfenced top-level arrays keep working, and genuinely unparseable replies still returnNone— the fail-closed contract is untouched.grapharc/runtime/parsing.py: one-line ranking change (sorted(spans, key=lambda span: (span[0] != "{", -len(span)))) plus a_balanced_spansdocstring update explaining why length alone was not sufficient.tests/test_parsing.py: regression tests — the repro above for both"supported": falseand"supported": true(fails onmain, which returns the citation list), and an array-answer reply with a non-parsing{}-like prose fragment to pin that arrays stay reachable.README.md: the "Closed" parsing note described the old "longest parse wins" ranking; it now states the object-before-array ordering.Verification
pytest -qfully green (whole suite)ruff check .cleanmainand pass with the fix;test_verifier_accepts_a_fenced_reply, the existing hijack tests, and both top-level-array cases pass unchanged.Fixes #21
🤖 Generated with Claude Code