Skip to content

Make structured output repair errors actionable - #709

Merged
brynary merged 3 commits into
mainfrom
fix/actionable-structured-output-errors
Aug 1, 2026
Merged

Make structured output repair errors actionable#709
brynary merged 3 commits into
mainfrom
fix/actionable-structured-output-errors

Conversation

@brynary

@brynary brynary commented Jul 31, 2026

Copy link
Copy Markdown
Member

Summary

  • preserve structured schema validation details instead of flattening them to strings
  • identify the exact JSON Pointer for required, type, enum, and additional-property errors
  • include bounded schema context in repair prompts
  • call out validation problems that remain unchanged across repair attempts
  • use the same actionable errors for command-stage schema failures

Why

A repair prompt could previously say that rationale was required without saying that the missing field belonged at /findings/0/rationale. A model could add the field at the wrong level and repeat the same failure. The new prompt identifies the exact target and explicitly flags an unchanged error on the next attempt.

Same-name misplaced-field detection is intentionally out of scope.

Testing

  • ulimit -n 4096 && cargo nextest run -p fabro-workflow — 1,323 passed, 31 skipped
  • cargo +nightly-2026-04-14 clippy -p fabro-workflow --all-targets -- -D warnings
  • cargo +nightly-2026-04-14 fmt --check --all
  • git diff --check

Copilot AI review requested due to automatic review settings July 31, 2026 20:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves structured-output repair UX in fabro-workflow by preserving JSON Schema validation structure (instance/schema pointers + keyword/detail) so repair prompts can point to the exact JSON Pointer that needs correction and can detect repeated, unchanged validation failures across repair attempts.

Changes:

  • Replaces flattened string validation errors with structured SchemaValidationIssue data (instance/schema/evaluation paths + keyword + typed details) and renders actionable messages including JSON Pointers and bounded schema fragments.
  • Extends repair prompting to (optionally) reference the previous failure and call out at least one unchanged schema issue across repair attempts.
  • Reuses the same actionable rendered validation errors for command-stage output_schema failures and adds tests verifying pointer reporting and repeated-error messaging.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
lib/components/fabro-workflow/src/handler/structured_output.rs Introduces structured schema-validation issue modeling + richer rendering and repair messaging; adds targeted tests for pointer accuracy and repeated-issue prompts.
lib/components/fabro-workflow/src/handler/llm/api.rs Threads previous validation error through LLM repair loops and adds an integration test asserting repeated-error callout behavior.
lib/components/fabro-workflow/src/handler/command.rs Updates command-stage schema failure reporting to use the new rendered, schema-aware validation messages.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread lib/components/fabro-workflow/src/handler/structured_output.rs Outdated
Follow-up review of the repair-error work. Behavior is the same or better;
the machinery is smaller.

Fixes a false "unchanged from your previous repair" nudge. same_problem_as
fell through to `_ => true`, so any two non-Required issues at the same
instance path, schema path and keyword compared equal. A model that removed
one unexpected property and added another was told it had changed nothing.
SchemaValidationIssue already derives PartialEq, so the 17-line comparison
is now `previous.contains(issue)`.

Drops the hand-written Type and Enum rendering. jsonschema already renders
both, and its messages name the offending value, which the hand-written
ones did not. Also switches masked() back to to_string(): masking replaced
the bad value with a placeholder, working against the goal of an actionable
message, and buys no privacy since the full response is already in the
prompt.

Resolves the schema fragment when the issue is captured rather than
threading Option<&OutputSchemaKind> through rendering. That reverts the
command.rs change and drops the test-only messages() shim. The fragment is
now attached only to Other, where it adds information; for required, type,
enum and additionalProperties it just repeated the prose.

Also: caps the model-controlled unexpected-property list so a wide object
cannot turn the repair prompt into megabytes; drops evaluation_path, which
was dead except under $ref, where it printed a pointer that does not
resolve; drops the keyword field, already named by the schema path; and
records the previous error only after the agent session accepted the
repair, since failover rebuilds the session from the original prompt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 1, 2026 13:04
@brynary

brynary commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Ran a reuse / quality / efficiency review pass over the branch and applied the findings. Net -160/+106 on top of the original commit; behavior is the same or better.

Bug fixed. same_problem_as fell through to _ => true, so any two non-Required issues sharing an instance path, schema path and keyword compared equal. A model that removed one unexpected property and added a different one was told "at least one validation problem is unchanged from your previous repair" — the opposite of what happened. SchemaValidationIssue already derives PartialEq, so the 17-line comparison is now previous.contains(issue). Added a test that locks this in.

Messages got more actionable, not less. The hand-written Type and Enum rendering is gone — jsonschema already renders both, and its messages name the offending value where the hand-written ones only named its type. At JSON Pointer /line: "85" is not of type "integer" instead of expected JSON type "integer", but got string. Relatedly, error.masked() is back to error.to_string(): masking substitutes a placeholder for the bad value, which works against the whole point of this PR, and buys no privacy since the model's full response is already in the prompt.

The schema fragment is resolved when the issue is captured, not threaded through rendering as Option<&OutputSchemaKind>. That reverts the command.rs change entirely and drops the test-only messages() shim. The fragment is now attached only to Other, where it adds something; for required, type, enum and additionalProperties it repeated the prose verbatim, doubling the length of every repair prompt.

Also removed evaluation_path (dead except under $ref, where the pointer it printed doesn't resolve against the schema), the keyword field (already the last segment of the schema path), and expected_json_types / format_expected_types / pointer_code. Capped the model-controlled unexpected-property list at 10 — .take(5) bounded the number of issues but not the size of one, so a wide object under additionalProperties: false could put a megabyte of model output into the repair prompt. bounded_json now truncates in one bounded pass. And in the agent loop, the previous error is recorded only after the session accepts the repair; if the send fails and failover rebuilds from the original prompt, the model never saw the repair message.

All 1324 fabro-workflow tests pass; clippy and rustfmt clean.

Two follow-ups I did not take here, both pre-existing:

  • On exhausted repairs, api.rs returns only output schema validation failed after N repair attempt(s) and drops the error detail this PR adds. command.rs includes it. Two tests pin the current string, so it's a separate change. handler/agent.rs:376 and handler/prompt.rs:194 are worse — they use if let Ok(..) else, which never binds the error at all.
  • bounded_json is the sixth copy of truncate-to-N-chars-with-ellipsis in the workspace (fabro-util/src/text.rs, check_report.rs, run_title_generation.rs, run_title.rs, cli/.../events.rs, cli/.../list.rs — the last has a latent underflow panic). Worth one shared helper in fabro-util, but that's a cross-crate change.

@brynary
brynary marked this pull request as ready for review August 1, 2026 13:04

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

Claude Code Review is paused for this repository. To reconnect it, an admin of this repository's GitHub organization (or the account owner, for personal repositories) who can also manage your Claude organization's Code Review settings needs to re-link GitHub in Code Review settings. This is a one-time step.

Tip: disable this comment in your organization's Code Review settings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Addresses Copilot review feedback on the repeated-failure check.

serde_json runs with preserve_order, and jsonschema builds the
additionalProperties `unexpected` list by walking the instance in
document order. So the same leftover keys emitted in a different order
produced a different Vec and compared as a different problem, which
suppressed the "unchanged from your previous repair" nudge.

Sorting at capture also makes the MAX_UNEXPECTED_PROPERTIES truncation
pick the same subset every time instead of an order-dependent one, and
stabilizes the rendered message.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 1, 2026 13:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@brynary
brynary merged commit f97156d into main Aug 1, 2026
15 checks passed
@brynary
brynary deleted the fix/actionable-structured-output-errors branch August 1, 2026 13:22
brynary added a commit that referenced this pull request Aug 1, 2026
One conflict, in StructuredOutputError::repair_message. main (#709)
added a `previous_error` parameter and richer validation-error
rendering; this branch had replaced the inline expectation match with
OutputSchemaKind::expectation().

Resolved by keeping both: main's new signature and section assembly,
calling schema.expectation() for the expectation text. The method
already supersedes main's inline match and carries this branch's intent
of embedding the resolved JSON Schema instead of naming it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants