Make structured output repair errors actionable - #709
Conversation
There was a problem hiding this comment.
Pull request overview
Improves structured-output repair UX in fabro-workflow by preserving JSON Schema validation structure (instance/schema pointers + keyword/detail) so repair prompts can point to the exact JSON Pointer that needs correction and can detect repeated, unchanged validation failures across repair attempts.
Changes:
- Replaces flattened string validation errors with structured
SchemaValidationIssuedata (instance/schema/evaluation paths + keyword + typed details) and renders actionable messages including JSON Pointers and bounded schema fragments. - Extends repair prompting to (optionally) reference the previous failure and call out at least one unchanged schema issue across repair attempts.
- Reuses the same actionable rendered validation errors for command-stage
output_schemafailures and adds tests verifying pointer reporting and repeated-error messaging.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| lib/components/fabro-workflow/src/handler/structured_output.rs | Introduces structured schema-validation issue modeling + richer rendering and repair messaging; adds targeted tests for pointer accuracy and repeated-issue prompts. |
| lib/components/fabro-workflow/src/handler/llm/api.rs | Threads previous validation error through LLM repair loops and adds an integration test asserting repeated-error callout behavior. |
| lib/components/fabro-workflow/src/handler/command.rs | Updates command-stage schema failure reporting to use the new rendered, schema-aware validation messages. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Follow-up review of the repair-error work. Behavior is the same or better; the machinery is smaller. Fixes a false "unchanged from your previous repair" nudge. same_problem_as fell through to `_ => true`, so any two non-Required issues at the same instance path, schema path and keyword compared equal. A model that removed one unexpected property and added another was told it had changed nothing. SchemaValidationIssue already derives PartialEq, so the 17-line comparison is now `previous.contains(issue)`. Drops the hand-written Type and Enum rendering. jsonschema already renders both, and its messages name the offending value, which the hand-written ones did not. Also switches masked() back to to_string(): masking replaced the bad value with a placeholder, working against the goal of an actionable message, and buys no privacy since the full response is already in the prompt. Resolves the schema fragment when the issue is captured rather than threading Option<&OutputSchemaKind> through rendering. That reverts the command.rs change and drops the test-only messages() shim. The fragment is now attached only to Other, where it adds information; for required, type, enum and additionalProperties it just repeated the prose. Also: caps the model-controlled unexpected-property list so a wide object cannot turn the repair prompt into megabytes; drops evaluation_path, which was dead except under $ref, where it printed a pointer that does not resolve; drops the keyword field, already named by the schema path; and records the previous error only after the agent session accepted the repair, since failover rebuilds the session from the original prompt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Ran a reuse / quality / efficiency review pass over the branch and applied the findings. Net -160/+106 on top of the original commit; behavior is the same or better. Bug fixed. Messages got more actionable, not less. The hand-written The schema fragment is resolved when the issue is captured, not threaded through rendering as Also removed All 1324 Two follow-ups I did not take here, both pre-existing:
|
There was a problem hiding this comment.
Claude Code Review
Claude Code Review is paused for this repository. To reconnect it, an admin of this repository's GitHub organization (or the account owner, for personal repositories) who can also manage your Claude organization's Code Review settings needs to re-link GitHub in Code Review settings. This is a one-time step.
Tip: disable this comment in your organization's Code Review settings.
Addresses Copilot review feedback on the repeated-failure check. serde_json runs with preserve_order, and jsonschema builds the additionalProperties `unexpected` list by walking the instance in document order. So the same leftover keys emitted in a different order produced a different Vec and compared as a different problem, which suppressed the "unchanged from your previous repair" nudge. Sorting at capture also makes the MAX_UNEXPECTED_PROPERTIES truncation pick the same subset every time instead of an order-dependent one, and stabilizes the rendered message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One conflict, in StructuredOutputError::repair_message. main (#709) added a `previous_error` parameter and richer validation-error rendering; this branch had replaced the inline expectation match with OutputSchemaKind::expectation(). Resolved by keeping both: main's new signature and section assembly, calling schema.expectation() for the expectation text. The method already supersedes main's inline match and carries this branch's intent of embedding the resolved JSON Schema instead of naming it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Summary
Why
A repair prompt could previously say that
rationalewas required without saying that the missing field belonged at/findings/0/rationale. A model could add the field at the wrong level and repeat the same failure. The new prompt identifies the exact target and explicitly flags an unchanged error on the next attempt.Same-name misplaced-field detection is intentionally out of scope.
Testing
ulimit -n 4096 && cargo nextest run -p fabro-workflow— 1,323 passed, 31 skippedcargo +nightly-2026-04-14 clippy -p fabro-workflow --all-targets -- -D warningscargo +nightly-2026-04-14 fmt --check --allgit diff --check