Skip to content

fix: expose output schemas to agents - #696

Merged
brynary merged 4 commits into
mainfrom
fix/expose-agent-output-schema
Aug 1, 2026
Merged

fix: expose output schemas to agents#696
brynary merged 4 commits into
mainfrom
fix/expose-agent-output-schema

Conversation

@brynary

@brynary brynary commented Jul 31, 2026

Copy link
Copy Markdown
Member

Summary

  • Add the resolved output contract to API-backed agent task instructions.
  • Limit the contract to the final response so intermediate tool use remains unrestricted.
  • Repeat the exact JSON Schema in structured-output repair turns.
  • Add regression coverage and document the behavior.

Why

One-shot prompt nodes pass the schema to the provider as a response format. Multi-turn agent nodes validated the final response but did not show the schema to the model. This could cause an agent to ask the user which object shape to return.

Applying a provider response format to every agent turn would interfere with tool use on some providers and is unsupported by others. The task-level final-output contract exposes the schema without changing intermediate agent turns.

Testing

  • cargo nextest run -p fabro-workflow (1,318 passed; 31 skipped)
  • cargo +nightly-2026-04-14 clippy --workspace --all-targets -- -D warnings
  • cargo +nightly-2026-04-14 fmt --check --all

Copilot AI review requested due to automatic review settings July 31, 2026 13:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the workflow engine’s structured-output behavior so API-backed agent tasks are explicitly shown the resolved output contract (routing or JSON Schema) in their task instructions, while keeping intermediate agent turns/tool calls unrestricted. It also ensures schema details are repeated during structured-output repair turns, adds regression assertions, and documents the behavior.

Changes:

  • Add the resolved output_schema contract to API-backed agent prompt instructions, scoped to the final response only.
  • Refactor structured-output messaging to reuse a shared expectation builder and include the full JSON Schema in repair prompts.
  • Strengthen regression tests and document how agent nodes receive and apply output contracts.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
lib/components/fabro-workflow/src/handler/structured_output.rs Refactors expectation text generation and adds agent prompt augmentation that embeds the final-output contract (including full JSON Schema).
lib/components/fabro-workflow/src/handler/llm/api.rs Tightens a repair-turn regression test to assert the JSON Schema content is actually present in the request payload.
lib/components/fabro-workflow/src/handler/agent.rs Parses output_schema once per execution, injects the contract into the agent prompt, and updates tests to assert prompt contents.
docs/public/reference/dot-language.mdx Documents that API-backed agent nodes receive the resolved output contract and that it applies only to the final response.
docs/public/agents/outputs.mdx Adds documentation clarifying agent-node task instructions include the resolved contract without restricting intermediate tool use.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Both contract helpers dispatched on OutputSchemaKind, so they belong on
the type. Moves expectation() and agent_prompt() into an impl block and
drops the free functions.

Splits the combined agent test: assertions no longer run inside the
backend's run(), where a failure surfaces as a panic from execute().
Adds coverage for the Routing branch of the contract, which was
previously untested.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 1, 2026 13:42
@brynary

brynary commented Aug 1, 2026

Copy link
Copy Markdown
Member Author

Ran a simplification pass over the branch.

Changed

  • agent_prompt_with_output_schema and output_expectation both dispatched on OutputSchemaKind, so they moved onto the type as agent_prompt() and expectation(). Call site is now schema.agent_prompt(&prompt).
  • Split the combined agent test. The five assert! calls were inside CustomOutputBackend::run(), where a failure surfaces as a panic from execute(). Prompt assertions now live in a dedicated test using the PromptCapturingBackend pattern already in that file; codergen_handler_custom_output_schema_updates_output_context_key goes back to asserting only the outcome.
  • Added coverage for the Routing branch of the contract, which had none — every output_schema="routing" agent node gets this new prompt text too.

Checked, no change needed

  • parse_node_output_schema is called in both agent.rs and api.rs, recompiling the validator twice per node execution. This is pre-existing on main, not introduced here — the PR only moved the agent.rs call earlier.
  • The no-schema path is already guarded: codergen_handler_no_preamble_when_empty asserts exact prompt equality, so an unconditional append would fail it.

Worth a second look before merge

  • Event::Prompt.text now carries the contract, so it shows up in the UI prompt panel, progress.jsonl, and the stages/<id>/prompt.md checkpoint artifact. That matches what was actually sent, but agent nodes will now display Fabro-authored text that prompt nodes do not.
  • Both docs pages say "API-backed agent nodes". The code appends the contract above backend dispatch; that is accurate today only because acp.rs:523 rejects output_schema outright.

cargo nextest run -p fabro-workflow — 1321 passed. Clippy and fmt clean.

@brynary
brynary marked this pull request as ready for review August 1, 2026 13:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

One conflict, in StructuredOutputError::repair_message. main (#709)
added a `previous_error` parameter and richer validation-error
rendering; this branch had replaced the inline expectation match with
OutputSchemaKind::expectation().

Resolved by keeping both: main's new signature and section assembly,
calling schema.expectation() for the expectation text. The method
already supersedes main's inline match and carries this branch's intent
of embedding the resolved JSON Schema instead of naming it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 1, 2026 13:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings August 1, 2026 14:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

@brynary
brynary merged commit 143e660 into main Aug 1, 2026
15 checks passed
@brynary
brynary deleted the fix/expose-agent-output-schema branch August 1, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants