Restore extract.ai web search and schema guidance - #1148
Merged
Conversation
pyarrow moved to requirements-full.txt as an optional dependency, but scripts/container_smoke.py imports it directly to validate NumPy/ Pandas/PyArrow wheel interop in the slim, compiler-free test image. Install it alongside the core package so the smoke check still passes.
mborodii-prog
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Restores the
extract.aiResponses API enhancements from #1142 after their revert in #1147, on a clean branch from currentmain.web_search_sourcesprovenanceextract.airecipe schema hintsrecord_examplesfrom field-levelexamplesinstructionswhile retainingmessagesas a compatibility aliasBehavior and API impact
web_search: trueenables the Responses web-search tool and always adds the reservedweb_search_sourcesoutput column. Each row receives a deduplicated list of{title, url}objects or an empty list.record_examplesaccepts complete input/output examples with optionalnameandnotes.examplesretains scalar/list compatibility and also accepts pairedinput/outputexamples with optionalnameandnotes.instructionsaccepts a string or list of strings. Existingmessagescallers remain supported as an alias; supplying both names is rejected.Compatibility
Web search remains disabled by default. Existing recipes that use
messages, top-levelexamples, or output-only field examples continue to work. The automatic provenance column is added only when web search is enabled.Validation
1557 passed, 112 deselectedusing the repository local/offline pytest configuration74 passedacross focused extraction, definition, cache, recipe, and schema testsextract.aicontract was inspectedpip check, andgit diff --checkpassedThe first broad local run encountered two failures from ignored SQLite databases left by an earlier run. After removing only those generated test artifacts, the clean suite passed.
Risk and rollback
The main behavioral risk is provider response-shape variation for web-search provenance. Unit tests cover source extraction, deduplication, empty-source behavior, retries, cache behavior, recipe output, and malformed JSON. Rollback is a normal revert of this PR; web search is opt-in and does not affect existing recipes by default.