New Search with AI Mode (serpAPI)

[ebhills]

Feature description

Add a new search.ai_mode wrangle backed by SerpAPI's Google AI Mode API.

The wrangle should combine the two steps we currently perform with search.find_links and search.retrieve_link_content: one AI Mode request should find relevant pages, return source/result records, and return the answer/content synthesized from those sources.

The default use case is industrial-product research. A query will normally contain a manufacturer name, one or more possible part codes, and optionally a description. The default prompt should search for authoritative product, manufacturer, supplier, and distributor pages and return confirmed product content and available pricing without guessing.

Draft implementation: branch search.ai_mode, feature commit 2b66af7.

Problem it solves

The current product-research flow requires separate search and URL-retrieval wrangles. That adds latency, provider calls, recipe complexity, and another failure boundary. Google AI Mode can search and synthesize cited content in one request, but its native response differs from the stable Wrangles search/retrieval contracts.

The new wrangle must therefore normalize AI Mode into a predictable Wrangles shape instead of exposing or heuristically flattening SerpAPI's provider-specific response.

Current branch review

The branch is a useful design spike and already establishes the intended layers:

• recipe/DataFrame wrapper in wrangles/recipe_wrangles/search.py;
• core Python orchestration in wrangles/search.py;
• SerpAPI client support in wrangles/clients/serp_api.py;
• readable dual-output formatting; and
• initial recipe tests.

The branch is not yet ready to merge. The implementation must address these gaps:

1. The normalized output does not yet match either existing contract. It introduces product_details, misc, content_like_results, validation, and an always-present raw_response rather than combining the stable search_results and extracted_content shapes.
2. The current one-level flattening classifies provider metadata, text_blocks, reconstructed_markdown, and shopping_results as product details. It does not actually extract product attributes, and a documented AI Mode response can produce an empty pricing object even when shopping results contain prices.
3. The focused branch tests currently produce 2 passes and 3 failures because wrangles.search resolves to wrangles.clients.serp_api, not the new core wrangles/search.py module. The direct Python API is therefore not exposed through the normal package surface.
4. Tests mock a hand-built post-normalization payload instead of a representative SerpAPI AI Mode response containing text_blocks, reconstructed_markdown, references, and shopping/product sections.
5. n_results is sent to AI Mode as num, although num is not documented for this API. n_results should be a local normalization/output limit.
6. google_domain is not documented for Google AI Mode. Do not expose it for this wrangle unless a focused live verification proves and documents the contract.
7. The draft sends the query verbatim and does not yet supply the required default industrial-product prompt.
8. Changes to classic search.find_links locale validation are mixed into the branch. The new wrangle must not change classic-search behavior unless that change is separately justified and regression-tested.

Proposed solution

1. Public API and recipe contract

Provide both supported entry paths:

wrangles.search.ai_mode(...)

wrangles:
  - search.ai_mode:
      queries: Product Search Query
      id: ID
      output:
        - AI Mode Results
        - AI Mode Text
      n_results: 10
      country: us
      language: en

Required behavior:

• queries follows the existing search.find_links convention: it names one query column or a list of query columns.
• Each cell may contain a scalar query or a list of queries.
• A scalar direct-Python query returns one dictionary; a list returns a list in input order.
• id identifies the input row and is copied to normalized source records as input_row_id.
• One output column returns structured dictionaries.
• Exactly two output columns return [structured_results, readable_text].
• Multiple query columns retain the existing one-input/one-output-column behavior.
• Missing or blank queries preserve row alignment and produce an empty cell result rather than a provider call.
• client defaults to serpapi; the API key may be supplied explicitly or via SERPAPI_API_KEY.
• Keep wrangles.clients.serp_api.SerpApiWranglesClient available. Fix or replace the current package alias so wrangles.search.ai_mode is actually callable, with compatibility coverage for the existing public surface.

2. Default industrial-product prompt

The query cell contains the product identity/evidence supplied by the caller. Typical content is:

Manufacturer: SKF
Potential part codes: 6205-2RS, 6205 2RS
Description: deep groove ball bearing

When prompt is omitted, prepend a maintained default instruction equivalent to:

Find authoritative manufacturer, product, supplier, and distributor pages for this industrial product. Confirm the manufacturer and exact part number where possible. Summarize the product description, important specifications/attributes, and available price, currency, vendor, availability, and quantity basis. Prefer exact identifier evidence, distinguish confirmed facts from inference, cite the supporting sources, and leave unknown values unknown rather than guessing.

Requirements:

• Add an explicit optional prompt parameter.
• The custom prompt replaces the default instruction but still receives the row's query/product evidence.
• Preserve manufacturer names and part-code punctuation in the request.
• The caller remains responsible for constructing the query column from manufacturer, codes, and description; direct acceptance of three separately named product columns is out of scope for the first release.
• The wrangle must remain usable for non-product research through a custom prompt.

3. Supported SerpAPI request parameters

First-release recipe parameters:

• queries, id, output, client, api_key, prompt, n_results, and threads;
• country as the friendly alias for SerpAPI gl;
• language as the friendly alias for SerpAPI hl;
• location or uule, but never both;
• device with desktop, tablet, or mobile; and
• no_cache.

Provider request requirements:

• Always send engine=google_ai_mode and structured JSON output.
• Apply n_results after normalization/deduplication; do not send it as undocumented num.
• Validate n_results >= 1 and threads >= 1.
• Do not silently pass unsupported recipe properties through **kwargs.
• Keep google_domain, async, continuable, subsequent_request_token, image_url, zero_trace, provider Markdown/HTML output, and json_restrictor out of scope for the first release. Add any of them later with an explicit contract and tests.

4. Default output contract

Each row output remains a list with one payload per query. Each query payload combines the familiar top-level shapes of find_links and retrieve_link_content:

{
  "search_metadata": {
    "query_index": 1,
    "query": "Manufacturer: SKF ...",
    "search_type": "ai_mode",
    "search_id": "provider search id",
    "status": "Success",
    "search_date": null,
    "response_time": null,
    "json_endpoint": null,
    "google_url": null,
    "language": "en",
    "country": "us",
    "location": null
  },
  "status": "Success",
  "error": null,
  "search_results": [
    {
      "input_row_id": "row id",
      "query_index": 1,
      "google_rank": 1,
      "result_type": "reference",
      "title": "Source title",
      "link": "https://example.com/product",
      "source": "Example",
      "snippet": "Supporting source snippet",
      "pricing": {
        "price": 12.5,
        "currency": "USD",
        "availability": null,
        "vendor": "Example"
      }
    }
  ],
  "extracted_content": {
    "answer_markdown": "The synthesized AI Mode answer with citations...",
    "text_blocks": []
  }
}

Contract rules:

• search_metadata and search_results intentionally mirror search.find_links.
• status, error, and extracted_content intentionally mirror the useful fields from search.retrieve_link_content.
• The source URLs are represented by search_results; do not invent a single retrieved_url for a multi-source AI answer.
• Do not include fabricated product_details, misc, or validation keys.
• A missing price is valid and does not make an otherwise successful response invalid.
• Add include_raw_response: false. When true, append raw_response; do not duplicate the full provider payload by default.
• The structured shape must be stable for success, empty, and error responses.
• All output must be JSON-serializable and preserve query/input order.

5. SerpAPI response mapping

Normalize documented AI Mode fields deliberately:

• reconstructed_markdown -> extracted_content.answer_markdown;
• text_blocks -> extracted_content.text_blocks;
• references -> search_results with result_type: reference;
• source-bearing quick_results, shopping_results, and inline_products -> the same search_results contract with an appropriate result_type;
• structured shopping prices -> the existing nested pricing shape;
• provider search_metadata.status and top-level error -> the normalized status/error contract.

Additional mapping requirements:

• Clean links and snippets using the existing web helpers.
• Deduplicate source records by normalized link plus title while preserving first-seen order.
• Assign a stable one-based google_rank after deduplication.
• Preserve provider/source provenance through result_type.
• Apply n_results only after deduplication.
• Do not infer arbitrary product fields by flattening unrelated top-level provider objects.

6. Empty, partial, and error behavior

• Blank/null/NaN queries: make no request and return the existing row-aligned empty result.
• Provider success with no references: status: Success, empty search_results, and the available extracted_content.
• Provider success with references but no price: preserve the sources and answer; pricing is empty for those records.
• Provider error/exception: return status: Failure, a useful error, empty search_results, and null/empty extracted_content.
• One failed query must not shift, drop, or corrupt results for other queries in the same row or batch.
• Do not convert provider errors into apparently valid product details.

7. Readable second output

The optional text output must include:

• query number and product query;
• success/failure status and error when present;
• the reconstructed AI answer;
• numbered source records with title, source, link, snippet, and structured pricing when available.

Update the formatter by dispatching on the normalized payload contract. Existing find_links and retrieve_link_content text output must remain unchanged.

8. Compatibility and scope

• Do not change search.find_links or search.retrieve_link_content behavior, schemas, output shapes, locale handling, or tests as part of this feature unless a separate, directly related compatibility fix is required.
• Preserve recipe scalar/list conventions, query ordering, row IDs, dual outputs, and DataFrame row alignment.
• Generate and validate the recipe schema for search.ai_mode.
• Keep the SerpAPI provider client behind the existing client factory.
• Add user-facing recipe documentation and release notes.
• Rebase/recreate the implementation on a freshly fetched origin/main before the first review-ready PR.

Use case

Given an input row containing manufacturer WESTFALIA, possible part code DN65, and description union nut, search.ai_mode should make one product-oriented AI Mode request and return:

• the synthesized answer/content;
• the cited manufacturer/product/supplier/distributor pages in the standard search-result shape;
• structured price data when SerpAPI supplies it;
• row and query identity for downstream scoring; and
• a readable text representation when requested.

The wrangle should reduce or replace the common two-step find_links -> retrieve_link_content recipe, while leaving those existing wrangles available for workflows that require explicit URL-by-URL retrieval.

Acceptance criteria

• search.ai_mode is callable through both the recipe/DataFrame path and wrangles.search.ai_mode.
• The direct Python API returns a dictionary for a scalar query and an ordered list for list input.
• Recipe inputs support scalar and list-valued cells, multiple rows, multiple queries per row, row IDs, one structured output, and the two-output structured/text form.
• The maintained default prompt implements the industrial-product search behavior; a custom prompt supports non-product use cases.
• A representative query containing manufacturer, potential part codes, and optional description is included in tests and documentation.
• Only documented first-release AI Mode request parameters are exposed and validated.
• n_results is enforced locally after deduplication and is not sent as num.
• The normalized payload uses search_metadata, status, error, search_results, and extracted_content as documented above.
• SerpAPI references, quick_results, shopping_results, and inline_products are normalized into stable source records without fabricating product details.
• reconstructed_markdown and text_blocks are preserved as extracted content.
• Structured shopping prices use the existing nested pricing shape.
• Empty, no-source, no-price, provider-error, and partial-batch-failure cases preserve row/query alignment.
• include_raw_response defaults to false and adds the raw provider payload only when requested.
• The optional text output includes the answer, sources, pricing, status, and errors.
• Unit tests use realistic documented SerpAPI-shaped fixtures and do not require network access.
• Client mapper tests, direct Python tests, recipe/DataFrame tests, schema-generation tests, and formatter regression tests are included.
• An optional live smoke test may run only when SERPAPI_API_KEY is available and must report separately from mocked/focused validation.
• Existing focused tests for search.find_links and search.retrieve_link_content remain green.
• The current 3 focused branch-test failures are resolved; no test patches the wrong public module.
• git diff --check is clean and new tracked text files follow the repository line-ending policy.
• Documentation identifies supported parameters, the default product prompt, output schema, error behavior, costs/caching considerations, and an end-to-end recipe example.

[ebhills]

Next steps

1. PR assignee: Rebuild the feature from current origin/main, implement issue 1073’s output and provider-mapping contract, and make the focused tests green.
2. AI agent: @codex address issue New Search with AI Mode (serpAPI) #1073 using the documented output contract, add realistic SerpAPI-shaped client/recipe/direct-API regression tests, and report focused versus live checks separately.
3. Reviewer: Verify the normalized payload, public Python API, unchanged classic-search behavior, and passing checks before approving.

[chatgpt-codex-connector]

To use Codex here, create an environment for this repo.

[ebhills]

@thomasstvr - I copied/pasted my local Codex 'next steps' as a comment and that kicked off Codex here in Copilot. I guess we'll see what it does with such a comprehensive issue description.