Skip to content

record_fact: published schema makes models fill the fact fields beside facts[], every call refused #409

Closed

Description

@tglemser

Summary

The record_fact tool schema advertises every fact field twice: at the top level (single-fact shape) and inside facts[] (batch shape), and lets the handler decide which shape a call is. A model that fills every declared property sends the single fact inside facts[] and a copy, or a placeholder, of each field beside it. Since #282 such a call is refused (field "summary" is not allowed at the top level in batch mode; only intent_id and facts are accepted), and the refusal text makes the model send a dummy value ("summary":"x") next, so the same refusal repeats until the intent ends. The schema text is the cause; the handler is correct.

Version

Upstream main at ab40834 (also c2ddd1c), record_fact after #282. Provider: an OpenAI Responses compatible endpoint through a model profile; the schema is passed as the function parameters without strict, so required and additionalProperties are instructions to the model, not guarantees.

What I ran

Two ordinary tasks (a port scan of a lab host, then a vulnerability review of the found services) with a worker/planner setup, reasoning effort high.

What happened

  • 212 record_fact calls, 209 refused. 41 calls carried an exact copy of facts[0] at the top level, 50 a copy with empty asset lists, 118 a dummy top-level summary such as "x" or "placeholder" (sent after the first refusal told the model that summary was not allowed there). Only 3 single-shape calls succeeded.
  • One intent called record_fact 156 times in a row with the same refused shape (7.3 M input tokens) before the intent ended; the planner redispatched the same work.
  • The published schema at that point, abbreviated:
{"type":"object","additionalProperties":false,"properties":{
  "summary":{"type":"string","description":"Required on every fact: ..."},
  "detail":{...},"evidence":{...},"confidence":{...},"source":{...},"asset_ids":{...},"tested_asset_ids":{...},
  "intent_id":{...},
  "facts":{"type":"array","minItems":1,"items":{"required":["summary"],"properties":{"summary":...,...}}}}}

A typical refused call (abbreviated):

{"summary":"<same text as facts[0].summary>","detail":"...","evidence":"...","confidence":"observed","source":"tool_output","asset_ids":[],
 "facts":[{"summary":"...","detail":"...","evidence":"...","confidence":"observed","source":"tool_output","intent_id":26}],
 "intent_id":26}

What I expected

One unambiguous call shape in the published schema, so the model has nothing to fill beside facts[]; the strict handler contract of #282 unchanged.

Proposal

  • Publish record_fact as {"facts":[{...}]} only (required: ["facts"], additionalProperties: false), keep the handler's strict refusal of anything else at the top level (with a message that names the fields and the one accepted shape), keep accepting the legacy single fact and the legacy top-level intent_id, repair stored and cached schemas to the new shape, and refresh the stored tool description once. An item without intent_id should default to the intent the worker is running, as report_finding already does.
  • Separately: stop a worker intent after two consecutive identical record_fact input errors instead of letting it loop (follow-up PR).

I have both changes ready with tests against an isolated PostgreSQL; the schema change comes first.

Activity

  1. github-actions commented on Nov 20, 2026

    @github-actions

    Thanks — this reads as a clear bug in the published record_fact schema (duplicate top-level fields next to facts[]), with a good behavioural trace. To turn it into a reproducible report, please add: (1) the exact model/provider and profile name plus reasoning effort, since you mention an OpenAI Responses compatible endpoint and tool-call reliability varies a lot by model; (2) where the schema actually comes from — the code path that builds it and the stored/cached row or description it is read from, as the directory summary here does not pinpoint that file; (3) a minimal reproduction: one worker prompt, the tool-call log showing the refused input and the refusal text returned by the handler, and the config used; (4) how the 212-call/209-refusal tally was collected, so the loop counts can be checked. Your proposal to publish only {"facts":[...]} with required:["facts"], keep the strict top-level refusal and legacy acceptance, and separately cap consecutive identical input errors looks sensible; the schema change and the loop guard should be reviewed separately.


    🤖 automated reply from the repo bot. A human maintainer reads everything. Ping me in any comment with @artex-bot.

  2. tglemser commented on Nov 20, 2026

    @tglemser
    ContributorAuthor

    @artex-bot Thanks, here are the four items.

    1. Model and profile. Provider format openai-responses (the built-in OpenAI Responses wire format, agent/provider.go → norma llm/openai_responses.go), base URL https://api.openai.com/v1, model gpt-6-sol, reasoning effort high, streaming on, no thinking/extra options, one profile for all agents (default profile, no per-agent bindings, no failover). The schema is sent as the function tool's parameters without strict (buildBody in norma copies InputSchema verbatim), so required/additionalProperties are instructions to the model, not guarantees. I have not reproduced it with other models; the pattern "fill every declared property" may well be model-specific, which is why the fix removes the duplicate declaration instead of tuning text.

    2. Where the schema comes from.

    • Built in code: agent/fact_input.go recordFactSchema() (top level) and factItemSchema() (the facts[] items); agent/tools.go recordFact() registers description + schema.
    • Stored once: agent/toolcatalog.go BuiltinToolSeeds() → db.SeedTool inserts the tools row key='record_fact' on first start (insert-only, operator-editable afterwards).
    • Served to the model: agent/toolcatalog.go ToolResolve/DecorateTool replace description and schema with the stored row, passing the stored schema through RepairRecordFactSchema (agent/fact_input.go); at startup server/fact_input_schema.go seedFactInputSchema repairs the stored row the same way (compare-and-swap).
    • On main all three places declare summary, detail, evidence, confidence, source, asset_ids, tested_asset_ids at the top level and on the items; the top-level summary description reads "Required on every fact: … Batch items do not inherit summary from the top level", which from the model's side says "also fill it here". The raw request body recorded by the LLM recorder confirms the model received exactly that schema (34 tools, record_fact with 9 top-level properties, no strict).

    3. Minimal reproduction.

    • Config: a fresh task with the profile above, default worker agent and tools, goal (German in my run, translated): "Scan host X completely over TCP ports 1–65535, document every open port with service and version evidence, and check the identified services for verifiable vulnerabilities." Nothing special; a plain port-scan goal is enough.
    • Tool-call log, first refused call of that intent (content shortened, addresses replaced):
    {"asset_ids":[],"confidence":"","detail":"","evidence":"",
     "facts":[{"asset_ids":[2],"confidence":"observed","detail":"Full TCP connect scan (-p1-65535) showed 5 open ports …","evidence":"nmap -sT -Pn -p1-65535 … → 'Not shown: 65530 closed tcp ports'","intent_id":26,"source":"tool_output","summary":"Full TCP scan of 203.0.113.11: five open services …"}],
     "intent_id":26,"source":"","summary":"","tested_asset_ids":[]}

    Handler result (is_error=true): invalid record_fact input: field "confidence" is not allowed at the top level in batch mode; only intent_id and facts are accepted; no facts were written

    • Next call of the same intent: identical, but with "summary":"x" (and later "placeholder", "test") at the top level; result: … field "summary" is not allowed at the top level …. This repeated 156 times in one intent and 86 times in another (which tops out at the intent's turn limit; the planner then redispatched the same work).
    • Unit-level reproduction on main without a model: call the tool from agent package tests with the input above (callGuardTool(tools.recordFact(), …)) → the same refusal. fix(agent): publish record_fact as a facts-only contract #410 adds that as TestRecordFactRefusesTopLevelCopiesBesideFacts and a synthetic replay of the shapes (TestRecordFactShapesReplay).

    4. How the tally was collected. Read-only SQL on the installation's PostgreSQL, activity table (one row per tool call and per tool result, joined on tool_use_id):

    select u.exploration_id, count(*) calls, count(*) filter (where r.is_error) refused
    from activity u join activity r on r.tool_use_id=u.tool_use_id and r.kind='tool_result'
    where u.kind='tool_use' and u.tool='record_fact' group by 1;

    → two tasks: 56 calls / 53 refused and 156 / 156 (the 3 accepted ones were single-shape calls). The per-shape split (41 exact copies of facts[0], 50 copies with empty asset lists, 118 placeholders/divergent) came from comparing each top-level field of u.detail with facts[0] in a script; the per-intent loop counts from group by u.node_id. Token figures are the llm_usage sums per task.

    If it helps the review I can attach the two anonymized tool-call sequences (input + handler result per call) as a file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions