Repository navigation
Description
Summary
The record_fact tool schema advertises every fact field twice: at the top level (single-fact shape) and inside facts[] (batch shape), and lets the handler decide which shape a call is. A model that fills every declared property sends the single fact inside facts[] and a copy, or a placeholder, of each field beside it. Since #282 such a call is refused (field "summary" is not allowed at the top level in batch mode; only intent_id and facts are accepted), and the refusal text makes the model send a dummy value ("summary":"x") next, so the same refusal repeats until the intent ends. The schema text is the cause; the handler is correct.
Version
Upstream main at ab40834 (also c2ddd1c), record_fact after #282. Provider: an OpenAI Responses compatible endpoint through a model profile; the schema is passed as the function parameters without strict, so required and additionalProperties are instructions to the model, not guarantees.
What I ran
Two ordinary tasks (a port scan of a lab host, then a vulnerability review of the found services) with a worker/planner setup, reasoning effort high.
What happened
- 212
record_factcalls, 209 refused. 41 calls carried an exact copy offacts[0]at the top level, 50 a copy with empty asset lists, 118 a dummy top-levelsummarysuch as"x"or"placeholder"(sent after the first refusal told the model thatsummarywas not allowed there). Only 3 single-shape calls succeeded. - One intent called
record_fact156 times in a row with the same refused shape (7.3 M input tokens) before the intent ended; the planner redispatched the same work. - The published schema at that point, abbreviated:
{"type":"object","additionalProperties":false,"properties":{
"summary":{"type":"string","description":"Required on every fact: ..."},
"detail":{...},"evidence":{...},"confidence":{...},"source":{...},"asset_ids":{...},"tested_asset_ids":{...},
"intent_id":{...},
"facts":{"type":"array","minItems":1,"items":{"required":["summary"],"properties":{"summary":...,...}}}}}A typical refused call (abbreviated):
{"summary":"<same text as facts[0].summary>","detail":"...","evidence":"...","confidence":"observed","source":"tool_output","asset_ids":[],
"facts":[{"summary":"...","detail":"...","evidence":"...","confidence":"observed","source":"tool_output","intent_id":26}],
"intent_id":26}What I expected
One unambiguous call shape in the published schema, so the model has nothing to fill beside facts[]; the strict handler contract of #282 unchanged.
Proposal
- Publish
record_factas{"facts":[{...}]}only (required: ["facts"],additionalProperties: false), keep the handler's strict refusal of anything else at the top level (with a message that names the fields and the one accepted shape), keep accepting the legacy single fact and the legacy top-levelintent_id, repair stored and cached schemas to the new shape, and refresh the stored tool description once. An item withoutintent_idshould default to the intent the worker is running, asreport_findingalready does. - Separately: stop a worker intent after two consecutive identical
record_factinput errors instead of letting it loop (follow-up PR).
I have both changes ready with tests against an isolated PostgreSQL; the schema change comes first.
Activity
github-actions commentedon Nov 20, 2026
Thanks — this reads as a clear bug in the published
record_factschema (duplicate top-level fields next tofacts[]), with a good behavioural trace. To turn it into a reproducible report, please add: (1) the exact model/provider and profile name plus reasoning effort, since you mention an OpenAI Responses compatible endpoint and tool-call reliability varies a lot by model; (2) where the schema actually comes from — the code path that builds it and the stored/cached row or description it is read from, as the directory summary here does not pinpoint that file; (3) a minimal reproduction: one worker prompt, the tool-call log showing the refused input and the refusal text returned by the handler, and the config used; (4) how the 212-call/209-refusal tally was collected, so the loop counts can be checked. Your proposal to publish only{"facts":[...]}withrequired:["facts"], keep the strict top-level refusal and legacy acceptance, and separately cap consecutive identical input errors looks sensible; the schema change and the loop guard should be reviewed separately.🤖 automated reply from the repo bot. A human maintainer reads everything. Ping me in any comment with
@artex-bot.tglemser commentedon Nov 20, 2026
@artex-bot Thanks, here are the four items.
1. Model and profile. Provider format
openai-responses(the built-in OpenAI Responses wire format,agent/provider.go→ normallm/openai_responses.go), base URLhttps://api.openai.com/v1, modelgpt-6-sol, reasoning efforthigh, streaming on, no thinking/extra options, one profile for all agents (default profile, no per-agent bindings, no failover). The schema is sent as the function tool'sparameterswithoutstrict(buildBodyin norma copiesInputSchemaverbatim), sorequired/additionalPropertiesare instructions to the model, not guarantees. I have not reproduced it with other models; the pattern "fill every declared property" may well be model-specific, which is why the fix removes the duplicate declaration instead of tuning text.2. Where the schema comes from.
agent/fact_input.gorecordFactSchema()(top level) andfactItemSchema()(thefacts[]items);agent/tools.gorecordFact()registers description + schema.agent/toolcatalog.goBuiltinToolSeeds()→db.SeedToolinserts thetoolsrowkey='record_fact'on first start (insert-only, operator-editable afterwards).agent/toolcatalog.goToolResolve/DecorateToolreplace description and schema with the stored row, passing the stored schema throughRepairRecordFactSchema(agent/fact_input.go); at startupserver/fact_input_schema.goseedFactInputSchemarepairs the stored row the same way (compare-and-swap).mainall three places declaresummary,detail,evidence,confidence,source,asset_ids,tested_asset_idsat the top level and on the items; the top-levelsummarydescription reads "Required on every fact: … Batch items do not inherit summary from the top level", which from the model's side says "also fill it here". The raw request body recorded by the LLM recorder confirms the model received exactly that schema (34 tools,record_factwith 9 top-level properties, nostrict).3. Minimal reproduction.
{"asset_ids":[],"confidence":"","detail":"","evidence":"", "facts":[{"asset_ids":[2],"confidence":"observed","detail":"Full TCP connect scan (-p1-65535) showed 5 open ports …","evidence":"nmap -sT -Pn -p1-65535 … → 'Not shown: 65530 closed tcp ports'","intent_id":26,"source":"tool_output","summary":"Full TCP scan of 203.0.113.11: five open services …"}], "intent_id":26,"source":"","summary":"","tested_asset_ids":[]}Handler result (
is_error=true):invalid record_fact input: field "confidence" is not allowed at the top level in batch mode; only intent_id and facts are accepted; no facts were written"summary":"x"(and later"placeholder","test") at the top level; result:… field "summary" is not allowed at the top level …. This repeated 156 times in one intent and 86 times in another (which tops out at the intent's turn limit; the planner then redispatched the same work).mainwithout a model: call the tool fromagentpackage tests with the input above (callGuardTool(tools.recordFact(), …)) → the same refusal. fix(agent): publish record_fact as a facts-only contract #410 adds that asTestRecordFactRefusesTopLevelCopiesBesideFactsand a synthetic replay of the shapes (TestRecordFactShapesReplay).4. How the tally was collected. Read-only SQL on the installation's PostgreSQL,
activitytable (one row per tool call and per tool result, joined ontool_use_id):→ two tasks: 56 calls / 53 refused and 156 / 156 (the 3 accepted ones were single-shape calls). The per-shape split (41 exact copies of
facts[0], 50 copies with empty asset lists, 118 placeholders/divergent) came from comparing each top-level field ofu.detailwithfacts[0]in a script; the per-intent loop counts fromgroup by u.node_id. Token figures are thellm_usagesums per task.If it helps the review I can attach the two anonymized tool-call sequences (input + handler result per call) as a file.