Skip to content

fix(agent): publish record_fact as a facts-only contract - #410

Merged
BaaRai merged 6 commits into
BaaRai:mainfrom
tglemser:fix/record-fact-facts-only-schema
Nov 20, 2026
Merged

BaaRai merged 6 commits into
BaaRai:mainfrom
tglemser:fix/record-fact-facts-only-schema

Conversation

@tglemser

Copy link
Copy Markdown
Contributor

Fixes #409.

Problem

The record_fact schema advertises every fact field twice: at the top level (single-fact shape) and inside facts[] (batch shape); the handler decides which shape a call is. A model that fills every declared property sends the single fact inside facts[] and a copy, or a placeholder ("", [], "x"), of each field beside it. Since #282 such a call is correctly refused, but the refusal text ("summary" is not allowed at the top level in batch mode) leads the model to send a dummy value next, so the same refusal repeats until the intent ends.

Observed with a model over an OpenAI Responses compatible endpoint (reasoning effort high): 209 of 212 record_fact calls in two tasks were refused; 41 carried an exact copy of facts[0] at the top level, 50 a copy with empty asset lists, 118 a dummy top-level summary. One intent repeated the refused call 156 times (7.3 M input tokens). The handler is right; the published schema is the cause.

Reproduction

On main (ab40834), call record_fact the way such a model does:

{"summary":"TLS 1.0 is offered on port 8443","confidence":"observed","source":"tool_output","asset_ids":[],
 "facts":[{"summary":"TLS 1.0 is offered on port 8443","confidence":"observed","source":"tool_output"}]}

Result: invalid record_fact input: field "summary" is not allowed at the top level in batch mode; only intent_id and facts are accepted; no facts were written. The schema the model received declares summary, detail, evidence, confidence, source, asset_ids, tested_asset_ids at the top level with "Required on every fact" wording, so from the model's side the call matched the schema.

Change

  • Published schema is facts-only: {"facts":[{...}]} with required: ["facts"] and additionalProperties: false; every fact field lives on the items. The description names that one shape. RepairRecordFactSchema republishes stored and cached schemas as facts-only and keeps the annotations (description/default) of facts and of its known item fields; the top-level fact fields of the earlier contract are dropped from the published schema.
  • Handler contract of fix: repair record_fact summary contracts and asset aliases #282 unchanged: unknown or mis-cased fields, invalid or foreign asset ids, conflicting alias lists, missing summaries, empty batches and any top-level field beside facts (other than the legacy intent_id) are refused by name before anything is written. The refusal now also names the accepted shape: top-level field(s) "summary" are not allowed beside facts (1 item(s)): record_fact takes only {"facts":[...]}; put every field of a fact inside its facts item and send nothing else at the top level. The legacy single fact and the legacy batch with a top-level intent_id are still accepted.
  • Default intent: an item without intent_id is linked to the intent the worker is running (SetOwnerNode), as report_finding already does; a planner ToolSet (owner = task root) writes an unlinked fact as before.
  • Harness schema check: record_fact opts out of norma's ValidateInput (Spec.RawInput, present in v0.4.11) because required: ["facts"] would otherwise make the SDK reject the legacy single shape with a generic message before the handler sees it. The catalog decorator (overriddenTool) forwards AcceptsRawInput(), which the embedded interface alone would hide. The handler validates every shape itself and answers with a Result.
  • Existing databases: seedFactInputSchema repairs the stored schema on every start (compare-and-swap, as before). A one-time refresh (tool_schema_refresh_record_fact_facts_only) replaces the stored tool description and the facts/item descriptions that referred to the top level when they equal the previous code defaults; operator edits stay and are named in the log. seedFactSourceSchema now adds the source enum on the items.

Not in this PR: the loop guard that ends an intent after two identical record_fact refusals (ready as a separate branch, independent of this change).

Tests

The new tests describe behaviour main does not have (the schema is not facts-only, AcceptsRawInput is false, the refusal wording differs, items without intent_id stay unlinked); the existing record_fact tests were adapted where they asserted the top-level fields.

  • TestRecordFactSchemaIsFactsOnly, TestRecordFactStoredContractRepair, TestRecordFactInputSchemaUpgrade, TestRecordFactSourceSchemaUpgrade, TestWireTools: published schema, repair of stored/edited/cached schemas, source enum on the items, RawInput behind the catalog decorator.
  • TestRecordFactStrictFields, TestRecordFactBatchTopLevelConflicts, TestRecordFactRefusesTopLevelCopiesBesideFacts, TestRecordFactSummaryContract: the fix: repair record_fact summary contracts and asset aliases #282 contract, including identical copies, empty placeholders and dummies beside facts, all refused by name with no writes.
  • TestRecordFactDefaultsToRunningIntent: running-intent default, legacy top-level intent_id, per-item intent_id, planner ToolSet.
  • TestRecordFactShapesReplay: synthetic replay of the call shapes (published, legacy single, legacy batch, copies, placeholders, dummies, conflicts) against the handler in the test database: one node per item, yields edge to the intent, asset anchors, nothing written on refusal.
  • TestRefreshRecordFactContract: one-time refresh from the previous default texts, operator-edited description kept, second start changes nothing.

Validation on an isolated PostgreSQL 16 (own database per package), Go 1.26:

gofmt -l ./agent ./server ./db   clean
go vet ./agent ./server ./db     clean
go test -p 1 -count=1 ./agent ./server ./db
ok  github.com/Autumn-27/artex/agent    6.316s
ok  github.com/Autumn-27/artex/server  64.584s
ok  github.com/Autumn-27/artex/db      29.528s

The change has been running on a local installation with the model described above: 79 model calls and 7 record_fact calls in three verification tasks, 0 refusals; before the change the same installation saw 209 refusals in 212 calls.

🤖 Generated with Claude Code

The schema the model sees for record_fact declared every fact field twice:
at the top level (single-fact shape) and inside facts[] (batch shape), with
the handler deciding which shape a call is. A model that fills every
declared property sends the single fact inside facts[] and a copy, or a
placeholder such as "", [] or "x", of each field beside it; since PR BaaRai#282
every such call is refused by name, and the refusal text ("summary is not
allowed at the top level") leads the model to send a dummy value next, so
the same refusal repeats until the intent ends. Observed with a model over
the OpenAI Responses API: 209 of 212 record_fact calls of two tasks were
refused, 156 of them identical in one intent.

The published schema is now {facts:[{...}]} and nothing else (required:
facts, additionalProperties false). The handler keeps the strict contract
of PR BaaRai#282: unknown or mis-cased fields, invalid or foreign asset ids,
missing summaries, empty batches and any top-level field beside facts
(other than the legacy intent_id) are refused before anything is written,
with an error that names the fields and the one accepted shape. It still
accepts the legacy single fact and the legacy batch with a top-level
intent_id. An item without intent_id is linked to the intent the worker is
running (SetOwnerNode), as report_finding already does.

record_fact opts out of the harness schema check (norma Spec.RawInput) so
the stricter published schema does not reject the legacy shapes before the
handler sees them; the catalog decorator forwards the opt-out.
RepairRecordFactSchema republishes stored and cached schemas as facts-only
and keeps the annotations of facts and its items. A one-time startup
refresh replaces the stored description and the texts that referred to the
top level when they equal the previous defaults; operator edits stay and
are logged. seedFactSourceSchema adds the source enum on the items.

Tests: the handler contract (TestRecordFactSummaryContract,
TestRecordFactStrictFields, TestRecordFactBatchTopLevelConflicts,
TestRecordFactRefusesTopLevelCopiesBesideFacts), the published schema and
its repair (TestRecordFactSchemaIsFactsOnly, TestRecordFactStoredContractRepair,
TestRecordFactInputSchemaUpgrade, TestRecordFactSourceSchemaUpgrade,
TestWireTools), the running-intent default, the one-time refresh
(TestRefreshRecordFactContract) and a synthetic replay of the call shapes
(TestRecordFactShapesReplay).

Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
record_fact opts out of the harness schema check (norma Spec.RawInput) and
validates its own input. The harness finds the opt-out with a type
assertion on the tool it is handed, and that tool is a wrapper: catalog
tools are metered (meteredTool), and guardPanic wraps every tool last
(guardedTool). Both embed CoreTool, which hides AcceptsRawInput, so the
opt-out was lost and a malformed call was answered by the SDK ("invalid
tool input: facts: expected array") instead of by the handler.

guardedTool and meteredTool now forward AcceptsRawInput, as overriddenTool
already does.

Test: TestFix410AcceptsRawInputSurvivesWrappers covers each wrapper and
record_fact through wireTools, AugmentTools and harness.Query.
… fails

refreshRecordFactContract wrote its one-time flag in a defer, so a start on
which GetTool, the schema decode or the UPDATE failed still spent the flag.
No later start retried and the database kept the earlier description.

The flag is now written once the outcome is known: after the UPDATE
succeeded (also when it changed no row), or when the row is not the
built-in one. A failed read, decode or UPDATE leaves it unset and the next
start retries.

Test: TestFix410RefreshFlagNotSetOnFailure (an UPDATE refused by a trigger,
and a stored schema that does not decode; the next start refreshes the
text and sets the flag).
… defaults

The releases v0.4.0 to v0.6.0 seed a record_fact description without the source bullet and their own facts property description. Neither was in the list of known defaults, so the one-time refresh kept both texts on a database seeded by a release and logged the description as an operator edit.

Both released texts are now known defaults, taken byte for byte from agent/tools.go at those tags, and the log line says the description does not match a known default.
@BaaRai
BaaRai merged commit 40a4358 into BaaRai:main Nov 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

record_fact: published schema makes models fill the fact fields beside facts[], every call refused

2 participants