Repository navigation
fix(agent): publish record_fact as a facts-only contract - #410
Merged
Merged
Conversation
The schema the model sees for record_fact declared every fact field twice: at the top level (single-fact shape) and inside facts[] (batch shape), with the handler deciding which shape a call is. A model that fills every declared property sends the single fact inside facts[] and a copy, or a placeholder such as "", [] or "x", of each field beside it; since PR BaaRai#282 every such call is refused by name, and the refusal text ("summary is not allowed at the top level") leads the model to send a dummy value next, so the same refusal repeats until the intent ends. Observed with a model over the OpenAI Responses API: 209 of 212 record_fact calls of two tasks were refused, 156 of them identical in one intent. The published schema is now {facts:[{...}]} and nothing else (required: facts, additionalProperties false). The handler keeps the strict contract of PR BaaRai#282: unknown or mis-cased fields, invalid or foreign asset ids, missing summaries, empty batches and any top-level field beside facts (other than the legacy intent_id) are refused before anything is written, with an error that names the fields and the one accepted shape. It still accepts the legacy single fact and the legacy batch with a top-level intent_id. An item without intent_id is linked to the intent the worker is running (SetOwnerNode), as report_finding already does. record_fact opts out of the harness schema check (norma Spec.RawInput) so the stricter published schema does not reject the legacy shapes before the handler sees them; the catalog decorator forwards the opt-out. RepairRecordFactSchema republishes stored and cached schemas as facts-only and keeps the annotations of facts and its items. A one-time startup refresh replaces the stored description and the texts that referred to the top level when they equal the previous defaults; operator edits stay and are logged. seedFactSourceSchema adds the source enum on the items. Tests: the handler contract (TestRecordFactSummaryContract, TestRecordFactStrictFields, TestRecordFactBatchTopLevelConflicts, TestRecordFactRefusesTopLevelCopiesBesideFacts), the published schema and its repair (TestRecordFactSchemaIsFactsOnly, TestRecordFactStoredContractRepair, TestRecordFactInputSchemaUpgrade, TestRecordFactSourceSchemaUpgrade, TestWireTools), the running-intent default, the one-time refresh (TestRefreshRecordFactContract) and a synthetic replay of the call shapes (TestRecordFactShapesReplay). Co-Authored-By: Claude Mythos 5.1 <noreply@anthropic.com>
record_fact opts out of the harness schema check (norma Spec.RawInput) and
validates its own input. The harness finds the opt-out with a type
assertion on the tool it is handed, and that tool is a wrapper: catalog
tools are metered (meteredTool), and guardPanic wraps every tool last
(guardedTool). Both embed CoreTool, which hides AcceptsRawInput, so the
opt-out was lost and a malformed call was answered by the SDK ("invalid
tool input: facts: expected array") instead of by the handler.
guardedTool and meteredTool now forward AcceptsRawInput, as overriddenTool
already does.
Test: TestFix410AcceptsRawInputSurvivesWrappers covers each wrapper and
record_fact through wireTools, AugmentTools and harness.Query.
… fails refreshRecordFactContract wrote its one-time flag in a defer, so a start on which GetTool, the schema decode or the UPDATE failed still spent the flag. No later start retried and the database kept the earlier description. The flag is now written once the outcome is known: after the UPDATE succeeded (also when it changed no row), or when the row is not the built-in one. A failed read, decode or UPDATE leaves it unset and the next start retries. Test: TestFix410RefreshFlagNotSetOnFailure (an UPDATE refused by a trigger, and a stored schema that does not decode; the next start refreshes the text and sets the flag).
… defaults The releases v0.4.0 to v0.6.0 seed a record_fact description without the source bullet and their own facts property description. Neither was in the list of known defaults, so the one-time refresh kept both texts on a database seeded by a release and logged the description as an operator edit. Both released texts are now known defaults, taken byte for byte from agent/tools.go at those tags, and the log line says the description does not match a known default.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #409.
Problem
The
record_factschema advertises every fact field twice: at the top level (single-fact shape) and insidefacts[](batch shape); the handler decides which shape a call is. A model that fills every declared property sends the single fact insidefacts[]and a copy, or a placeholder ("",[],"x"), of each field beside it. Since #282 such a call is correctly refused, but the refusal text ("summary" is not allowed at the top level in batch mode) leads the model to send a dummy value next, so the same refusal repeats until the intent ends.Observed with a model over an OpenAI Responses compatible endpoint (reasoning effort high): 209 of 212
record_factcalls in two tasks were refused; 41 carried an exact copy offacts[0]at the top level, 50 a copy with empty asset lists, 118 a dummy top-levelsummary. One intent repeated the refused call 156 times (7.3 M input tokens). The handler is right; the published schema is the cause.Reproduction
On
main(ab40834), callrecord_factthe way such a model does:{"summary":"TLS 1.0 is offered on port 8443","confidence":"observed","source":"tool_output","asset_ids":[], "facts":[{"summary":"TLS 1.0 is offered on port 8443","confidence":"observed","source":"tool_output"}]}Result:
invalid record_fact input: field "summary" is not allowed at the top level in batch mode; only intent_id and facts are accepted; no facts were written. The schema the model received declaressummary,detail,evidence,confidence,source,asset_ids,tested_asset_idsat the top level with "Required on every fact" wording, so from the model's side the call matched the schema.Change
{"facts":[{...}]}withrequired: ["facts"]andadditionalProperties: false; every fact field lives on the items. The description names that one shape.RepairRecordFactSchemarepublishes stored and cached schemas as facts-only and keeps the annotations (description/default) offactsand of its known item fields; the top-level fact fields of the earlier contract are dropped from the published schema.facts(other than the legacyintent_id) are refused by name before anything is written. The refusal now also names the accepted shape:top-level field(s) "summary" are not allowed beside facts (1 item(s)): record_fact takes only {"facts":[...]}; put every field of a fact inside its facts item and send nothing else at the top level. The legacy single fact and the legacy batch with a top-levelintent_idare still accepted.intent_idis linked to the intent the worker is running (SetOwnerNode), asreport_findingalready does; a planner ToolSet (owner = task root) writes an unlinked fact as before.record_factopts out of norma'sValidateInput(Spec.RawInput, present in v0.4.11) becauserequired: ["facts"]would otherwise make the SDK reject the legacy single shape with a generic message before the handler sees it. The catalog decorator (overriddenTool) forwardsAcceptsRawInput(), which the embedded interface alone would hide. The handler validates every shape itself and answers with aResult.seedFactInputSchemarepairs the stored schema on every start (compare-and-swap, as before). A one-time refresh (tool_schema_refresh_record_fact_facts_only) replaces the stored tool description and thefacts/item descriptions that referred to the top level when they equal the previous code defaults; operator edits stay and are named in the log.seedFactSourceSchemanow adds thesourceenum on the items.Not in this PR: the loop guard that ends an intent after two identical
record_factrefusals (ready as a separate branch, independent of this change).Tests
The new tests describe behaviour
maindoes not have (the schema is not facts-only,AcceptsRawInputis false, the refusal wording differs, items withoutintent_idstay unlinked); the existing record_fact tests were adapted where they asserted the top-level fields.TestRecordFactSchemaIsFactsOnly,TestRecordFactStoredContractRepair,TestRecordFactInputSchemaUpgrade,TestRecordFactSourceSchemaUpgrade,TestWireTools: published schema, repair of stored/edited/cached schemas, source enum on the items,RawInputbehind the catalog decorator.TestRecordFactStrictFields,TestRecordFactBatchTopLevelConflicts,TestRecordFactRefusesTopLevelCopiesBesideFacts,TestRecordFactSummaryContract: the fix: repair record_fact summary contracts and asset aliases #282 contract, including identical copies, empty placeholders and dummies besidefacts, all refused by name with no writes.TestRecordFactDefaultsToRunningIntent: running-intent default, legacy top-levelintent_id, per-itemintent_id, planner ToolSet.TestRecordFactShapesReplay: synthetic replay of the call shapes (published, legacy single, legacy batch, copies, placeholders, dummies, conflicts) against the handler in the test database: one node per item,yieldsedge to the intent, asset anchors, nothing written on refusal.TestRefreshRecordFactContract: one-time refresh from the previous default texts, operator-edited description kept, second start changes nothing.Validation on an isolated PostgreSQL 16 (own database per package), Go 1.26:
The change has been running on a local installation with the model described above: 79 model calls and 7
record_factcalls in three verification tasks, 0 refusals; before the change the same installation saw 209 refusals in 212 calls.🤖 Generated with Claude Code