Repository navigation
feat(tools): slim the ToC in the agent's build_datasheet result (0.47.0) - #41
Merged
Merged
Conversation
The manifest stays in the conversation and is re-sent on every later turn, and on large documents its ToC was most of the agent's context: 270K chars on PIC16F887, 192K on ESP32 TRM, with titles only 13% of the node data. The build_datasheet handler now drops breadcrumb, level, node_id, has_tables, cross_references and footnote_markers from each node, and table_count when it is zero. breadcrumb and has_tables restate other fields, no tool takes a node_id, and level differs from the nesting only when an outline skips a level. Cross-references and footnote markers are in the page text, and a resolved section reference is still findable by its number in the titles. The description points the agent at json_path for all three. A denylist, so new enrichments still reach the agent. get_artifact_manifest() and the ToC JSON are unchanged. A/B with Sonnet 4.6 and GPT-5.1 on footnote and cross-reference questions: no measurable accuracy or turn change, prompt tokens per question down 2.7-2.9x.
…ed ToC fields PR review: no tool reads the ToC JSON and its path may not resolve on the agent's side (the same wall the preamble hits), so the build_datasheet description now says breadcrumbs, cross-references and footnote markers are left out and to read them in the page text. search_text no longer calls its breadcrumb "the ToC breadcrumb", which the agent's ToC no longer has. Tests: the slim-shape test now asserts the unslimmed tree had every dropped field, so it cannot pass vacuously; the continued_tables fixture uses the real list[str] shape; the description test follows the new wording.
A/B re-run at 360 runs (Sonnet 4.6, GPT-5.1, Qwen 3.6 27B; 20 questions over 16 documents, 3 runs per question per arm): 175/180 correct in both arms, every miss read by hand and none involving the dropped fields; prompt tokens per question down 2.5-2.7x. scripts/token_economy.py priced get_artifact_manifest(), which is no longer what the agent is sent. It now goes through _agent_manifest(), the function the build_datasheet handler calls, so the study cannot drift from the tool. Regenerated without LLM credentials and with default scans: median first answer 9,637 -> 4,293 tokens (3.1x -> 7.3x cheaper than the document).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
build_datasheetresult stays in the agent's conversation and is re-sent on every later turn. On large documents its ToC was most of the agent's context: 270K chars on PIC16F887, 200K on ATmega328, 192K on ESP32 TRM. In a probe, a lookup on ESP32 TRM that read one 2.7K-char page peaked near 80K prompt tokens. Section titles were only 13% of the ToC node data.What changes
The
build_datasheettool handler (tools/defs.py:_agent_toc) sends the agent a slimmed copy of each ToC node:breadcrumbcross_referencesfootnote_markershas_tablestable_count > 0node_idleveltable_countis omitted when zero. Every node stays, withtitle,start_page,end_page,nodes,boilerplate_category,continued_tables,summary. It is a denylist, so a future enrichment reaches the agent by default.Unchanged:
DatasheetTools.get_artifact_manifest()(Python API), the ToC JSON artifact, andsearch_textbreadcrumbs. The tool description says breadcrumbs, cross-references and footnote markers are left out and sends the agent to the page text (not tojson_path, which no tool reads and which may not resolve where the agent runs).Manifest size: PIC16F887 270K -> 79K chars, ESP32 TRM 192K -> 54K, RP2040 124K -> 32K, MSP430F5529 55K -> 18K.
Evidence
Before/after A/B: 20 questions over 16 documents (15 answered from a footnote or a cross-referenced table, 5 controls), Sonnet 4.6, GPT-5.1 and Qwen 3.6 27B, 3 runs per question per arm, 360 runs:
175/180 in both arms. Every miss was read by hand; none involves the dropped fields. GPT misses the same questions in both arms (answer from priors, a value the text layer puts on the wrong line, register defaults read instead of the per-pin table, a curve read instead of a table), and Qwen's one miss is that same register misread. Grading normalises exotic spaces, hyphens and markdown; one question was graded by hand. An earlier 10-question, 2-model run agreed.
Token economy (
docs/token-economy.md, regenerated without LLM credentials): the script now prices the manifest the agent is sent, via the handler's own_agent_manifest(). Median first answer (map + one section) 9,637 -> 4,293 tokens, i.e. 3.1x -> 7.3x cheaper than the document; PIC16F887's manifest 92K -> 27K tokens. The further-answer median moved 625 -> 760 because sections grew in earlier releases, not here.Tests
tests/test_defs.py: slim node shape through the handler, Python API keeps the full ToC, unknown fields pass through without mutating the cached tree, description checked against the denylist.Version 0.47.0, CHANGELOG updated.