Skip to content

feat(tools): slim the ToC in the agent's build_datasheet result (0.47.0) - #41

Merged
qye-inf merged 4 commits into
mainfrom
feat/slim-agent-manifest
Nov 9, 2026
Merged

qye-inf merged 4 commits into
mainfrom
feat/slim-agent-manifest

Conversation

@qye-inf

@qye-inf qye-inf commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Why

The build_datasheet result stays in the agent's conversation and is re-sent on every later turn. On large documents its ToC was most of the agent's context: 270K chars on PIC16F887, 200K on ATmega328, 192K on ESP32 TRM. In a probe, a lookup on ESP32 TRM that read one 2.7K-char page peaked near 80K prompt tokens. Section titles were only 13% of the ToC node data.

What changes

The build_datasheet tool handler (tools/defs.py:_agent_toc) sends the agent a slimmed copy of each ToC node:

Dropped Share of node data Why
breadcrumb 31% restates the nesting (equal on all 3,734 corpus nodes)
cross_references 17.5% text is in the pages read; resolved section refs are findable by number in the titles
footnote_markers 8.4% bare marker strings; footnote text is in the pages read
has_tables 5.9% restates table_count > 0
node_id 5.4% no tool takes one
level 3.2% nesting carries the hierarchy

table_count is omitted when zero. Every node stays, with title, start_page, end_page, nodes, boilerplate_category, continued_tables, summary. It is a denylist, so a future enrichment reaches the agent by default.

Unchanged: DatasheetTools.get_artifact_manifest() (Python API), the ToC JSON artifact, and search_text breadcrumbs. The tool description says breadcrumbs, cross-references and footnote markers are left out and sends the agent to the page text (not to json_path, which no tool reads and which may not resolve where the agent runs).

Manifest size: PIC16F887 270K -> 79K chars, ESP32 TRM 192K -> 54K, RP2040 124K -> 32K, MSP430F5529 55K -> 18K.

Evidence

Before/after A/B: 20 questions over 16 documents (15 answered from a footnote or a cross-referenced table, 5 controls), Sonnet 4.6, GPT-5.1 and Qwen 3.6 27B, 3 runs per question per arm, 360 runs:

Model Correct before -> after Turns Prompt tokens / question
Sonnet 4.6 60/60 -> 60/60 3.67 -> 3.50 129K -> 48K
GPT-5.1 55/60 -> 56/60 4.00 -> 3.93 121K -> 46K
Qwen 3.6 27B 60/60 -> 59/60 5.42 -> 5.38 202K -> 82K

175/180 in both arms. Every miss was read by hand; none involves the dropped fields. GPT misses the same questions in both arms (answer from priors, a value the text layer puts on the wrong line, register defaults read instead of the per-pin table, a curve read instead of a table), and Qwen's one miss is that same register misread. Grading normalises exotic spaces, hyphens and markdown; one question was graded by hand. An earlier 10-question, 2-model run agreed.

Token economy (docs/token-economy.md, regenerated without LLM credentials): the script now prices the manifest the agent is sent, via the handler's own _agent_manifest(). Median first answer (map + one section) 9,637 -> 4,293 tokens, i.e. 3.1x -> 7.3x cheaper than the document; PIC16F887's manifest 92K -> 27K tokens. The further-answer median moved 625 -> 760 because sections grew in earlier releases, not here.

Tests

  • New tests in tests/test_defs.py: slim node shape through the handler, Python API keeps the full ToC, unknown fields pass through without mutating the cached tree, description checked against the denylist.
  • Full local suite including vendor-PDF tests: 1405 passed, 1 skipped.

Version 0.47.0, CHANGELOG updated.

The manifest stays in the conversation and is re-sent on every later turn,
and on large documents its ToC was most of the agent's context: 270K chars on
PIC16F887, 192K on ESP32 TRM, with titles only 13% of the node data.

The build_datasheet handler now drops breadcrumb, level, node_id, has_tables,
cross_references and footnote_markers from each node, and table_count when it
is zero. breadcrumb and has_tables restate other fields, no tool takes a
node_id, and level differs from the nesting only when an outline skips a
level. Cross-references and footnote markers are in the page text, and a
resolved section reference is still findable by its number in the titles.
The description points the agent at json_path for all three. A denylist, so
new enrichments still reach the agent. get_artifact_manifest() and the ToC
JSON are unchanged.

A/B with Sonnet 4.6 and GPT-5.1 on footnote and cross-reference questions:
no measurable accuracy or turn change, prompt tokens per question down
2.7-2.9x.
…ed ToC fields

PR review: no tool reads the ToC JSON and its path may not resolve on the
agent's side (the same wall the preamble hits), so the build_datasheet
description now says breadcrumbs, cross-references and footnote markers are
left out and to read them in the page text. search_text no longer calls its
breadcrumb "the ToC breadcrumb", which the agent's ToC no longer has.

Tests: the slim-shape test now asserts the unslimmed tree had every dropped
field, so it cannot pass vacuously; the continued_tables fixture uses the
real list[str] shape; the description test follows the new wording.
A/B re-run at 360 runs (Sonnet 4.6, GPT-5.1, Qwen 3.6 27B; 20 questions over
16 documents, 3 runs per question per arm): 175/180 correct in both arms,
every miss read by hand and none involving the dropped fields; prompt tokens
per question down 2.5-2.7x.

scripts/token_economy.py priced get_artifact_manifest(), which is no longer
what the agent is sent. It now goes through _agent_manifest(), the function
the build_datasheet handler calls, so the study cannot drift from the tool.
Regenerated without LLM credentials and with default scans: median first
answer 9,637 -> 4,293 tokens (3.1x -> 7.3x cheaper than the document).
@qye-inf
qye-inf merged commit 3d7063b into main Nov 9, 2026
7 checks passed
@qye-inf
qye-inf deleted the feat/slim-agent-manifest branch November 9, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant