Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,35 @@

All notable changes to this project will be documented in this file.

## [0.47.0] - 2026-11-09

### Changed
- **Slimmer ToC in the agent's `build_datasheet` result.** Each ToC node the
agent receives now omits `breadcrumb`, `level`, `node_id`, `has_tables`,
`cross_references` and `footnote_markers`, and omits `table_count` when it is
zero. Every node is still there, with its title, page range and nesting,
along with `boilerplate_category`, `continued_tables`, `summary` and any field
not on that list. The manifest stays in the conversation and is re-sent on
every turn; on large documents it was most of the agent's context (270K
characters on PIC16F887, 192K on ESP32 TRM) and is now about 3.5x smaller.
On a before/after test of 360 runs (Sonnet 4.6, GPT-5.1, Qwen 3.6 27B; 20
questions, most answered from footnotes and cross-referenced tables),
accuracy was 175/180 in both arms, turns fell slightly, and prompt tokens
per question fell 2.5-2.7x.
- **Token economy re-measured** (`docs/token-economy.md`, README). The
script now prices the manifest the agent is actually sent, through the same
function the tool handler uses, rather than `get_artifact_manifest()`. The
median first answer (map plus one section) is now 4,293 tokens, 7.3x
cheaper than the document (was 9,637, 3.1x). The further-answer median
moved from 625 to 760 tokens because sections grew in earlier releases, not
in this one.
- Applies to the MCP server, `create_datasheet_tool_defs()` and the Agent SDK
server. `DatasheetTools.get_artifact_manifest()` and the ToC JSON artifact
are unchanged and keep every field; `search_text` hits still carry their
breadcrumb, and the `build_datasheet` description tells the agent that
breadcrumbs, cross-references and footnote markers are left out and to read
them in the page text.

## [0.46.0] - 2026-11-05

### Added
Expand Down
20 changes: 11 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,11 @@ credentials. Full results, per document, are in
| Median across the corpus | |
|---|---|
| Whole document in context | 33,973 tokens |
| First answer (map + one section) | 9,637 tokens |
| Every further answer about the same part | 625 tokens |
| First answer vs. the document | **3.1x** cheaper |
| Further answer vs. the document | **54x** cheaper |
| Build | ~6s cold, 0.01s warm (cached artifact) |
| First answer (map + one section) | 4,293 tokens |
| Every further answer about the same part | 760 tokens |
| First answer vs. the document | **7.3x** cheaper |
| Further answer vs. the document | **47x** cheaper |
| Build | ~5s cold, 0.01s warm (cached artifact) |

Each ratio is the **median of the per-document ratios**, so it is deliberately
not the quotient of the two token medians above: every document is priced on
Expand All @@ -76,10 +76,10 @@ an order of magnitude and a cold/warm ratio.

**The two ratios are different claims, and the first one is the weaker.** The
first answer pays for the enriched ToC as well as the section, and on a large
document that map is most of the cost -- for the 322-page PIC16F887 it is half
the document's tokens on its own. The compounding is in the second row: the
document that map is most of the cost -- for the 322-page PIC16F887 it is 27K
of a 28K-token first answer, though only 15% of the document. The compounding is in the second row: the
map is paid once, so the tenth question about a part costs a section alone.
That is 45x on a 31-page datasheet and 728x on a 784-page reference manual,
That is 39x on a 31-page datasheet and 518x on a 784-page reference manual,
which is the shape to expect -- the bigger the document, the more there is not
to read.

Expand Down Expand Up @@ -442,7 +442,9 @@ You can run the local MCP server directly from the repository. It exposes these
tools for the bound PDF source:

- `build_datasheet` - build and save the `.json` / `.txt` artifacts, and return
the manifest: source info, total pages, ToC quality, the full enriched ToC,
the manifest: source info, total pages, ToC quality, every ToC node (title,
page range, nonzero table count, boilerplate and continued-table flags --
breadcrumbs, cross-references and footnote markers stay in the `.json`),
and a bounded `figures` digest naming the pages that carry figure entries
(see "Figure indexing and captions"). For a PDF with no usable ToC the
manifest also carries a `hint` telling the agent to navigate by `search_text`
Expand Down
55 changes: 55 additions & 0 deletions docs/datasheetindex_architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -1529,6 +1529,61 @@ any one agent framework:
single pattern or a list and tags each hit with its section breadcrumb;
`get_section_text` returns a page range prefixed with a position header.

**The agent's ToC is slimmer than the artifact's.** The `build_datasheet`
handler (`tools/defs.py:_agent_toc`) drops `breadcrumb`, `level`, `node_id`,
`has_tables`, `cross_references` and `footnote_markers` from every node, and
`table_count` where it is zero; `DatasheetTools.get_artifact_manifest()` and
the ToC JSON keep them all. The manifest stays in the conversation and is
re-sent on every later turn, and on large documents it was most of the agent's
context: 270K chars on PIC16F887 and 192K on ESP32 TRM, where a probe lookup
that read one 2.7K-char page peaked near 80K prompt tokens. Titles were 13% of
the node data; the dropped fields were 72%. `breadcrumb` and `has_tables`
restate other fields (equal on all 3,734 corpus nodes), no tool takes a
`node_id`, and `level` differs from the nesting only when an outline skips a
level. The other two are a judgement: both are in the page text the agent
reads. Only section references are ever resolved (`target_node_id`, 988 of
2,248), and only to a section whose title starts with the referenced number,
so the target is still findable by title; table and figure references carry
only their label. The tool description says these fields are left out and
sends the agent to the page text -- not to `json_path`, which no tool reads
and which may not resolve where the agent runs. It is a denylist so that a
new enrichment reaches the agent unless someone decides it should not.

Measured before merging (2026-11-09): 20 questions across 16 documents, 15 of
them chosen so the answer sits in a footnote or a cross-referenced table, 3
models, 3 runs per question per arm -- 360 runs, 180 per arm, full manifest
(`main`) against slim (this change):

| Model | Correct before -> after | Turns | Prompt tokens / question |
|---|---|---|---|
| Sonnet 4.6 | 60/60 -> 60/60 | 3.67 -> 3.50 | 129K -> 48K |
| GPT-5.1 | 55/60 -> 56/60 | 4.00 -> 3.93 | 121K -> 46K |
| Qwen 3.6 27B | 60/60 -> 59/60 | 5.42 -> 5.38 | 202K -> 82K |

175/180 correct in both arms. Every miss was read by hand, and none involves
the dropped fields: GPT misses the same four questions in both arms (an
answer from priors instead of the note on the page, a value the text layer
puts on the wrong line, a register's generic defaults read instead of the
per-pin table, a curve read instead of a table), and Qwen's one miss is that
same register misread. Turns fell slightly in every model, and prompt tokens
per question fell 2.5-2.7x. Grading is by string match with exotic spaces,
hyphens and markdown normalised; one question (ESP32 GPIO2 reset state) was
graded by hand, because correct answers quote the register defaults before
concluding. An earlier, smaller run of the same design (10 questions, 2
models) agreed.

The measurement was prompted by Belcak et al., "You don't need long context"
(arXiv:2611.05306, 2026). Their agent reads long inputs selectively, with grep
and bounded windows of about 6,000 characters, instead of holding the whole
input in context, and does as well or better: 84.0% vs 65.0% on AA-LCR with
Opus 4.5. That supports this library's split, where the agent navigates with
tools rather than ingesting the document. The paper does not test PDFs, tables
or any pre-processing, so it says nothing either way about the enriched ToC.
Its specific remedy did not carry over: a read-size cap was measured here and
not adopted, because agents already read narrowly (largest read 47K
characters, most under 10K) and the context was being filled by the manifest
ToC instead.

`search_text` also carries the text-layer limitation on **both** surfaces its
consumer reads: in the tool description, and — when a search returns nothing on
a document that holds raster regions — as a `note` on the result itself. The
Expand Down
58 changes: 29 additions & 29 deletions docs/token-economy.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,44 +5,44 @@ Tokens counted with the `o200k_base` encoding. 25 documents, 24 priced.

| Document | Pages | Full doc | Manifest | Section (med) | Section (p90) | First answer | vs full | Further answer | vs full | Build (cold) | Build (warm) |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| vishay_1n4001.pdf | 5 | 2,683 | 386 | 0 | 0 | n/a | n/a | n/a | n/a | 0.9s | 0.01s |
| raspi_pico.pdf | 31 | 13,856 | 4,917 | 307 | 877 | 5,224 | 2.7x | 307 | 45x | 5.0s | 0.03s |
| ti_ina219.pdf | 38 | 23,860 | 7,100 | 634 | 1,050 | 7,734 | 3.1x | 634 | 38x | 5.1s | 0.01s |
| ti_ne555.pdf | 39 | 17,830 | 5,561 | 556 | 941 | 6,117 | 2.9x | 556 | 32x | 4.1s | 0.01s |
| ti_sn74hc595.pdf | 41 | 16,273 | 3,437 | 532 | 926 | 3,969 | 4.1x | 532 | 31x | 4.4s | 0.01s |
| ti_tcan1044a.pdf | 42 | 20,040 | 7,502 | 545 | 945 | 8,047 | 2.5x | 545 | 37x | 4.0s | 0.01s |
| ti_lm317.pdf | 44 | 15,411 | 6,831 | 408 | 746 | 7,239 | 2.1x | 408 | 38x | 4.8s | 0.01s |
| ti_opa2340.pdf | 48 | 23,025 | 6,856 | 786 | 1,089 | 7,642 | 3.0x | 786 | 29x | 4.7s | 0.01s |
| bosch_bmp280.pdf | 49 | 20,829 | 7,851 | 609 | 744 | 8,460 | 2.5x | 609 | 34x | 4.7s | 0.01s |
| ti_tps54360.pdf | 50 | 26,643 | 8,363 | 760 | 1,242 | 9,123 | 2.9x | 760 | 35x | 6.0s | 0.01s |
| ti_ads1115.pdf | 57 | 33,973 | 10,096 | 776 | 992 | 10,872 | 3.1x | 776 | 44x | 5.8s | 0.01s |
| bosch_bme280.pdf | 60 | 27,975 | 10,722 | 550 | 792 | 11,272 | 2.5x | 550 | 51x | 5.9s | 0.02s |
| ti_lm358.pdf | 68 | 46,506 | 4,931 | 778 | 1,630 | 5,709 | 8.1x | 778 | 60x | 7.2s | 0.01s |
| ti_cc2640r2f.pdf | 72 | 37,132 | 9,463 | 689 | 992 | 10,152 | 3.7x | 689 | 54x | 10.9s | 0.01s |
| esp32c3_ds.pdf | 76 | 32,595 | 12,899 | 462 | 746 | 13,361 | 2.4x | 462 | 71x | 5.7s | 0.01s |
| esp32_ds.pdf | 78 | 40,373 | 9,958 | 617 | 1,391 | 10,575 | 3.8x | 617 | 65x | 6.5s | 0.01s |
| infineon_psc3.pdf | 90 | 55,792 | 7,766 | 737 | 1,839 | 8,503 | 6.6x | 737 | 76x | 9.6s | 0.01s |
| ti_tlv9061.pdf | 99 | 41,159 | 5,369 | 750 | 1,944 | 6,119 | 6.7x | 750 | 55x | 9.8s | 0.02s |
| esp8266_trm.pdf | 111 | 53,385 | 13,715 | 515 | 912 | 14,230 | 3.8x | 515 | 104x | 6.5s | 0.01s |
| bosch_bmi160.pdf | 114 | 57,582 | 16,811 | 581 | 827 | 17,392 | 3.3x | 581 | 99x | 28.1s | 0.01s |
| ti_msp430f5529.pdf | 143 | 101,951 | 20,469 | 906 | 1,568 | 21,375 | 4.8x | 906 | 113x | 11.3s | 0.02s |
| micro_atmega328.pdf | 294 | 199,004 | 69,229 | 742 | 1,062 | 69,971 | 2.8x | 742 | 268x | 18.3s | 0.03s |
| micro_pic16f887.pdf | 322 | 185,565 | 92,177 | 680 | 1,047 | 92,857 | 2.0x | 680 | 273x | 30.3s | 0.03s |
| raspi_rp2040.pdf | 642 | 400,095 | 46,518 | 689 | 2,796 | 47,207 | 8.5x | 689 | 581x | 36.8s | 0.03s |
| esp32_trm.pdf | 784 | 401,974 | 64,763 | 552 | 1,512 | 65,315 | 6.2x | 552 | 728x | 86.9s | 0.03s |
| vishay_1n4001.pdf | 5 | 2,683 | 359 | 0 | 0 | n/a | n/a | n/a | n/a | 0.8s | 0.01s |
| raspi_pico.pdf | 31 | 13,856 | 2,152 | 357 | 1,118 | 2,509 | 5.5x | 357 | 39x | 4.1s | 0.04s |
| ti_ina219.pdf | 38 | 23,860 | 2,744 | 745 | 1,238 | 3,489 | 6.8x | 745 | 32x | 3.1s | 0.01s |
| ti_ne555.pdf | 39 | 17,830 | 2,132 | 598 | 981 | 2,730 | 6.5x | 598 | 30x | 3.2s | 0.01s |
| ti_sn74hc595.pdf | 41 | 16,273 | 1,599 | 635 | 1,022 | 2,234 | 7.3x | 635 | 26x | 3.6s | 0.01s |
| ti_tcan1044a.pdf | 42 | 20,040 | 2,643 | 621 | 1,070 | 3,264 | 6.1x | 621 | 32x | 3.2s | 0.01s |
| ti_lm317.pdf | 44 | 15,411 | 2,956 | 408 | 765 | 3,364 | 4.6x | 408 | 38x | 4.1s | 0.01s |
| ti_opa2340.pdf | 48 | 23,025 | 2,456 | 916 | 1,518 | 3,372 | 6.8x | 916 | 25x | 3.9s | 0.01s |
| bosch_bmp280.pdf | 49 | 20,829 | 3,633 | 657 | 1,042 | 4,290 | 4.9x | 657 | 32x | 4.0s | 0.02s |
| ti_tps54360.pdf | 50 | 26,643 | 3,122 | 779 | 1,448 | 3,901 | 6.8x | 779 | 34x | 4.8s | 0.01s |
| ti_ads1115.pdf | 57 | 33,973 | 3,976 | 821 | 1,301 | 4,797 | 7.1x | 821 | 41x | 4.0s | 0.01s |
| bosch_bme280.pdf | 60 | 27,975 | 4,348 | 608 | 1,258 | 4,956 | 5.6x | 608 | 46x | 5.0s | 0.02s |
| ti_lm358.pdf | 68 | 46,506 | 2,062 | 801 | 1,772 | 2,863 | 16.2x | 801 | 58x | 5.9s | 0.02s |
| ti_cc2640r2f.pdf | 72 | 37,132 | 3,837 | 776 | 1,373 | 4,613 | 8.0x | 776 | 48x | 5.8s | 0.02s |
| esp32c3_ds.pdf | 76 | 32,595 | 3,817 | 656 | 1,116 | 4,473 | 7.3x | 656 | 50x | 6.4s | 0.01s |
| esp32_ds.pdf | 78 | 40,373 | 3,463 | 834 | 1,530 | 4,297 | 9.4x | 834 | 48x | 6.1s | 0.01s |
| infineon_psc3.pdf | 90 | 55,792 | 3,370 | 908 | 2,110 | 4,278 | 13.0x | 908 | 61x | 6.9s | 0.02s |
| ti_tlv9061.pdf | 99 | 41,159 | 2,464 | 934 | 2,057 | 3,398 | 12.1x | 934 | 44x | 8.3s | 0.02s |
| esp8266_trm.pdf | 111 | 53,385 | 4,624 | 579 | 1,011 | 5,203 | 10.3x | 579 | 92x | 4.7s | 0.02s |
| bosch_bmi160.pdf | 114 | 57,582 | 6,927 | 695 | 1,151 | 7,622 | 7.6x | 695 | 83x | 23.3s | 0.01s |
| ti_msp430f5529.pdf | 143 | 101,951 | 6,774 | 1,026 | 1,639 | 7,800 | 13.1x | 1,026 | 99x | 8.5s | 0.02s |
| micro_atmega328.pdf | 294 | 199,004 | 18,259 | 801 | 1,270 | 19,060 | 10.4x | 801 | 248x | 15.3s | 0.03s |
| micro_pic16f887.pdf | 322 | 185,205 | 27,452 | 725 | 1,078 | 28,177 | 6.6x | 725 | 255x | 27.0s | 0.03s |
| raspi_rp2040.pdf | 642 | 400,095 | 12,184 | 891 | 3,173 | 13,075 | 30.6x | 891 | 449x | 31.2s | 0.03s |
| esp32_trm.pdf | 784 | 401,974 | 19,301 | 776 | 1,727 | 20,077 | 20.0x | 776 | 518x | 38.8s | 0.04s |

## Headline

- Median full document: **33,973** tokens
- Median **first** answer (manifest + one section): **9,637** tokens, **3.1x** cheaper than the document (range 2.0x-8.5x)
- Median **further** answer about the same part (one section, manifest already in context): **625** tokens, **54x** cheaper
- Median build: **6.01s** cold, **0.01s** warm (25 of 25 cache hits)
- Median **first** answer (manifest + one section): **4,293** tokens, **7.3x** cheaper than the document (range 4.6x-30.6x)
- Median **further** answer about the same part (one section, manifest already in context): **760** tokens, **47x** cheaper
- Median build: **5.04s** cold, **0.01s** warm (25 of 25 cache hits)

## How to read this

**`Full doc` is a conservative baseline.** It is the page-matched text file -- what you paste in to read the document. Attaching the PDF itself costs more, because the pages arrive as images too.

**The two ratios answer different questions.** The first answer pays for the map as well as the section, and on large documents the map is most of the cost: the enriched ToC of a 322-page datasheet is itself half that document's tokens. Every question after it about the same part pays for a section alone, which is where the ratio gets large. An agent that asks one thing and leaves sees the first ratio; an agent doing real extraction work sees the second.
**The two ratios answer different questions.** The first answer pays for the map as well as the section, and on large documents the map is most of the cost: for a 322-page datasheet it is 27K of a 28K-token first answer, though only 15% of the document. Every question after it about the same part pays for a section alone, which is where the ratio gets large. An agent that asks one thing and leaves sees the first ratio; an agent doing real extraction work sees the second.

**The ratios are medians of per-document ratios, not the quotient of the medians above.** Each document is priced on its own and the ratios are then summarised, which is why dividing the two token medians gives a different -- and less representative -- number: a median of the per-document ratios is not the ratio of the medians, and the token medians are taken over different sets, since an unpriced document still has a full-document count.

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "datasheetindex"
version = "0.46.0"
version = "0.47.0"
description = "Agent-first parameter extraction from technical datasheets"
readme = "README.md"
requires-python = ">=3.13"
Expand Down
14 changes: 9 additions & 5 deletions scripts/token_economy.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@
the datasheet". This is a *conservative* baseline: attaching the PDF
itself costs more, since the pages arrive as images too.
``manifest``
What ``get_artifact_manifest`` actually hands the agent -- the enriched
ToC and the build's signals.
What the ``build_datasheet`` agent tool actually hands the agent -- the
enriched ToC, slimmed as the tool slims it, and the build's signals.
Measured through the same function the tool handler calls, so it is not
the Python API's fuller ``get_artifact_manifest``.
``section``
What ``get_section_text`` returns for one leaf ToC section, summarised
across every leaf as a median and a p90.
Expand Down Expand Up @@ -217,6 +219,7 @@ def measure_document(
the published ones.
"""
from datasheetindex import DatasheetTools
from datasheetindex.tools.defs import _agent_manifest

pdf_path = Path(pdf_path)
notes: list[str] = []
Expand All @@ -230,7 +233,7 @@ def measure_document(
)
cold_build_s = time.perf_counter() - started

manifest = tools.get_artifact_manifest()
manifest = _agent_manifest(tools)
# ``json_path``/``text_path`` are absolute and machine-specific. They
# are a handful of tokens and the agent really is sent them, so they
# stay in the count rather than being trimmed to make the number look
Expand Down Expand Up @@ -446,8 +449,9 @@ def render_markdown(
"",
"**The two ratios answer different questions.** The first answer pays "
"for the map as well as the section, and on large documents the map "
"is most of the cost: the enriched ToC of a 322-page datasheet is "
"itself half that document's tokens. Every question after it about "
"is most of the cost: for a 322-page datasheet it is 27K of a "
"28K-token first answer, though only 15% of the document. Every "
"question after it about "
"the same part pays for a section alone, which is where the ratio "
"gets large. An agent that asks one thing and leaves sees the first "
"ratio; an agent doing real extraction work sees the second.",
Expand Down
Loading
Loading