Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,16 @@ LITELLM_BASE_URL=https://your-litellm-gateway.example.com
# before any caption text is written.
# DATASHEETINDEX_VISION_MODEL=

# Optional. Folders the MCP server searches with its find_documents tool,
# separated like PATH (";" on Windows, ":" elsewhere). Unset or blank means the
# tool is not offered. Read by the MCP server only.
# DATASHEETINDEX_LIBRARY_ROOTS=

# ---------------------------------------------------------------------------
# Advanced: gateway compatibility. Rarely needed -- every setting below has a
# working default; set one only when your gateway or model needs it.
# ---------------------------------------------------------------------------

# Optional generation settings. Text (including structured ToCs) and vision
# reasoning are INDEPENDENT; unset/blank means omit the field, not "none".
# Supported spellings: none, minimal, low, medium, high, xhigh, max. Only set
Expand Down
30 changes: 30 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,36 @@

All notable changes to this project will be documented in this file.

## [0.46.0] - 2026-11-05

### Added
- **`find_documents` MCP tool.** Lists PDFs under the folders named in the new
`DATASHEETINDEX_LIBRARY_ROOTS` setting (separated like `PATH`), matching every
query term against the file name and folder path, so an agent whose shell
cannot see a network share can still find the PDF to pass to
`build_datasheet`. Results carry the full server-side path, last-modified date
and size, and a status per root.
- **On-disk library cache.** One file per root, valid for one day, refreshed in
a background thread at server startup and on `refresh=true`. An unreachable
root or unreadable folder is reported in the response, never raised; a PDF
that cannot be read or whose name is not valid UTF-8 is left out and counted
in `skipped_files`.
- `DATASHEETINDEX_LIBRARY_ROOTS` (and `DATASHEETINDEX_OUTPUT_DIR`, which places
the cache) can be set in `.env` on a source checkout (the server loads it at
startup); an installed server takes them from the MCP client config's `env`
block.
- Nothing changes when `DATASHEETINDEX_LIBRARY_ROOTS` is unset or blank: the
tool is not registered and the server serves its five tools as before. The
tool is MCP-server-only and is not in `create_datasheet_tool_defs()`.

### Changed
- **Shorter configuration docs.** The README and `.env.example` now lead with
the five settings most deployments need (`LITELLM_BASE_URL`,
`LITELLM_MASTER_KEY`, `DATASHEETINDEX_MODEL`, `DATASHEETINDEX_VISION_MODEL`,
`LITELLM_TLS_VERIFY`). The gateway-tuning variables (reasoning effort, output
budgets, token-limit spelling, `store`, timeout and retries) moved to an
"Advanced: gateway compatibility" section. No variable was removed or renamed.

## [0.45.0] - 2026-11-03

### Fixed
Expand Down
94 changes: 73 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Agent-first parameter extraction from technical datasheets.
- [What it does](#what-it-does) · [Philosophy](#philosophy) · [Supported products](#supported-products)
- [Token economy](#token-economy) · [Benchmark](#benchmark) · [Links](#links)
- **Getting started** — [Setup](#setup) · [Development](#development) · [Input sources](#input-sources)
- **Using it from an agent** — [Hand the MCP server to an agent](#hand-the-mcp-server-to-an-agent) · [Realize the tools without the Claude Agent SDK](#realize-the-tools-without-the-claude-agent-sdk) · [Run a local MCP server](#run-a-local-mcp-server)
- **Using it from an agent** — [Hand the MCP server to an agent](#hand-the-mcp-server-to-an-agent) · [Realize the tools without the Claude Agent SDK](#realize-the-tools-without-the-claude-agent-sdk) · [Run a local MCP server](#run-a-local-mcp-server) · [Find documents in a library](#find-documents-in-a-library)
- **What the artifacts tell an agent** — [Datasheets without a ToC](#datasheets-without-a-toc) · [Knowing where the ToC came from](#knowing-where-the-toc-came-from) · [Asking for a better ToC](#asking-for-a-better-toc) · [Datasheets that cover a product family](#datasheets-that-cover-a-product-family) · [Figure indexing and captions](#figure-indexing-and-captions)
- **Reference** — [Python API](#python-api) · [CLI](#cli) · [Project structure](#project-structure) · [License](#license)

Expand Down Expand Up @@ -177,15 +177,19 @@ uv sync --extra mcp
uv pip install claude-agent-sdk
```

For LLM-backed ToC fallback, summaries and figure captions, configure
`LITELLM_BASE_URL` and `LITELLM_MASTER_KEY` (see `.env.example`). Two optional
variables name the models, since which ones a gateway serves is a property of
your deployment rather than of this library:
For LLM-backed ToC fallback, summaries and figure captions, these are the only
settings most deployments need (see `.env.example`):

| variable | names | default |
| variable | sets | default |
|---|---|---|
| `DATASHEETINDEX_MODEL` | summaries and the ToC fallback | `qwen3.8-27b` |
| `DATASHEETINDEX_VISION_MODEL` | figure captions only | follows the text model |
| `LITELLM_BASE_URL` | the gateway URL | required |
| `LITELLM_MASTER_KEY` | your gateway key | required |
| `DATASHEETINDEX_MODEL` | model for summaries and the ToC fallback | `qwen3.8-27b` |
| `DATASHEETINDEX_VISION_MODEL` | model for figure captions only | follows the text model |
| `LITELLM_TLS_VERIFY` | gateway certificate verification | on |

The model names are settings because which models a gateway serves is a
property of your deployment rather than of this library.

`build_datasheet`'s `model` argument overrides `DATASHEETINDEX_MODEL` for one
call; the vision model is deployment-level only, and deliberately not a tool
Expand All @@ -195,10 +199,22 @@ Deployments that do not serve `qwen3.8-27b` must explicitly name an available
model with `DATASHEETINDEX_MODEL`. Unless separately overridden, captions use
that same model and require image-input support.

### Chat Completions configuration
TLS verification is **on** by default. `LITELLM_TLS_VERIFY=false` (also `0`,
`no`, `off`) turns it off, for a gateway whose certificate the local trust store
cannot accept -- an internal CA missing from the image, or a self-signed cert on
localhost. Adding that CA to the trust store is the better fix: `LITELLM_MASTER_KEY`
travels on this channel. A certificate that fails verification raises
`LlmTlsVerificationError` (exported from the package root) naming the gateway
and both remedies, rather than degrading to an index with no ToC entries. Figure
captioning is the exception: it still degrades, because the artifact is complete
without captions.

All library LLM calls use Chat Completions, including images and strict JSON
schemas; there is no automatic fallback to a different API or different model.
### Advanced: gateway compatibility

Rarely needed: these settings exist for gateways or models that need different
request parameters, and every one has a working default. All library LLM calls
use Chat Completions, including images and strict JSON schemas; there is no
automatic fallback to a different API or different model.

| variable | controls | default |
|---|---|---|
Expand Down Expand Up @@ -273,16 +289,6 @@ threads, so use a thread-safe collector in applications. Callback exceptions are
ignored with a payload-free warning. Optional metadata fields remain `None` when
the gateway does not supply them.

TLS verification is **on** by default. `LITELLM_TLS_VERIFY=false` (also `0`,
`no`, `off`) turns it off, for a gateway whose certificate the local trust store
cannot accept -- an internal CA missing from the image, or a self-signed cert on
localhost. Adding that CA to the trust store is the better fix: `LITELLM_MASTER_KEY`
travels on this channel. A certificate that fails verification raises
`LlmTlsVerificationError` (exported from the package root) naming the gateway
and both remedies, rather than degrading to an index with no ToC entries. Figure
captioning is the exception: it still degrades, because the artifact is complete
without captions.

`claude-agent-sdk` is only required for the SDK-flavored handoff
(`create_datasheet_tools_server`). The tool logic itself is framework-neutral --
non-SDK hosts get it via `create_datasheet_tool_defs()` with no SDK import (see
Expand Down Expand Up @@ -513,6 +519,51 @@ server object inside another Python runtime, use
`create_datasheet_tools_server()` instead; it exposes the same tool surface, with
the document loaded at runtime via the `build_datasheet` tool.

### Find documents in a library

`build_datasheet` needs a PDF path the server can open. When the datasheets sit
on a network share, the agent's own shell often cannot see it (a Linux dev
container, say), so it cannot tell which PDF to load -- but the server can.
Set `DATASHEETINDEX_LIBRARY_ROOTS` and the server adds a sixth tool,
`find_documents`, which lists PDFs under those folders. Unset or blank, the
tool is not registered and the server serves exactly its five datasheet tools.

```json
{
"servers": {
"datasheetindex": {
"command": "uvx",
"args": ["--from", "datasheetindex[mcp]", "datasheetindex-mcp-server"],
"env": {
"DATASHEETINDEX_LIBRARY_ROOTS": "\\\\server\\share\\Datasheets;D:\\Manuals"
}
}
}
}
```

- Several roots are separated like `PATH`: `;` on Windows, `:` on Linux and macOS.
A server running under WSL (or any Linux) therefore takes `/mnt/z/...`-style
roots: a `Z:\...` root would split at its colon. Quotes around a root and
blank entries are tolerated. On a source checkout the variable may also be set
in `.env`; an installed server takes it from the MCP client config's `env`
block.
- A mapped drive (`Z:\...`) works only if the server process itself sees the
mapping; a UNC path does not depend on that.
- Arguments: `query` (whitespace-separated terms; a document matches when its
path below a root contains every term, case-insensitively), `limit` (default
50, maximum 500) and `refresh` (rescan now).
- Results are path matches, not content matches: only file and folder names are
searched, and the root itself never matches. Each result has `path`,
`modified` (date) and `size_kb`; pass `path` to `build_datasheet` unchanged.
The response also carries `total_matches`, `truncated` and a status per root.
- The list is cached on disk for one day (fixed), so a restarted server does not
rescan. The server refreshes a missing or stale list in the background at
startup, and `refresh=true` forces a rescan. An unreachable root or an
unreadable folder is reported in the response, never raised. A PDF that
cannot be read, or whose name is not valid UTF-8, is left out and counted in
its root's `skipped_files`.

### Datasheets without a ToC

The enriched ToC comes from the PDF's own bookmarks, and plenty of real
Expand Down Expand Up @@ -796,6 +847,7 @@ src/datasheetindex/
defs.py # create_datasheet_tool_defs (framework-neutral defs)
registry.py # Claude Agent SDK adapter (re-exports DatasheetTools)
mcp_server.py # Local stdio/HTTP MCP server entry point
library.py # find_documents: PDF listing for MCP library roots
llm/
client.py # LLM client factory (free-text + structured_json + vision)
toc_fallback.py # LLM-based ToC generation fallback
Expand Down
52 changes: 52 additions & 0 deletions docs/datasheetindex_architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -1726,6 +1726,56 @@ So the SDK server, the local MCP server, and non-SDK hosts all present identical
tool names, descriptions, and JSON schemas -- a change to a tool def propagates to
every surface from one place.

### Library discovery: `find_documents`

`mcp_server.py` registers a sixth tool, `find_documents`, only when
`DATASHEETINDEX_LIBRARY_ROOTS` names at least one folder (roots separated by
`os.pathsep`). It lists PDFs by file name and folder path, never by content. The
reason it exists: an agent can only call `build_datasheet` on a path it can name,
and when the agent's shell and the server see different filesystems (a dev
container against a Windows share) only the server can do the finding.

- **MCP-only.** It is added in `mcp_server.py`, not in
`create_datasheet_tool_defs()`. Embedders of the shared defs did not opt into a
filesystem lister, and the five-tool surface stays true for them. Unset or
blank, the tool is not registered at all (not a tool that answers "not
configured"), so nothing changes for anyone who does not use it.
- **Full paths.** Each result carries the full server-side path, in the server's
own format (`\\server\share\...`), which the agent passes to
`build_datasheet` unchanged. `build_datasheet`'s contract is untouched, and
there is no root-plus-relative join for an agent on a different OS to get wrong.
- **Matching.** Every whitespace-separated term must appear, case-insensitively,
in the path relative to its root (folders and file name, either separator).
The root is excluded, or a root named `Datasheets` would match `data`. Results
are sorted by relative path, capped by `limit`, and the response says
`total_matches` and `truncated`. Files under overlapping roots are listed once.
- **Cache.** One JSON file per root at `<output dir>/library`, where the output
dir is the artifacts' (`DATASHEETINDEX_OUTPUT_DIR`, else the temp default),
written atomically, valid for one day (not configurable). It is loaded and, if
missing or stale, rescanned in a background thread at server startup, so the
slow scan of a share happens before the agent asks. A list whose `listed_at`
lies in the future (a host with its clock ahead sharing the folder) counts as
stale. The server loads `.env` before reading either setting, since both are
read once, at startup. `load_dotenv()` finds `.env` by walking up from the
installed package, not the CWD, so this works on a source checkout only; an
installed server takes both from the MCP client config's `env` block.
- **Failure rules.** One bad root or folder never becomes a tool error. An
unreadable subfolder is skipped and counted (`skipped_folders`). A PDF that
cannot be stat'ed, or whose path is not valid UTF-8 (a non-UTF-8 name on POSIX,
an unpaired surrogate on Windows), is skipped and counted (`skipped_files`):
no cache file or JSON-RPC message can carry such a path, and returning one
killed the stdio transport. A root whose rescan fails keeps its previous list,
if any: its `status` stays `ok` (or `refreshing`) and an `error` field says
what failed; it is `unavailable` only when there is no list at all. A stale
list is served at once while a rescan runs (`refreshing`). Only a root with no
list makes a call wait, for at most 30 seconds, after which it reports
`scanning` with the elapsed time. A corrupt, damaged (non-finite numbers, deep
nesting) or old-format cache is treated as missing. Symlinks and junctions are not followed. Scans are de-duplicated
within one process only; several server processes may each scan a stale share.
- **Threads, not processes.** A process pool inside the MCP stdio server
deadlocks on Windows (see the scan-worker notes), and the scan is I/O bound, so
scans run in daemon threads that never hold the state lock while scanning.

### Page-cut truncation signal

A section's ToC page range does not always contain the whole of its table. In
Expand Down Expand Up @@ -2113,6 +2163,8 @@ datasheetindex/
│ │ # bounded by max_figure_captions
│ └── untrusted.py # Framing for document text sent to an LLM
│ # (PDF text is untrusted input, not instructions)
├── mcp_server.py # Local MCP server (stdio / streamable-http / sse)
├── library.py # find_documents: library scan, disk cache, matching
├── index.py # Main DatasheetIndex class
└── models.py # Data models
```
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "datasheetindex"
version = "0.45.0"
version = "0.46.0"
description = "Agent-first parameter extraction from technical datasheets"
readme = "README.md"
requires-python = ">=3.13"
Expand Down
Loading
Loading