fix(pixel): read PDF text in the shared page reader; binary bodies are never page reads

[Lightheartdevs]

Review fixes (d7268e0a9)

Blocker: a small crafted PDF crashed the whole gateway. The extractor ran in a worker thread with a 192 MB heap limit. In Node, a worker that runs out of heap during a large allocation takes the whole process down (FATAL ERROR: Reached heap limit, exit 134). I reproduced the review's two cases locally (Windows, node v24.14.1) through pixel_ods_web_extract on the old head a2dcacbaf:

Document	Size	Old head	This head
One content stream of ASCII85 z groups (29 MB decoded)	7,300,657 bytes	process exit 134, 3 of 3 runs	exit 0, receipt not_read: too-large; the extraction process exits 0
Three Type0 fonts, each ToUnicode map a bfrange list of 2.2 million codes	73,433 bytes	process exit 134, 3 of 3 runs	exit 0, receipt not_read: no-text; the extraction process exits 0
1.9 million empty dictionaries, inside every parser bound (found while fixing)	7,600,411 bytes	exit 134 when calling extractPdfText directly; through the tool, 3 of 3 runs ended only the worker	the extraction process runs out of heap (exit 134) and only it ends; receipt not_read: too-large

What changed:

• Extraction runs in its own Node process, not a worker thread (pdf-text-worker.mjs). It is started from process.execPath with --max-old-space-size=192, an empty environment (so no NODE_OPTIONS) and no inherited flags. The document goes in on stdin and one JSON line comes back on stdout. The process is killed with SIGKILL at the deadline or when the caller aborts. If it runs out of heap, only it ends, and the PDF is reported as too-large. If it cannot start (spawn error, missing script), the PDF is reported as unavailable, not unsupported.
• Decoders and font tables stop at their bounds before they grow.
  • ASCII85 writes into a preallocated array capped at the stream's share of the decoded-bytes bound (16 MB per stream, 48 MB per document). It refuses as soon as the next group would pass the cap.
  • A ToUnicode map keeps at most 65,536 codes: bfchar and list-form bfrange together, with list targets read one at a time instead of collected. It also keeps at most 10,000 ranges and 256 code space ranges. One Unicode value is at most 512 bytes.
  • All maps and CID width tables of one document share a budget of 500,000 entries; running out means too-large.
  • Objects from object streams now count toward maxObjects, shown text stops growing past the text bound, and the glyph cache is capped per document. Each of these could also grow without limit before.
• At most two extraction processes run at once in the gateway. The others wait, and the wait counts toward their own deadline. This bounds the total when search_read, the host citation check and parallel web_extract calls all read PDFs.
• Wording. The docs and the module header no longer say the extractor has "no network access, no file access". The process is not a sandbox: it has the gateway's file and network access, and the extractor simply does not use them. The receipt for too-large now says "larger than the PDF reader's size or memory bounds". The docs also say that PDFs encrypted with only an owner password are reported as encrypted.
• RTF and PostScript are text formats and are no longer treated as binary bodies.

Tests added (all in tests/pdf_reader.test.mjs, fixtures generated in tests/fixtures/pdf-fixtures.mjs):

• Real bombs, real gateway process. A separate Node process runs pixel_ods_web_extract on the ASCII85 and ToUnicode bombs through the real reader and extraction. The test asserts that process exits 0 with the exact not-read receipt. On the old head this exits 134.
• Real out-of-memory. The empty-dictionary document really exhausts the extraction process's heap. The test checks the process died of a heap abort (exit 134 or SIGABRT), that the result is too-large, and that a gateway process reading it through pixel_ods_web_extract exits 0. An abort writes a crash report on Linux and macOS (apport on tower2), so this test runs by default only on Windows, or with ODS_PDF_OOM_TEST=1. On tower2 it is skipped.
• Bounds without processes. ASCII85 with z groups and a partial last group still decodes. The ASCII85 bomb is too-large before any large allocation. The ToUnicode bomb finishes in well under a second as no-text. Ten maps pass the per-document budget.
• The process itself (stubbed spawn): flags, empty environment, stdin, the child's own deadline, SIGKILL on timeout and abort, and how each exit is reported (SIGABRT, 134 and a foreign SIGKILL are too-large; exit 1 and spawn failures are unavailable; a forged or oversized message is unsupported). There are also two real cases: a missing worker script and a missing executable, both unavailable.
• Concurrency. Two processes run and the third waits. A waiting read that is aborted or times out never starts a process. The third starts once the first process has exited.

Fail-before: I ran the new test file against the old head's plugin code. 7 of 22 tests fail: the bounds test, both process tests, the gateway-survival test, the real out-of-memory test, the RTF/PostScript assertion, and the oversized-PDF test (only because of the new receipt wording).

What was wrong

A PDF fetched as raw bytes was listed as a page read. OpenClaw's web_fetch decodes any 2xx body as text. On 2026-09-25 it fetched the NVIDIA RTX 5070 user guide (https://www.nvidia.com/content/geforce-gtx/geforce-rtx-5070-user-guide-r1.pdf) in two fleet runs:

Host	Session / row	Result
tower2, round 061	b63ee849, row 276 (0-indexed; 277 counting from 1), 11:23:35Z	HTTP 200, contentType: application/pdf, extractor: raw, 15,994 characters, 2,444 U+FFFD
tower1	f702a671, row 327, 15:09:36Z	same PDF, same shape, 2,722 U+FFFD

In both, the body starts %PDF-1.7 … <</Linearized 1/L 5381674: the bytes of a 5.4 MB PDF, never decoded. I read these from the session transcripts, read-only.

completion-assurance.mjs counted any 2xx web_fetch with non-empty text as a page read. So the PDF went into the host's "Pages Pixel read successfully" list, the stop-synthesis sources and the cited-page check. The existing round-061 replay test in progress_finalization.test.mjs expected exactly this: its list of eight pages read included the PDF.

Three related gaps:

• The shared reader (createPublicPageReader, used by pixel_ods_web_extract and the host citation check) turned down every PDF as "not a supported text document". Nothing in ODS could read a PDF.
• A binary receipt did not say what the body was (kind, size) or why it was not read.
• The GitHub repository-research check (canonicalWebFetchSucceeded) also accepted any 2xx web_fetch, so a binary file from the repository counted as a source read.

What changed

• plugin/document-body.mjs (new). Recognizes a PDF or other binary body from its content type or from the bytes themselves: a PDF signature, a NUL, or mostly replacement and control characters. The same check is used in three places: the reader, the read receipts and the web_fetch projection.
• Shared reader (web-extract.mjs).
  • PDFs, including ones served as application/octet-stream or with no type, have their text extracted.
  • An image, archive or other binary body gets a not-read receipt without its body being read. The receipt has kind, contentType, bytes (from Content-Length or the bytes read) and the reason.
  • A body with a text type that is really binary is not a page.
  • application/pdf is added to the reader's type sets. That covers pixel_ods_web_extract, the host citation check and pixel_ods_search_read from feat(pixel): search and read public pages in one call (pixel_ods_search_read) #6698.
  • pixel_ods_web_extract cites PDF evidence as Targeted evidence from <url> (PDF text, pages 1-N of M), with details.document = {kind, read, bytes, pages_read, page_count, text_truncated}. It reports a PDF it could not read as pdf-text with not_read set to too-large, timeout, encrypted, no-text, unsupported or unavailable.
  • The tool description gains one sentence saying it reads PDFs.
• plugin/pdf-text.mjs and pdf-text-worker.mjs (new): bounded PDF text extraction.
  • Written in plain JavaScript; it uses only node:zlib and node:child_process. Each extraction runs in its own short-lived Node process (see the review fixes above), at most two at a time.
  • Limits: 8 MB of body, the first 10 pages, 300k characters, 48 MB of decoded data (16 MB per stream), 250,000 objects, 500,000 font table entries, and a 192 MB heap for the process.
  • Time limit: at most 5 s, and at most half of the read's timeout. That gives 2 s inside the host check's 4 s, and 5 s inside search_read's 12 s per read and 15 s per call. Time spent waiting for a free process counts.
  • The process is killed at the time limit or when the caller aborts. It stops itself a little earlier and keeps the pages it finished.
  • It reads classic and object-stream layouts, Flate/ASCII85/ASCIIHex streams, simple fonts (with their encodings and Differences), composite fonts through ToUnicode, and form XObjects.
  • It reports as not read: encrypted files (including owner-password-only ones), scans or outlined text, and composite fonts without a ToUnicode map. It never guesses their glyphs.
  • Nothing in a document is run.
• Read receipts (completion-assurance.mjs). A binary web_fetch body no longer counts as a read or as a returned source. That covers cited-page attribution, the host's list of pages read and the stop synthesis.
• Steering (web-result-projection.mjs, and two lines in tool-loop-guard.mjs).
  • The bound PDF or binary web_fetch result is saved as a short receipt instead of the bytes. This applies to native calls and to Tool Search calls.
  • For a PDF, the receipt names the exact pixel_ods_web_extract call: {"url": …}, plus a short query.
  • Saved tool results reach the live model request through the ODS runtime repair (createPersistedToolResultModelProjection), so the run that made the call sees the receipt, not the bytes.
• GitHub source check. A binary file from the owner's repository no longer counts as a repository source read.
• Docs. COMPLETION-RELIABILITY.md and docs/PIXEL.md.

No new dependency. The plugin's package.json has no dependencies and there is no lockfile. OpenClaw bundles clawpdf, which is PDFium compiled to WASM (4.3 MB). I did not use it: it cannot be resolved from the plugin directory, and it is not pure JavaScript.

Tests

Full plugin JS suite on tower2 (host node v22.23.1, throwaway dir, remote-js-test.sh, removed afterwards):

Ref	Tests	Pass	Fail
this branch, d7268e0a9	3134	3095	0
previous head, a2dcacbaf	3130	3092	0
d7268e0a9 merged with #6698 1f0158f2d (import lines kept from both)	3203	3163	0

The tests that did not pass are skipped/todo: the same 38 as on the previous head, plus the real out-of-memory test, which is skipped on Linux. /var/crash on tower2 was still empty after these runs.

Locally (Windows, node v24.14.1): pdf_reader.test.mjs 22 of 22, including the real out-of-memory test. The related files together (pdf_reader, completion_assurance, host_citation_verification, partial_citation_delivery, progress_finalization, public_page_request, web_extract, web_extract_performance, web_result_projection): 186 of 186.

Process cost on tower2: 40-72 ms per extraction including process start (8 in a row). Five at once through the two slots took 166 ms.

Tests in tests/pdf_reader.test.mjs (22). The PDF fixtures are generated inside the test (tests/fixtures/pdf-fixtures.mjs), so every byte can be reviewed. Besides the review tests above, they cover:

• A generated PDF in six producer layouts (standard font, subset font with ToUnicode, kerned text, object and xref streams with a linearization header): text extracted, cited, and counted as a read.
• Only the first 10 pages are read.
• Encrypted, image-only, unmapped-font and garbage PDFs: not read, each with its reason.
• Decompression, text, size and time limits.
• A real process is killed at its deadline while the event loop keeps running.
• An oversized PDF, with and without Content-Length: a not-read receipt, and reading stops at the limit.
• An image body: not read, with a receipt.
• Binary bodies labelled as text.
• The time limit fits each caller's budget, and an abort ends the read at once.
• The host citation check verifies a cited PDF.
• web_fetch receipts shaped like the tower1/tower2 rows: PDF and image are not reads; readable text served as octet-stream still is, and so are RTF and PostScript.
• Projection of Tool Search and native web_fetch results.
• The GitHub source check.

Changed test. The round-061 replay now feeds row 276 as the PDF it actually was. The host's list of pages read has seven entries instead of eight.

Earlier checks (first version, before the review fixes). I broke ten parts of the change one at a time; each break was caught by at least one failing test. Text was extracted from real PDFs (local, statistics only, not committed): Chrome/Edge print-to-PDF, a 379-page Word document, a LaTeX spec, earnings-call transcripts, a 3.8 MB SEC proxy statement, ReportLab output. An image-only slide deck and a business card with outlined text came back as no-text.

Conflicts with open PRs

I checked with git merge-tree against current heads, for the old head and the new one. The review fixes add no conflict with any open PR.

• Clean: fix(pixel): count a passing inspection of the snapshot awaiting its byte check #6726, fix(pixel): check that owner-requested text is visible when the preview loads #6723, fix(pixel): let the turn continue after an in-project rm -rf is refused #6722, fix(host-agent): one redactor for every log excerpt shown to the owner or Pixel #6721, fix(extensions): re-enable curated library recipes that build locally #6720, host-agent: stop warning when Pixel reproof is only waiting on a model download #6719, fix(extensions): one shared compose policy for both extension validators #6718 (compose-validator hardening), fix(pixel): stop whole-file rewrites and test bisection after failed tool results (output efficiency C1-C4) #6716, fix(pixel): recover owner-turn context overflows by trimming tool output when that fits #6715, fix(pixel): one system prompt for every Pixel chat (drop session key and id from the Runtime line) #6711, fix(installer): reload rendered LiteLLM routes before Pixel setup #6701, Replace 2^n GPU subset enumeration with a tier-based early-exit search #6700, fix(pixel): refresh runtime groups during fresh installation #6684, fix(portal): enforce team scheduling limits on retry and answer #6659.
• feat(pixel): search and read public pages in one call (pixel_ods_search_read) #6698 (1f0158f2d): conflicts only in the import lines of plugin/completion-assurance.mjs. feat(pixel): search and read public pages in one call (pixel_ods_search_read) #6698 adds the externalContentBody and source-kind imports; this branch adds webFetchBinaryBody. Keep both. With that resolution the full suite passes (table above), and the binary exclusion in openedSourceUrls and sourceUrls survives the merge.
• fix(models): keep the selected model fully on the GPU #6706, fix(pixel): keep Pixel's prompt append-only across owner messages #6702, feat(doctor,dashboard): report when the running model is partly on the CPU #6697 and drafts fix(pixel): keep the system prompt byte-stable across owner messages #6654, Improve Pixel task execution, evidence reuse, and visible progress #6645, fix(pixel): restore extension discovery and multiline clarification #6564: their conflicts are in files this branch does not change.

Risks

• A process per PDF read. About 40-70 ms on tower2 (Linux, node 22), about 150-300 ms on Windows. Each process can use up to its 192 MB heap plus Node's baseline and at most 48 MB of decoded buffers. At most two run at once.
• Waiting counts toward the deadline. Under load, a PDF read can come back as timeout because it waited for a free process. The host check reads up to four cited URLs in parallel; if all four are PDFs, two wait inside its 2 s extraction budget.
• Heap aborts still leave a crash report. A document that gets past the in-process bounds (the empty-dictionary case is one) still aborts its extraction process. On Linux or macOS the host's crash handler may record that (apport, systemd-coredump, ReportCrash), depending on host configuration and whether the gateway runs in a container. The gateway keeps running.
• The real out-of-memory test is skipped on Linux and macOS by default, including tower2 and Linux CI. There, the same path is covered by the bomb documents (bounded, no abort) and by a stub process that exits with SIGABRT.
• The process is not a sandbox. It runs with the gateway's user, files and network. Only the empty environment, the heap limit and the kill are enforced; the extractor simply opens nothing.
• It needs to start process.execPath. The macOS gateway sandbox allows process-exec, and the plugin already starts processes (provider-lease-worker.mjs). If starting fails, the receipt says unavailable.
• Coverage of the extractor. Text order and word spacing come from positions and glyph widths, so odd layouts can merge or split words. These are reported as not read:
  • LZW or RunLength content streams;
  • encrypted PDFs, including the many vendor spec sheets that have only an owner password (pdf.js would read those);
  • composite fonts without a ToUnicode map.
  • ToUnicode maps are cut at 65,536 codes (no font has more glyphs), and a document with more than 500,000 font table entries is reported as too-large.
• Time budget. The 5 s / half-of-timeout limit does not subtract network time already spent on the read. The caller's own deadline or abort still ends the extraction: search_read's 15 s per call and the host check's 4 s.
• The host citation check now reads cited PDFs, within its unchanged 4 s budget.
• Wording changes. Content-type refusals from pixel_ods_web_extract are worded differently; they still contain "not a supported text document". A body with a text type that has a NUL or a PDF signature is now refused.
• Binary detection without a type. In the fleet row, U+FFFD is 15.3% of the text, below binaryText's 20% threshold. That row is caught by its content type and by the %PDF at the start of the body. A binary body with no type and no PDF signature would be caught only by NULs. Unchanged here.
• Runtimes without the ODS repair. The receipt only reaches the live model request through the ODS runtime repair. On a runtime without it, the replacement affects only the saved transcript. Receipt accounting does not depend on it.
• Not run live. I did not run this against the fleet, which is read-only during qualification, and I did not re-fetch the NVIDIA guide. It is 5,381,674 bytes, so it is within the size limit. Whether its fonts carry ToUnicode maps is unknown; if they do not, the result is an honest no-text receipt.

🤖 Generated with Claude Code