fix(pixel): show schema.org event and offer data from a page's markup in the shared reader

[Lightheartdevs]

Summary

Many event and shop pages build their visible text with JavaScript, which Pixel's page reader does not run. The HTML they serve often already contains the facts as schema.org data: JSON-LD scripts, or microdata attributes. This PR reads that data, without running anything, and shows a short, fixed set of fields at the top of the page text.

Honest scope first. On the fleet corpus this rarely supplies a fact that was missing. It does not touch the main causes of unread sources:

• bot walls (403/405);
• JavaScript-built listings that publish no schema.org data;
• invented or dead URLs.

Those need the render work or model-side changes. See the replay below.

What changes

plugin/structured-data.mjs (new) is called from the shared reader, createPublicPageReader in web-extract.mjs. The reader change is one import and one line. The block is added:

• after bot-challenge detection, so it never makes a challenge page look like a real one;
• before the visible text of an HTML response. Plain text, markdown and JSON responses are unchanged.

The block is part of text, so every reader consumer sees the same thing:

• pixel_ods_web_extract returns it inside the existing untrusted-content markers;
• host citation verification checks the same text. A date that appears only in the markup can verify a citation. A cancelled event does not verify, because the word "cancelled" sits next to its date;
• extension repository context uses the tool;
• after feat(pixel): search and read public pages in one call (pixel_ods_search_read) #6698, pixel_ods_search_read gets it too (checked in a local composition, see below).

The tool description is unchanged, so the prompt does not change.

A real block (The Met Philadelphia shows page, read from tower2). Main's reader gets 56 characters from this page, the title only:

Structured data published in this page's markup (schema.org; not part of the visible page text):
- MusicEvent: CHELSEA HANDLER: THE HIGH AND MIGHTY TOUR | start: 2026-09-26 20:00 UTC-0400 | location: The Met Presented by Highmark, 858 N Broad St, Philadelphia, PA, 19130, US | url: https://www.ticketmaster.com/chelsea-handler-the-high-and-mighty-philadelphia-pennsylvania-09-26-2026/event/02006471E2A6EBC0
- MusicEvent: Kany Garcia: Puerta Abierta Tour | start: 2026-09-27 20:00 UTC-0400 | location: ...
...
- (8 more schema.org items not shown)

Only these fields are read

type	fields shown
Event and its subtypes (MusicEvent, TheaterEvent, SportsEvent, Festival, ...)	name, start, end (dropped when it is the start's day), status only when cancelled / postponed / rescheduled / moved online, location (place name and postal address), organizer name, url
Product (and ProductModel, ProductGroup, ...)	name, sku, gtin (8-14 digits only), up to 3 offers
Offer, AggregateOffer	price or low-high, currency (3 capital letters), availability, condition, seller name, priceValidUntil
Organization and some business subtypes	name, postal address, url. Shown only when it has an address, and never when it is credited as a publisher, author, brand, manufacturer and similar (on news pages that is the site itself)

Nothing else is read: no descriptions, reviews, ratings, images or performer bios. A test checks that a prompt-injection line in description never appears.

Enumerations are shown as fixed words ("in stock", "new", "cancelled"). Any other value is dropped.

ISO times are written as 2026-10-02 20:00 UTC-0400. The zone has no colon, so the citation verifier cannot read an offset such as -04:00 as the time 4:00.

Bounds and safety

• Nothing is executed. JSON-LD is parsed with JSON.parse. Comments are skipped, and so is the content of other script, style, noscript, template, textarea and iframe elements, so a <script type="application/ld+json"> written inside a JavaScript string or an HTML comment is ignored. Microdata is read from tag attributes and text by a simplified tag walk.
• A schema.org vocabulary is required. For JSON-LD that means a schema.org @context or full schema.org type IRIs. For microdata it means a schema.org itemtype.
• Values are cleaned. Each value is one line of plain text: entities decoded; tags, control, zero-width and bidi characters removed; capped at 160 characters.
• URLs are filtered. Only http(s) URLs are kept, never with credentials, with utm_ tags removed, at most 300 characters.
• Size limits:
  • at most 16 JSON-LD scripts, 256 KB each, 512 KB in total;
  • 4,000 JSON nodes, depth 12;
  • 60,000 tags for microdata, of which 4,000 may carry microdata attributes;
  • at most 8 events/products/offers and 2 organizations, 2,400 characters. A final line counts the items left out.
• On an item's own page (its url is the page), related items are left out, so an event page does not look like a listing.
• Linear time. Hostile 1 MB inputs (unterminated scripts, 200k [, 15k nested itemscope, unclosed quotes, stray end tags) each finish in under 25 ms locally. The test fails above 1 s.

Evidence: replay on tower2

This is a live replay from tower2 (OpenClaw 2026.6.33 SDK, the guarded fetch, the commit's reader). It covers two sets of URLs:

• A: the 142 URLs from the render-scout corpus: every URL the research journeys and the Hermes/OpenCode comparators failed to read in rounds 070-092;
• B: the other 117 pages the fleet read in those rounds.

The needed-fact rules are the scout's (facts.mjs, per journey). For retail pages I also counted "549.99 USD" with a stock word as a USD price with stock state; it changed nothing.

For each page read, the same response was also passed through main's reader offline. Main's text was identical to this branch's visible text on 171/171 HTML pages.

	set A (scout corpus)	set B (other fleet reads)	all
URLs	142	117	259
read (HTTP 2xx, not a challenge)	63	111	174
pages with a block	12	23	35
needed fact in visible text (= main)	42	76	118
needed fact with the block	42	77	119
newly readable	0	1 (themetphilly.com/shows)	1
needed fact also inside the block	9	18	27

Set A has no gain. Of its 79 unread URLs, JSON-LD cannot reach any, because nothing is read:

• the 403/405 bot walls (TechSpot, Notebookcheck, JamBase, Shazam and others);
• challenges;
• dead or invented URLs.

The JavaScript-built pages that are read carry no schema.org fact:

• BestBuy search has no JSON-LD;
• Newegg has only breadcrumbs and FAQ;
• NVIDIA's family page JSON-LD has two offers ($549 and $749, "in stock") but not the VRAM or board power the task needs.

This matches the scout's raw-HTML probe: JSON-LD had the fact on 0 of 25 JavaScript-built pages.

Where the block does appear (35 pages, almost all event pages), 27 already restate the needed fact in a compact form: date, time, venue address and the event's own URL. That makes answers easier to cite and host verification easier to pass. It is not a coverage gain.

Added latency is the block's own CPU time on the fetched body, measured on tower2 (median of 5 runs per page, 171 HTML pages):

	p50	p95	max
block extraction	0.49 ms	1.57 ms	3.7 ms
same body through the reader, this branch minus main	0.38 ms	2.1 ms	3.7 ms
for scale: main's reader without network	4.5 ms	20 ms
for scale: a live read with network	123 ms	1.5 s

A run before the final fixups gave the same counts: 0 in A, and Met Philly as the only new page in B.

Tests

• New file tests/structured_data.test.mjs, 13 tests:
  • event fields and whitelist; @id references; status words; end dropped on the same day;
  • products and offers, including AggregateOffer, priceSpecification, invalid price, currency and GTIN, and offers deduplicated against their product;
  • organization rules (address required, publisher/author/brand excluded);
  • own-page vs listing (including utm_ tags and a differently encoded query), and the listing cap with an "N more" line;
  • value cleaning and URL filtering;
  • which scripts count as JSON-LD (comments, JS strings, noscript, template, textarea, other vocabularies, CDATA, raw newlines, broken and oversized JSON);
  • microdata (meta, link, time, the content attribute, nested items, link text as a name, li siblings, other vocabularies);
  • hostile-input timing;
  • reader integration (block before the text; pages without data unchanged; text/plain untouched; a challenge page with JSON-LD is still a challenge);
  • pixel_ods_web_extract in overview and query mode (the block only inside the markers);
  • host citation verification (a markup-only date verifies, a wrong time does not, a cancelled event does not).
• Full plugin suite on tower2 (node --test tests/*.test.mjs):

ref	tests	fail	skipped
main 8ed4d293a	3121	0	38
this branch 2583ed3d0	3134 (+13, all new)	0	38
this branch merged with #6698 (1f0158f2d)	3203	0	39

A test-name diff against main shows only the 13 new tests.

Coordination

• git merge-tree is clean against feat(pixel): search and read public pages in one call (pixel_ods_search_read) #6698 (1f0158f2d) and fix(pixel): read PDF text in the shared page reader; binary bodies are never page reads #6724 (d7268e0a9).
  • The reader change sits after the challenge check, away from the lines both PRs edit.
  • The import sits between the two existing imports, where neither PR inserts.
  • The PIXEL.md paragraph is a new one after the exact-download paragraph, not in the pixel_ods_web_extract bullet that both PRs edit.
• feat(pixel): search and read public pages in one call (pixel_ods_search_read) #6698 and fix(pixel): read PDF text in the shared page reader; binary bodies are never page reads #6724 conflict with each other in completion-assurance.mjs, independently of this PR.
• Composed with feat(pixel): search and read public pages in one call (pixel_ods_search_read) #6698. A JSON-LD-only event page read through pixel_ods_search_read (urls mode, focus "date venue") earns a receipt: match facts, with the start time and venue inside the boundary. The same page without JSON-LD is "opened, nothing relevant found; not citable".
• Merging with fix(pixel): read PDF text in the shared page reader; binary bodies are never page reads #6724: PDFs are not HTML and never reach this code.

Risks

• Markup data can be stale or wrong, and Pixel may repeat it. The block says the data comes from markup, but the model may still cite it. Examples seen in the replay:
  • The Verge's RTX 5070 review carries affiliate offers at 549.99 USD (Best Buy, NVIDIA, Newegg) with no date or stock state;
  • philadelphiaevents.guide gives a Phillies game a 03:30 UTC-0400 start;
  • Reading Terminal Market's festival starts 2026-07-01 in its markup.
• Host verification can accept a claim backed only by markup. That is intended, but a wrong markup date on the cited page can verify. A stated time that disagrees with the markup still fails.
• Listing pages lose some overview space. Up to about 2.5 KB goes before the visible text on those pages. pixel_ods_web_extract overview mode shows 6,000 characters, so the block takes some of that space. Query mode searches the whole text.
• With feat(pixel): search and read public pages in one call (pixel_ods_search_read) #6698's search_read:
  • dated block lines count toward listing detection. That is correct for listings (5 or more events), and own pages show one item;
  • search_read strips bare URLs from excerpts, so a block line ends in url: there.
• Microdata is a simplified tag walk, not a full HTML parser. Odd markup can drop a property or attach it to the wrong item. The failure mode is a missing or thinner line.
• Gateway-thread CPU is added per HTML read: median 0.5 ms, p95 about 2 ms, max 3.7 ms on the corpus.

Replay artifacts are on tower2 in /mnt/bulk/claude-tests/reader-jsonld/: replay.jsonl, replay-run1.jsonl, blocks/, replay.mjs. That is a throwaway dir; no ods-* container was touched.

🤖 Generated with Claude Code