Skip to content

fix(extraction): read stickers at any orientation, and flag what it can't - #142

Merged
iiamit merged 2 commits into
mainfrom
fix/rotated-stickers
Aug 9, 2026
Merged

fix(extraction): read stickers at any orientation, and flag what it can't#142
iiamit merged 2 commits into
mainfrom
fix/rotated-stickers

Conversation

@iiamit

@iiamit iiamit commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Customer report: pages carrying an upright sticker and one rotated 90° came back with only the upright entry.

The rotated entry wasn't wrong — it was absent. And an absent entry has no low-confidence field to catch the eye, so the owner had no way to know it was missing. That's a worse failure than a misread value, and it's the part this PR is really about.

Why it happened

Three things in EXTRACTION_SYSTEM_PROMPT combined:

  1. Stickers were described only as content to merge into a single entry — never as entries in their own right.
  2. "return one object per entry, in top-to-bottom order" bakes in one vertical reading order. A sideways sticker has no slot in it.
  3. Orientation was never mentioned at all.

Stage 1 — prompt (no extra cost)

States that shops affix stickers wherever there's room, at any orientation (90°, upside down, angled); that each sticker is normally its own entry; and that ordering never justifies omitting one.

Stage 2 — make a miss visible

The model now reports unread_rotated_content when it can see rotated content it couldn't fully read. It's required in the schema, so it can't quietly default to false.

That triggers one targeted second pass, and if that still comes up short the page keeps the flag and Review shows an amber warning pointing at "Add an entry the extractor missed".

The second pass re-reads the same bytes with a narrower prompt rather than rotating the image. A vision model reads rotated text when told to look — and re-encoding would mean importing sharp into app code, which package.json deliberately avoids: it's pinned for a CVE precisely because user-uploaded images reach it via image optimization, and the comment there notes that nothing in the app imports it.

Gated on the model asking, so the extra call lands on the few pages that need it rather than tripling the cost of every page against the per-user daily cap and global dollar ceiling. Best-effort throughout — a failed retry never costs the first pass's entries.

The bit most worth reviewing

stillUnreadAfterRetry() is a separate pure function with its own test, because inverting it would suppress the warning on exactly the pages that need it:

return second.unread_rotated_content || second.entries.length === 0;

An empty retry is not evidence the page is clean. It errs toward keeping the warning.

Migration

0052_page_rotated_content.sql adds page.unread_rotated_content, default false — documented as "nothing to flag", not "verified clean", since pages extracted before this were never assessed.

Apply to prod and test before merging. 462 unit tests, typecheck, lint and build clean. /help updated.

Worth asking the customer

A photo of one of those pages would be worth more than my reasoning about which stickers get missed — and it becomes the regression fixture. Also worth telling them "Add an entry the extractor missed" already exists on the review screen as a manual escape hatch today.

https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R

iiamit added 2 commits August 9, 2026 13:44
…an't

Customer report: pages carrying an upright sticker AND one rotated 90°
came back with only the upright entry. The rotated one wasn't wrong, it
was ABSENT — and an absent entry has no low-confidence field to catch the
eye, so the owner had no way to know it was missing.

The prompt explains it. Stickers were described only as content to MERGE
into a single entry, entries were ordered "top-to-bottom", and orientation
was never mentioned at all. A sideways sticker had no slot in any of that.

Stage 1 — prompt. States that shops affix stickers at any orientation
(90°, upside down, angled), that EACH sticker is normally its own entry,
and that ordering never justifies omitting one. Costs nothing extra.

Stage 2 — turn a silent miss into a visible one. The model now reports
unread_rotated_content when it can see rotated content it couldn't fully
read (required in the schema, so it can't quietly default to false). That
triggers ONE targeted second pass over the same image, and if that still
comes up short the page keeps the flag and Review shows an amber warning
pointing at "Add an entry the extractor missed".

The second pass re-reads the same bytes with a narrower prompt rather than
rotating the image: a vision model reads rotated text when told to look,
and re-encoding would mean importing sharp into app code — which
package.json deliberately avoids, since it's pinned for a CVE precisely
because user-uploaded images reach it via image optimization.

Gated on the model asking, so the extra call lands on the few pages that
need it rather than tripling the cost of every page against the per-user
daily cap and the global dollar ceiling. Best-effort throughout: a failed
retry never costs the first pass's entries.

stillUnreadAfterRetry() is a separate pure function with its own test
because inverting it would suppress the warning on exactly the pages that
need it. An empty retry is NOT evidence the page is clean.

Migration 0052 adds page.unread_rotated_content (default false — pages
extracted before this were never assessed, so false means "nothing to
flag", not "verified clean").

Claude-Session: https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R
Semgrep (unsafe-formatstring, blocking) caught pipeline.ts passing an
interpolated template AND a second argument to console.error — that makes
it a util.format call, so an injected format specifier could forge the
log line.

Fixed there, and in the two denial logs added by #141, which semgrep did
NOT flag only because they pass a single argument. Those are the riskier
pair: `path` comes from the request URL, so it is attacker-influenced.
Values are now arguments to a constant format string.

Claude-Session: https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R
@iiamit
iiamit merged commit df1a147 into main Aug 9, 2026
6 of 7 checks passed
@iiamit
iiamit deleted the fix/rotated-stickers branch August 9, 2026 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant