fix(extraction): read stickers at any orientation, and flag what it can't - #142
Merged
Conversation
…an't Customer report: pages carrying an upright sticker AND one rotated 90° came back with only the upright entry. The rotated one wasn't wrong, it was ABSENT — and an absent entry has no low-confidence field to catch the eye, so the owner had no way to know it was missing. The prompt explains it. Stickers were described only as content to MERGE into a single entry, entries were ordered "top-to-bottom", and orientation was never mentioned at all. A sideways sticker had no slot in any of that. Stage 1 — prompt. States that shops affix stickers at any orientation (90°, upside down, angled), that EACH sticker is normally its own entry, and that ordering never justifies omitting one. Costs nothing extra. Stage 2 — turn a silent miss into a visible one. The model now reports unread_rotated_content when it can see rotated content it couldn't fully read (required in the schema, so it can't quietly default to false). That triggers ONE targeted second pass over the same image, and if that still comes up short the page keeps the flag and Review shows an amber warning pointing at "Add an entry the extractor missed". The second pass re-reads the same bytes with a narrower prompt rather than rotating the image: a vision model reads rotated text when told to look, and re-encoding would mean importing sharp into app code — which package.json deliberately avoids, since it's pinned for a CVE precisely because user-uploaded images reach it via image optimization. Gated on the model asking, so the extra call lands on the few pages that need it rather than tripling the cost of every page against the per-user daily cap and the global dollar ceiling. Best-effort throughout: a failed retry never costs the first pass's entries. stillUnreadAfterRetry() is a separate pure function with its own test because inverting it would suppress the warning on exactly the pages that need it. An empty retry is NOT evidence the page is clean. Migration 0052 adds page.unread_rotated_content (default false — pages extracted before this were never assessed, so false means "nothing to flag", not "verified clean"). Claude-Session: https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R
Semgrep (unsafe-formatstring, blocking) caught pipeline.ts passing an interpolated template AND a second argument to console.error — that makes it a util.format call, so an injected format specifier could forge the log line. Fixed there, and in the two denial logs added by #141, which semgrep did NOT flag only because they pass a single argument. Those are the riskier pair: `path` comes from the request URL, so it is attacker-influenced. Values are now arguments to a constant format string. Claude-Session: https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Customer report: pages carrying an upright sticker and one rotated 90° came back with only the upright entry.
The rotated entry wasn't wrong — it was absent. And an absent entry has no low-confidence field to catch the eye, so the owner had no way to know it was missing. That's a worse failure than a misread value, and it's the part this PR is really about.
Why it happened
Three things in
EXTRACTION_SYSTEM_PROMPTcombined:"return one object per entry, in top-to-bottom order"bakes in one vertical reading order. A sideways sticker has no slot in it.Stage 1 — prompt (no extra cost)
States that shops affix stickers wherever there's room, at any orientation (90°, upside down, angled); that each sticker is normally its own entry; and that ordering never justifies omitting one.
Stage 2 — make a miss visible
The model now reports
unread_rotated_contentwhen it can see rotated content it couldn't fully read. It's required in the schema, so it can't quietly default tofalse.That triggers one targeted second pass, and if that still comes up short the page keeps the flag and Review shows an amber warning pointing at "Add an entry the extractor missed".
The second pass re-reads the same bytes with a narrower prompt rather than rotating the image. A vision model reads rotated text when told to look — and re-encoding would mean importing
sharpinto app code, whichpackage.jsondeliberately avoids: it's pinned for a CVE precisely because user-uploaded images reach it via image optimization, and the comment there notes that nothing in the app imports it.Gated on the model asking, so the extra call lands on the few pages that need it rather than tripling the cost of every page against the per-user daily cap and global dollar ceiling. Best-effort throughout — a failed retry never costs the first pass's entries.
The bit most worth reviewing
stillUnreadAfterRetry()is a separate pure function with its own test, because inverting it would suppress the warning on exactly the pages that need it:An empty retry is not evidence the page is clean. It errs toward keeping the warning.
Migration
0052_page_rotated_content.sqladdspage.unread_rotated_content, defaultfalse— documented as "nothing to flag", not "verified clean", since pages extracted before this were never assessed.Apply to prod and test before merging. 462 unit tests, typecheck, lint and build clean.
/helpupdated.Worth asking the customer
A photo of one of those pages would be worth more than my reasoning about which stickers get missed — and it becomes the regression fixture. Also worth telling them "Add an entry the extractor missed" already exists on the review screen as a manual escape hatch today.
https://claude.ai/code/session_01XBNGwWrPih2Xgu6MVrcd6R