Skip to content

research(051): catalog product retrieval from one photo + confidence gate - #137

Merged
dompm merged 4 commits into
research/delightingfrom
research/delighting-051-catalog-retrieval
Jul 16, 2026
Merged

research(051): catalog product retrieval from one photo + confidence gate#137
dompm merged 4 commits into
research/delightingfrom
research/delighting-051-catalog-retrieval

Conversation

@dompm

@dompm dompm commented Jul 16, 2026

Copy link
Copy Markdown
Owner

What

Can we RECOGNIZE which catalog glass product a user's uploaded photo shows (CTO product idea: identify the sheet → look up cached per-SKU relief presets + metadata), and can we CALIBRATE a "confidently in-catalog" gate for out-of-catalog uploads (fallback = study 050's photo-only detection)?

Benchmark: the report-033 real cross-capture pairs, wild→clean direction — 674 window/shop queries over 176 Delphi products, index = 490 clean captures (226 products) + the 1,281-image clean corpus as realistic distractors. DINOv2-small on MPS (verified runnable first, per the 028 lesson). All 033 contamination screens applied. The 033 raw-image cache was restored via the idempotent harvest_033.py under the original approved posture — zero attrition: 254/254 products, 1,491/1,491 images, 100% capture-label agreement with the frozen manifest.

Findings

run top-1 top-5 gate AUC
raw full-frame (primary) 12.6% 26.3% 0.523
delighted-T query (extract.py) 9.8% 23.7% 0.515
019 luma-quotient 12.2% 26.4% 0.523
center-crop 50% 26.6% 44.4% 0.552
center-crop 50%, no distractors 33.2% 54.7% 0.566
  • Composition, not illumination, is half the gap: a dumb central crop DOUBLES accuracy; delighting the query actively hurts. Product lesson: ship a sheet-isolation/crop stage before any backbone or preprocessing work.
  • The confidence gate is the honest negative: in-catalog vs out-of-catalog (leave-product-out) score distributions are near-inseparable — the catalog's sibling colorways make OOC queries look like in-catalog queries of a sibling SKU. Only a ~3%-recall ultra-confident tier (cosine ≥ 0.913, measured 100% top-1 precision) survives calibration. Recommended interface: auto-confirm that tier; everything else = top-5 suggestions, with 050's photo-only path always-on (retrieval is metadata enrichment, never a switch that turns 050 off).
  • VLM top-5 verification (40 sonnet calls): 16/16 perfect rank-2..5 promotion, 78.6% top-1 confirm, 70% "none-of-these" specificity — estimated end-to-end ~38.7% top-1 at crop50, and the only measured signal that meaningfully rejects hopeless shortlists.
  • Failure mode: retrieval finds the look, not the SKU — among misses, top-1 shares the brand 37% / relief family 23% (e.g. clear English Muffle showing green foliage through it retrieves Noble Brass/Sage/Emerald Muffle). Clear glass IS its background — the color signal isn't in the photo.
  • Relief cache (scope 6): key presets by (manufacturer, texture family) — family survives the dominant wrong-colorway miss. 19.8% of the shipped registry is texture-NAMED (preset = pure metadata lookup; ~15 families cover it); the smooth-named 80% is where 050's auto-detected presets slot in. Delphi realpairs: 55.9% texture-named.

Evidence

  • reports/051-catalog-retrieval.md — full write-up
  • results/051/ — summary_all.json, summary_crop.json, per-run metrics, failure_decomposition.json, relief_name_audit.json, vlm_verify.json, qualitative boards (downscaled thumbnails only: board_raw.jpg, board_crop50.jpg)
  • catalog051/ — index build + query + benchmark + gate + VLM scripts; embeddings/caches gitignored, regenerate with run_all.py → run_crop_ablation.py (~40 min MPS)

Raw corpus/realpairs photography stays local-only per the 033 posture. DO NOT MERGE without lab-lead review.

🤖 Generated with Claude Code

dompm and others added 4 commits July 16, 2026 16:06
@dompm @claude
research(051): catalog-retrieval scaffold — DINOv2 index, benchmark+g…
7af7522 
…ate harness, relief-name audit

Infrastructure for wild->clean catalog product retrieval (report 051):
- embed.py / embed_cache.py: DINOv2-small (384-d) MPS embedder, verified
  runnable (report 028 lesson); path-keyed cache with per-representation keys.
- build_clean_index.py + clean_index_meta.json: 1,281-image clean-corpus index
  (the shipped registry / realistic distractor pool). Embeddings gitignored.
- retrieve.py: cosine + product-level max-pool scoring, leave-product-out.
- rp_data.py / realpairs_bench.py: wild->clean benchmark on the 033 realpairs
  (query=window/shop, target=closeup/lightbox) + calibrated in-catalog vs
  out-of-catalog confidence gate (AUC + precision/recall).
- transforms.py: raw / delighted-T (extract.py) / 019 luma-quotient reprs.
- vlm_verify051.py: claude-CLI top-k 'same product?' verification (sonnet).
- make_board.py: qualitative board (query | top-3 | correct).
- relief_name_audit.py + json: scope-6 per-SKU relief-cache metadata audit.

Realpairs raw images restored via idempotent harvest_033.py (033 posture);
LOCAL-ONLY, gitignored. Metrics/report to follow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dompm @claude
feat(051): any-capture leave-1-out diagnostic + run_all orchestrator
9f5b9e9 
- retrieve.py: per-query row exclusion (leave-one-image-out).
- realpairs_bench.py: any-capture diagnostic isolating the clean-reference
  penalty from raw same-product matching; board data with resolved paths.
- run_all.py: one-pass suite (raw/delight/quotient x distractors, holdout
  scope, diagnostic) -> results/051/summary_all.json.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dompm @claude
results(051): full benchmark metrics — wild->clean retrieval + gate +…
7df042f 
… ablations

Headline (674 wild queries, 176 Delphi products, 1,771-entry index w/ clean-
corpus distractors, DINOv2-small):
- raw top1=12.6% top5=26.3%; gate AUC 0.523 (no usable separation)
- delighted-T HURTS (9.8%); 019 luma-quotient no effect (12.2%)
- center-crop50 DOUBLES accuracy: 26.6%/44.4% (33.2%/54.7% w/o distractors)
  -> composition/scene, not illumination, is half the gap
- any-capture diagnostic == clean-target (12.8%) -> no clean-reference penalty
- ultra-confident tier: cosine>=0.913 covers 2.7% of queries at 100% top-1
- failure decomposition: among crop50 misses top-1 shares brand 37%, relief
  family 23% -> finds the look, not the SKU (colorway/line confusion)

Boards: board_raw.jpg / board_crop50.jpg (downscaled thumbnails only).
Heavy per-query/curve/board JSONs + embedding caches gitignored (regenerable
via catalog051/run_all.py + run_crop_ablation.py).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dompm @claude
research(051): report — catalog retrieval from one photo + confidence…
e0f5216 
… gate

Verdict: exact-SKU recognition is a suggestions feature (crop50 26.6% top-1 /
44.4% top-5 at realistic scale; naive full-frame 12.6%); the cosine confidence
gate does NOT separate in- vs out-of-catalog (AUC 0.52-0.57) — only a ~3%
ultra-confident tier (>=0.913, measured 100% top-1) survives calibration.
Center-crop doubles accuracy (composition, not illumination, is the gap);
delighting the query hurts. VLM top-5 verification: 16/16 rerank promotion,
70% none-of-these specificity -> estimated end-to-end ~38.7% top-1. Relief
cache keys off texture family, which survives the dominant same-family-wrong-
colorway miss mode; 20% of registry SKUs are texture-named (pure metadata
lookup), realpairs 56%.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 16, 2026

Copy link
Copy Markdown

Deploying vitrai with  Cloudflare Pages  Cloudflare Pages

Latest commit: e0f5216
Status: ✅  Deploy successful!
Preview URL: https://93eca913.vitrai.pages.dev
Branch Preview URL: https://research-delighting-051-cata.vitrai.pages.dev

View logs

@dompm
dompm merged commit a0e4b7b into research/delighting Jul 16, 2026
2 checks passed
@dompm
dompm deleted the research/delighting-051-catalog-retrieval branch July 16, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dompm