Skip to content

fix(ios): stop camera labels resolving to unrelated foods and record failed Vision requests - #8

Draft
SamGu-NRX wants to merge 13 commits into
feat/minimal-product-20261007from
fix/recognition-precision
Draft

SamGu-NRX wants to merge 13 commits into
feat/minimal-product-20261007from
fix/recognition-precision

Conversation

@SamGu-NRX

Copy link
Copy Markdown
Owner

Stacked on #4. Base: feat/minimal-product-20261007.

What changed

Camera labels no longer turn into unrelated foods

When a Vision classification label missed the curated lexicon, the catalog resolver fell back to prefix matching. Generic labels became specific foods: "drink" became Strawberry Kefir, "seasonings" became Salt Pork, and "raw_glass" became Soy Vermicelli.

IngredientCatalogResolver.resolve(_:matching:) now takes a required mode. Classification labels use .exact: names and aliases are searched together, and a match counts only when exactly one ingredient answers. OCR text and recipe generation pass .allowPrefix, which keeps their old behaviour, and each call site states its choice.

Label text split across two lines is read as one food

Cans often print a name on two lines. "BLACK" and "BEANS" each failed the lexicon on their own, and "BEANS" then fell through to the catalog as Shellie Beans.

IngredientOCRLineJoining pairs a line with the nearest horizontally aligned line directly below it. It joins them only if the combined phrase is a curated food and neither line is one alone. It never chains a third line, and it can't skip a line in between: "BLACK / KIDNEY / BEANS" doesn't become black beans. Within the same photo, catalog matches for a join's separate words are dropped, so "OLIVE OIL" no longer also adds a second olive oil from a lone "OLIVE".

The alignment limits (70% horizontal overlap, gap from -25% to 100% of the taller line) were chosen on the five bundled demo photos and haven't been tuned on anything else.

Each failed Vision request is recorded

passErrors recorded a crop only when both classification and OCR failed on it, so a scan where classification failed everywhere but OCR worked looked healthy. Now:

  • Each failed request is stored as a typed ScanRequestFailure in ScanDiagnostics.requestFailures and the saved scan run. Older saved runs decode with an empty list.
  • The scan report sheet shows "Request failures: classification N · text M".
  • passErrors keeps its old meaning, so benchmark validity hasn't changed.
  • A scan where every request fails still throws, but the error now carries the failures, and the benchmark report keeps them.

This explained the simulator gap. On the iOS 26.5 simulator, every VNClassifyImageRequest fails with "Failed to create espresso context", so only OCR runs there. On macOS the same photo yields 11 detections. Simulator scans can't be used to judge recognition quality.

Why

Wrong identities are worse than misses, because the review pre-checks confident items and they go straight into the Kitchen and recipe matching.

Verification

Replay of saved Vision observations from the five bundled photos, through this branch's resolver, lexicon and joiner. This is not fresh inference or a benchmark.

Photo Visible foods found Detections Not in photo
Asian stir-fry 5 → 5 9 → 7 4 → 2
Garlic fried rice 3 → 3 5 → 4 2 → 1
Mediterranean 5 → 5 10 → 8 5 → 3
Quick breakfast 2 → 2 4 → 3 2 → 1
Taco night 5 → 6 11 → 10 5 → 3
Total 20/31 → 21/31 39 → 32 18 → 10
  • New unit tests cover exact-mode ambiguity across names and aliases, all six orderings of a three-line stack, suppression of joined parts across crops, typed failure recording, old-record decoding, and the fully failed path.
  • The whole-app iOS typecheck passes (194 files).
  • Full simulator suite at c632663: 192 of 192 passed on iPhone 17 (iOS 26.5).
  • One test drives VisionService end to end with injected Vision results, across all six crops. It checks that a lone "BEANS" in another crop is dropped once "BLACK / BEANS" is joined, and that the classification result is kept.
  • Not run: fresh Vision inference on a device.

Not in this PR

  • A scan where every request fails still isn't saved to the scan report. The error now carries the failures, but ScanView drops it, so the report sheet has nothing to show.

Two changes recovered more foods on the same photos but cost too much:

Change Foods found Not in photo Cost
3×3 crop grid 20 → 22 18 → 20 2.5× Vision requests; latency unmeasured
Confidence threshold 0.1 → 0.03 20 → 22 18 → 49 31 more wrong detections

Checklist

  • The scope is focused; related changes belong together
  • I explained what changed and why
  • N/A before/after images: the only UI change is one diagnostic caption in the developer scan report
  • N/A video: no motion

🤖 Generated with Claude Code

SamGu-NRX and others added 11 commits October 7, 2026 10:10
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…iases

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…parts

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The report's pass-error count only covers crops where both requests
failed, so a classification failure with working text recognition was
invisible. List classification and text failures separately.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… files

- Await the GRDB write inside the async VisionService test; the simulator
  build rejected the synchronous call.
- Apply the repository's swift-format configuration to the files this
  branch added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 7, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…r the macos-26 CI workflow

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Prevent descriptor and truncated OCR tokens from becoming ingredient identities through catalog prefix aliases. Preserve prefix matching for multi-word windows and cover curated olive oil and black beans recognition.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant