Skip to content

fix(ios): scan the user's own photos in onboarding instead of inventing a kitchen - #9

Draft
SamGu-NRX wants to merge 8 commits into
feat/minimal-product-20261007from
fix/onboarding-real-scan
Draft

SamGu-NRX wants to merge 8 commits into
feat/minimal-product-20261007from
fix/onboarding-real-scan

Conversation

@SamGu-NRX

Copy link
Copy Markdown
Owner

Stacked on #4. Base: feat/minimal-product-20261007.

What changed

Onboarding scans the photos the user takes

Onboarding asks the user to photograph their fridge and pantry, but the review step never looked at those photos. After a short delay it showed 14 hard-coded ingredients with made-up confidences labelled as Vision results, and "Add to My Kitchen" wrote them into the real Kitchen. A stir-fry photo produced bread, eggs and butter at 94%.

The review now runs the same recognition as Scan Ingredients (VisionService.scan) on the captured photos, fridge and pantry separately. ConfidenceRouter decides what starts checked, as in the main review. OnboardingKitchenDemoDetections is deleted; nothing else used it. The logic is in OnboardingKitchenScan.swift, and the screen is OnboardingKitchenReviewStep in OnboardingInventorySteps.swift.

Empty, failed and partial scans say what happened

Situation What the review shows
No photos taken "No photos to scan", Continue
Photos scanned, nothing found "We didn't find ingredients in these photos. You can add them from Kitchen any time.", Continue
Scan failed "Couldn't read your photos", Try Again or Continue; nothing is saved
One location failed The other location's items stay reviewable; the failed section has its own Try Again
Some photos unreadable Found items plus "Some fridge photos couldn't be read.", Try Again

Unreadable means every Vision request failed on a crop. On the simulator, classification always fails while OCR works, and that doesn't raise this warning.

Confirming twice never duplicates, loses or revives food

The user can go back and confirm again any number of times.

  • One transaction per run: each confirmation is one ingestConfirmedScan call under a single session reference for the onboarding run, so it saves completely or not at all. An item that moves between the fridge and pantry sections on a rescan stays one lot. Food that was cooked in between isn't added back.
  • Lots filed by photo: lots are stored under the location of the photo they came from (ScanLotLocation.photographed(byIngredient:)). They used to land under "Other". If an added ingredient has no location, intake throws IntakeError.missingLocation instead of guessing. The main scan passes .inferredFromIngredient and behaves as before.
  • What a scan can't speak for is kept: ingestConfirmedScan takes a required preserving set, and lots for those ingredients are left exactly as they are. Onboarding preserves an earlier-confirmed item when its section's latest scan failed, had unreadable photos, or has no photos now, as long as the item isn't shown and the user hasn't unchecked it. The main scan passes [].
  • Choices last the run: check and uncheck choices survive rescans, failures and retries. Only foods seen for the first time start checked.
  • Failed saves: a failed save shows "Couldn't save to your Kitchen" with Try Again and Continue Without Saving, the same pattern as the main review. Try Again retries the confirmation the user made, even if a late photo import has refreshed the review behind the alert.
  • Navigation: a photo that finishes importing after the review opens triggers a rescan instead of leaving the screen stuck, and leaving mid-scan stops the remaining work. Back from "You're ready" now returns to the review; it used to replay the setup animation and come straight back.

Capture and accessibility fixes found on the way

  • Library imports now go through ScanImagePreprocessor.prepare, so orientation and the 1600 px cap apply.
  • FLCaptureView records whether each photo came from the shutter or the library, through a new FLCapturedPhoto. The old [UIImage] initialiser still works for its other callers.
  • Remove buttons on photo thumbnails, in onboarding and the shared camera screen, have 44 pt targets and names like "Remove fridge photo 2".
  • VoiceOver announces when the scan finishes or fails, and focus moves to the review heading.
  • The "0 of N selected" row shows an empty circle instead of a check, and a section with one result says "1 item".

Verification

  • Simulator walkthrough at 1ead827 (iPhone 17, iOS 26.5):
    • Fridge photo: the taco scenario. Pantry photo: the stir-fry scenario.
    • The review showed the real detections: cheese and beans at 55% unchecked, soy sauce at 90% checked.
    • After confirming, the database held cheese (120 g, estimate) filed under fridge and soy sauce (30 g, estimate) under pantry.
    • Back from "You're ready" returned to the review, and confirming again added nothing.
    • Kitchen showed Fridge 1 and Pantry 1 with "≈ 120g est." and "≈ 30g est.".
  • Earlier walkthrough at 6829370: skipping both photos showed "No photos to scan", and unchecking an item on a revisit retired its lot.
  • Screenshots (local): ~/.long-run/fridgeluck/evidence/lead-verify/ (onboarding-review-taco.png, kitchen-after-onboarding-revisit.png, kitchen-final-onboarding.png).
  • Full simulator suite at 1ead827: 192 of 192 passed. The 42 tests in OnboardingKitchenReviewTests drive OnboardingKitchenIntake.commit against a real migrated database, covering:
    • an item moving between sections, untouched, partly consumed and fully consumed
    • failed, unreadable and photo-less sections
    • a forced write failure that leaves nothing changed
    • a retry after a refresh
  • 21dd039 only changes the section-count text; the whole-app typecheck passes on it.
  • Not checked: the save-failure alert by hand, VoiceOver announcement timing, Dynamic Type layout and motion.

Checklist

  • The scope is focused; related changes belong together
  • I explained what changed and why
  • N/A uploaded before/after images: gh can't attach images. The screenshots listed above are local files
  • N/A video: no new motion

🤖 Generated with Claude Code

SamGu-NRX and others added 7 commits October 7, 2026 10:37
- Run the main scan's recognition on the fridge and pantry photos
  separately, route results through ConfidenceRouter, and preselect only
  sure items. The 14 hard-coded detections are gone, so nothing the photos
  didn't show can reach the Kitchen.
- Show honest states: no photos, nothing found, and a failed scan with
  Try Again and Continue. If one location fails, the other's items stay
  reviewable and only the failed one is retried. A failed scan commits
  nothing.
- Keep one intake session per onboarding run, so coming back and adding
  again reconciles the Kitchen instead of adding a second copy. Choices
  for items already shown survive a rescan.
- Record whether each photo came from the camera or the library in
  FLCaptureView, and prepare library imports like other scan inputs.
- Give the photo remove and retry controls 44 pt targets and specific
  names, and announce scan results to VoiceOver.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The remove button was a 16 pt icon offset outside the thumbnail. It now
has a 44 pt target inside the thumbnail and names the photo it removes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… failures

- Reconcile fridge and pantry as separate intake sessions, and only for
  locations whose latest scan succeeded, so a failed rescan can't remove
  food the user already confirmed.
- File onboarding lots under the location they were photographed in.
  Intake now takes an explicit location; the main scan keeps inferring it.
- Keep the user's check and uncheck choices for the whole run, so a failed
  scan and retry don't reset them.
- Rescan when a late photo import changes the photos, drop stale results,
  and stop work when the user leaves mid-scan.
- Warn when some photos couldn't be read instead of reporting nothing found.
- Back from the final step returns to the review instead of replaying setup.
- Show an empty circle, not a check, when nothing is selected.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Use one intake session for the whole onboarding run and one call per
  confirmation. An item whose higher score moves between fridge and
  pantry on a rescan keeps its one lot, and food already cooked is
  never added back.
- File each new lot under the section its detection sits in, through
  a per-ingredient location map. Intake throws a specific error rather
  than guessing when the map has no entry; the main scan still infers.
- Add a required `preserving` set to scan intake. Onboarding uses it
  to leave earlier confirmations alone when their section's photos
  failed or weren't all read this time, unless the review shows them
  again or the user unchecked them.
- Show "Couldn't save to your Kitchen" with Try Again and Continue
  Without Saving when the write fails, instead of moving on silently.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A section with no photos says nothing about its food, so onboarding now
treats it like a failed or unread scan: items an earlier confirmation
added from it, and not shown now, are preserved unless the user
unchecked them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When saving the kitchen review fails, keep the exact scan and choices
the user confirmed. Try Again retries that confirmation, so a late
import that publishes new preselections behind the alert can't be
saved unconfirmed. Continue Without Saving discards it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Real scans often find a single item per section, which read "1 items".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 7, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

… the macos-26 CI workflow

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SamGu-NRX added a commit that referenced this pull request Oct 11, 2026
Folds fix/intake-inference-semantics @ 35c358b (based on #9's branch) into the app integration trunk, fold 2 of 12.

Co-authored-by: obvious-autobuild[bot] <262744130+obvious-autobuild[bot]@users.noreply.github.com>
SamGu-NRX added a commit that referenced this pull request Oct 11, 2026
Folds obv/fridgeluck-056 @ 4f51d21 (based on #9's branch) into the app integration trunk, fold 7 of 12.

Co-authored-by: obvious-autobuild[bot] <262744130+obvious-autobuild[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant