Side - #7
Open
ibrahim-khan12 wants to merge 4 commits into
Open
Conversation
The AI's system prompt never told the model whether a menu came from a photo, a website, or a web search. So during ordinary conversation (not just explicit "where did this come from" questions, which the provenance intent-answer system already handled), it could casually claim things like "since this is the most up-to-date menu" or "found online" even for a menu the guest just photographed in person. Feed the same MenuProvenance facts already computed for the provenance intent answers (lib/provenance.ts) into buildSystemPrompt for every turn, with an explicit guard against online/recency claims when sourceType is 'photo'. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The guided scanner only checked lighting, glare, blur, and steadiness — it
had no awareness of framing, so it would fire the shutter on a menu photo
taken extremely close, cropped, or crooked without any coaching to fix it.
Extract the per-frame analysis into a pure, unit-testable
computeFrameMetrics() and add three new signals:
- Too close: detected content's bounding box touches opposite frame edges
- Too far: bounding box is small relative to the frame
- Skewed/tilted: mean edge-gradient angle offset from the nearest axis
exceeds 12deg, via a signed-gradient orientation heuristic
These block capture the same way dark/glare/blur already do (no best-shot
bypass) — a badly framed photo is more likely to fail menu extraction than a
slightly soft one.
Known limitation, documented in tests: an exact 90deg rotation (phone held
sideways) is mathematically indistinguishable from level using local
edge-gradient orientation alone. True 90deg detection needs document
boundary/aspect-ratio detection, out of scope here. Since the app is
portrait-locked, users are far more likely to hold the phone at a moderate
unintentional tilt than a deliberate 90deg turn.
9 new unit tests in tests/scanner.test.ts synthesize grayscale frames with
known bounding boxes and rotation angles.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Added zoom: { ideal: 0.5 } as a soft hint to the initial getUserMedia
constraints, so devices with an ultra-wide lens have the best chance of
starting on it. The initial zoom now explicitly targets 0.5x (clamped to
whatever the device's native zoom range actually supports) instead of just
using the hardware's reported minimum.
Known limitation: true sub-1x zoom requires a physical ultra-wide lens
exposed through the standard `zoom` capability. Devices/browsers without
that silently fall back to 1x, same as before — there is no way to digitally
fake a wider field of view than the sensor's native capture.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The live auto-capture scanner coaches for lighting/blur/framing WHILE shooting, but that only applies in auto mode. Manual "Take photo" shots and uploaded library photos got zero quality feedback — a real gap for a user who can't glance at a thumbnail to judge whether a shot came out blurry, dark, or cropped. Export the scanner's tuned thresholds (LUM_DARK, GLARE_FRAC, SHARP_MIN, EDGE_MIN, TOO_FAR_BBOX, SKEW_WARN_DEG) as one shared source of truth for both live coaching and post-capture checks. lib/photoQuality.ts: evaluateQuality() is a pure, unit-testable function that turns frame metrics into a verdict. assessPhotoQuality() is the DOM-dependent wrapper — decodes a captured/uploaded JPEG, downsamples it to the same analysis scale the live scanner uses, and runs it through the same computeFrameMetrics(). Fails open on any decode error. CaptureScreen.tsx: every photo (manual, auto, or uploaded) is assessed immediately, with issues spoken right after capture. New "Retake last photo" button removes only the most recent shot. Tapping "Read menu" with any flagged photo announces a one-time confirmation before proceeding, same two-tap-confirm pattern already used for Saved delete / Settings sign-out. Deliberately not built: a visual thumbnail gallery. This is a voice-first accessibility app; a thumbnail grid isn't usable by the target audience, so the fix is spoken feedback plus a same-photo redo, not a visual review screen. 9 new unit tests in tests/photoQuality.test.ts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@ibrahim-khan12 is attempting to deploy a commit to the avitaldrel's projects Team on Vercel. A member of the Team first needs to authorize it. |
Collaborator
Author
|
@copilot resolve the merge conflicts in this pull request |
ibrahim-khan12
added a commit
that referenced
this pull request
Jul 21, 2026
Bug #11's changes fixed the underlying behavior; this proves it, since extractSessionLearnings/chatReply can't run under the plain Node test runner (they rely on import.meta.env and relative fetch() paths that only resolve inside Vite). Ran two ways this session: 1. Directly in the running app's browser console, importing the real compiled src/lib/openai.ts and calling the actual production functions with a live API key. 2. scripts/verify-order-learning.mjs, a standalone script (same env-loading and duplicated-prompt pattern as test-find-menu.mjs and add-partner.mjs) that anyone can re-run later, e.g. after a future prompt or model change, without needing a live browser session. Both runs, against the real model, confirmed every acceptance criterion from bugs #11 and #12: - 'I'll get the salmon' -> orders: ['salmon'] - asking about ingredients/allergens, 'just checking' -> orders: [] - an unconfirmed recommendation ('I'll think about it') -> orders: [] - 'Remember that I chose the pasta' -> orders: ['pasta'] - at an unrelated Thai restaurant, a past salmon order surfaced as a soft, non-forced steer toward a relevant dish, never a literal 'get the salmon' demand for a dish not on the menu - mid-conversation, 'I'll get the pad thai' produced exactly one warm confirmation sentence, matching the rewritten prompt Review/removal of pastOrders was already covered by #11's Settings section (dislikes already had it). No further UI change needed. Bug #7 (VoiceOver detection): searched every plausible naming pattern across the codebase and git history. No such feature exists here to remove — reporting as already satisfied rather than fabricating a diff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ibrahim-khan12
added a commit
that referenced
this pull request
Jul 21, 2026
One section per bug from MENUVOICE-INTERNAL-BUGS-TO-FIX.md, matching the original numbering: what was broken, the fix, its commit, and how it was verified. Notes #7 as already satisfied (nothing existed to remove) and #14/#15 as needing a physical iPhone, with a suggested next-steps list covering the new GOOGLE_CLIENT_ID / SESSION_SECRET Vercel env vars #20 needs before deploying. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.