feat(benchmarks): dish-ambiguity-v1 — taxonomy, manifests, and evaluated classifier pins

[obvious-autobuild]

Port onto alpha (2026-10-15, head e6fa5b3)

The original base feat/routing-eval-harness is stale: alpha (the current integration branch) is 64 commits ahead of it. The change is ported onto alpha by merging origin/alpha @ 0309dd1 into obv/fl-l2-dish-ambiguity (merge commit e6fa5b3, no conflicts; the PR base is retargeted to alpha). The merge brings in alpha's iOS/CI/skills changes only - it does not touch benchmarks/dish-ambiguity-v1.

Evidence preservation, verified on the merged head:

• git diff a6c921a..e6fa5b3 -- benchmarks/dish-ambiguity-v1 is empty: the 21-file study diff, all committed predictions/scored/report artifacts, and the pinned weight/output hashes are byte-identical to the reviewed head a6c921a.
• check_manifest.py → exit 0 on the merged tree.
• python3 -m pytest benchmarks/dish-ambiguity-v1/tests -q → 18 passed.
• python3 benchmarks/dish-ambiguity-v1/score.py --verify-report → verify OK (scored.json and report.md byte-identical to committed).
• Frozen manifest hashes re-hashed on the merged head, unchanged: test manifest.json 85ae3fb06242bf542df026ba03e9e2e53faae690fe50b53eba7e12a984a12ec2, development dev_manifest.json 90dd25dbc785a22a90b5efd6b98608ef25b38b62d36311f9a96b6701d71615c6 (both match results/run-metadata.json).
• Hosted macOS CI on the merged head (run 38738162582): build-and-test pass (6m40s) and dead-code-advisory pass (3m10s).

Everything below (the original dish-ambiguity experiment) is unchanged.

What changed and why

benchmarks/dish-ambiguity-v1 answers the routing question: when a fridge photo is classified into one of Food-101's 101 dish labels, when can the app safely show a specific bundled recipe? Food-101 classes and the 166-recipe bundled catalog overlap only partially, and some labels are genuinely ambiguous (e.g. apple_pie vs cheesecake-family renders), so a blind "show recipe for the predicted dish" policy can render the wrong recipe with high confidence.

Three commits, one per milestone, then review-fix commits:

1. Taxonomy freeze (f3fe481): every Food-101 class is mapped to the bundled catalog with status exact (one recipe answer), coarse (an equivalence class of recipes), ambiguous (multiple non-equivalent renders; no answer claimable), or unsupported (nothing close in the catalog). 11 exact / 3 coarse / 2 ambiguous / 85 unsupported. check_manifest.py validates coverage, recipe existence, uniqueness, and no-overlap rules.
2. Manifests (05c6ef0): seeded stratified samples from the official Food-101 splits - 25 test images/class (2,525) and 6 dev images/class (606), with per-image sha256 + dHash; 5 dev near-duplicates of test images excluded and refilled. Rebuild is byte-identical.
3. Evaluation (12c1f23): both pinned classifiers run over both manifests on CPU; scorer computes accuracy + a dev-selected abstention threshold (suggest only exact-status predictions above tau, targeting 0.90 suggestion precision on dev, coverage-maximizing), and the committed report/scored JSON are re-verifiable byte-for-byte via score.py --verify-report.
4. Review fixes (9d84084): emitted paths canonicalized (repo-relative inside the checkout, bare names otherwise; no /home/user under the committed results), --verify-report confirmed from a different working directory, and the report/README lead with unrestricted suggestion precision.
5. Review fix follow-up (a6c921a): the README Handoff section (the one reported for 9d84084 had not landed in that commit) with the frozen manifest hashes.

Headline results

• nateraw/vit-base-food101 @ 55859a2 is the only viable pin: top1 0.8075, top5 0.9620 (2,525 test images). At the dev-selected tau (0.1), unrestricted suggestion precision is 175/183 = 0.9563 (coverage 0.0725) across all test images. Restricted to the 350 eligible test images (true class exact/coarse), precision is 175/175 = 1.000 at 0.500 coverage. Recommendation: adopt it for recipe suggestion with this abstention policy; abstain otherwise.
• rajkr/mobilenet-v2-food101 @ 0dea82e is broken: top1 0.0614 via a confident collapse - five labels absorb 2,196/2,525 predictions, often with top1 probability ~1.0, so no threshold can separate right from wrong. Cross-checked against the canonical HF image-classification pipeline at the same revision (identical outputs), so the checkpoint is at fault, not our runner.
• Practical caveat: even the good ViT is underconfident (most correct predictions sit far below 0.5), so the threshold is dev-selected, not hand-picked, and its test-side precision (0.9563 unrestricted) is reported alongside dev (0.9111).

How I checked it

Re-verified in this review round at head a6c921a, from a fresh independent worktree of origin/obv/fl-l2-dish-ambiguity:

• cwd-independent report verification: cd /tmp && python3 /home/user/work/fridgeluck-57-review/benchmarks/dish-ambiguity-v1/score.py --verify-report → score: verify OK - scored.json and report.md byte-identical to committed (exit 0). The same command from the checkout root gives the identical output and exit 0.
• Path canonicalization: grep -rn "/home/user" benchmarks/dish-ambiguity-v1/results benchmarks/dish-ambiguity-v1/*.py benchmarks/dish-ambiguity-v1/README.md → no matches. The report emits Results: benchmarks/dish-ambiguity-v1/results (repo-relative). git diff 12c1f23..a6c921a shows the only changes to prediction payloads and run-metadata.json are path strings (/home/user/... → repo-relative/bare); model repo/revision/weights-sha256 pins, dataset tarball sha256, and manifest sha256s are byte-identical.
• Report emphasis: the report's per-model blocks list unrestricted suggestion precision first with its own denominator (ViT: precision 175/183 = 0.9563, coverage 0.0725 over 2,525), and the eligible-only statistic on a separate line with a separate denominator (175/175 = 1.0, coverage 0.5 over the 350 eligible images); the README Results section leads with unrestricted 175/183 for the recommended pin. The two claims are never blended.
• python3 -m pytest benchmarks/dish-ambiguity-v1/tests -q → 18 passed (validator planted failures + scoring hand controls); re-run at head.
• python3 check_manifest.py → exit 0 (check_manifest: OK); re-run at head.
• Manifest rebuild determinism → md5sum -c OK, byte-identical (at evaluation time).
• Full run: both models × both manifests, 0 recorded failures; weights verified by sha256 against the pinned HF revisions (hashes in results/run-metadata.json).
• Runner vs canonical pipeline cross-check on 30 images → identical label behavior (this is how the MobileNet checkpoint was confirmed broken).
• Food-101 archive verified: sha256 d97d15e4…4053a4, 101 classes, no cheeseburger class in this distribution.

Handoff

For whoever wires recipe suggestion into the app:

• Pin: nateraw/vit-base-food101 @ 55859a2a1349 (weights sha256 61f707d9d423461b..., recorded in results/run-metadata.json). Do not pin rajkr/mobilenet-v2-food101 @ 0dea82e - the checkpoint is broken (see Headline results); if a small mobile model is wanted, re-run this benchmark over a replacement candidate before trusting it.
• Policy: suggest the mapped recipe only when (a) the predicted class has status exact in mapping.food101-v1.json, and (b) its top1 probability >= 0.1. Expected test behavior: unrestricted suggestion precision 175/183 = 0.9563 (coverage 0.0725), eligible-only precision 175/175 = 1.000 at 0.500 coverage on the 350 eligible images - keep the two denominators separate when quoting. Show nothing otherwise; unsupported/ambiguous classes should render dish identity only.
• Re-verify after any change to model, mapping, or thresholds: python3 benchmarks/dish-ambiguity-v1/score.py --verify-report (works from any directory; byte-compares the committed report - confirmed from /tmp at head a6c921a). Full reproduction: run.py --dataset-root <food-101> --out benchmarks/dish-ambiguity-v1/results, then score.py; validate first with check_manifest.py, tests with python3 -m pytest benchmarks/dish-ambiguity-v1/tests -q.
• Frozen manifest hashes (sha256, at head a6c921a, must match results/run-metadata.json): test manifest.json (2,525 images) 85ae3fb06242bf542df026ba03e9e2e53faae690fe50b53eba7e12a984a12ec2; development dev_manifest.json (606 images) 90dd25dbc785a22a90b5efd6b98608ef25b38b62d36311f9a96b6701d71615c6. Both are committed; changing either is a reproducibility break, not a tweak - rebuild with build_manifest.py and re-pin deliberately. A rebuild that hashes differently is a different sample - do not rescore the committed results against it.
• Resume from a clean checkout: python3 benchmarks/dish-ambiguity-v1/check_manifest.py (exit 0 = taxonomy/manifest consistency; re-verified at head), python3 -m pytest benchmarks/dish-ambiguity-v1/tests -q (18 passed at head), then score.py --verify-report above.
• Open items: per-class thresholds instead of one global tau; a tau transfer check on future data (dev precision 0.9111 vs test 0.9563 is one sample); macOS CI has not run on this PR - the benchmark is Linux-runnable Python only and touches no Swift code.

What I could not check

• No Xcode/macOS in this environment: nothing here builds or runs the iOS app; the benchmark is Python-only and does not touch Swift code. Hosted macOS CI has not run on this PR yet.
• The ViT checkpoint claims ~85-90% accuracy in its model card; I measured 0.8075 top1 on my 2,525-image official-split sample (this distribution's test split). I did not investigate the delta beyond confirming the pipeline agreement.
• tau transfer risk: dev (606 images) selects tau; test precision (0.9563 unrestricted) is a point estimate from one sample, with a Wilson-style uncertainty not computed. The 1.000 precision over 175 eligible-image suggestions has a non-trivial confidence interval; treat "never wrong" as "0 wrong in this sample".

Assumptions

• Class-level mapping decisions (exact/coarse/ambiguous/unsupported) were curated from bundled recipe titles and dish identity, not ingredient overlap (imported ingredient mappings are unreliable).
• This distribution's meta/classes.txt lacks cheeseburger; the burger family is represented by hamburger alone, so the frozen class list is authoritative.
• The abstention policy (single global probability threshold on exact-status predictions only) is the minimal deployable policy; per-class thresholds are future work.
• Food-101 dataset and model weights are not committed (hashes only), per the benchmark's reproducibility contract.

Human author: Sam Gu (sgu07966@gmail.com)

🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: fe5414da-fe59-497d-9cf6-ed47f1c4c5bf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}