Repository navigation
feat(evaluation): audit legacy recipes that mislead or slip past the safety boundary - #209
Conversation
The plan and its screens see ingredient lines; anything the steps name
but the lines omit never reaches them. This audit checks all 166 legacy
recipes line-by-line against that boundary and ships the resulting hide
list:
- (a) missing headline: a food named in the title, or an animal/gluten
food named in the steps, with no matching ingredient line
- (b) diet-tag contradiction: vegan/vegetarian/low_carb/gluten_free vs
the families present in title, steps or lines
- (c)/(d) implausible required mass and sodium: review-only bands
- (e) third-party source and (f) rights quarantine: recorded, quarantine
always hides
- (g) every (a)/(b) firing was hand-checked against the full recipe
text; 36 false positives across 25 recipes were dropped with the
reason recorded (serve-with suggestions, tool names, optional
add-ins, similes, 'cauliflower steaks')
Matching uses curated term families with longest-match-wins span
blanking ('butter beans' never fires dairy 'butter'), plural variants,
and generic-sibling suppression (a salmon line satisfies 'fish').
Regeneration is byte-identical; 128 of 166 recipes hide.
Co-authored-by: Sam Gu <sgu07966@gmail.com>
28 tests run the audit against the real bundled data.json and quarantine.jsonl: phrase-over-inner-word matching, plural variants, generic-sibling suppression, title/steps missing-headline behavior, tag contradictions, the mass/sodium bands, the drop mechanism on the hand-checked ids, quarantine mapping, and byte-stable regeneration whose output matches the committed JSONL. Co-authored-by: Sam Gu <sgu07966@gmail.com>
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Note Review skipped Codemetrics is installed on this repository, but it hasn't been added to your team yet. To enable AI code reviews for this repo, go to your dashboard and add FridgeLuck to your team's repositories. |
What changed and why
Audits all 166 legacy recipes in
apps/ios/Resources/data.jsonagainst the household safety boundary from PR #204: the plan and its screens see ingredient lines, so anything the steps name but the lines omit never reaches them. Ships a pinned hide list (data/recipes/legacy-audit.jsonl, 128 rows) plus the rules engine (evaluation/legacy-recipe-audit/audit.py), tests, and report so the list can be regenerated and consumed by future evaluation tooling.Verdict: 128 of 166 recipes (77%) must be hidden — 110 on content, 18 on rights quarantine. The worst patterns: 41 recipes carry a diet tag their own text contradicts (18 "vegan" recipes name cheese, meat, fish, egg, honey, cream or yogurt — e.g. id 61 "Cod with… salsa" tagged vegetarian+vegan), and the BBC-imported block routinely ships without its headline food (id 53 "Chicken madras" has no chicken line; id 43 cooks orecchiette and ricotta and ships oil + garlic; 28 recipes miss cheese, 23 miss chicken, 18 miss pasta).
Rules, one per failure mode (full detail in
evaluation/legacy-recipe-audit/REPORT.md):quarantine.jsonl→ bundled ids 1–25)Matching uses curated term families with three structural guards, each added to kill an observed false-positive class: longest-match-wins with span blanking ("butter beans" never fires dairy "butter", "coconut milk" never "milk", "egg noodle" carries both families), plural variants ("peanuts", "anchovies", "egg noodles"), and generic-sibling suppression (a salmon line satisfies a generic "fish" mention).
The hand-check. Every (a)/(b) firing was read against the full recipe text; 36 findings across 25 recipes were dropped, each with its reason recorded in the row's
dropped_false_positivesfield and in the script'sDROPPED_FALSE_POSITIVEStable. Six classes: "serve with" sides (naan/rice/couscous/flatbread), either/or alternatives ("lettuce leaves or bread", "honey or sugar", "if you like"), tool names ("potato masher" is not a potato), negligible quantities (a 1-tbsp flour dusting does not break low_carb), optional add-ins (id 154's "Choose your toppings" menu fired seven flags on an otherwise correctly-vegan one-pot), and non-ingredient text ("cauliflower steaks" is a cut, "wrap up like a cracker" is a simile, "some nut butters are thicker than others" is prose about an ingredient already on a line). Four recipes (49, 92, 102, 163) drop out of the hide list entirely.How I checked it
python3 evaluation/legacy-recipe-audit/audit.py→recipes=166 hides=128 (a)162 (b)41 (c)30 (d)4 (e)141 (f)25 (g)36, wrotedata/recipes/legacy-audit.jsonl; byte-identical across consecutive runs (test-pinned).python3 -m pytest evaluation/legacy-recipe-audit/ -q→ 28 passed (14.9 s): phrase-over-inner-word matching, plurals, generic-sibling suppression, missing-headline and tag-contradiction behavior, mass/sodium bands, the drop mechanism on ids 15/48/92/105/154, quarantine mapping, regeneration determinism, and committed-JSONL-matches-current-rules.What I couldn't check
Assumptions
Human author: Sam Gu (sgu07966@gmail.com)
🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact