Short version: the early numbers are encouraging, but the full evaluation hasn't been run, so Forkery makes no accuracy claims yet. The site is live as a preview only.
Accuracy is measured against comments labeled by hand, using written rules for the hard cases. For example: someone whose dough is rising overnight hasn't cooked it yet, but someone whose rolls are in the oven has.
The full plan:
- 600 comments from 30 recipes (20 blog posts across 10 sites, 10 YouTube videos), labeled by hand
- 100 of them labeled twice, a week apart, to see how consistent the labeling itself is
- two baselines: simple keyword rules ("made this", "turned out"), and a frontier chat model asked the same questions, compared on accuracy, cost and speed
- confidence thresholds tuned on 20 recipes, then checked on the other 10, and only those 10 reported
What gets measured:
| Measure | Why it matters |
|---|---|
| Accuracy on confident answers, per question | The main quality number |
| Coverage: how many comments get a confident answer | A strict threshold is accurate but leaves cards empty |
| Calibration: is "90% sure" right about 90% of the time? | The thresholds only make sense if it is |
| Card-level error: Forkery's success rate vs the hand-labeled one | What a reader actually sees |
| Cooks per recipe | Feeds the kill criteria |
Round 1 (23 September 2026): 200 comments from 10 recipes. To get started quickly, these were labeled by another AI model, not by hand. So this round measured two models agreeing, not accuracy:
| Measure | Result |
|---|---|
| "Did they cook it?", all comments | 91% agreement (182 of 200) |
| "Did they cook it?", confident answers only | 99% (163 of 165) |
| "How did it turn out?" | 96% (91 of 95 cooks) |
| Cooks per recipe (median) | 87 |
| Time per Jev call | 145 ms median, 246 ms at the slow end |
Nearly all the disagreements were answers Jev wasn't sure about, which is what the confidence thresholds are for. This round also caught the badly shaped ingredient question (see How Forkery uses Jev).
Hand labels (24 September 2026): 101 of those 200 comments, labeled by me.
| Question | Comments | Jev agrees with my labels |
|---|---|---|
| Did they cook it? | 101 | 80% |
| How did it turn out? | 56 cooks | 96% |
| Which ingredient did they change? | 56 cooks | 82% |
| Did they swap an ingredient? | 56 cooks | 82% |
| Would they make it again? | 56 cooks | 75% |
| Did they cook it longer or shorter? | 56 cooks | 98% |
These count Jev's raw answer, sure or not, so they're a floor rather than the accuracy of what the card actually shows. I changed 1 of the 101 labels after seeing Jev's answer.
"Did they cook it?" at 80% is below the 90% target, and it matters most, because it decides who counts at all. Several of the misses were people part-way through: yeast proofed, dough in the fridge, rolls in the oven. My notes on those cases went into the next version of the questions.
- "Added something" is over-reported. In round 1 it caught every real addition but flagged about twice as many as the labels did.
- Problem flags are untested. The 200 comments held only three reports of a problem (too salty, dry and so on), which is too few to say anything either way.
- the full 600 comments from 30 recipes
- the double-labeled 100
- both baselines
- threshold tuning on a held-out set, coverage and calibration
- the kill check against hand labels
The live site was built before this finished, which reverses the plan's own order. That's why it runs as a soft launch, with no accuracy claims, until the evaluation is done.
Stop or rethink if, after the full run on 30 recipes:
- the median recipe has fewer than 10 comments from people who made it, or
- "did they cook it?" accuracy on confident answers is below 85%.
Round 1 passed both, but against another model's labels, which doesn't count.