docs(readme): measured comparison of the judging backends - #4
Merged
Merged
Conversation
Add a "Judging backends: measured comparison" section to README.md and its Chinese mirror, carrying the numbers that used to live only in the experiment records: the 20-item accuracy table (hosted Jev, both Kev 4B row limits, the GGUF 4B default, the two 4B candidates, Kev 0.8B, the GGUF 0.8B and the trivial positional baseline), what each costs in latency, disk, memory and money, how the set was measured, the 15 real browser runs, and the three goal_done bars with their measured basis. Every figure is read from the repo's own records; nothing is estimated and unmeasured cells say so. The hosted-Jev-default statement is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One new README section —
## Judging backends: measured comparisoninREADME.md, mirrored as## 判定后端实测对比inREADME.zh-CN.md— carrying the measured comparison of the judgingbackends that until now lived only in
experiments/**anddocs/**.It contains: the 20-item accuracy table (hosted Jev, Kev 4B at both row limits, the GGUF 4B
local-readoutdefault, the two other 4B candidates, Kev 0.8B, the GGUF 0.8B and the trivialpositional baseline); what each costs in per-item latency, disk, memory and money; a short "how this
was measured"; the 15 real browser runs; and the three
goal_donebars with their measured basis.No code, tests or config changed. The existing "hosted Jev is the default" statement is untouched —
the new section restates it. The only edit outside the new section is
lives in SKILL.md→also lives in SKILL.md(and见→同时见), so the pointer above the new tables no longer reads asif the numbers were only in SKILL.md.
Sources for every figure
experiments/gguf-provider/results/ceiling-jev-20items.json(overall,by_kind); restated inexperiments/gguf-provider/RESULTS.mdandresults/local-models-4b.md§0results/ceiling-jev-20items.json(latency_ms,cost_usd_total)experiments/kev-4b/README.md(Measured: Kev 4B / Memory); cross-checked inresults/local-models-4b.md§0docs/local-kev-bringup.md§0,experiments/kev-4b/README.md(verification log)experiments/kev-4b/README.md(Measured: Kev 0.8B, Row limit §3)experiments/gguf-provider/RESULTS.md,results/local-models-4b.md§0/§2results/local-models-4b.md§0/§2results/local-models-4b.md§0/§2experiments/gguf-provider/RESULTS.mdresults/baseline-first-option.json+RESULTS.mdclick_targetresults/local-models-4b.md§0.2,RESULTS.mde8/e5/e1, P≈0.99experiments/gguf-provider/RESULTS.mdsuccess/ 8stuck/ 2needs_user/ 1max_steps; 39 step requests; 0 timeouts; 0LOW_LABEL_MASS; min label mass 0.870docs/local-backend-run-smoke.md§0/§2/§3click_target0.71–0.78 on a 65-element page;typelast of four (0.105 vs 0.430); 3 prematurestops; 8 of 9 non-success runs stop onactiondocs/local-backend-run-smoke.md§0/§5docs/local-backend-run-smoke.md§4/§9,experiments/kev-4b/README.md(Threshold replay)Nothing is estimated. Cells the records do not measure say
not measured; cells that do not existfor a backend say
—.Verification
balanced, and every relative link in the new section resolves to a file that exists.
npm test(mock mode, Ubuntu + headless Chrome) — docs-only change.