Sync model-table latency to the eval receipt; surface Quickstart at the top - #5
Merged
Merged
Conversation
…t teaser The README model table claimed 1.3s (gpt-5.4-mini) and 0.7s (gpt-4.1-nano) single-turn latency, but the cited receipt docs/model-eval-results.json records p50ms 2821 and 1999. The 1.3s/0.7s figures traced to hardcoded "clean single-turn" values in scripts/generate-chart.mjs with no receipt behind them (both table and chart landed in the same commit 40cd6b6 as the receipt). - README table latency column now uses the receipt byModel.p50ms values, relabeled "p50, full turn", with a note on what the receipt measures - generate-chart.mjs now plots the receipt p50 values; chart regenerated - Takeaways prose updated to match (2.8s / 3.4-6.3s, reasoning tokens) - Add a compact Quickstart near the top linking down to the full Quick start section (previously buried ~1,180 lines in) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What / why
The README model table claimed 1.3s (gpt-5.4-mini) and 0.7s (gpt-4.1-nano) single-turn latency, but the cited receipt
docs/model-eval-results.jsonrecords p50ms 2821 and 1999. Quality and cost columns already matched the receipt — only latency diverged.Root cause (git evidence)
The table, the chart, and the receipt all landed in the same commit (
40cd6b6, 2026-07-04). The 1.3s/0.7s numbers came from a hardcoded array inscripts/generate-chart.mjslabeled "clean single-turn latency ... (parallel eval latency is inflated by contention)" — an unrecorded side measurement with no receipt anywhere in the repo. A number pointing at a receipt that contradicts it is worse than a slower honest number.Changes
p50msmeasures (full completion wall clock, contended eval run — treat as relative).scripts/generate-chart.mjsnow plots the same receipt p50 values (axes rescaled);docs/model-chart.svgregenerated so the embedded chart no longer contradicts the table.Verified on this branch: no
1.3s/0.7sremain in the raw README; corrected table and relabeled chart render on the branch.Do not merge without a quick look at the rendered README.
🤖 Generated with Claude Code