Add matplotlib charts to README, generated by scripts/gen_readme_charts.py - #17
Merged
Merged
Conversation
Adds five SVG charts to README.md and the stdlib-only generator that
produces them, scripts/gen_readme_charts.py:
1. routing demo — success rate per failure type, both benchmark arms
2. per-corpus accuracy — training vs held-out, in scoring order
3. held-out recall by group across corpora C/D/E — the flat
routing-sensitive line through a full tuning cycle
4. Rules vs LLM vs Hybrid — recall bought, precision spent
5. corpus E by code family — where the structural signal reaches
The generator does not transcribe the README's tables. It re-scores
corpora A-E through the same RulesClassifier path
classifier_accuracy.py uses, and derives the routing demo from
bench_synthetic.py's own _task_body, so a rules.py change that moves a
held-out number moves the charts on the next run. The two figures that
cannot be recomputed — corpus C's pre-tuning v1.0 score, and the
LLMClassifier/HybridClassifier runs that need a live model — are marked
HISTORICAL constants naming the document that recorded them.
No matplotlib and no new dependency: the SVGs are written by hand, each
with its own prefers-color-scheme block so one asset reads correctly in
both GitHub themes. Layout bands are measured from calibrated text
metrics rather than guessed, and row labels raise instead of silently
overlapping the plot.
Also adds the corpus E row (69% recall, 100% precision) to the block
table, which had stopped at corpus D, and corrects "Nine-block
measurement" to ten.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VYCX2qtGYVNsSUmtN6cipS
…awn SVG Replaces the stdlib-only SVG renderer in scripts/gen_readme_charts.py with matplotlib + seaborn (whitegrid theme, deep palette, 200 DPI), so the five README charts read like plots from a real experiment rather than custom-drawn graphics. Output is now PNG. Data sourcing is unchanged: every chart is still scored live against the frozen corpora and derived live from bench_synthetic.py, with the two truly historical figures (corpus C's pre-tuning score, the LLM/ Hybrid runs) still marked HISTORICAL with their source. matplotlib/seaborn are added to the dev extra only — not a runtime dependency of triage itself, same footing as anthropic/openai for the other scripts. README image links updated from .svg to .png. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VYCX2qtGYVNsSUmtN6cipS
…es-q6zzjw Resolves two conflicts, both independently-added entries at the same list position (no logical overlap): - CHANGELOG.md: kept both "Added" entries (charts, MAST pilot corpus) - scripts/README.md: kept both additions (Charts section, MAST pilot paragraph), reordered so the MAST paragraph continues its existing section before the new Charts section starts Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VYCX2qtGYVNsSUmtN6cipS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change and why?
Adds five measured matplotlib/seaborn charts to
README.mdthat visualize the classifier's performance across error corpora and failure modes. The charts are generated by a new scriptscripts/gen_readme_charts.pythat:RulesClassifierpathscripts/classifier_accuracy.pyusesscripts/bench_synthetic.py's own_task_bodysuccess logicHISTORICAL(corpus C's pre-tuning v1.0 score and LLMClassifier/HybridClassifier runs) that cannot be recomputed without a live modelThe charts cannot drift from the README's tables — a
rules.pychange that moves a held-out number moves the charts on the next run.Charts added:
Related issue
None
Type of change
Checklist
pytest tests/ -x --tb=shortpasses locallyruff check .,ruff format --check ., andmypy triage/ --strictare all cleantriage/core —matplotlib/seabornadded only todevextra inpyproject.tomlREADME.mdandCHANGELOG.mdupdated with the new charts and script documentationscripts/README.mdupdated with chart generation instructionsTest Plan
The script re-scores the frozen corpora in
tests/data/through the sameRulesClassifierpath the test suite uses, so the numbers it produces are deterministic and matchscripts/classifier_accuracy.py. Run:The five PNG files are written to
docs/assets/charts/and embedded inREADME.md. Verify the charts render correctly in the README and that the accuracy numbers match the tables in the same document.Anything reviewers should look at closely?
The
routing_demo()function dynamically importsscripts/bench_synthetic.pyto extract itsTASKSand_task_body— this is intentional to avoid duplicating the synthetic task definitions. The function calls_task_bodytwice per task (once with no hint, once with the triage hint) to measure the recovery gap, matching the benchmark's own success logic.The two
HISTORICALconstants (corpus C's v1.0 pre-tuning score and LLMClassifier/HybridClassifier runs) are documented in the module docstring and marked in the code. They are sourced fromdocs/known-limitations.mdand cannot be recomputed without a live model or re-tuning against held-out data.https://claude.ai/code/session_01VYCX2qtGYVNsSUmtN6cipS