Skip to content

Repository navigation

docs(exp-18): clarify origin of the 35% hesitation gate #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ghchinoy merged 1 commit into from
Sep 30, 2026
Merged

docs(exp-18): clarify origin of the 35% hesitation gate #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/experiments/exp-18-mizan-judge-capability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ v0.1.0, 2026-09-29. n = 100 per suite; accuracy in %. The 09-26 value is in pare
- No single dgem suite changed significantly.
- The Gemini judges were just as stable across the two dates.
- **No-model metrics.** mizan's local `computation` kind agrees item for item with Vertex `EvaluateInstances` on exact match, tool-call and trajectory metrics (350/350). It also reproduces sacrebleu and rouge_score exactly. Vertex's own BLEU and ROUGE-Lsum differ from those libraries by up to 0.12 and 0.23, although they rank items almost identically (Spearman 0.99).
- **Entropy cascade** (dgem G4 escalating to gemini-3.8-flash at a fixed 35% hesitation threshold). Each figure is cascade accuracy, followed by the share of items escalated:
- **Entropy cascade** (dgem G4 escalating to gemini-3.8-flash at a fixed 35% hesitation threshold). 35% reuses the value of dgem's earlier raw-entropy default (0.35 nats, EXP-04/05) on the normalized scale; it is not a Studio hesitation band (16% / 50%). It was fixed before the data was seen. On these runs it sits at the knee of the curve: pooled accuracy over 799 paired items is flat (77.0–77.6%) for gates from 5% to 35% while escalation falls from 68% to 36%. Each figure is cascade accuracy, followed by the share of items escalated:

| Suite | Cascade accuracy | Items escalated |
| :--- | :---: | :---: |
Expand Down
2 changes: 1 addition & 1 deletion docs/experiments/exp-18-mizan-judge-capability.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ v0.1.0, 2026-09-29. n = 100 per suite; accuracy in %. The 09-26 value is in pare
- No single dgem suite changed significantly.
- The Gemini judges were just as stable across the two dates.
- **No-model metrics.** mizan's local `computation` kind agrees item for item with Vertex `EvaluateInstances` on exact match, tool-call and trajectory metrics (350/350). It also reproduces sacrebleu and rouge_score exactly. Vertex's own BLEU and ROUGE-Lsum differ from those libraries by up to 0.12 and 0.23, although they rank items almost identically (Spearman 0.99).
- **Entropy cascade** (dgem G4 escalating to gemini-3.8-flash at a fixed 35% hesitation threshold). Each figure is cascade accuracy, followed by the share of items escalated:
- **Entropy cascade** (dgem G4 escalating to gemini-3.8-flash at a fixed 35% hesitation threshold). 35% reuses the value of dgem's earlier raw-entropy default (0.35 nats, EXP-04/05) on the normalized scale; it is not a Studio hesitation band (16% / 50%). It was fixed before the data was seen. On these runs it sits at the knee of the curve: pooled accuracy over 799 paired items is flat (77.0–77.6%) for gates from 5% to 35% while escalation falls from 68% to 36%. Each figure is cascade accuracy, followed by the share of items escalated:

| Suite | Cascade accuracy | Items escalated |
| :--- | :---: | :---: |
Expand Down
Loading