Skip to content

Set reproduction tolerances from measured noise; test manifest and download - #15

Merged
jmxpearson merged 1 commit into
mainfrom
test-tolerances
Oct 23, 2026
Merged

jmxpearson merged 1 commit into
mainfrom
test-tolerances

Conversation

@jmxpearson

Copy link
Copy Markdown
Member
  • Tolerances: each pinned DAF metric now allows ~3 SD of its spread across 10 evaluation seeds of the shipped model (SD noted inline). Latency widens from ±2.5 to ±5.0 ms, since it was tighter than the noise (SD 1.5 ms). The extra-spike and baseline pins tighten.
  • Duration: 47.5 ms at the test seed, but 45.0 ± 1.3 ms over seeds 42–51, so it passes the bird check only at this seed; the comment says so.
  • New offline tests: test_manifest.py (6) and test_download.py (8).

197 tests pass; ruff clean. No code changes.

🤖 Generated with Claude Code

…wnload

- Each pinned DAF metric's tolerance is now ~3 SD across 10 evaluation seeds
  of the shipped model (SD noted inline). Latency widens 2.5 -> 5.0 ms (it was
  tighter than the noise); the extra-spike pins tighten.
- tests/test_manifest.py: write/verify round-trip, changed and missing
  artifacts, comments, absent manifest.
- tests/test_download.py: archive flattening, skip-if-present, no-WAV
  archive, TLS and other network errors, empty body, User-Agent, CA bundle;
  all offline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jmxpearson
jmxpearson marked this pull request as ready for review October 23, 2026 14:45
@jmxpearson
jmxpearson merged commit 167c391 into main Oct 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant