Skip to content

eval: publish Ornith 1.5 9B and 35B sweeps - #148

Merged
antoinezambelli merged 3 commits into
mainfrom
az/ornith
Aug 20, 2026
Merged

eval: publish Ornith 1.5 9B and 35B sweeps#148
antoinezambelli merged 3 commits into
mainfrom
az/ornith

Conversation

@antoinezambelli

Copy link
Copy Markdown
Owner

Summary

  • publish complete Ornith 1.5 9B and 35B evaluation sweeps;
  • append 5,200 generation-3 rows to the canonical v0.9.0 corpus;
  • add the corresponding sampling defaults and managed batch configurations;
  • normalize both Ornith families in reports;
  • update the deterministic publication contract; and
  • regenerate the Markdown reports and HTML dashboard.

Evaluation setup

  • Models:
    • Ornith-1.5-9B-Q4_K_M
    • Ornith-1.5-35B-Q4_K_M
  • Backend: llama-server with native tool calling
  • Quantization: Q4_K_M
  • Reasoning replay: none
  • Reasoning level: default
  • Generation: 3
  • Coverage: 26 scenarios ? 50 runs in both Reforged and Bare modes
  • Total new rows: 5,200

Results

Model Configuration Score Validated accuracy Completion Avg time
Ornith 1.5 35B Reforged 97.8% 97.8% 100.0% 7.1s
Ornith 1.5 35B Bare 60.3% 90.8% 66.4% 9.0s
Ornith 1.5 9B Reforged 94.2% 94.2% 99.9% 11.1s
Ornith 1.5 9B Bare 62.2% 91.2% 68.2% 12.5s

Publication

The canonical v0.9.0 source grows from 28,600 to 33,800 unique rows. The new rows preserve the existing source byte-for-byte and append four complete 1,300-row configurations.

The refreshed publication contract contains:

  • history: 552,500 rows;
  • snapshot: 412,100 rows; and
  • latest: 293,800 rows.

Reports group the new models as ornith-1.5-9b and ornith-1.5-35b, with Q4_K_M recognized as the quantization. The Markdown reports and HTML dashboard were regenerated from all six canonical JSONLs.

The Hugging Face Parquet bundle will be rebuilt and published separately from merged main.

Validation

  • strict six-source publication audit: passed
  • focused publication, reporting, and configuration suite: 106 passed
  • full suite: 1,538 passed, 1 skipped, 42 deselected
  • v0.9.0 corpus: 33,800 rows with 33,800 unique evaluation identities
  • LFS pointer and attributes verified
  • git diff --check: clean
  • generated dashboard: human-reviewed and approved

Release

No package-version bump is included. This is an evaluation corpus and reporting update.

?? Generated with Codex and human-reviewed.

Add the Ornith 35B sampling and batch configuration, fold the generation-3 9B and 35B rows into the v0.9 dataset, and remove the standalone 9B intermediate file.
@antoinezambelli
antoinezambelli merged commit 484050c into main Aug 20, 2026
3 checks passed
@antoinezambelli
antoinezambelli deleted the az/ornith branch August 20, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant