Skip to content

eval: publish Ornith 1.5 9B and 35B sweeps #148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
antoinezambelli merged 3 commits into from
Aug 20, 2026
Merged

eval: publish Ornith 1.5 9B and 35B sweeps #148

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8ee8ac6
eval: add Ornith 1.5 9B sweep
antoinezambelli Aug 20, 2026
a967700
eval: publish Ornith 1.5 35B sweep
antoinezambelli Aug 20, 2026
1e5ebd6
eval: finalize Ornith publication surfaces
antoinezambelli Aug 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/MODEL_REGISTRY.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ Sampling values are sourced from the model's HuggingFace card unless noted. Valu
| Qwen3.5 35B-A3B | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | — | 1.5 | [HF](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) |
| Qwen3.6 27B | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | — | 0.0⁶ | [HF](https://huggingface.co/Qwen/Qwen3.6-27B) |
| Qwen3.6 35B-A3B UD | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | — | 1.5 | [HF](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) |
| Ornith 1.5 9B | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | 1.0 | 1.5 | [HF](https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF) |
| Ornith 1.5 35B-A3B | Q4_K_M | 0.6 | 0.95 | 20 | — | — | — | [HF](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B-GGUF) |
| Nemotron-3 Nano 30B-A3B | Q4_K_M | 0.6 | 0.95 | — | — | — | —⁷ | [HF](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) |
| Claude Haiku 4.5⁵ | — | — | — | — | — | — | — | (SDK-managed) |
| Claude Sonnet 4.6⁵ | — | — | — | — | — | — | — | (SDK-managed) |
Expand Down
3 changes: 2 additions & 1 deletion docs/results/dashboard.html
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/results/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ For model and backend recommendations, see [Model Guide](../MODEL_GUIDE.md).
- [native-vs-prompt.md](raw/native-vs-prompt.md) — llama-server native FC vs prompt-injected, reforged only
- [reasoning-replay.md](raw/reasoning-replay.md) — reasoning_replay policy comparison (none / keep-last / full) per config

*Generated 2026-08-17 17:14*
*Generated 2026-08-20 07:41*
2 changes: 1 addition & 1 deletion docs/results/raw/native-vs-prompt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,4 +324,4 @@ Eval generations (older runs carried forward, superscript-tagged):
¹ gen 1 — v0.6.0 suite — incl. Anthropic ablation (commit 2b05dc4, 2026-05-08)
² gen 2 — v0.7.0 lineup refresh (8–14B) + 32GB tier debut (v0.7.4) (commit 655e1f6, 2026-05-22)

*Generated 2026-08-17 17:14*
*Generated 2026-08-20 07:41*
2 changes: 1 addition & 1 deletion docs/results/raw/reasoning-replay.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -693,4 +693,4 @@ Eval generations (older runs carried forward, superscript-tagged):
¹ gen 1 — v0.6.0 suite — incl. Anthropic ablation (commit 2b05dc4, 2026-05-08)
² gen 2 — v0.7.0 lineup refresh (8–14B) + 32GB tier debut (v0.7.4) (commit 655e1f6, 2026-05-22)

*Generated 2026-08-17 17:14*
*Generated 2026-08-20 07:41*
24 changes: 23 additions & 1 deletion docs/results/raw/reforged-vs-bare.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,17 @@ Muse-Glimmer-30B-UD-Q4_K_XL LS/N [bare:full@xhigh] 86.5% 95.7%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```

## Ornith-1.5-35B-Q4_K_M (llamaserver/native)

```
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Model/Backend Scr VAcc Cmp Eff Wst Spd N rel arg tsl b2s s3s crt srn err dgr dge art grs iar rel_s arg_s tsl_s b2s_s s3s_s crt_s srn_s err_s dgr_s dge_s art_s grs_s iar_s
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Ornith-1.5-35B-Q4_K_M LS/N [reforged] 97.8% 97.8% 100.0% 100% 0.4 7.1s 50 100 100 100 100 100 100 100 100 100 94 98 86 98 100 100 100 100 100 100 100 96 100 94 100 82 94
Ornith-1.5-35B-Q4_K_M LS/N [bare] 60.3% 90.8% 66.4% 100% 0.0 9.0s 50 4 98 0 0 100 100 100 0 98 62 100 76 86 4 98 0 0 100 100 100 0 94 68 100 80 0
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```

## Qwen3.6-35B-A3B-UD-Q4_K_M (llamaserver/native)

```
Expand Down Expand Up @@ -125,6 +136,17 @@ claude-haiku-4-5-20251001 AN/N [bare+any] 74.0% 80.2% 92.3% 100% 0.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```

## Ornith-1.5-9B-Q4_K_M (llamaserver/native)

```
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Model/Backend Scr VAcc Cmp Eff Wst Spd N rel arg tsl b2s s3s crt srn err dgr dge art grs iar rel_s arg_s tsl_s b2s_s s3s_s crt_s srn_s err_s dgr_s dge_s art_s grs_s iar_s
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Ornith-1.5-9B-Q4_K_M LS/N [reforged] 94.2% 94.2% 99.9% 100% 0.3 11.1s 50 100 100 100 98 100 100 100 62 98 78 94 90 94 100 100 100 98 100 100 100 58 100 88 100 96 94
Ornith-1.5-9B-Q4_K_M LS/N [bare] 62.2% 91.2% 68.2% 100% 0.0 12.5s 50 2 100 2 84 100 86 100 0 72 32 96 84 96 4 100 0 84 100 96 98 0 68 40 94 78 2
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```

## Qwen3.5-27B-Q4_K_M (llamaserver/native)

```
Expand Down Expand Up @@ -902,4 +924,4 @@ Eval generations (older runs carried forward, superscript-tagged):
¹ gen 1 — v0.6.0 suite — incl. Anthropic ablation (commit 2b05dc4, 2026-05-08)
² gen 2 — v0.7.0 lineup refresh (8–14B) + 32GB tier debut (v0.7.4) (commit 655e1f6, 2026-05-22)

*Generated 2026-08-17 17:14*
*Generated 2026-08-20 07:41*
4 changes: 3 additions & 1 deletion docs/results/raw/reforged/all.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,15 @@ Qwen3.8-27B-UD-Q4_K_XL LS/N [reforged@medium] 98.5%
Qwen3.8-27B-UD-Q4_K_XL LS/N [reforged@low] 98.8% 98.8% 100.0% 100% 0.1 11.8s 50 100 100 100 100 100 100 100 100 100 96 100 92 100 100 100 100 100 100 100 100 100 100 100 88 94 100
claude-opus-4-6 AN/N [reforged:full]¹ 99.2% 99.8% 99.4% 100% 0.0 15.6s 50 100 100 100 100 100 98 100 100 100 100 98 94 98 100 100 100 100 100 100 100 96 100 100 98 100 98
Muse-Glimmer-30B-UD-Q4_K_XL LS/N [reforged@high] 98.8% 99.8% 98.9% 83% 1.1 8.4s 50 100 100 100 100 100 100 100 100 100 98 86 100 100 100 100 100 100 100 100 100 98 100 100 86 100 100
Ornith-1.5-35B-Q4_K_M LS/N [reforged] 97.8% 97.8% 100.0% 100% 0.4 7.1s 50 100 100 100 100 100 100 100 100 100 94 98 86 98 100 100 100 100 100 100 100 96 100 94 100 82 94
Muse-Glimmer-30B-UD-Q4_K_XL LS/N [reforged@low] 97.8% 98.5% 99.2% 91% 0.8 4.2s 50 100 100 100 100 100 100 100 100 100 82 88 100 100 100 100 100 100 100 100 100 98 100 84 90 100 100
Muse-Glimmer-30B-UD-Q4_K_XL LS/N [reforged@medium] 98.5% 99.3% 99.2% 86% 1.0 6.1s 50 100 100 100 100 100 100 100 100 100 96 86 100 100 100 100 100 100 100 100 100 94 100 94 90 100 100
Muse-Glimmer-30B-UD-Q4_K_XL LS/N [reforged:full@xhigh] 97.9% 99.3% 98.6% 82% 1.2 9.0s 50 100 100 100 100 100 100 100 100 100 100 84 100 100 100 100 100 100 98 100 100 86 100 100 80 100 98
Muse-Glimmer-30B-UD-Q4_K_XL LS/N [reforged@xhigh] 97.5% 99.3% 98.2% 83% 1.1 8.8s 50 100 100 100 100 100 100 100 100 100 98 80 98 100 100 100 100 100 100 100 100 90 100 100 72 100 98
Muse-Glimmer-30B-UD-Q4_K_XL LS/N [reforged:keep-last@xhigh] 97.5% 99.4% 98.2% 83% 1.1 8.8s 50 100 100 100 100 100 100 100 100 100 98 74 100 100 100 100 100 100 100 100 100 88 100 98 78 100 100
Qwen3.6-35B-A3B-UD-Q4_K_M LS/N [reforged:full] 95.2% 95.2% 99.9% 100% 0.5 7.2s 50 100 100 100 100 100 98 96 100 100 60 76 94 100 100 100 100 100 100 100 98 100 100 74 82 98 98
claude-haiku-4-5-20251001 AN/N [reforged] 94.2% 94.2% 99.9% 100% 0.3 6.6s 50 100 100 100 100 100 100 98 100 100 74 74 98 100 100 100 100 100 100 100 100 100 100 72 38 94 100
Ornith-1.5-9B-Q4_K_M LS/N [reforged] 94.2% 94.2% 99.9% 100% 0.3 11.1s 50 100 100 100 98 100 100 100 62 98 78 94 90 94 100 100 100 98 100 100 100 58 100 88 100 96 94
Qwen3.5-122B-A10B-Q4_K_M LS/N [reforged:full] 94.5% 94.5% 100.0% 100% 0.2 52.8s 50 100 100 100 100 100 100 100 100 100 94 36 100 100 100 100 100 100 100 100 100 100 100 90 40 96 100
Qwen3.6-35B-A3B-UD-Q4_K_M LS/N [reforged:keep-last] 92.7% 92.7% 100.0% 100% 0.4 6.6s 50 100 100 100 100 100 100 98 100 100 46 78 78 100 100 100 100 98 100 98 98 100 98 52 86 80 100
Qwen3.6-27B-Q4_K_M LS/N [reforged:keep-last] 92.5% 93.0% 99.5% 100% 0.1 17.6s 50 100 100 100 100 100 100 100 96 98 44 74 98 100 100 100 100 100 100 100 100 92 98 40 68 98 100
Expand Down Expand Up @@ -150,4 +152,4 @@ Eval generations (older runs carried forward, superscript-tagged):
¹ gen 1 — v0.6.0 suite — incl. Anthropic ablation (commit 2b05dc4, 2026-05-08)
² gen 2 — v0.7.0 lineup refresh (8–14B) + 32GB tier debut (v0.7.4) (commit 655e1f6, 2026-05-22)

*Generated 2026-08-17 17:14*
*Generated 2026-08-20 07:41*
2 changes: 1 addition & 1 deletion docs/results/raw/reforged/by-backend.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,4 +151,4 @@ Eval generations (older runs carried forward, superscript-tagged):
¹ gen 1 — v0.6.0 suite — incl. Anthropic ablation (commit 2b05dc4, 2026-05-08)
² gen 2 — v0.7.0 lineup refresh (8–14B) + 32GB tier debut (v0.7.4) (commit 655e1f6, 2026-05-22)

*Generated 2026-08-17 17:14*
*Generated 2026-08-20 07:41*
22 changes: 21 additions & 1 deletion docs/results/raw/reforged/by-family.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,16 @@ Muse-Glimmer-30B-UD-Q4_K_XL LS/N [reforged:keep-last@xhigh] 97.5% 99.4%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```

## ornith-1.5-35b

```
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Model/Backend Scr VAcc Cmp Eff Wst Spd N rel arg tsl b2s s3s crt srn err dgr dge art grs iar rel_s arg_s tsl_s b2s_s s3s_s crt_s srn_s err_s dgr_s dge_s art_s grs_s iar_s
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Ornith-1.5-35B-Q4_K_M LS/N [reforged] 97.8% 97.8% 100.0% 100% 0.4 7.1s 50 100 100 100 100 100 100 100 100 100 94 98 86 98 100 100 100 100 100 100 100 96 100 94 100 82 94
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```

## qwen3.6-35b-a3b

```
Expand All @@ -76,6 +86,16 @@ Qwen3.5-122B-A10B-Q4_K_M LS/N [reforged] 91.3% 91.4% 99.9%
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```

## ornith-1.5-9b

```
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Model/Backend Scr VAcc Cmp Eff Wst Spd N rel arg tsl b2s s3s crt srn err dgr dge art grs iar rel_s arg_s tsl_s b2s_s s3s_s crt_s srn_s err_s dgr_s dge_s art_s grs_s iar_s
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Ornith-1.5-9B-Q4_K_M LS/N [reforged] 94.2% 94.2% 99.9% 100% 0.3 11.1s 50 100 100 100 98 100 100 100 62 98 78 94 90 94 100 100 100 98 100 100 100 58 100 88 100 96 94
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```

## qwen3.5-27b

```
Expand Down Expand Up @@ -368,4 +388,4 @@ Eval generations (older runs carried forward, superscript-tagged):
¹ gen 1 — v0.6.0 suite — incl. Anthropic ablation (commit 2b05dc4, 2026-05-08)
² gen 2 — v0.7.0 lineup refresh (8–14B) + 32GB tier debut (v0.7.4) (commit 655e1f6, 2026-05-22)

*Generated 2026-08-17 17:14*
*Generated 2026-08-20 07:41*
4 changes: 2 additions & 2 deletions eval_results_v0.9.0.jsonl
View file
Open in desktop
Git LFS file not shown
6 changes: 6 additions & 0 deletions src/forge/clients/sampling_defaults.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,15 @@
# presence_penalty (0.0 vs 1.5) for thinking-mode general tasks.
"qwen3.6:27b-q4_K_M": {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0}, # https://huggingface.co/Qwen/Qwen3.6-27B
"Qwen3.6-27B-Q4_K_M": {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0}, # https://huggingface.co/Qwen/Qwen3.6-27B
# Ornith 1.5 9B — publisher's general-tasks profile. The precise-coding
# alternative changes temperature to 0.6 and presence_penalty to 0.0.
"ornith-1.5:9b": {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5, "repeat_penalty": 1.0}, # https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF
"Ornith-1.5-9B-Q4_K_M": {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5, "repeat_penalty": 1.0}, # https://huggingface.co/ornith-ai/Ornith-1.5-9B-GGUF
# Qwen3.8-27B — Unsloth's thinking-mode profile. xhigh is the model's
# documented default reasoning effort for complex coding and agentic work.
"Qwen3.8-27B-UD-Q4_K_XL": {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repeat_penalty": 1.0, "chat_template_kwargs": {"reasoning_effort": "xhigh"}}, # https://unsloth.ai/docs/models/qwen3.8
# Ornith 1.5 35B-A3B — model-card general-task profile.
"Ornith-1.5-35B-Q4_K_M": {"temperature": 0.6, "top_p": 0.95, "top_k": 20}, # https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B-GGUF
# Qwen3-Coder — non-thinking instruct; card does not mention min_p or presence_penalty
"qwen3-coder:30b-a3b-instruct-q4_K_M": {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "repeat_penalty": 1.05}, # https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct
"Qwen3-Coder-30B-A3B-Instruct-Q4_K_M": {"temperature": 0.7, "top_p": 0.8, "top_k": 20, "repeat_penalty": 1.05}, # https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct
Expand Down
2 changes: 2 additions & 0 deletions tests/eval/batch_eval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,9 @@ def _glimmer_server_recipe(reasoning_strength: str) -> _BatchServerRecipe:
("Qwen3.5-35B-A3B-Q4_K_M.gguf", _REASONING_SERVER_RECIPE),
("Qwen3.6-27B-Q4_K_M.gguf", _REASONING_SERVER_RECIPE),
("Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", _REASONING_SERVER_RECIPE),
("Ornith-1.5-9B-Q4_K_M.gguf", _REASONING_SERVER_RECIPE),
("Qwen3.8-27B-UD-Q4_K_XL.gguf", _QWEN38_SERVER_RECIPE),
("Ornith-1.5-35B-Q4_K_M.gguf", _REASONING_SERVER_RECIPE),
("Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf", _REASONING_SERVER_RECIPE),
("Muse-Glimmer-30B-UD-Q4_K_XL.gguf", _GLIMMER_SERVER_RECIPE),
# Gemma-4 large (rig-04, az/eval-large): 26B-A4B MoE + 31B dense. Native FC;
Expand Down
20 changes: 10 additions & 10 deletions tests/eval/publication.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,20 +57,20 @@
VIEW_NAMES = ("history", "snapshot", "latest")

PINNED_VIEW_COUNTS = {
"history": 547_300,
"snapshot": 406_900,
"latest": 288_600,
"history": 552_500,
"snapshot": 412_100,
"latest": 293_800,
}
PINNED_VIEW_METRIC_COUNTS = {
"history": (341_192, 456_242, 456_242),
"snapshot": (252_245, 339_699, 339_699),
"latest": (201_092, 255_014, 255_014),
"history": (345_280, 460_591, 460_591),
"snapshot": (256_333, 344_048, 344_048),
"latest": (205_180, 259_363, 259_363),
}
PINNED_SNAPSHOT_SOURCE_IDENTITY_SHA256 = (
"f19679e627ff3ca5e339c38f6564dfc9f215f95adbe824036dad58667f0bfe39"
"405a6d71e092d2af847f0bbd049a0b60e21ee307649b960e48fd99ff22fbacee"
)
PINNED_SNAPSHOT_SOURCE_PAYLOAD_SHA256 = (
"6b0a9e85b6d4b863c19117b549f92e7127b41e564945d722f0c6e5c037717c98"
"8508da22521cdb821f4245e9538c36019cae905e006745a46300ee17a572ddc8"
)


Expand Down Expand Up @@ -337,8 +337,8 @@ class SourceSpec:
"v0.9.0",
3,
CANONICAL_DIALECT,
28_600,
"50b2ae49a8b9161711289c60dc060a3403dbf2534ec1cbb6d5834e3f384fa8c3",
33_800,
"e36079168689159362b1aac23901ac121615f38380ccbed5f9a1e90f324a9c8e",
),
)

Expand Down
3 changes: 3 additions & 0 deletions tests/eval/report.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -583,6 +583,9 @@ def compute_config_metrics(
"Muse-Glimmer-30B-UD-Q4_K_XL": {"family": "muse-glimmer-30b", "cross_backend_key": "muse-glimmer-30b-q4_K_XL"},
# DeepSeek V4 Flash 0731 (managed llama.cpp RPC; UD is a quant variant)
"DeepSeek-V4-Flash-0731-UD-Q4_K_XL": {"family": "deepseek-v4-flash", "cross_backend_key": "deepseek-v4-flash-0731-q4_K_XL"},
# Ornith 1.5 (llama-server only)
"Ornith-1.5-9B-Q4_K_M": {"family": "ornith-1.5-9b", "cross_backend_key": "ornith-1.5-9b-q4_K_M"},
"Ornith-1.5-35B-Q4_K_M": {"family": "ornith-1.5-35b", "cross_backend_key": "ornith-1.5-35b-q4_K_M"},
# lfm2.5 8b-a1b (llama-server only)
"LFM2.5-8B-A1B-Q4_K_M": {"family": "lfm2.5-8b-a1b", "cross_backend_key": "lfm2.5-8b-a1b-q4_K_M"},
# mellum2 12b-a2.5b (llama-server only; instruct + thinking share a family)
Expand Down
Loading
Loading