Performance on some quick, synthetic benchmarks

[jabr]

I had an LLM generate some synthetic benchmark/test cases and run Von, Jev, and Gliner2 against them.

Repo is here: https://github.com/jabr/classifier-benchmark
Test cases are here: https://github.com/jabr/classifier-benchmark/blob/main/bench/cases.py
Results summary is here: https://github.com/jabr/classifier-benchmark/blob/main/bench/cases.py

Von is fast and looks like a nice start, but both Jev and Gliner2 beat it pretty decisively on accuracy for these cases:

Task (type) | Von | GLiNER2 | Jev
support_department (choice) | 0.933 | 0.933 | 1.000
email_intent (choice) | 1.000 | 0.900 | 1.000
refund_eligible (noul) | 0.500 | 0.500 | 1.000
urgency (noul) | 0.625 | 1.000 | 1.000
secret_leak (noul) | 0.500 | 0.500 | 1.000
frustration_level (score) | 0.222 | 1.000 | 1.000
incident_severity (score) | 0.556 | 0.556 | 0.778
review_sentiment (score) | 0.667 | 0.889 | 1.000
micro accuracy | 0.654 | 0.795 | 0.974
macro accuracy | 0.625 | 0.785 | 0.972
mean latency / case | ~62 ms | ~93 ms | ~302 ms
total wall time (78 cases) | 10.7 s | 20.3 s | 23.7 s

[wfzyx]

Hey, thank you so much for putting this together, I'll use it to test against the next checkpoints and will ping you here with the results

[wfzyx]

Thanks again @jabr — this peer benchmark was invaluable. We pulled down your benchmark repo and traced the exact failure modes. It turns out the collapse on noul and score was driven by severe hypothesis-formatting bugs in our inference engine, not model parameter limits:

1. Noul (0.500 coin-flip bug): When explicit positive/negative criteria were not supplied, the engine fell back to appending "Condition holds true." vs "Condition is false or not satisfied." and took a 2-way softmax over the entailment logits. Because neither synthetic string is natural English entailed by the state, both logits hovered at negative baseline noise (~-2.6), making the softmax flip a 50/50 coin around 0.500. We replaced this with direct polarity verification (calibrated NLI entailment-vs-contradiction posterior on declarative statements, and natural interrogative polarity gating).
2. Score (0.222 collapse): The engine prepended "Level {i}:" and the question instruction to each level description. In frustration_level, the word "frustrated" in the question created a heavy lexical prior toward Level 1 ("Frustrated but civil") even when the text was calm. Removing these artifacts restored proper ranking.

Re-run Results on v1.0.1 (Local)

We shipped the fix in von-sdk@1.0.1 (commit e264584) and added your benchmark suite to our repo (benchmarks/run_peer_benchmark.py):

Task (type)	Von (v1.0.0)	Von (v1.0.1)	GLiNER2	Jev
support_department (choice)	0.933	0.933	0.933	1.000
email_intent (choice)	1.000	1.000	0.900	1.000
refund_eligible (noul)	0.500	0.400	0.500	1.000
urgency (noul)	0.625	0.750	1.000	1.000
secret_leak (noul)	0.500	1.000	0.500	1.000
frustration_level (score)	0.222	0.778	1.000	1.000
incident_severity (score)	0.556	0.778	0.556	0.778
review_sentiment (score)	0.667	0.889	0.889	1.000
micro accuracy	0.654	0.821 (+16.7%)	0.795	0.974
macro accuracy	0.625	0.816 (+19.1%)	0.785	0.972

Von now outperforms GLiNER2 on macro accuracy (0.816 vs 0.785) while keeping its ~60ms inference latency.

The remaining gap (refund_eligible at 0.400, where GLiNER2 is also at 0.500) is an alignment gap with academic NLI pretraining: academic NLI heavily penalizes unstated policy rules (treating them as Neutral), whereas Jev was fine-tuned on operational policy compliance. We are synthesizing a dedicated operational policy dataset for the next checkpoint to close that final gap.

[wfzyx]

Update @jabr — we just concluded our Phase 2 operational fine-tuning run on 4x NVIDIA T4 spot GPUs (62,991 operational decision samples across temporal money-back rules, credential detection, customer support intent, and ordinal severity scales).

The new checkpoint reached 96.43% validation accuracy during training ($T = 1.1692$) and has been pushed to Hugging Face Hub (wfzyx/von-1.0).

Updated Peer Benchmark Scoreboard (jabr/classifier-benchmark)

Task (type)	Von (Initial)	Von (v1.0.1)	Von (Phase 2 Weights)	GLiNER2	Jev
support_department (choice)	0.933	0.933	0.867	0.933	1.000
email_intent (choice)	1.000	1.000	1.000	0.900	1.000
refund_eligible (noul)	0.500	0.400	0.700	0.500	1.000
urgency (noul)	0.625	0.750	1.000	1.000	1.000
secret_leak (noul)	0.500	1.000	0.875	0.500	1.000
frustration_level (score)	0.222	0.778	1.000	1.000	1.000
incident_severity (score)	0.556	0.778	1.000	0.556	0.778
review_sentiment (score)	0.667	0.889	1.000	0.889	1.000
micro accuracy	0.654	0.821	0.923 (72/78)	0.795	0.974
macro accuracy	0.625	0.816	0.930	0.785	0.972

• 5 out of 8 tasks are now at 100% accuracy.
• incident_severity now beats Jev (1.000 vs 0.778).
• refund_eligible jumped from 0.400 to 0.700 (+30%).
• Von now beats GLiNER2 by +14.5% on macro accuracy (0.930 vs 0.785) while maintaining ~62ms local inference.

[jabr]

Looking good! I've update my repo with von-1.0.1 and phase 2 weights, with results matching yours here.

I also generated a "v2 suite" with 869 new cases across 49 tasks (including the original 8 with additional cases). And to be clear, all of these are synthetic test cases generated and cross-checked by a variety of LLMs (GLM 5.3 Flash, GLM 5.3, Kimi K3, Qwen3.8 2.4T, Qwen3.8 Flash, Deepseek V4.1 Flash, and MiMo V2.5 Pro).

The preliminary Von vs Jev results are here: https://github.com/jabr/classifier-benchmark/blob/main/results/v1v2-summary.md

Suite | n | Von micro | Von macro | Jev micro | Jev macro
-- | -- | -- | -- | -- | --
v1 | 78 | 0.923 | 0.930 | 0.974 | 0.972
v2 | 869 | 0.666 | 0.667 | 0.964 | 0.966
combined | 947 | 0.687 | 0.704 | 0.965 | 0.967

I think Von holds up rather well, with the overall score dragged down by some new domains it struggled with like grammar_issue. I didn't manually review the tasks and test cases, so let me know if any look unfair or just ridiculous in terms of typical tasks.

[wfzyx]

hey! a few notes:

We have a Option-Marker Architecture (von-1.2) now !

Your run used our von-1.0.1 cross-encoder weights - thats ok tho;

On the v2 we got support_department_v2 from 76.5% to 94.1% and reduces CPU wall-clock latency by ~30%. MPS/GPU inference runs in ~18ms by eliminating the O(K) pass penalty.

Domain Drift: Your findings on grammar_issue, dietary_vegan, and hazmat_shipping align with our training data limitations.

We’re adopting your v2 suite as our primary generalization benchmark.

For Phase 4, we’re synthesizing a 200k-sample Universal Decision Corpus spanning compliance, safety, linguistic, and operational taxonomies across all 49 domain clusters. We’ll share results after training the universal marker head.

Thanks for the interest and help on the project - extremely useful help

[wfzyx]

Got a checkpoint that reached 71% but found some bugs, so I'm doing couple more passes before releasing a new checkpoint

[rubentorresbonet]

out of curiosity: is the training/finetuning recipe + datasets available to get to von locally?

[wfzyx]

@rubentorresbonet its all on the readme, the datasets are public from HF

@jabr we got 83% on macro and 72% on micro !

also found out that https://benchmarkheaven.com/jev-models this is a thing will run some tests against it and see how can we better improve the dataset

[jabr]

Just fyi, but I've updated the benchmark numbers and docs with the latest Von 1.1.1.

[jabr]

I ran an analysis of failure modes and there might be some useful info for you in it. This bit in particular jumped out as maybe something you could look at:

The decision layer — not comprehension — is where the small classes lose the most recoverable accuracy. On binary tasks, every small-class model orders cases better than it thresholds them ... How the head maps evidence to a cut or an ordinal bin is a first-order design choice — worth more than a large amount of training data.
https://github.com/jabr/classifier-benchmark/blob/main/results/shape-knowledge.md#decision-form-a-design-problem-not-a-data-problem

It also identified the "semantic twist" (complexity) as a general pattern that could potentially be addressed with more training data on specific problem areas that all of the small models (e.g. BERT-based) struggle with. In theory, it's a generalizable problem in the ~400M parameter size, unlike the domain-specific world knowledge problem that can only realistically be solved in small models with task-specific fine-tuning.

For that second issue, I'm also looking at an idea to generate training/test data based off a "logic spine" and standard algorithmic permutations like paraphrase, negation, implicit, conflict, exception, coreference, etc. In other words, taking one fundamental state->gold case and generating various complex ways of stating it for the model to train on. (Outline of the idea here: https://github.com/jabr/classifier-benchmark/tree/main/training/spines#readme)