Give classes with no holdout support a neutral decision threshold

[LeoGrin]

Summary

find_optimal_classification_thresholds tuned a threshold for every class in range(n_classes), including classes with no positive rows in the tuning holdout. Rare classes can be absent entirely: StratifiedKFold allocates folds by class count, so a class with a handful of samples deterministically receives zero holdout rows.

With no positive rows every threshold scores identically, so the search returned an arbitrary value — which then fed the divisive reweight in _maybe_reweight_probas (probas / threshold, then renormalize). Per metric, the threshold produced for the absent class:

metric	behaviour	result
roc_auc	nan at every threshold; argmin on an all-nan array returns index 0	smallest threshold in the grid → ~24× boost relative to the tuned classes
log_loss	raised before #1140 (shipped in v8.3.0); now returns a finite loss at every threshold	~1.4× suppression (was a ValueError before v8.3.0)
f1, balanced_accuracy	degenerate plateau	~2.6× suppression relative to the tuned classes
accuracy	—	roughly neutral by coincidence

The roc_auc case is the sharpest: the class with the least evidence receives the largest boost, silently. On synthetic 6-class data that moved an absent class from 0.1% to 49.8% of predictions.

Note the log_loss row is why this PR is still needed after #1140 (shipped in v8.3.0). That PR fixes the crash by passing labels=[0, 1], which is correct — but it turns a loud failure into a quiet one: the search then runs, finds no signal, and suppresses the class. The guard here is what actually resolves the underlying problem.

Fix

Classes with no positive (or no negative) holdout rows are skipped and assigned the geometric mean of the successfully tuned thresholds — thresholds act as divisive reweights, so only their ratios matter and the geometric mean carries the average multiplier on the log scale.

A fixed constant would not work: tuned thresholds sit near 0.6 for accuracy but near 0.2 for f1, so any constant boosts the class under one metric and suppresses it under another. When no class is tunable the thresholds come out uniform, which cancels in the caller's renormalization and leaves predictions untouched.

select_robust_optimal_threshold additionally ignores non-finite losses rather than letting them reach np.argmin, where they silently selected the first — most aggressive — threshold.

Why neutral rather than suppression

Suppressing a class absent from the holdout looks conservative but is not. The model already under-predicts rare classes heavily (in the test setup, a class with 3.2% true frequency is predicted 0.12% of the time). The threshold layer exists to counteract that: under balanced_accuracy the tuned threshold correlates with class frequency at +0.99 — rarer classes get a larger boost. An absent class is the rarest of all, so suppressing it inverts the treatment every other rare class receives in the same fit.

Validated on real datasets with real TabPFN probabilities (digits K=10, wine K=3, covtype K=7 subsampled to 6k; classes geometrically subsampled so genuine rare classes exist; 8 seeds; holdout deliberately small and unstratified so rare classes can miss it). Paired per-seed, fix vs pre-fix main:

dataset	accuracy	balanced_acc	f1	roc_auc	log_loss
digits	+0.0023 (W2/L0)	+0.0169 (W3/L0)	+0.0114 (W2/L0)	-0.0001 (W1/L2)	main crashed 8/8
wine	0.0000 (T8)	0.0000 (T8)	0.0000 (T8)	0.0000 (T8)	+0.0000 (T6)
covtype	0.0000 (T8)	0.0000 (T8)	+0.0122 (W1/L0)	-0.0000 (W1/L7)	main crashed 8/8

No meaningful regression anywhere: the only negative cells are -0.0001 and -0.0000 on roc_auc. And on real imbalanced multiclass data the log_loss crash is not an edge case -- it fired on 8/8 seeds for both digits and covtype.

Behaviour impact

Only affects fits with tune_decision_thresholds=True and a class with no holdout support. Thresholds for tunable classes are unchanged, so fits where every class appears in the holdout are bit-for-bit identical.

Tests

Nine regression tests, all failing on main:

• find_optimal_classification_thresholds parametrized over all five metrics — the absent class must get the geometric mean of the tuned thresholds, and every tunable class must match a direct single-class search (covers the log_loss mis-thresholding and the roc_auc 0.01 collapse).
• A single-class holdout must produce uniform thresholds that leave predictions untouched.
• select_robust_optimal_threshold must fall back to the midpoint when all losses are non-finite, and must never let a nan beat a finite loss — including when all finite losses are positive (the log_loss regime), where an internal zero-fill would win the argmin.

Not addressed here

metric_name="roc_auc" remains tunable in this PR. It is handled separately in #1153, which rejects the combination outright: the threshold search scores thresholded labels, so it was silently optimizing balanced accuracy rather than ROC AUC (identical threshold vectors), and the reweighting measurably lowers macro OvR AUC.

Best merged in that order -- this PR first, #1153 on top -- since landing #1153 first makes the roc_auc portion of the motivation here unreachable through the classifier path. The guard in this PR still stands on its own for the other metrics, notably the log_loss mis-thresholding (whose pre-v8.3.0 crash fired on 8/8 seeds on digits and covtype) and the f1/balanced_accuracy gains.

Also not addressed (pre-existing on main, surfaced during review): the sibling rare-class arrangement still crashes fit() upstream of this code. When the rare class's rows land in a used tuning fold's holdout, that fold's sub-classifier trains without the class and emits K-1 logit columns — under auto tuning settings this fails at the np.concatenate in _compute_holdout_validation_data (classifier.py:1353), and with tuning_n_folds=1 it is an IndexError at y_pred_probas[:, i]. Since auto settings use every fold for n ≤ 5000, the holdout union covers the whole training set there, so the absent-from-holdout path this PR fixes is reachable only for n > 5000 (auto uses 3 of 5 folds) or an explicit tuning_n_folds below the fold count; below that, the singleton-class scenario hits the upstream crash instead. Deserves its own issue and an end-to-end rare-class tuning test once fixed.

Prior art, and the choice of fallback value

Neither sklearn nor AutoGluon offers multiclass decision-threshold tuning: sklearn's TunedThresholdClassifierCV raises ValueError("Only binary classification is supported") (multiclass is open issue scikit-learn#30970), and AutoGluon's calibrate_decision_threshold asserts problem_type == 'binary'.

That is not evidence against what we do here, because we are not doing the thing they decline. They refuse multiclass thresholding because per-class cutoffs do not compose into a decision rule — rows appear where two classes both clear their threshold, or none do, with no principled tiebreak. This code never thresholds at predict time: _maybe_reweight_probas turns the tuned values into per-class reweighting factors (probas / t, renormalize, argmax), which is always well defined.

So the mechanism is closer to learned class-prior reweighting (class_weight='balanced') than to threshold tuning, with weights coming from a validation search rather than training frequencies. Two consequences worth noting:

• The values are tuned as cutoffs (probas[:, i] >= t) but applied as divisors. That mismatch costs almost nothing in practice: a direct coordinate search over the divisors actually used scores 0.3491 vs 0.3480 for the current approach on held-out balanced accuracy (untuned baseline 0.2416), because the two are monotonically related and argmax after renormalization only depends on relative magnitudes.
• Since these behave as class weights, "rarer class gets a larger boost" is exactly what balanced weighting does — which strengthens the case for min below.

For the fallback value itself there is no established practice to copy. The nearest reference point is sklearn's compute_class_weight, which derives weights from training frequencies and refuses to produce one for a class absent from the data rather than inventing a value.

mean is what this PR originally shipped, and the evidence in the table below favours min; review resolved this on the geometric mean — see the end of this section. On real data, restricted to seeds where a class was genuinely absent (8/8 seeds on both datasets):

dataset	metric	mean	min	min - mean	W/L/T
digits	balanced_accuracy	0.7901	0.8265	+0.0364	5/2/1
digits	f1	0.7682	0.7947	+0.0264	4/3/1
covtype	balanced_accuracy	0.4195	0.4549	+0.0354	3/3/2
covtype	f1	0.3843	0.3804	-0.0039	3/3/2

The rationale for min: under balanced_accuracy the tuned threshold correlates with class frequency, so the absent class -- the rarest of all -- belongs at the bottom of the tuned range rather than its middle.

The caution: the win/loss records are near even (3/3, 3/3, 4/3). min wins bigger than it loses rather than winning more often, which is the high-variance signature expected of an order statistic. At n=8 the per-seed evidence is weak even though the mean effect is consistent across two unrelated datasets.

Resolution (after review): the PR now ships the geometric mean of the tuned thresholds. The thresholds are applied as divisors and only their ratios survive renormalization, so neutrality lives on the log scale: the arithmetic mean of thresholds is the harmonic mean of the multipliers 1/t — the most suppressive classical mean, with AM/GM reaching ~1.45 on realistic balanced_accuracy threshold spreads — while the geometric mean carries exactly the average multiplier. It is derivable from the mechanism rather than fitted to n=8 seeds, and it sits between mean and min, capturing part of min's expected gain without betting on an order statistic.