Reject eval_metric='roc_auc' with tune_decision_thresholds=True

[LeoGrin]

Summary

Threshold tuning under eval_metric='roc_auc' never optimized ROC AUC. It silently optimized balanced accuracy, and separately degraded the metric the user asked for.

The search scores thresholded 0/1 predictions:

y_pred_tuned = (y_pred_probas >= threshold).astype(int)
current_loss = compute_metric_to_minimize(metric_name, y_true, y_pred_tuned)

With a binary score vector the ROC curve has a single interior point, so the area is FPR·TPR/2 + (1−FPR)(TPR+1)/2 = (TPR + TNR)/2 — exactly balanced accuracy. The two metrics are therefore the same objective, and the search returns bit-identical thresholds:

metric_name="roc_auc"            [0.26868  0.248782  0.194061  0.129391  0.069695  0.049797]
metric_name="balanced_accuracy"  [0.26868  0.248782  0.194061  0.129391  0.069695  0.049797]
identical: True

Why this is rejected rather than repaired

Passing probabilities instead would be worse. ROC AUC does not depend on the threshold, so the objective would be constant across the grid, every candidate would score identically, and select_robust_optimal_threshold would return the midpoint of a full-width plateau regardless of the data.

Thresholds cannot improve a ranking metric. At predict time the values are applied as divisors (probas / t, renormalized). For binary targets that is a strictly monotone transform of the positive-class probability:

q₁ = p₁t₀ / (t₁(1−p₁) + t₀p₁),    dq₁/dp₁ = t₀t₁/(…)² > 0

so the ranking, and therefore ROC AUC, is provably unchanged. Verified identical to 10 decimal places across thresholds from (0.5, 0.5) to (0.01, 0.99).

For multiclass the ranking does move, but negligibly. The renormalizer mixes in other classes, so q_i is not monotone in p_i. However D = Σⱼwⱼpⱼ is a convex combination of the weights, so it varies only mildly across rows and the reranking is second order. Measured:

	digits (K=10)	covtype (K=7)
headroom from a deliberate per-class distortion (0.37×–2.7×)	+0.0000	+0.0011
coordinate search aimed directly at held-out macro OvR AUC, test gain	−0.0000	+0.0002

The search was validated by a positive control — a known per-class distortion was injected, and the optimizer recovered 110% of the resulting headroom on covtype (4/4 seeds). So the null result reflects absence of signal, not a weak optimizer. Caveat: digits' AUC is ~0.997, at the ceiling, so its control was uninformative; the validation rests on covtype.

Meanwhile the reweighting is applied inside _predict_proba, and measurably lowers macro OvR AUC: −0.0128 on covtype, losing on 7 of 8 seeds.

Scope — only one case changes

eval_metric has exactly one functional effect: selecting the threshold-tuning objective. find_optimal_temperature takes no metric_name and always optimizes log loss.

config	before	after
no tuning_config (the default)	inert, no warning	unchanged
tuning_config with neither flag	inert, no warning	unchanged
calibrate_temperature only	warned	still allowed, warning reworded
tune_decision_thresholds=True	warned, silently optimized balanced accuracy	raises ValueError

The error message names both escape routes: eval_metric='balanced_accuracy' to keep the previous behaviour exactly, or tune_decision_thresholds=False.

The old warning was misleading

"ROC AUC is independent of these tunings and they will not improve this metric."

Wrong on three counts: the reweight is applied inside _predict_proba so AUC is not independent of it; the effect is not neutral but negative; and it never mentions the metric substitution, which is the important part. It read as "harmless, optional cleanup." The temperature-calibration warning is reworded to state the accurate fact — that calibration optimizes log loss regardless of eval_metric.

Not affected

FinetunedTabPFNClassifier(eval_metric='roc_auc') is a different parameter on a different class. It scores roc_auc_score(y_val, probabilities) with multi_class='ovr' for checkpoint selection — a correct use of the metric, deliberately untouched.

Tests

The test asserting the removed warning is replaced by two: one asserting the raise for tune_decision_thresholds, one asserting the calibration path still warns and still fits. Full test_classifier_interface.py: 163 passed, 63 skipped.

Relation to #1152

Independent, different files (classifier.py here, inference_tuning.py there). #1152 fixes classes with no holdout support, where roc_auc was one of several affected metrics; it stands on its own for log_loss (crashes on 8/8 seeds on real data) and f1/balanced_accuracy. Best merged after #1152, since landing this first makes some of that PR's roc_auc motivation unreachable through the classifier path.

🤖 Generated with Claude Code