When an operator overrides a segment classification, the database records what the model originally predicted alongside its confidence score. It would be super helpful to see an "active learning" summary that ranks corrections where the model was most confidently wrong. This immediately points out which heuristics or thresholds are miscalibrated and shows us the best candidates to turn into unit test traces.
Suggested fix:
Add a summary section to calibrate (or a dedicated feedback.py command) that sorts corrections by model confidence on overridden calls, grouped by original_type -> corrected_type.
Definition of Done:
The report outputs a top-N list of confident misclassifications along with their feature values.
When an operator overrides a segment classification, the database records what the model originally predicted alongside its confidence score. It would be super helpful to see an "active learning" summary that ranks corrections where the model was most confidently wrong. This immediately points out which heuristics or thresholds are miscalibrated and shows us the best candidates to turn into unit test traces.
Suggested fix:
Add a summary section to calibrate (or a dedicated feedback.py command) that sorts corrections by model confidence on overridden calls, grouped by original_type -> corrected_type.
Definition of Done:
The report outputs a top-N list of confident misclassifications along with their feature values.