Skip to content

Add macro precision, recall and F1 to the classification validator - #883

Open
velu-97 wants to merge 2 commits into
LibreYOLO:devfrom
velu-97:852-classify-macro-metrics
Open

velu-97 wants to merge 2 commits into
LibreYOLO:devfrom
velu-97:852-classify-macro-metrics

Conversation

@velu-97

@velu-97 velu-97 commented Sep 19, 2026

Copy link
Copy Markdown

What: the classification validator reports macro precision, recall and F1 from a confusion matrix.
Why: it only had top-1/top-5. Lands part of #852 focused on the classifier changes.

  • ClassifyValidator: per-batch confusion matrix (in-place index_add_, cost independent of class count). New keys metrics/precision, metrics/recall, metrics/f1: unweighted mean over classes present in the validation targets; precision is 0 for a never-predicted class. Same key names as the point validator.
  • Existing keys, their values and fitness (top-1) are unchanged, so best-checkpoint selection and early stopping are unchanged.
  • Targets outside the head's class range are skipped by the matrix and still count as wrong for top-1.
  • Output changes: the validator's result log gains one line; libreyolo val classify output gains precision, recall, f1 in --json and in the text line, only when the validator returned them (that branch whitelists keys); results.csv and loggers see three new metric columns.
  • Docs: val() docstring, a "Validation metrics" section in docs/classification_training.md (the only classify training doc today), and a CHANGELOG entry under Unreleased.

Shared code: ClassifyValidator is where classify metrics are computed for every family, so the change has to live there. It is inherited by the CLIP, SigLIP2 and ViT classify validators and used by every classify trainer and exported backend. Keys are additive and nothing enumerates classify keys with a fixed list. Under DDP, classify validation already runs on rank 0 over the whole split, so the matrix needs no reduction.

Check: the macro definition in _macro_precision_recall_f1 (classes absent from the targets are excluded, which differs from a union-of-labels macro average on subset splits); the out-of-range target mask in _update_metrics; key names.
Verified: full PR gate locally (macOS, CPU), 7437 passed, plus 14 distributed.
Not verified: no GPU run.

Code provenance

Original code written for this PR against LibreYOLO's own first-party code; no third-party code ported, adapted, or introduced; no GPL/AGPL/LGPL/non-commercial/unknown-license material involved.

RetriggerConfidence Score: 5/5

The PR appears safe to merge, with the previous quadratic-memory problem fully resolved and no new actionable defects identified.

Summary

This PR adds macro precision, recall, and F1 to classification validation while preserving top-1 as the fitness and checkpoint-selection metric.

  • Accumulates true positives, prediction counts, and target counts in three per-class vectors.
  • Excludes classes absent from valid validation targets from macro averaging.
  • Exposes the new metrics through the Python API, CLI text and JSON output, logs, and result columns.
  • Documents the metric definitions and adds regression coverage for calculation, accumulation, invalid targets, empty inputs, CLI formatting, and linear memory usage.
  • The previous quadratic-memory finding is fully addressed by replacing the dense class-by-class confusion matrix with linear-size count vectors.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Classification logits and targets] --> B[Top-1 prediction]
  B --> C[Valid target-range mask]
  C --> D[Per-class true positives]
  C --> E[Per-class prediction counts]
  C --> F[Per-class target counts]
  D --> G[Per-class precision recall and F1]
  E --> G
  F --> G
  G --> H[Macro average over classes present in targets]
  H --> I[Python API logs CLI JSON and text]
Loading

Reviews (2) · Last reviewed commit: "Accumulate per-class confusion counts in..."

Comment thread libreyolo/validation/classify_validator.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant