Skip to content

Add zero-shot object detection schema and evaluator to winml eval #1147

Description

@ssss141414

Problem

winml eval has no schema or evaluator for zero-shot-object-detection, so valid open-vocabulary detectors such as OWLv2 cannot produce a task metric. Reusing the existing fixed-label object-detection evaluator is not semantically valid: zero-shot detection must derive text queries from category names, post-process grounded detections, and map query-local results back to dataset category IDs.

Reproduction

$MODEL_ID = 'google/owlv2-base-patch16-finetuned'
$REV = 'd69b086b07123308a4e198343ab2824e1af7774a'
$SNAPSHOT = 'checkpoint_snapshot'
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='$MODEL_ID', revision='$REV', local_dir='$SNAPSHOT')"
winml eval -m model.onnx --model-id $SNAPSHOT --task zero-shot-object-detection --dataset detection-datasets/coco --dataset-revision cf0b22332314a937e9dc8a1957b21725430bb41d --split val --samples 1 --no-shuffle --streaming --ep cpu --device cpu --output eval.json --overwrite --format json --no-color

Actual result:

Error: Evaluation failed: Task 'zero-shot-object-detection' is not supported.

The task is absent from both the evaluation task schema and evaluator registry, so execution stops before dataset or model evaluation.

Expected capability

Add a generalized zero-shot-object-detection evaluation schema/evaluator that:

  1. derives candidate text queries from dataset category names;
  2. runs the model processor's grounded object-detection post-processing;
  3. maps query-local detections to dataset category IDs; and
  4. reports a standard detection metric such as COCO mAP.

This should remain distinct from fixed-label object-detection evaluation.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions