Problem
winml eval has no schema or evaluator for zero-shot-object-detection, so valid open-vocabulary detectors such as OWLv2 cannot produce a task metric. Reusing the existing fixed-label object-detection evaluator is not semantically valid: zero-shot detection must derive text queries from category names, post-process grounded detections, and map query-local results back to dataset category IDs.
Reproduction
$MODEL_ID = 'google/owlv2-base-patch16-finetuned'
$REV = 'd69b086b07123308a4e198343ab2824e1af7774a'
$SNAPSHOT = 'checkpoint_snapshot'
python -c "from huggingface_hub import snapshot_download; snapshot_download(repo_id='$MODEL_ID', revision='$REV', local_dir='$SNAPSHOT')"
winml eval -m model.onnx --model-id $SNAPSHOT --task zero-shot-object-detection --dataset detection-datasets/coco --dataset-revision cf0b22332314a937e9dc8a1957b21725430bb41d --split val --samples 1 --no-shuffle --streaming --ep cpu --device cpu --output eval.json --overwrite --format json --no-color
Actual result:
Error: Evaluation failed: Task 'zero-shot-object-detection' is not supported.
The task is absent from both the evaluation task schema and evaluator registry, so execution stops before dataset or model evaluation.
Expected capability
Add a generalized zero-shot-object-detection evaluation schema/evaluator that:
- derives candidate text queries from dataset category names;
- runs the model processor's grounded object-detection post-processing;
- maps query-local detections to dataset category IDs; and
- reports a standard detection metric such as COCO mAP.
This should remain distinct from fixed-label object-detection evaluation.
Problem
winml evalhas no schema or evaluator forzero-shot-object-detection, so valid open-vocabulary detectors such as OWLv2 cannot produce a task metric. Reusing the existing fixed-labelobject-detectionevaluator is not semantically valid: zero-shot detection must derive text queries from category names, post-process grounded detections, and map query-local results back to dataset category IDs.Reproduction
Actual result:
The task is absent from both the evaluation task schema and evaluator registry, so execution stops before dataset or model evaluation.
Expected capability
Add a generalized
zero-shot-object-detectionevaluation schema/evaluator that:This should remain distinct from fixed-label
object-detectionevaluation.