Skip to content

feat: Add export override options to winml eval command#1141

Open
xieofxie wants to merge 1 commit into
mainfrom
hualxie/dyn_to_eval
Open

feat: Add export override options to winml eval command#1141
xieofxie wants to merge 1 commit into
mainfrom
hualxie/dyn_to_eval

Conversation

@xieofxie

Copy link
Copy Markdown
Contributor

Summary

Brings winml eval to parity with winml build and winml perf by adding four export override options:

  • --shape-config — JSON shape overrides for the auto-generated HuggingFace export config
  • --input-specs — JSON input tensor specs merged into the HuggingFace export config (symbolic dims infer dynamic axes)
  • --export-config — JSON ONNX export config overrides (opset version, constant folding, etc.)
  • --dynamic-axes — JSON dynamic axes mapping for HuggingFace ONNX export

These options shape the ONNX export that eval generates when -m is a HuggingFace model ID. When -m is a pre-built .onnx file there is no export step, so the flags are ignored and the command prints a warning.

Changes

  • eval/config.py — added shape_config and export_overrides fields to WinMLEvaluationConfig, JSON-safe serialization of InputTensorSpec values, and to_dict/from_dict support.
  • eval/evaluate.py_load_model threads {"export": ...} overrides plus shape_config into WinMLAutoModel.from_pretrained on the HuggingFace build path; the from_onnx path is unchanged.
  • commands/eval.py — added the four option decorators/params and an _apply_export_overrides helper that parses the JSON and warns + skips for pre-built ONNX inputs.
  • docs/commands/eval.md — documented the four flags and added a pitfall note.
  • Tests — added TestEvalExportOverrides (10 cases) and updated the existing from_pretrained assertion for the new shape_config kwarg.

Verification

  • uv run pytest tests/unit/commands/test_eval.py tests/unit/eval/test_eval.py → 103 passed
  • uv run ruff check → clean
  • Doc flag/code alignment check → passing

Bring winml eval to parity with build/perf by adding --shape-config,
--input-specs, --export-config, and --dynamic-axes. These shape the
ONNX export when eval builds from a HuggingFace model ID and are
ignored (with a warning) for pre-built ONNX inputs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@xieofxie
xieofxie requested a review from a team as a code owner July 20, 2026 06:40
@xieofxie xieofxie changed the title Add export override options to winml eval command feat: Add export override options to winml eval command Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant