Summary
winml eval does not support the exact document-question-answering task contract. LayoutLM v1 document-QA models expose an extractive span head (input_ids, OCR-derived bbox, attention_mask, token_type_ids → start_logits, end_logits), but end-to-end task evaluation also requires OCR, word/box normalization and token alignment, answer-span decoding, and a document-QA dataset/metric adapter.
Current behavior
winml eval --schema --task document-question-answering
Error: Unsupported task 'document-question-answering'.
The generic inference task registry describes image + question inputs, but checkpoints such as DmitrySpartak/layoutlm-invoices have a tokenizer only and consume pre-tokenized OCR/layout tensors; they do not accept a PDF or image directly.
Desired behavior
Define an explicit document-QA inference/evaluation contract that:
- preserves
document-question-answering rather than silently substituting plain question-answering;
- supplies or integrates OCR and normalized bounding-box preprocessing;
- aligns question/document tokens and boxes;
- decodes
start_logits / end_logits to answer spans;
- provides a representative dataset and task metric schema;
- handles architecture differences such as LayoutLMv1 (text + boxes) versus LayoutLMv3 (text + boxes + pixels).
Scope note
This is separate from ONNX exporter registration tracked by #134. Export/build/perf can work on prepared tensors without providing an end-to-end document image-to-answer pipeline.
Summary
winml evaldoes not support the exactdocument-question-answeringtask contract. LayoutLM v1 document-QA models expose an extractive span head (input_ids, OCR-derivedbbox,attention_mask,token_type_ids→start_logits,end_logits), but end-to-end task evaluation also requires OCR, word/box normalization and token alignment, answer-span decoding, and a document-QA dataset/metric adapter.Current behavior
The generic inference task registry describes image + question inputs, but checkpoints such as
DmitrySpartak/layoutlm-invoiceshave a tokenizer only and consume pre-tokenized OCR/layout tensors; they do not accept a PDF or image directly.Desired behavior
Define an explicit document-QA inference/evaluation contract that:
document-question-answeringrather than silently substituting plainquestion-answering;start_logits/end_logitsto answer spans;Scope note
This is separate from ONNX exporter registration tracked by #134. Export/build/perf can work on prepared tensors without providing an end-to-end document image-to-answer pipeline.