Skip to content

Add end-to-end document-question-answering inference and eval contract #1132

Description

@ssss141414

Summary

winml eval does not support the exact document-question-answering task contract. LayoutLM v1 document-QA models expose an extractive span head (input_ids, OCR-derived bbox, attention_mask, token_type_idsstart_logits, end_logits), but end-to-end task evaluation also requires OCR, word/box normalization and token alignment, answer-span decoding, and a document-QA dataset/metric adapter.

Current behavior

winml eval --schema --task document-question-answering
Error: Unsupported task 'document-question-answering'.

The generic inference task registry describes image + question inputs, but checkpoints such as DmitrySpartak/layoutlm-invoices have a tokenizer only and consume pre-tokenized OCR/layout tensors; they do not accept a PDF or image directly.

Desired behavior

Define an explicit document-QA inference/evaluation contract that:

  • preserves document-question-answering rather than silently substituting plain question-answering;
  • supplies or integrates OCR and normalized bounding-box preprocessing;
  • aligns question/document tokens and boxes;
  • decodes start_logits / end_logits to answer spans;
  • provides a representative dataset and task metric schema;
  • handles architecture differences such as LayoutLMv1 (text + boxes) versus LayoutLMv3 (text + boxes + pixels).

Scope note

This is separate from ONNX exporter registration tracked by #134. Export/build/perf can work on prepared tensors without providing an end-to-end document image-to-answer pipeline.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions