feat(layoutlm): add document question answering support

[ssss141414]

Summary

Adds class-wide LayoutLM v1 document-question-answering export support and verified CPU fp32/fp16 recipes for DmitrySpartak/layoutlm-invoices. This matters because current main misresolves this checkpoint to next-sentence prediction and cannot build it. The shipped contribution is Effort L1 / Outcome L1 and independently reaches Goal L2 PASS for both required tuples; end-to-end task evaluation remains CLI-blocked and is tracked by #1132.

Model metadata

What the model does

This pinned LayoutLM v1 checkpoint is an extractive document-question-answering model for invoices and other documents. Its model forward consumes token IDs, normalized token bounding boxes, an attention mask, and token-type IDs, then emits per-token start and end logits. It does not consume a PDF or image directly: OCR, word/box construction, token alignment, and answer decoding are external pipeline stages. The resolved stock Transformers class ignores the checkpoint's extra token_classifier_head weights, so the model card's advertised non-consecutive-token behavior is not verified on this concrete runtime path. Confidence: verified.

Durable sources: pinned model card, pinned checkpoint config, and LayoutLMForQuestionAnswering in Transformers 4.57.6.

Primary user stories

• A user supplies an invoice document and a question such as “What is the invoice number?” to obtain an answer span for document review or extraction; an external OCR/layout pipeline must first turn the document into words and normalized boxes. Confidence: verified.
• A user supplies OCR-derived words, token bounding boxes, and a natural-language question to obtain start/end token scores for extracting an answer from an invoice. Confidence: verified.

Sources: the pinned model card, pinned checkpoint config, and Transformers 4.57.6 implementation.

Supported tasks

• document-question-answering — supported by the checkpoint and Transformers model head. Confidence: verified. WinML export/config support is added by this PR; WinML end-to-end evaluation is not yet supported and is tracked by #1132. Plain question-answering was deliberately not substituted because it would erase the document-layout contract.

Model architecture

LayoutLMForQuestionAnswering
├── LayoutLMModel
│   ├── Embeddings (word + 1D position + x/y/h/w 2D layout + token type; hidden 768)
│   └── Encoder stack × 12
│       ├── Multi-head self-attention (12 heads)
│       ├── Feed-forward (768 → 3072 → 768, GELU)
│       └── Residual + LayerNorm
└── QA span head (Linear 768 → 2)
    ├── start_logits [batch, sequence]
    └── end_logits [batch, sequence]

• Source/confidence: pinned checkpoint config and LayoutLMForQuestionAnswering source in Transformers 4.57.6 (verified).

Validation and support evidence

Baseline

• Baseline: e7509b1e908c74beff0a5655b8f8d7de69c5afae, WinML 0.2.0.
• Exact-task config: FAIL, exit 2 — Task 'document-question-answering' not supported by TasksManager; no requested-task config artifact produced.
• Automatic config behavior: it succeeds only by misresolving the checkpoint to next-sentence-prediction / AutoModelForNextSentencePrediction; this diagnostic output is not a valid starting recipe.
• Recipe-free build: FAIL, exit 2 — Unrecognized LayoutLMConfig for AutoModelForNextSentencePrediction; no model.onnx produced.
• Perf floor: FAIL, exit 2 — ONNX file not found because baseline build failed.
• Eval floor: FAIL, exit 2 — ONNX file not found; independent schema probe also rejects exact task document-question-answering as unsupported. Plain question-answering schema exists but was not substituted.
• Optimum probe for layoutlm: vendor tasks and post-WinML tasks are both feature-extraction, fill-mask, text-classification, and token-classification; WinML adds no task. Verdict for document-question-answering: UNREGISTERED.

Goal

• Committed Effort: L1
• Committed Goal ceiling: L2
• Committed Outcome: L1
• Success definition: Both CPU/cpu fp32 and fp16 build structurally, execute 100-iteration perf with valid named LayoutLM inputs, and compare start/end logits to pinned PyTorch on identical OCR-derived boxes.
• No ceiling change was issued.

Outcome

• Shipped tier: L1
• Highest Goal verdict: L2 PASS
• Coverage: full
• Deferred tuples: none
• Handoff: PASS — learner-owned findings are durably shipped in ModelKitArtifacts Lane A commit 7e939ae638193e0f83f717d7bee4f9b5b4311bf1, tracked by ModelKitArtifacts PR #159.
• Shipped recipes:
  • document-question-answering_fp32_config.json
  • document-question-answering_fp16_config.json
• Shipped generalized code and focused tests are enumerated under Delta.
• Model findings captured and pushed in ModelKitArtifacts Lane A commit 7e939ae638193e0f83f717d7bee4f9b5b4311bf1 on ModelKitArtifacts PR #159: layoutlm-002 architecture-driven exact-task resolution; layoutlm-003 input bounds and generic high-exclusive range propagation; layoutlm-004 fp32/fp16 build, perf, parity, and analysis; layoutlm-005 exact eval blocker and issue Add end-to-end document-question-answering inference and eval contract #1132.
• Methodology: the producer/tester repair loop encountered the already-known generic winml perf value-range friction documented by _meta-017. No new methodology finding or verdict shape was required; no skill edit is part of this model PR.

Per-EP/device/precision results — including perf and eval data

Goal ladder

Tier	Verdict	Evidence
L0	PASS	Both required CPU/cpu precisions independently configured, built, and structurally validated.
L1	PASS	Both tuples completed 10 warmups + 100 timed CLI iterations using named semantic NPZ input; exact numbers below.
L2	PASS	Pinned PyTorch LayoutLMForQuestionAnswering comparison on identical named 512-token inputs with 33 nonzero boxed document tokens; start/end argmaxes preserved.

Build and structural coverage

EP / Device	Precision	Verdict	Structural evidence
CPUExecutionProvider / cpu	fp32	PASS	config exit 0; build exit 0; Build complete; onnx.checker PASS; IR=8; opset=17; nodes=442; inputs=input_ids, bbox, attention_mask, token_type_ids; outputs=start_logits, end_logits; initializers: FLOAT 208, INT64 12; final artifact 508,972,442 bytes with external data.
CPUExecutionProvider / cpu	fp16	PASS	config exit 0; build exit 0; Build complete; onnx.checker PASS; IR=8; opset=17; nodes=444; inputs=input_ids, bbox, attention_mask, token_type_ids; outputs=start_logits, end_logits; initializers: FLOAT16 208, INT64 12; final artifact 254,570,982 bytes with external data.

Perf — named semantic document inputs

Each row used normalized OCR boxes, all-zero token_type_ids, 10 warmups, and 100 timed iterations on CPUExecutionProvider.

EP / Device	Precision	Verdict	Mean	p50	p90	p95	Throughput	RAM load Δ	RAM inference Δ	RAM total Δ	Local VRAM Δ
CPUExecutionProvider / cpu	fp32	PASS	208.185 ms	201.974 ms	236.443 ms	245.382 ms	4.8 samples/s	334.71 MB	103.98 MB	438.69 MB	0.0 MB
CPUExecutionProvider / cpu	fp16	PASS	256.448 ms	251.709 ms	289.198 ms	297.091 ms	3.9 samples/s	347.75 MB	117.7 MB	465.46 MB	0.0 MB

Perf — default generated inputs after the generic range repair

These rows used no --input-data; each completed 10 warmups and 100 timed iterations.

EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM total Δ
CPUExecutionProvider / cpu	fp32	PASS	213.624 ms	206.071 ms	4.68 samples/s	441.04 MB
CPUExecutionProvider / cpu	fp16	PASS	252.585 ms	245.067 ms	3.96 samples/s	467.14 MB

Default winml perf now honors config-derived ranges for both repaired artifacts; generic integer, float, singleton, and unspecified-range behavior passes independently.

L2 numeric parity

The identical semantic input contains 33 nonzero document-box tokens, bbox range 0–755, monotonic boxes, and only token type 0. Reference/ONNX argmaxes are preserved at 16 for start_logits and 22 for end_logits.

EP / Device	Precision	Output	Verdict	Shape	Cosine	Max abs	Mean abs	Ref argmax	ONNX argmax
CPUExecutionProvider / cpu	fp32	start_logits	PASS	[1, 512]	0.9999999999999564	1.9073486328125e-05	3.6600977182388306e-06	16	16
CPUExecutionProvider / cpu	fp32	end_logits	PASS	[1, 512]	0.9999999999999577	1.9073486328125e-05	3.725290298461914e-06	22	22
CPUExecutionProvider / cpu	fp16	start_logits	PASS	[1, 512]	0.9999999880818508	0.009905815124511719	0.002066316083073616	16	16
CPUExecutionProvider / cpu	fp16	end_logits	PASS	[1, 512]	0.999999960350131	0.010828971862792969	0.003938476555049419	22	22

Eval

Schema command verdict: CLI-BLOCKED, exit 2.

Error: Task 'document-question-answering' is not supported by winml eval. Supported tasks: depth-estimation, feature-extraction, fill-mask, image-classification, image-feature-extraction, image-segmentation, image-to-text, keypoint-detection, mask-generation, next-sentence-prediction, object-detection, question-answering, sentence-similarity, sequence-classification, text-classification, token-classification, zero-shot-classification, zero-shot-image-classification.

EP / Device	Precision	Verdict	Exit	Dataset / revision / subset	Metric	Exact error
CPUExecutionProvider / cpu	fp32	CLI-BLOCKED	1	— / — / —	—	Error: Evaluation failed: Task 'document-question-answering' is not supported. Supported tasks: compare-tensor, depth-estimation, feature-extraction, fill-mask, image-classification, image-feature-extraction, image-segmentation, image-to-text, keypoint-detection, mask-generation, next-sentence-prediction, object-detection, question-answering, sentence-similarity, sequence-classification, text-classification, token-classification, zero-shot-classification, zero-shot-image-classification.
CPUExecutionProvider / cpu	fp16	CLI-BLOCKED	1	— / — / —	—	Error: Evaluation failed: Task 'document-question-answering' is not supported. Supported tasks: compare-tensor, depth-estimation, feature-extraction, fill-mask, image-classification, image-feature-extraction, image-segmentation, image-to-text, keypoint-detection, mask-generation, next-sentence-prediction, object-detection, question-answering, sentence-similarity, sequence-classification, text-classification, token-classification, zero-shot-classification, zero-shot-image-classification.

No dataset or metric was produced. This is an eval task-family blocker, not an export, precision, or model-quality failure. The checkpoint forward graph is only the neural span scorer; OCR, PDF/image decoding, word/box normalization and alignment, and answer decoding remain external. The exact generalized gap is #1132 — Add end-to-end document-question-answering inference and eval contract (open).

Delta

Recipes

• Baseline recipe: none. Comparison is NOT-COMPARABLE because exact-task config exited 2 before producing a config artifact; automatic config's next-sentence-prediction misresolution was diagnostic evidence, not a recipe baseline.
• After the fix, explicit and automatic generated configs resolve the requested task/class/I/O contract and are byte-semantically identical to the checked-in fp32/fp16 recipes.
• fp32 resolved build config: optim={}, quant=null, compile=null.
• fp16 resolved build config: optim={}, quant.mode=fp16, compile=null.
• No manual recipe-only override or JSON-pointer old/new delta exists.
• Recipe-free fp32 and fp16 acceptance is required and passed with the exact four-input/two-output contract.
• The production recipe README remains untouched.

Generalized code and tests

Path	Symbols / behavior change
src/winml/modelkit/loader/resolution.py	Adds generic model_type + architectures[0] registry lookup before Optimum inference with explicit TaskSource.ARCHITECTURE_MAPPING provenance. There is no model-ID or LayoutLM branch in shared resolution code.
src/winml/modelkit/models/hf/layoutlm.py	Adds class-wide LayoutLMForQuestionAnswering → document-question-answering, AutoModelForDocumentQuestionAnswering, bbox-preserving four-input export, start_logits/end_logits, type-vocabulary-derived token types, and conditional RoBERTa position-offset handling.
src/winml/modelkit/models/hf/__init__.py	Imports LayoutLM registration and aggregates its class and architecture-task metadata.
src/winml/modelkit/commands/perf.py	Generic generate_random_inputs now reads io_config.input_value_ranges per input and forwards valid metadata ranges. Canonical ranges stay high-exclusive; integer maxima are reduced by one only at the legacy generator's inclusive-max adapter boundary. Float ranges, shapes, tensor classification, and unspecified ranges retain prior behavior.
tests/unit/loader/test_layoutlm_resolution.py	Covers architecture-derived exact-task resolution, exact document auto class, and non-QA head preservation.
tests/unit/export/test_layoutlm_onnx_config.py	Covers registration, exact I/O, token-type bounds, one-time conditional position adjustment, untouched non-RoBERTa convention, and invalid usable length.
tests/unit/commands/test_perf_cli.py	Adds model-agnostic regressions for high-exclusive integer ranges, arbitrary input names, singleton nonzero integer and float ranges, and unchanged defaults when no range is declared.

The fix remains reducible and consistent with the frozen charter: it is class-of-models L1 code, metadata-driven, recipe-free accepted, and contains no checkpoint-only branching. Other LayoutLM heads remain on existing Optimum task paths.

Analyze summary — component level and op level

ANALYZE-PARTIAL-SUCCESS (exit 1): OpenVINO plugin registration failed because onnxruntime_providers_shared.dll was missing (Error 126); emitted JSON remained complete for all 11 EP/device rows. This is static rule analysis, not runtime execution on those accelerator EPs.

Component-level summary

Artifact	Architecture coverage	Mapping	Confidence	Actionable EP findings
fp32	embeddings; 12x encoder attention/FFN; QA span head	442 mapped, 0 unmapped	mapped	QNNExecutionProvider/NPU and QNNExecutionProvider/GPU: partial=Div,Erf,Add,Mul; unsupported=none
fp16	embeddings; 12x encoder attention/FFN; QA span head	444 mapped, 0 unmapped	mapped	QNNExecutionProvider/NPU and QNNExecutionProvider/GPU: partial=Div,Erf,Add,Mul; unsupported=none

Built-artifact ONNX mapping was absent from the frozen planner report; the tester derived it from emitted full hierarchy tags.

Op-level summary

Artifact	Graph	Dominant operators	Grouped EP outcomes
fp32	442 operators / 16 unique types	Add 129; MatMul 97; Reshape 48; Transpose 48; Mul 37; LayerNormalization 25	Fully supported by NvTensorRTRTX/GPU and OpenVINO NPU/GPU/CPU; QNN NPU/GPU partial on Div,Erf,Add,Mul, unsupported=none; all types unknown for CUDA/GPU, MIGraphX/GPU, Dml/GPU, CPU/CPU, VitisAI/NPU.
fp16	444 operators / 16 unique types	Add 129; MatMul 97; Reshape 48; Transpose 48; Mul 37; LayerNormalization 25	Fully supported by NvTensorRTRTX/GPU and OpenVINO NPU/GPU/CPU; QNN NPU/GPU partial on Div,Erf,Add,Mul, unsupported=none; all types unknown for CUDA/GPU, MIGraphX/GPU, Dml/GPU, CPU/CPU, VitisAI/NPU.

The host exposed DmlExecutionProvider and CPUExecutionProvider; only CPU runtime measurements are claimed. Static rows for NvTensorRTRTX, CUDA, MIGraphX, QNN, OpenVINO, DML, CPU, and VitisAI do not imply runtime validation.

Reproduce commands

The sequence below is portable PowerShell. It embeds the tester-supplied L2 harness verbatim and uses the versioned public rules-v0.2.0.zip archive with SHA-256 verification.

$MODEL='DmitrySpartak/layoutlm-invoices'
$REVISION='ce2422049c250384731eccef90bf6d92e846b09c'
$BASELINE_COMMIT='e7509b1e908c74beff0a5655b8f8d7de69c5afae'
$OUT='temp/layoutlm-invoices-repro'
New-Item -ItemType Directory -Force -Path $OUT | Out-Null
$OUT=(Resolve-Path $OUT).Path
winml --version
git rev-parse HEAD

winml config -m $MODEL --task document-question-answering --ep cpu --device cpu --precision fp32 --no-compile -o (Join-Path $OUT 'generated_fp32.json') --overwrite --no-color
winml config -m $MODEL --task document-question-answering --ep cpu --device cpu --precision fp16 --no-compile -o (Join-Path $OUT 'generated_fp16.json') --overwrite --no-color
winml build -c examples/recipes/DmitrySpartak_layoutlm-invoices/cpu/cpu/document-question-answering_fp32_config.json -m $MODEL -o (Join-Path $OUT 'fp32') --ep cpu --device cpu --precision fp32 --no-analyze --rebuild --no-color
winml build -c examples/recipes/DmitrySpartak_layoutlm-invoices/cpu/cpu/document-question-answering_fp16_config.json -m $MODEL -o (Join-Path $OUT 'fp16') --ep cpu --device cpu --precision fp16 --no-analyze --rebuild --no-color

$HARNESS=Join-Path $OUT 'l2_layoutlm_parity.py'
@'
"""Portable LayoutLM ONNX-vs-PyTorch parity harness with semantic document boxes."""

from __future__ import annotations

import argparse
import hashlib
import json
from pathlib import Path

import numpy as np
import onnxruntime as ort
import torch
from transformers import AutoModelForDocumentQuestionAnswering, AutoTokenizer

MODEL_ID = "DmitrySpartak/layoutlm-invoices"
REVISION = "ce2422049c250384731eccef90bf6d92e846b09c"
OUTPUT_NAMES = ("start_logits", "end_logits")


def cosine(a: np.ndarray, b: np.ndarray) -> float:
    x = a.astype(np.float64).ravel()
    y = b.astype(np.float64).ravel()
    denom = np.linalg.norm(x) * np.linalg.norm(y)
    return float(np.dot(x, y) / denom) if denom else float(x.shape == y.shape and np.array_equal(x, y))


def make_inputs() -> tuple[dict[str, np.ndarray], dict[str, object]]:
    question_words = "what is the invoice number".split()
    document_words = [
        "ACME", "SUPPLIES", "Invoice", "Number", "INV-2026-0719", "Invoice", "Date",
        "July", "19", "2026", "Bill", "To", "Contoso", "Total", "$1,245.00",
    ]
    # Normalized OCR boxes in reading order, each satisfying 0 <= x0 <= x1 <= 1000
    # and 0 <= y0 <= y1 <= 1000. The invoice-number answer occupies one real box.
    document_boxes = [
        [70, 55, 245, 95], [255, 55, 430, 95], [70, 145, 165, 180],
        [175, 145, 285, 180], [300, 145, 520, 180], [70, 205, 165, 240],
        [175, 205, 255, 240], [270, 205, 335, 240], [345, 205, 375, 240],
        [385, 205, 445, 240], [70, 300, 125, 335], [135, 300, 170, 335],
        [185, 300, 300, 335], [70, 720, 140, 755], [155, 720, 290, 755],
    ]
    tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, revision=REVISION, use_fast=True)
    encoded = tokenizer(
        question_words,
        document_words,
        is_split_into_words=True,
        truncation=True,
        max_length=512,
        padding="max_length",
        return_attention_mask=True,
        return_token_type_ids=True,
        return_tensors="np",
    )
    word_ids = encoded.word_ids(batch_index=0)
    sequence_ids = encoded.sequence_ids(batch_index=0)
    boxes = []
    for word_id, sequence_id in zip(word_ids, sequence_ids, strict=True):
        boxes.append(document_boxes[word_id] if sequence_id == 1 and word_id is not None else [0, 0, 0, 0])
    inputs = {
        "input_ids": encoded["input_ids"].astype(np.int32),
        "bbox": np.asarray([boxes], dtype=np.int32),
        "attention_mask": encoded["attention_mask"].astype(np.int32),
        "token_type_ids": np.zeros_like(encoded["input_ids"], dtype=np.int32),
    }
    answer_token_indices = [
        i for i, (wid, sid) in enumerate(zip(word_ids, sequence_ids, strict=True))
        if sid == 1 and wid == 4
    ]
    semantics = {
        "question": " ".join(question_words),
        "document_words": document_words,
        "document_boxes": document_boxes,
        "answer_text": "INV-2026-0719",
        "answer_token_indices": answer_token_indices,
        "nonzero_document_box_tokens": int(np.count_nonzero(np.asarray(boxes).sum(axis=1))),
        "bbox_min": int(inputs["bbox"].min()),
        "bbox_max": int(inputs["bbox"].max()),
        "bbox_monotonic": bool(np.all(inputs["bbox"][..., 0] <= inputs["bbox"][..., 2]) and np.all(inputs["bbox"][..., 1] <= inputs["bbox"][..., 3])),
        "token_type_unique": np.unique(inputs["token_type_ids"]).tolist(),
    }
    return inputs, semantics


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--fp32", type=Path, required=True)
    parser.add_argument("--fp16", type=Path, required=True)
    parser.add_argument("--output", type=Path, required=True)
    args = parser.parse_args()
    inputs, semantics = make_inputs()

    model = AutoModelForDocumentQuestionAnswering.from_pretrained(MODEL_ID, revision=REVISION)
    model.eval()
    pt_inputs = {name: torch.from_numpy(value.astype(np.int64)) for name, value in inputs.items()}
    with torch.inference_mode():
        pt = model(**pt_inputs)
    reference = {"start_logits": pt.start_logits.cpu().numpy(), "end_logits": pt.end_logits.cpu().numpy()}

    artifacts = []
    for precision, path in (("fp32", args.fp32), ("fp16", args.fp16)):
        session = ort.InferenceSession(str(path), providers=["CPUExecutionProvider"])
        actual_names = [item.name for item in session.get_inputs()]
        if actual_names != list(inputs):
            raise RuntimeError(f"named input mismatch: {actual_names} != {list(inputs)}")
        onnx_outputs = dict(zip(OUTPUT_NAMES, session.run(list(OUTPUT_NAMES), inputs), strict=True))
        output_rows = []
        for name in OUTPUT_NAMES:
            ref = reference[name]
            got = onnx_outputs[name]
            output_rows.append({
                "name": name,
                "shape": list(got.shape),
                "cosine": cosine(ref, got),
                "max_abs": float(np.max(np.abs(ref.astype(np.float64) - got.astype(np.float64)))),
                "mean_abs": float(np.mean(np.abs(ref.astype(np.float64) - got.astype(np.float64)))),
                "reference_argmax": int(np.argmax(ref)),
                "onnx_argmax": int(np.argmax(got)),
            })
        artifacts.append({
            "precision": precision,
            "path": str(path),
            "sha256": hashlib.sha256(path.read_bytes()).hexdigest(),
            "providers": session.get_providers(),
            "input_specs": [{"name": x.name, "shape": x.shape, "type": x.type} for x in session.get_inputs()],
            "output_specs": [{"name": x.name, "shape": x.shape, "type": x.type} for x in session.get_outputs()],
            "outputs": output_rows,
        })

    result = {
        "model_id": MODEL_ID,
        "revision": REVISION,
        "reference_class": model.__class__.__name__,
        "seedless_deterministic_input": True,
        "semantics": semantics,
        "named_inputs": {name: {"shape": list(value.shape), "dtype": str(value.dtype)} for name, value in inputs.items()},
        "named_outputs": list(OUTPUT_NAMES),
        "artifacts": artifacts,
    }
    args.output.parent.mkdir(parents=True, exist_ok=True)
    args.output.write_text(json.dumps(result, indent=2) + "\n", encoding="utf-8")
    print(json.dumps(result, indent=2))


if __name__ == "__main__":
    main()
'@ | Set-Content -Path $HARNESS -Encoding utf8

python $HARNESS --fp32 (Join-Path $OUT 'fp32/model.onnx') --fp16 (Join-Path $OUT 'fp16/model.onnx') --output (Join-Path $OUT 'l2.json')

$INPUTS=Join-Path $OUT 'semantic_layoutlm_inputs.npz'
python -c "import importlib.util,numpy as np; s=importlib.util.spec_from_file_location('h',r'$HARNESS'); m=importlib.util.module_from_spec(s); s.loader.exec_module(m); x,_=m.make_inputs(); np.savez(r'$INPUTS',**x)"
winml perf -m (Join-Path $OUT 'fp32/model.onnx') --ep cpu --device cpu --precision fp32 --input-data $INPUTS --iterations 100 --warmup 10 --memory --output (Join-Path $OUT 'perf_fp32.json') --overwrite --format json --no-color
winml perf -m (Join-Path $OUT 'fp16/model.onnx') --ep cpu --device cpu --precision fp16 --input-data $INPUTS --iterations 100 --warmup 10 --memory --output (Join-Path $OUT 'perf_fp16.json') --overwrite --format json --no-color
winml perf -m (Join-Path $OUT 'fp32/model.onnx') --ep cpu --device cpu --precision fp32 --iterations 100 --warmup 10 --memory --output (Join-Path $OUT 'perf_default_fp32.json') --overwrite --format json --no-color
winml perf -m (Join-Path $OUT 'fp16/model.onnx') --ep cpu --device cpu --precision fp16 --iterations 100 --warmup 10 --memory --output (Join-Path $OUT 'perf_default_fp16.json') --overwrite --format json --no-color

python -m pytest tests/unit/commands/test_perf_cli.py -k 'honors_exclusive_integer_value_range or ranges_and_unspecified_defaults_preserve_existing_generation' -q
python -m pytest tests/unit/loader tests/unit/export/test_layoutlm_onnx_config.py tests/unit/commands/test_perf_cli.py -q

$RULES_ZIP=Join-Path $OUT 'rules-v0.2.0.zip'
Invoke-WebRequest 'https://github.com/microsoft/winml-cli/releases/download/v0.2.0/rules-v0.2.0.zip' -OutFile $RULES_ZIP
if((Get-FileHash $RULES_ZIP -Algorithm SHA256).Hash.ToLowerInvariant() -ne '6dcabbe7f5493fbc2bd23de195ab6e35b3578d510f18c2e220bc5538a1f232cc'){throw 'rules checksum mismatch'}
$RULES=Join-Path $OUT 'rules-v0.2.0'; Expand-Archive $RULES_ZIP $RULES -Force; $env:WINMLCLI_RULES_DIR=$RULES
winml analyze --model (Join-Path $OUT 'fp32/model.onnx') --ep all --device all --htp-metadata (Join-Path $OUT 'fp32/export_htp_metadata.json') --output (Join-Path $OUT 'analyze_fp32.json') --overwrite --format json --no-color
winml analyze --model (Join-Path $OUT 'fp16/model.onnx') --ep all --device all --htp-metadata (Join-Path $OUT 'fp16/export_htp_metadata.json') --output (Join-Path $OUT 'analyze_fp16.json') --overwrite --format json --no-color
Remove-Item Env:WINMLCLI_RULES_DIR

winml eval --schema --task document-question-answering --no-color
winml eval -m (Join-Path $OUT 'fp32/model.onnx') --model-id $MODEL --task document-question-answering --ep cpu --device cpu --precision fp32 --output (Join-Path $OUT 'eval_fp32.json') --overwrite --no-color
winml eval -m (Join-Path $OUT 'fp16/model.onnx') --model-id $MODEL --task document-question-answering --ep cpu --device cpu --precision fp16 --output (Join-Path $OUT 'eval_fp16.json') --overwrite --no-color

$BASELINE=Join-Path (Split-Path (Get-Location) -Parent) 'winml-cli-layoutlm-baseline'
git worktree add --detach $BASELINE $BASELINE_COMMIT
Push-Location $BASELINE
uv run winml config -m $MODEL --task document-question-answering --ep cpu --device cpu --precision fp32 --no-compile -o (Join-Path $OUT 'baseline_config_fp32.json') --overwrite --no-color
uv run winml build -m $MODEL -o (Join-Path $OUT 'baseline_build') --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild --no-color
Pop-Location

[ssss141414]

Independent reviewer verdict: REQUEST_CHANGES

The Lane B implementation and independently reproduced model evidence pass, but the required learner-owned Lane A knowledge shipment has not landed. This is a fail-closed workflow blocker.

Blocking item

1. Learner / orchestrator — persist and push the claimed LayoutLM findings in Lane A.
   • The PR Outcome says layoutlm-002 through layoutlm-005 were captured.
   • Independent inspection of ModelKitArtifacts found copilot-skills/dev_skill/adding-model-support/model_knowledge/layoutlm.json still modified in the working tree. Its latest committed history does not contain this run's findings, and no pushed Lane A commit/PR is cited.
   • Required fix: learner commits and pushes the existing model-specific findings in a ModelKitArtifacts Lane A commit/PR; explainer cites that durable commit/PR in the model PR evidence/handoff; orchestrator then requests a fresh independent review.
   • Keep the winml-cli Lane B diff unchanged; do not copy skill or knowledge files into this PR.

Independently verified gates

• Freshness/state: PR head e65450dcdbb57774f373ad93d806905a79fce33b; PR base and current origin/main both e7509b1e908c74beff0a5655b8f8d7de69c5afae; OPEN, DRAFT, MERGEABLE, with model-scale-by-skill.
• CI: all 9 reported checks completed successfully: CodeQL analysis/result, lint, analyze/models/optim/commands/remaining tests, and CLA.
• Discussion inventory immediately before verdict: 0 issue comments, 0 reviews, 0 inline comments, 0 GraphQL review threads, 0 open threads.
• Scope/design: generalized L1 architecture/task/class registration, LayoutLM document-QA ONNX config, and generic perf range adapter are metadata-driven; no checkpoint-ID branch; non-QA LayoutLM resolution remains unchanged.
• Tests/lint: 87 affected tests passed. PR and current main have the same six repository-wide Ruff diagnostics, so this PR introduces no Ruff regression.
• Fresh builds: fp32 and fp16 config/build both exited 0 and emitted Build complete; generated configs are semantically identical to the checked-in recipes.
• Artifacts: both pass ONNX checker with IR 8, opset 17, four int32 inputs (input_ids, bbox, attention_mask, token_type_ids), and two outputs (start_logits, end_logits). fp32 has 208 FLOAT initializers and a 508,803,072-byte final external-data file; fp16 has 208 FLOAT16 initializers, zero FLOAT initializers, and a 254,401,536-byte final external-data file.
• Fresh runtime: all four independent 10-warmup/100-iteration runs exited 0. Default p50: fp32 219.291 ms, fp16 262.563 ms. Semantic p50: fp32 253.669 ms, fp16 260.496 ms. Both precision and named-input contracts were preserved.
• Fresh L2 parity: fp32 start/end cosine 0.9999999999999564 / 0.9999999999999577, max abs 1.9073486328125e-05; fp16 cosine 0.9999999880818508 / 0.999999960350131, max abs 0.009905815124511719 / 0.010828971862792969. Start/end argmaxes remain 16/22.
• Exact L3 blocker: winml eval independently rejects document-question-answering; plain question-answering is not a semantic substitute. Issue Add end-to-end document-question-answering inference and eval contract #1132 is OPEN and exactly tracks the missing OCR/inference/eval contract.
• Public evidence/body: required hierarchy, architecture tree, compact static-analysis summary, durable baseline/config/L2/rules reproduction, and no inaccessible local evidence references were verified.

No producer code or recipe change is requested. Models 3–8 were not inspected or started.

[ssss141414]

Resolved the REQUEST_CHANGES blocker exactly as requested:

• The learner-owned LayoutLM findings are committed and pushed in ModelKitArtifacts Lane A commit 7e939ae638193e0f83f717d7bee4f9b5b4311bf1 on branch shzhen/model-breakdown-skill, tracked by ModelKitArtifacts PR #159.
• Remote verification of that exact commit confirms layoutlm-002, layoutlm-003, layoutlm-004, and layoutlm-005 are present in the committed LayoutLM knowledge file.
• PR feat(layoutlm): add document question answering support #1133's Outcome / Handoff and findings entry now cite the durable Lane A commit and PR, with no local paths exposed.
• The winml-cli Lane B head and diff remain unchanged. No code, recipe, or Lane A edits were made in this resolution; no tests or self-review were run.
• PR feat(layoutlm): add document question answering support #1133 remains OPEN and DRAFT with model-scale-by-skill preserved.

Requesting a fresh independent review.

[ssss141414]

Independent reviewer verdict: APPROVE

The prior reviewer blocker is resolved. The Lane B head remains unchanged, all applicable gates pass, and this PR must remain DRAFT.

REQUEST_CHANGES resolution

• Remote ModelKitArtifacts commit 7e939ae638193e0f83f717d7bee4f9b5b4311bf1 resolves through the GitHub API. It modifies the LayoutLM knowledge file and is the current head of OPEN ModelKitArtifacts PR #159; the exact commit appears in that PR's remote commit list.
• Reading the LayoutLM blob at that exact SHA independently confirms layoutlm-002, layoutlm-003, layoutlm-004, and layoutlm-005 are committed.
• PR feat(layoutlm): add document question answering support #1133's Outcome/Handoff and findings entry cite that exact commit and PR P1-INFRA-003: CI/CD Pipeline — ADO + GitHub Actions + Self-hosted Runners #159. The public body retains reproducible commands and public evidence rather than inaccessible reviewer scratch paths.

Fresh state and discussion inventory

• PR head and checked-out head: e65450dcdbb57774f373ad93d806905a79fce33b (unchanged since the prior review).
• PR base, current origin/main, and merge base: e7509b1e908c74beff0a5655b8f8d7de69c5afae; one commit ahead, zero behind, clean working tree.
• OPEN, DRAFT, label model-scale-by-skill present. Approval does not authorize promotion from draft.
• Current inventory before verdict: 2 conversation comments (the prior REQUEST_CHANGES and its remediation handoff), 0 submitted reviews, 0 inline comments, 0 GraphQL review threads, 0 open threads. The sole prior blocker is disposed by the remote verification above.
• All 9 current GitHub checks are completed successfully: Analyze (Python), lint, five test partitions, CodeQL, and CLA.

Diff and engineering review

• Full diff re-read: 9 files, 573 insertions and 7 deletions; two CPU recipes, generalized LayoutLM/task-resolution/perf support, registration, and focused tests. Production recipe README is unchanged; git diff --check passes.
• The repair is architecture/task and metadata driven: no checkpoint-ID branch; LayoutLMForQuestionAnswering resolves to exact document QA while token/sequence-classification heads retain existing resolution. The ONNX config follows existing registration patterns and declares the four named int32 inputs plus start/end logits. The generic perf range adapter preserves high-exclusive recipe ranges and is covered for singleton/arbitrary integer and float ranges.
• Focused post-remediation rerun: 87 passed. uv run ruff check reports the same six repository-wide diagnostics as current main; a same-executable comparison on clean PR/main worktrees was byte-identical, and none references a changed file.

Independent model evidence rechecked

• Fresh fp32/fp16 config and recipe builds both completed; generated configs are semantically identical to the checked-in recipes.
• Both artifacts pass ONNX structural validation (IR 8, opset 17, four correctly named int32 inputs, start/end outputs). fp16 uses FLOAT16 initializers and approximately half the external-data size of fp32.
• All four 10-warmup/100-iteration CPU perf runs (default and semantic inputs, fp32 and fp16) completed with memory evidence.
• L2 remains strong: fp32 start/end cosine 0.9999999999999564 / 0.9999999999999577, max abs 1.9073486328125e-05; fp16 cosine 0.9999999880818508 / 0.999999960350131, max abs 0.009905815124511719 / 0.010828971862792969; start/end argmaxes remain 16/22.
• Exact L3 remains correctly CLI-BLOCKED: document-question-answering is absent from winml eval; substituting plain question-answering would erase the OCR/layout contract. Issue Add end-to-end document-question-answering inference and eval contract #1132 remains the exact tracked gap.
• Reachable/deferred split: CPU fp32 and fp16 are freshly verified through L2; no required tuple is deferred and coverage is full.

No code, body, recipe, or draft-state mutation was made during this review. Models 3–8 were not inspected or started.