Add ZhengPeng7/BiRefNet image segmentation support

[ssss141414]

Summary

This adds WinML support for the pinned ZhengPeng7/BiRefNet image-segmentation checkpoint, including metadata-derived trusted-remote model resolution, a BiRefNet ONNX export contract, export-only deformable-convolution lowering, and CPU fp32/fp16 recipes. The shipped implementation scope is Effort L1 with Outcome L1. The committed Goal L2 is fully reached for both required CPU tuples through build/structure, runtime performance, and independent pinned-PyTorch numerical parity; L3 dataset-metric evaluation is explicitly DATA-BLOCKED and is not claimed.

Model metadata

What the model does

BiRefNet is a high-resolution binary foreground segmentation model. A user supplies an RGB image, normally resized and ImageNet-normalized to 1024 x 1024, and the model returns a one-channel foreground logit map; applying sigmoid yields a mask used for dichotomous segmentation and background removal.

• Evidence: the model card, handler.py, and birefnet.py at checkpoint revision e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4 document the official DIS/background-removal purpose, ImageNet normalization, 1024 x 1024 default, and full-resolution foreground-logit output.
• Confidence: verified.

Primary user stories

• Obtain a high-resolution foreground mask from a photograph for background removal or foreground extraction with alpha transparency. Evidence: pinned model-card usage and handler.py alpha composition. Confidence: verified.
• Obtain a dichotomous segmentation mask for hard-to-separate foregrounds and DIS evaluation. Evidence: the pinned model card identifies DIS-TR training and DIS-TEs/DIS-VD validation. Confidence: verified.
• Obtain a binary object mask for salient-object or camouflaged-object detection. Evidence: the pinned model card states use on HRSOD and COD in addition to DIS. Confidence: verified.

Supported tasks

• image-segmentation — checkpoint and Transformers surfaces. Evidence: Hub pipeline_tag=image-segmentation, the pinned custom_pipelines.image-segmentation and auto_map.AutoModelForImageSegmentation metadata, and the pinned model-card loading example with trusted remote code. Confidence: verified.
• background-removal — checkpoint surface. Evidence: Hub background-removal/mask-generation tags and the pinned handler's alpha-mask use. Confidence: verified.
• dichotomous-image-segmentation — checkpoint surface. Evidence: the official pinned model card identifies the checkpoint as standard dichotomous image segmentation trained on DIS-TR. Confidence: verified.

Model architecture

BiRefNet
├── RGB input (normally 1 x 3 x 1024 x 1024, ImageNet normalized)
├── Bilateral multi-scale encoder
│   ├── Swin-v1-large at original scale (patch 4, window 12; stages 2/2/18/2)
│   ├── Shared Swin-v1-large at half scale
│   └── Concatenate four paired feature scales (384/768/1536/3072 channels)
├── Multi-scale context fusion into deepest feature
├── Squeeze module: BasicDecBlk x 1
└── Bilateral-reference decoder (4 coarse-to-fine stages)
    ├── Direct image-patch input branches at five scales
    ├── ASPP-deformable decoder blocks + lateral skip fusion
    ├── Gradient-aware attention at three coarse scales
    └── 1-channel full-resolution foreground logits (sigmoid downstream)

• Source/confidence: pinned birefnet.py configuration and BiRefNet.forward_enc / forward_ori / Decoder.forward execution paths at revision e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4 (verified). Despite the remote config model type, this is a custom-code PreTrainedModel, not a SegFormer implementation.

Validation and support evidence

1. Baseline

The frozen baseline is WinML 0.2.0 at current-main commit 4daf0f19097d03e92aeb3c4f9713b7d037e0be3d.

• Recipe-free build: FAIL, exit 2 after 17.123 s. Task auto-resolution selected next-sentence-prediction, then export failed because no OnnxConfig was registered for model type SegformerForSemanticSegmentation. Without trusted remote code, it failed earlier at the Hugging Face custom-code guard. No ONNX model or node count was produced.
• Starting auto-config behavior: an image-segmentation config could only be forced through a SegFormer-shaped workaround. That workaround treated BiRefNet as SegFormer, specified pixel_values at [1, 3, 64, 64] in [0, 1], and could not repair unsupported deformable convolution.
• Perf floor: FAIL, exit 2 because the failed baseline build produced no ONNX model; therefore no baseline latency or memory metric exists.
• Eval floor: FAIL, exit 1 because image segmentation has no default dataset. The schema expects image and annotation columns, but no authoritative checkpoint-compatible dataset/default was configured.
• Optimum probe: UNREGISTERED before and after WinML; WinML added no vendor registration.

2. Goal

• Effort: L1
• Committed Goal: L2
• Outcome: L1
• Goal success definition: both required CPU fp32/fp16 tuples pass build/structure, perf, and independent pinned-PyTorch numerical parity.
• Ceiling history: charter revision 1 set L2; no ceiling change or downgrade occurred.

3. Outcome

The shipped tier is L1 and the highest reached Goal verdict is L2 PASS, with full coverage and no deferred required tuples. The shipped surface comprises:

• CPU fp32/fp16 recipes under examples/recipes/ZhengPeng7_BiRefNet/cpu/cpu/.
• Metadata-derived trusted-remote class/task resolution in src/winml/modelkit/loader/resolution.py.
• The BiRefNet ONNX contract and export-only deformable-convolution replacement in src/winml/modelkit/models/hf/birefnet.py, registered from src/winml/modelkit/models/hf/__init__.py.
• Export patch discovery fallback in src/winml/modelkit/export/htp/exporter.py.
• Focused exporter, loader-resolution, and BiRefNet ONNX-config tests.
• A model-specific birefnet optional dependency extra plus a generated lockfile and platform markers for pre-existing Windows-only runtime distributions.

The current PR head is b174e95604039e25140e6ee39416cf374cecd248. Relative to the previously tested head 64fa85bf96b4aca47c1be943a33e67c8cd8517b0, it removes only the redundant test-side import winml.modelkit.models # noqa: F401 identified by CodeQL; it does not change runtime, export, loader, dependency, lock, or recipe behavior. A fresh subprocess proved registration remains explicit and observable: birefnet was not loaded before _get_onnx_config(), was loaded afterward, and resolved BiRefNetIOConfig with the x -> logits contract. The direct used BiRefNetIOConfig import and _get_onnx_config() registration path preserve the test intent without the unused side-effect import.

Knowledge capture refreshed segformerforsemanticsegmentation-002 and retained model findings segformerforsemanticsegmentation-001 through -006. Methodology finding _meta-090 records the dependency/lock blocker as resolved; _meta-089 records the L3 DATA-BLOCKED verdict shape. Those skill-level updates remain isolated in the separate Draft PR gim-home/ModelKitArtifacts#170, not in this model PR.

L3 remains DATA-BLOCKED: no pinned owner-authoritative, checkpoint-compatible DIS dataset revision/split, binary-mask target semantics, preprocessing/resize policy, or metric contract exists. No L3 dataset metric is claimed.

4. Per-EP/device/precision results — including perf and eval data

Goal ladder

Tier	Evidence	fp32 CPU	fp16 CPU	Verdict
L0	Fresh recipe build and structural contract	233.836824 s; 7,980 nodes; opset 17; ONNX/input/output/precision/external-data checks PASS	254.9124443 s; 7,982 nodes; opset 17; ONNX/input/output/precision/external-data checks PASS	PASS
L1	CPU runtime perf using named semantic input	PASS	PASS	PASS
L2	Independent pinned-PyTorch reference comparison	PASS	PASS	PASS

Both artifacts use x float32 [1, 3, 1024, 1024] input and logits [1, 1, 1024, 1024] output. fp32 has 738 FLOAT and 191 INT64 initializers; fp16 has 738 FLOAT16 and 191 INT64 initializers.

CPU performance

EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM total Δ	VRAM local total Δ
CPUExecutionProvider / cpu	fp32	PASS	17,910.671 ms	17,849.948 ms	0.06 samples/s	+6,822.11 MB	0.0 MB
CPUExecutionProvider / cpu	fp16	PASS	19,086.671 ms	19,052.808 ms	0.05 samples/s	+6,359.3 MB	0.0 MB

Independent PyTorch parity (Goal L2)

The reference is PyTorch float32 for ZhengPeng7/BiRefNet revision e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4; both rows are finite and explicitly reject self-comparison.

EP / Device	Precision	Verdict	Raw-logit cosine	Raw max abs	Raw mean abs	Sigmoid-probability cosine	Sigmoid max abs	Sigmoid mean abs	Inference
CPUExecutionProvider / cpu	fp32	PASS	0.9999999999992928	0.0002460479736328125	1.469493054173654e-05	0.9999999998250598	6.935860640311961e-09	1.428778516955221e-10	18.337445300014224 s
CPUExecutionProvider / cpu	fp16	PASS	0.9999976252917562	0.6679306030273438	0.019459117600490572	0.9997865562680887	4.76461613367988e-06	1.5381412280325682e-07	19.136855699995067 s

Dataset evaluation

EP / Device	Precision	Verdict	Dataset / revision / subset	Task metric	Evidence
CPUExecutionProvider / cpu	fp32	DATA-BLOCKED	—	—	Exit 1: no default dataset; only unverified DIS5K mirrors were found, with no authoritative checkpoint-compatible dataset revision or binary-mask metric contract.
CPUExecutionProvider / cpu	fp16	DATA-BLOCKED	—	—	Exit 1: no default dataset; only unverified DIS5K mirrors were found, with no authoritative checkpoint-compatible dataset revision or binary-mask metric contract.

The two supplementary eval rows are blocked by data provenance, not passed, failed, or omitted because of the L2 ceiling. No L3 metric is claimed.

5. Delta

The baseline workaround-to-supported-contract JSON-pointer changes are exact:

JSON pointer	Baseline workaround	Shipped/generated contract
/export/input_tensors/0/name	pixel_values	x
/export/input_tensors/0/shape	[1, 3, 64, 64]	[1, 3, 1024, 1024]
/export/input_tensors/0/value_range	[0, 1]	[-2.1179039301310043, 2.64]
/loader/model_type	segformer	SegformerForSemanticSegmentation

The fp32 checked-in recipe is semantically identical to the standard generated fp32 config. The fp16 checked-in recipe is semantically identical to the standard generated fp16 config; relative to fp32, its only precision-intent delta is /quant, generated by --precision fp16 with float32 I/O preservation. The recipes contain exact derived I/O metadata rather than a recipe-only workaround.

The class-wide code fix is metadata/architecture-derived and contains no model-ID hardcoding: trusted remote architectures and auto_map metadata select the canonical AutoModel class/task; the ONNX contract defines x -> logits at 1024 x 1024 and applies a pure-PyTorch deformable-convolution replacement only during export; export patch discovery recovers config_class metadata when custom code overwrites model.config. This is reducibility-consistent with the charter. Required recipe-free acceptance now PASSes (exit 0, 236.7961552 s). The production recipe README remains untouched.

Dependency/lock proof:

• pyproject.toml declares einops>=0.8 and kornia>=0.8 only in the model-specific birefnet optional extra because trusted remote code imports them during loading/export.
• The lockfile was generated by uv 0.11.17, not hand-edited, and resolves einops==0.8.2, kornia==0.8.3, and kornia-rs==0.1.14 with the birefnet extra requirements recorded.
• uv lock --check passed with 275 packages; frozen all-extras/all-groups sync and imports passed. At head b174e95604039e25140e6ee39416cf374cecd248, 12 focused tests, mypy on three changed source files, scoped Ruff check and format on all seven changed Python files, and diff checks passed.
• Full-repository Ruff is not claimed as passing: independent ruff 0.15.21 runs found the same five pre-existing, unchanged findings in scripts/sam3_reference_check.py on current main 4daf0f19097d03e92aeb3c4f9713b7d037e0be3d and the PR. These findings are unrelated to this contribution and are not a PR-introduced lint regression.
• An isolated Windows AMD64 frozen birefnet install/import passed. A frozen Linux x86_64 resolution selected einops/kornia while excluding onnxruntime-genai-winml, onnxruntime-windowsml, and windowsml.
• The three pre-existing exact-pinned Windows runtime distributions now have sys_platform == 'win32' markers, proven true on Windows and false on Linux; existing Windows AMD64 and Linux x86_64 environments remain otherwise unchanged.

6. Analyze summary — component level and op level

ANALYZE-PARTIAL-SUCCESS: both fp32 and fp16 JSON artifacts contain complete metadata and classifications. The CLI returned exit 1 because it reported unsupported operators, not because output was truncated. This is static rule compatibility analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	Foreground-logit head; gradient-aware attention; direct image-patch branches; bilateral multi-scale Swin-L encoder; squeeze block; coarse-to-fine decoder	7,959 mapped; 21 unmapped; confidence partial	TensorRT GPU: Pad partial. QNN GPU: Pad, Reshape, Transpose, Div, Erf, Add, Mul partial. OpenVINO GPU: Pad partial. No unsupported types reported in these actionable groups.
fp16	Same six architecture regions	7,959 mapped; 23 unmapped; confidence partial	TensorRT GPU: Pad partial. QNN NPU: Pad, Reshape, Transpose, Div, Erf, Add, Mul, Gather partial. QNN GPU: Pad, Reshape, Transpose, Div, Erf, Add, Mul partial. OpenVINO NPU: Pad, Reshape, Transpose, Gather partial. OpenVINO GPU: Pad partial. OpenVINO CPU: Pad partial; GlobalAveragePool unsupported.

For both artifacts, input.rgb and encoder.context_fusion remain explicit mapping gaps because input and functional outer-forward regions do not retain unique module scopes after export/optimization; they were not inferred.

Op-level summary

Artifact	Graph	Dominant operators	EP roll-up
fp32	7,980 operators / 23 unique types	Gather 1,384; Unsqueeze 1,200; Mul 1,067; Add 1,011; Sub 600; Reshape 504; Concat 477; Slice 359	TensorRT GPU: Pad partial. QNN GPU: Pad, Reshape, Transpose, Div, Erf, Add, Mul partial. OpenVINO GPU: Pad partial. No fully supported EP group.
fp16	7,982 operators / 24 unique types	Gather 1,384; Unsqueeze 1,200; Mul 1,067; Add 1,011; Sub 600; Reshape 504; Concat 477; Slice 359	TensorRT GPU: Pad partial. QNN NPU: Pad, Reshape, Transpose, Div, Erf, Add, Mul, Gather partial. QNN GPU: Pad, Reshape, Transpose, Div, Erf, Add, Mul partial. OpenVINO NPU: Pad, Reshape, Transpose, Gather partial. OpenVINO GPU: Pad partial. OpenVINO CPU: Pad partial; GlobalAveragePool unsupported. No fully supported EP group.

Rule-less/all-unknown groups: fp32 — CUDA GPU, MIGraphX GPU, and DML GPU; fp16 — CUDA GPU, MIGraphX GPU, DML GPU, CPU, and VitisAI NPU.

7. Reproduce commands

Run this from a fresh checkout with no reused project environment. The sequence creates one dedicated environment and keeps the exact same frozen --all-extras --all-groups selection for sync, the provider probe, and every WinML invocation. Do not narrow or otherwise resynchronize this environment with another extras/groups selection: overlapping onnxruntime distributions share a Python namespace, and narrowing a previously provisioned environment can remove files required by OrtHardwareDeviceType.

$OUT = 'temp/birefnet-public-repro'
New-Item -ItemType Directory -Force $OUT | Out-Null
$env:UV_PROJECT_ENVIRONMENT = "$OUT/venv"
if (Test-Path $env:UV_PROJECT_ENVIRONMENT) { throw "Use a fresh environment: $env:UV_PROJECT_ENVIRONMENT already exists" }
uv sync --frozen --all-extras --all-groups
uv run --frozen --all-extras --all-groups winml --version
uv run --frozen --all-extras --all-groups python -c "import onnxruntime as ort; print({'version': ort.__version__, 'has_OrtHardwareDeviceType': hasattr(ort, 'OrtHardwareDeviceType'), 'providers': ort.get_available_providers()})"
uv run --frozen --all-extras --all-groups winml config -m ZhengPeng7/BiRefNet -t image-segmentation --ep cpu --device cpu --precision fp32 --trust-remote-code --no-compile -o $OUT/config-fp32.json --overwrite --no-color
uv run --frozen --all-extras --all-groups winml build -m ZhengPeng7/BiRefNet -o $OUT/build-recipe-free-fp32 --ep cpu --device cpu --precision fp32 --trust-remote-code --no-analyze --no-optimize --no-quant --no-compile --rebuild --no-color
uv run --frozen --all-extras --all-groups winml build -c examples/recipes/ZhengPeng7_BiRefNet/cpu/cpu/image-segmentation_fp32_config.json -m ZhengPeng7/BiRefNet -o $OUT/fp32 --ep cpu --device cpu --precision fp32 --trust-remote-code --no-analyze --no-compile --rebuild --no-color
uv run --frozen --all-extras --all-groups winml build -c examples/recipes/ZhengPeng7_BiRefNet/cpu/cpu/image-segmentation_fp16_config.json -m ZhengPeng7/BiRefNet -o $OUT/fp16 --ep cpu --device cpu --precision fp16 --trust-remote-code --no-analyze --no-compile --rebuild --no-color
uv run --frozen --all-extras --all-groups winml perf -m $OUT/fp32/model.onnx --ep cpu --device cpu --precision fp32 --iterations 5 --warmup 1 --memory --output $OUT/perf_fp32.json --overwrite --format json --no-color
uv run --frozen --all-extras --all-groups winml perf -m $OUT/fp16/model.onnx --ep cpu --device cpu --precision fp16 --iterations 5 --warmup 1 --memory --output $OUT/perf_fp16.json --overwrite --format json --no-color
$env:WINMLCLI_RULES_DIR='<populated-public-rules-directory>'
uv run --frozen --all-extras --all-groups winml analyze --model $OUT/fp32/model.onnx --ep all --device all --output $OUT/analyze_fp32.json --overwrite --format json --no-color
uv run --frozen --all-extras --all-groups winml analyze --model $OUT/fp16/model.onnx --ep all --device all --output $OUT/analyze_fp16.json --overwrite --format json --no-color
uv run --frozen --all-extras --all-groups winml eval --model $OUT/fp32/model.onnx --model-id ZhengPeng7/BiRefNet --task image-segmentation --ep cpu --device cpu --precision fp32 --trust-remote-code --samples 1 --split validation --streaming --no-shuffle --output $OUT/eval_fp32.json --overwrite --format json --no-color
Remove-Item Env:WINMLCLI_RULES_DIR
Remove-Item Env:UV_PROJECT_ENVIRONMENT

[ssss141414]

Reviewer findings are addressed at head b174e95604039e25140e6ee39416cf374cecd248:

1. The PR body now uses a fresh dedicated UV_PROJECT_ENVIRONMENT, keeps --frozen --all-extras --all-groups on the provider probe and every WinML invocation, and contains no bare winml reproduction. It explicitly warns not to narrow/resynchronize the environment, preventing the reproduced overlapping onnxruntime namespace transition. Tester-r3 verified this portable selection with winml --version, the OrtHardwareDeviceType/provider probe, config, and recipe-free build.
2. The CodeQL unused test import was removed in b174e95604039e25140e6ee39416cf374cecd248. Tester-r3 independently proved registration still transitions from unloaded to loaded and resolves BiRefNetIOConfig (x -> logits); 12 focused tests and scoped changed-file quality checks passed. A fix/evidence reply has been posted on the CodeQL thread and the thread has been resolved through GraphQL.
3. Quality prose is now scoped: changed-file Ruff check/format passed. Full-repository Ruff is not claimed as passing; independent Ruff 0.15.21 evidence found the same five pre-existing unchanged findings in scripts/sam3_reference_check.py on current main and the PR, so there is no PR-introduced lint regression.

All existing build, perf, parity/eval, and component/op analyze evidence remains unchanged. The PR remains Draft with model-scale-by-skill. Please re-review this fresh head.