feat(loader): support Qualcomm Segment Anything Model 2

[ssss141414]

Summary

This L2 contribution adds class-wide support for release-asset ONNX composites and uses it to build and run qualcomm/Segment-Anything-Model-2 as a prompt-bearing image encoder plus embedding-only mask decoder. It ships four exact CPU recipes covering fp32/fp16 encoder and decoder artifacts, with no model-ID hardcoding. Fresh tester-r2 evidence at 448b3508740f1e3b7fc4bb0c004af6020cad9a8b reached L3 PASS with full planned CPU fp32/fp16 coverage. Static analysis remains ANALYZE-PARTIAL-SUCCESS: its prior complete results are reused only because every analyzed model payload has the same SHA-256 at the repaired head.

Model metadata

What the model does

Qualcomm's SAM 2.1 tiny release is a promptable image-segmentation pipeline: it accepts a 1024 x 1024 RGB image plus positive/negative point prompts, computes reusable image and prompt embeddings, and returns a low-resolution binary-mask logit map with an estimated mask quality score. Confidence: verified.

Evidence: the pinned model card identifies SAM 2 and sam2.1_hiera_t; Qualcomm AI Hub Models v0.58.0 defines SAM2Encoder.forward(image, norm_coords, labels) and SAM2Decoder.forward(image_embeddings, high_res_features1, high_res_features2, sparse_embedding); the exact published float graphs expose that encoder/decoder boundary and return masks as FLOAT[1,1,256,256] plus scores as FLOAT[1,1].

Primary user stories

• A user supplies an image and foreground/background point prompts to isolate the selected object for interactive segmentation. The pinned model card describes promptable zero-shot segmentation, and the pinned publisher source defines labels 1=positive, 0=negative, and -1=no prompt. Confidence: verified.
• An application runs the encoder once and reuses its image features and sparse prompt embeddings so the lightweight decoder can produce an updated mask. The exact release is split into encoder.onnx and decoder.onnx with matching image_embeddings, high_res_features1, high_res_features2, and sparse_embedding boundaries. Confidence: mapped.

Supported tasks

• image-segmentation — checkpoint surface. The pinned Hub repository declares pipeline_tag=image-segmentation, and Qualcomm metadata classifies the use case as semantic segmentation. Confidence: verified.
• mask-generation — Transformers/WinML surfaces. The equivalent upstream facebook/sam2.1-hiera-tiny family is a Transformers mask-generation model; WinML already registers sam2/sam2_video mask generation, but frozen main cannot resolve this config-less Qualcomm release as that family. Confidence: mapped.

Model architecture

SAM2 (WorkbenchModelCollection; sam2.1_hiera_t)
|-- SAM2Encoder -> encoder.onnx (33.5M parameters; published metadata)
|   |-- RGB normalization (image FLOAT[1,3,1024,1024])
|   |-- Hiera image encoder + multi-scale feature preparation
|   |-- Point-prompt encoder (coordinates + labels)
|   `-- Embedding outputs at 256x256, 128x128, and 64x64
|-- SAM2Decoder -> decoder.onnx (6.22M parameters; published metadata)
|   |-- Dense no-mask embedding + sparse prompt embedding
|   |-- Two-way transformer mask decoder
|   `-- Mask head + IoU score head -> masks[1,1,256,256], scores[1,1]
`-- Runtime postprocessing: resize mask logits to source-image resolution

• Source/confidence: pinned Qualcomm AI Hub Models v0.58.0 SAM2/shared SAM2 source at 3b3d83fb21782fac65b75464e4f9e118be835534, exact release graph I/O, ONNX checker, and CPU Runtime execution. The exported-component mapping is mapped, not cryptographically source-to-graph verified; runtime pre/postprocessing boundaries are verified.

Validation and support evidence

Baseline

• Frozen baseline: microsoft/winml-cli main 4daf0f19097d03e92aeb3c4f9713b7d037e0be3d, winml, version 0.2.0.
• Bare Hub-ID build: FAIL (exit 2). Frozen main rejected the repository before build because it has no config.json/model_type: Unrecognized model ... should have a model_type key in config.json.
• Auto-config: unavailable. Config generation failed before producing portable component recipes. Optimum probe: UNREGISTERED; neither Optimum nor WinML normalized the exact config-less repository to a registered architecture.
• Supplementary exact-asset controls showed that manually extracting each publisher graph and its external-data sidecar let frozen main build the encoder and decoder independently. This is not out-of-box checkpoint support and does not change the baseline build verdict.
• Supplementary fp32 component baseline on CPU: encoder mean/p50 1837.981/1836.950 ms, 0.54 samples/s, 1334.95 MB RSS total delta; decoder mean/p50 72.833/72.959 ms, 13.73 samples/s, 132.63 MB RSS total delta.
• Baseline eval: FAIL (exit 1 before dataset iteration). The frozen evaluator rejected encoder inputs image, unnorm_coords, and labels; it expected only an image input or pixel_values. No baseline task metric is claimed.
• Baseline Goal floor was L0 through the manually extracted exact-asset control, while the bare repository remained unsupported.

Goal

• Effort: L2 — generalized source resolution, external-data-aware acquisition, composite-role handling, and mask-generation evaluator support.
• Committed Goal ceiling: L3.
• Outcome target: L2.
• Success definition: CPU fp32/fp16 composite build, perf, exact-publisher numeric parity, and pinned task metric.
• The charter was not re-issued or downgraded.

Outcome

• Shipped tier: L2.
• Highest Goal verdict: L3 PASS.
• Coverage: full for the frozen CPU fp32/fp16 plan; deferred tuples: none.
• Shipped four exact recipes under examples/recipes/qualcomm_Segment-Anything-Model-2/cpu/cpu/, plus generalized release-asset/composite loader, build/config, evaluation, task, SAM, and pattern-analysis changes with tests.
• Producer-r2 repairs at the tested head add schema-v2 cache provenance and revalidation, recursive ONNX external-data discovery, shape-safe encoder/decoder pairing, offline-safe optional discovery, and both CodeQL cleanups.
• Learner-r2 refreshed sam-007 through sam-010; sam-011 required no claim change because all recipe hashes remain identical. _meta-091 and _meta-087 were refreshed separately in Lane A Draft PR gim-home/ModelKitArtifacts#170; no skill files are included here.

Per-EP/device/precision results — including perf and eval data

Goal ladder

Tier	CPU fp32	CPU fp16	Evidence
L0 build/structure	PASS	PASS	Fresh recipe-free and candidate builds; two components per tuple; ONNX checker, I/O, precision, and schema-v2 provenance passed after real cache tamper/reacquisition.
L1 semantic composite runtime	PASS	PASS	Fresh end-to-end semantic-route measurements below.
L2 publisher parity	PASS	PASS	Compared with the SHA-pinned Qualcomm v0.58.0 published float ONNX pair, not another WinML build.
L3 pinned task metric	PASS	PASS	Deterministic first two dataset rows, no shuffle, zero skipped samples.

L1 semantic composite runtime — CPUExecutionProvider

Two warmups and ten measured iterations. The route sends image, unnorm_coords, and labels into the encoder, then routes the four named embeddings into the decoder.

Precision	Mean	p50	Throughput	RSS total delta	Verdict
fp32	1748.85515 ms	1766.34005 ms	0.5718026447187464 samples/s	662.16796875 MB	PASS
fp16	2075.66157 ms	2070.39425 ms	0.4817741073271401 samples/s	920.21484375 MB	PASS

On this CPU, fp16 was 18.6869% slower by mean, 17.2138% slower by p50, 15.7447% lower in throughput, and used 258.046875 MB (38.9700%) more RSS delta. This is a performance/resource result, not an accuracy failure.

L2 publisher ONNX parity

Reference: SHA-pinned Qualcomm AI Hub Models v0.58.0 published float pair, executed with CPUExecutionProvider.

Precision	Masks cosine	Masks max abs	Masks mean abs	Scores cosine	Scores max abs	Verdict
fp32	1.0000000000000002	0	0	1.0	0	PASS — exact
fp16	0.9999993592392288	0.08708381652832031	0.009807090507536032	1.0	0.000209808349609375	PASS

All compared tensors were finite and shape-identical.

L3 pinned two-sample task metric

Dataset provenance is fixed to mattmdjaga/human_parsing_dataset revision db120bb5c18c146a8fbd2160f7575a288269fe7d, train split, first 2 rows, --no-shuffle, no subset, seed 42, and zero skipped samples.

EP / device	Precision	Samples	mIoU	Dice	Verdict
CPUExecutionProvider / cpu	fp32	2/2	0.3036502038704683	0.459494362253176	PASS
CPUExecutionProvider / cpu	fp16	2/2	0.30301711039773194	0.4586453067822188	PASS

Schema-v2 provenance, cache integrity, and security

• Pinned Hub revision: b79dd7131b5409ff0c74d1c2793656305403476c; release manifest SHA-256: 8d05f332a0b78994adbbf60589b70fbfc2d33688d14a9f80c34bab79c9205641; archive SHA-256: 6c47f7e4eacfad4212d512c434277dfab60d980fea467b4b13d4110baa7f25cf.
• Provenance schema v2 records and revalidates the exact archive hash, two-graph inventory and hashes, recursively discovered sidecar inventory, and each sidecar hash on cache reuse. Invalid extracted content is removed and atomically reconstructed from a verified archive; an invalid archive is removed and reacquired.
• Publisher graph hashes: encoder d0e5eee1603ac256babf93058997b5a90354470595349f6a5433a112b3caf157; decoder bcf847720ec944a87704e3965556602d137f1523deb78b5d2ec7e2c29759fea6. Sidecar hashes: encoder 97fdcb37bdb1723aa1187356fc152d1c864666ebda4d3e6d7abf1895b635785f; decoder eb0ddcfbd041fce511ef5ee57b58fce7858299d2cd56214b817d201cfb852818.
• Both real isolated fp32/fp16 cache archives were deliberately corrupted and reacquired to the pinned archive hash before candidate builds. Each resulting cache contained exactly two recorded graphs and two recorded sidecars, all with matching hashes.
• Eleven independent adversarial checks passed: connected static and repeated-symbol shape conflicts; nested tensor-attribute sidecar discovery; missing/escaping sidecars; extracted graph, sidecar, and inventory tampering; archive tampering/reacquisition; and both optional offline-discovery fallback paths.
• Shared-port pairing now requires matching dtype/rank and compatible static/symbolic dimensions across every same-named connection, with global repeated-symbol unification.
• A real offline regression also passed: cached microsoft/resnet-50 config exited 0 under HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, producing SHA-256 2b7f28c35ebd8a88467f3e973d7ca2a1cc178842cb9f6693587b6754c26bb339.

Artifact and recipe hashes

All eight fresh model payloads are byte-identical to the prior tester artifacts; acquisition/provenance nevertheless reran under schema v2, and runtime/perf/parity/eval were refreshed.

Precision	Role	Model SHA-256	Recipe SHA-256
fp32	image-encoder	a0b0814788a62e149de019592fe2410fb26dafbeb4b404eb49990316c080de06	cc5557e9d311b395fa0eaf69bc88a3473974fd918e66f1859baadc04101a1d9d
fp32	prompt-decoder	ca482f77f063955c7a2ac40a2f27920732cfa91dc3f356aef083a095982b5f2c	8336ad34070ed8e003e926d28764ef99173a5db27a98bf809ff41eb9149d4113
fp16	image-encoder	4256c472d1934b4680cb6867c5f0871d6ac020753eb55f551d6053de88fa5942	cf929b58cae6c6ea966391490ad84c0ad0630066fd7222467d1766bf4138c2b2
fp16	prompt-decoder	f83e0da8b5153b513fc8ab5a7b7036f965faf610f75c484660cf1f66b1eeb8db	26381e648acf214f204341110506d5074764ad68420fd8a8a9f8ad126eafe2cd

Each checked-in recipe is semantically and byte-for-byte identical to its tester-passed producer candidate.

Quality and CI

• Tester-r2: 40 passed focused repaired-contract tests and 760 passed broader affected tests.
• Producer-r2 exact CI groups: commands 2386 passed, 8 skipped, 2 deselected; models/loader/export 1369 passed, 6 skipped, 1 xfailed. Producer's focused blocker regression set: 41 passed.
• Full ModelKit mypy: Success; 407 source files. Changed-file Ruff check and format, uv lock --check, and git diff --check passed.
• Repository-wide Ruff still exits 1 only for five byte-unchanged findings in scripts/sam3_reference_check.py; this PR does not claim a clean repository-wide Ruff run.
• Hosted checks at head 448b3508740f1e3b7fc4bb0c004af6020cad9a8b: CodeQL, Analyze (Python), lint, test (commands), test (analyze), test (models), test (optim), test (remaining), and license/cla all SUCCESS. Merge state remains blocked only by required review; this PR remains Draft.

Delta

Recipe delta

Frozen-main config generation produced no recipe, so the comparison is honestly NOT-COMPARABLE, not an invented old/new diff.

• fp32 encoder: /export = null, /optim = {}, /quant = null, /compile = null, /loader/task = image-feature-extraction, /loader/component_name = image-encoder.
• fp32 decoder: same pass-through fp32 settings, /loader/task = mask-generation, /loader/component_name = prompt-decoder.
• fp16 encoder/decoder preserve the same role contracts and set /quant/mode = fp16 with /quant/fp16_keep_io_types = true.

All four promoted recipes passed portable rebuilds. examples/recipes/README.md remains untouched.

Generalized code delta

Area	Class-wide behavior change
Release acquisition	Resolves a pinned release manifest, safely acquires/extracts HTTPS ZIP assets, recursively discovers ONNX external data, and persists schema-v2 archive/graph/sidecar provenance. Every cache reuse revalidates exact inventories and hashes; invalid extracted content is rebuilt atomically and an invalid archive is reacquired.
Composite selection	Selects a single promptable encoder/decoder pair only when every same-named shared port has compatible dtype, rank, static dimensions, and globally unified symbolic constraints; malformed or ambiguous published contracts fail closed.
Resolution/config	Explicit (model_type, task) preflight avoids unnecessary checkpoint/Hub probing. Optional release discovery preserves the established path on offline/connectivity failures while malformed published metadata still fails closed.
Build/eval	Supports config-less release repositories, isolated component roles, consistent precision, prompt routing to the declaring graph, embedding-only decoders, rank-4 masks, and metadata-derived preprocessing.
CodeQL cleanup	Removes the unused release-assets logger and uses one idiomatic from onnx import ... style in the release-asset tests.

The behavior is metadata/graph-contract derived, contains no qualcomm/Segment-Anything-Model-2 source hardcoding, is reducible consistently with the L2 charter, and passed recipe-free class acceptance for fp32/fp16.

Analyze summary — component level and op level

ANALYZE-PARTIAL-SUCCESS (hash-reused): the complete prior static results are preserved because the analyzer consumes model.onnx bytes and the repaired head reproduces the exact same set of four analyzed model SHA-256 values: 4256c472..., a0b08147..., ca482f77..., and f83e0da8.... Schema-v2 provenance JSON changed but is not an analyzer input. The prior four commands each exited 1 while retaining complete metadata and all 11 requested target classifications; this is static rule analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable static EP findings
fp32 image encoder	complete exported image + point-prompt encoder	526 mapped, 0 partial, 0 unmapped; mapped confidence	Pad partial on NvTensorRT GPU, QNN NPU/GPU, and OpenVINO NPU/GPU/CPU
fp32 prompt decoder	complete exported prompt-conditioned mask decoder	842 mapped, 0 partial, 0 unmapped; mapped confidence	no partial/unsupported type
fp16 image encoder	complete exported image + point-prompt encoder	534 mapped, 0 partial, 0 unmapped; mapped confidence	Pad partial on NvTensorRT GPU, QNN NPU/GPU, and OpenVINO NPU/GPU/CPU
fp16 prompt decoder	complete exported prompt-conditioned mask decoder	848 mapped, 0 partial, 0 unmapped; mapped confidence	no partial/unsupported type

The frozen overview maps each complete exported graph by tensor boundary but does not provide exhaustive node-name partitions for nested Hiera backbone, prompt encoder, two-way transformer, and output heads. That mapping gap remains explicit rather than guessed.

Op-level summary

Artifact	Graph	Dominant operators	Static target roll-up
fp32 image encoder	526 ops / 22 types	Transpose 178; Reshape 116; Conv 79; Add 33; MatMul 28; LayerNormalization 24; Mul 15; Softmax 12	Six rule-backed targets: Pad partial; Resize/Mul unknown. Five targets all-unknown.
fp32 prompt decoder	842 ops / 17 types	Conv 187; Transpose 132; MatMul 120; Squeeze 116; Reshape 91; Div 56; Softmax 56; Add 25	Six rule-backed targets: Concat unknown; no partial/unsupported. Five targets all-unknown.
fp16 image encoder	534 ops / 22 types	Transpose 178; Reshape 116; Conv 79; Add 33; MatMul 28; LayerNormalization 24; Mul 15; Softmax 12	Six rule-backed targets: Pad partial; Resize/Mul unknown. Five targets all-unknown.
fp16 prompt decoder	848 ops / 18 types	Conv 187; Transpose 132; MatMul 120; Squeeze 116; Reshape 91; Div 56; Softmax 56; Add 25	Six rule-backed targets: Concat/Cast unknown; no partial/unsupported. Five targets all-unknown.

The six partially classified targets are NvTensorRT/GPU, QNN/NPU, QNN/GPU, and OpenVINO/NPU/GPU/CPU. CUDA/GPU, MIGraphX/GPU, DML/GPU, CPU/CPU, and VitisAI/NPU have no matching public rules here, so every operator type is unknown. All analyzer rows report runtime_support=false; the measured CPU runtime/eval rows above are the only runtime claims.

Reproduce commands

Run from a fresh Windows PowerShell checkout. The checkout is detached at the exact tested head so later branch changes cannot silently alter the evidence. The targeted cache-integrity test covers cache reuse, graph/sidecar/inventory tamper repair, and archive tamper/reacquisition; the isolated $env:HF_HOME then exercises fresh recipe-free and checked-in recipe acquisition.

git clone https://github.com/microsoft/winml-cli.git
Set-Location winml-cli
git fetch origin pull/1166/head:pr-1166
git checkout --detach 448b3508740f1e3b7fc4bb0c004af6020cad9a8b
$env:UV_PROJECT_ENVIRONMENT = Join-Path $PWD '.venv-sam2-repro'
uv sync --frozen --all-extras --all-groups
& "$env:UV_PROJECT_ENVIRONMENT\Scripts\Activate.ps1"

$OUT='temp/sam2-pr1166-repro'
New-Item -ItemType Directory -Force $OUT | Out-Null
$env:HF_HOME=(Resolve-Path $OUT).Path+'\hf-home'

# Portable cache validation and reacquisition regression.
python -m pytest tests/unit/loader/test_release_assets.py::test_acquisition_records_pinned_provenance_and_reuses_cache tests/unit/loader/test_release_assets.py::test_external_data_recurses_into_nested_graph_tensor_attributes tests/unit/loader/test_onnx_composite_hub.py tests/unit/commands/test_config_composite_resolution.py -q

# Fresh isolated acquisition and recipe-free acceptance.
winml build -m qualcomm/Segment-Anything-Model-2 -o $OUT/recipe-free-fp32 --ep cpu --device cpu --precision fp32 --no-analyze --no-optimize --no-compile --rebuild --no-color --no-quant
winml build -m qualcomm/Segment-Anything-Model-2 -o $OUT/recipe-free-fp16 --ep cpu --device cpu --precision fp16 --no-analyze --no-optimize --no-compile --rebuild --no-color

# Checked-in exact CPU recipes.
winml build -c examples/recipes/qualcomm_Segment-Anything-Model-2/cpu/cpu/mask-generation_fp32_image-encoder_config.json -m qualcomm/Segment-Anything-Model-2 -o $OUT/candidate-fp32-image-encoder --ep cpu --device cpu --precision fp32 --no-analyze --no-optimize --no-compile --rebuild --no-color --no-quant
winml build -c examples/recipes/qualcomm_Segment-Anything-Model-2/cpu/cpu/mask-generation_fp32_prompt-decoder_config.json -m qualcomm/Segment-Anything-Model-2 -o $OUT/candidate-fp32-prompt-decoder --ep cpu --device cpu --precision fp32 --no-analyze --no-optimize --no-compile --rebuild --no-color --no-quant
winml build -c examples/recipes/qualcomm_Segment-Anything-Model-2/cpu/cpu/mask-generation_fp16_image-encoder_config.json -m qualcomm/Segment-Anything-Model-2 -o $OUT/candidate-fp16-image-encoder --ep cpu --device cpu --precision fp16 --no-analyze --no-optimize --no-compile --rebuild --no-color
winml build -c examples/recipes/qualcomm_Segment-Anything-Model-2/cpu/cpu/mask-generation_fp16_prompt-decoder_config.json -m qualcomm/Segment-Anything-Model-2 -o $OUT/candidate-fp16-prompt-decoder --ep cpu --device cpu --precision fp16 --no-analyze --no-optimize --no-compile --rebuild --no-color

# Pinned task metrics.
winml eval -m image-encoder=$OUT/candidate-fp32-image-encoder/model.onnx -m prompt-decoder=$OUT/candidate-fp32-prompt-decoder/model.onnx --model-id qualcomm/Segment-Anything-Model-2 --task mask-generation --dataset mattmdjaga/human_parsing_dataset --dataset-revision db120bb5c18c146a8fbd2160f7575a288269fe7d --split train --samples 2 --no-shuffle --device cpu --ep cpu --output $OUT/eval-fp32.json --overwrite --format json -vv --no-color
winml eval -m image-encoder=$OUT/candidate-fp16-image-encoder/model.onnx -m prompt-decoder=$OUT/candidate-fp16-prompt-decoder/model.onnx --model-id qualcomm/Segment-Anything-Model-2 --task mask-generation --dataset mattmdjaga/human_parsing_dataset --dataset-revision db120bb5c18c146a8fbd2160f7575a288269fe7d --split train --samples 2 --no-shuffle --device cpu --ep cpu --output $OUT/eval-fp16.json --overwrite --format json -vv --no-color

# Quality gates.
python -m pytest tests/unit/loader/test_release_assets.py tests/unit/loader/test_onnx_composite_hub.py tests/unit/commands/test_config_composite_resolution.py -q
$env:MYPYPATH=(Resolve-Path src).Path
python -m mypy -p winml.modelkit
Remove-Item Env:MYPYPATH
uv lock --check
git diff --check origin/main...HEAD

[ssss141414]

Reviewer-blocker follow-up — re-review requested

The Draft PR body and evidence are refreshed for head 448b3508740f1e3b7fc4bb0c004af6020cad9a8b. The prior blocker areas are addressed as follows:

1. Cache/provenance: schema v2 records and revalidates the archive, exact graph inventory/hashes, recursively discovered sidecar inventory, and sidecar hashes. Invalid extracted content is atomically reconstructed; an invalid archive is reacquired. Real isolated fp32/fp16 archives were corrupted and reacquired to 6c47f7e4eacfad4212d512c434277dfab60d980fea467b4b13d4110baa7f25cf.
2. Recursive external data: discovery covers initializers, sparse initializer values/indices, tensor and sparse-tensor attributes, nested graphs/subgraphs, tensor collections, and model-local function nodes. Missing and escaping nested sidecars fail closed.
3. Composite contract: every same-named shared port must match dtype/rank and compatible static/symbolic constraints, with repeated symbols unified across all ports. Independent static and repeated-symbol conflicts were rejected.
4. Resolution/offline behavior: explicit (model_type, task) preflight avoids Hub probing; optional release discovery preserves established offline/connectivity behavior while malformed or ambiguous published metadata still fails closed. A cached offline microsoft/resnet-50 config command exited 0.
5. CodeQL: the unused logger and mixed ONNX import causes were removed; both outdated threads have fix-evidence replies and are resolved. Hosted CodeQL is SUCCESS.

Fresh tester-r2 evidence remains L3 PASS for CPU fp32/fp16: semantic runtime, publisher parity, and pinned two-sample eval were rerun. Analyze is explicitly hash-reused only because all analyzed model.onnx SHA-256 values are unchanged and schema-v2 provenance is not an analyzer input. Focused repaired contracts: 40 passed; broader affected tests: 760 passed; full ModelKit mypy: 407 source files; all nine hosted checks: SUCCESS.

Please re-review this exact head. The PR intentionally remains Draft and retains model-scale-by-skill; no readiness or final verdict is asserted here.