recipe(ast-audioset): add CPU fp32 and fp16 audio-classification recipes

[codykk]

Add verified CPU (fp32 + fp16) recipes for MIT/ast-finetuned-audioset-10-10-0.4593, the Audio Spectrogram Transformer fine-tuned on AudioSet for audio event classification. Effort L0 (recipe-only); Goal ceiling L2 reached — numeric parity with PyTorch confirmed (cosine = 1.0). L3 is CLI-BLOCKED because winml eval does not yet support the audio-classification task.

Model metadata

What the model does

Audio Spectrogram Transformer (AST) fine-tuned on AudioSet-10-10 for multi-label audio event classification across 527 categories. Processes mel-spectrogram patches (1024 x 128) through a ViT-style encoder and outputs per-class logits.

• Evidence: HuggingFace model card, config.json (num_labels: 527, architectures: ["ASTForAudioClassification"])
• Confidence: verified

Primary user stories

• A developer supplies a raw audio spectrogram to obtain class probabilities for 527 AudioSet event categories for audio tagging / content analysis.
• Evidence: checkpoint task (audio-classification), model card
• Confidence: verified

Supported tasks

Task	Support surfaces	Evidence	Confidence
audio-classification	checkpoint, transformers, optimum-onnx, winml	Optimum vendor-native; winml inspect confirms default support	verified

Model architecture

ASTForAudioClassification
├── Embeddings
│   ├── Patch embedding (Conv2d 16x16, stride 10x10 → 768-d)
│   ├── CLS + distillation tokens
│   └── Positional embedding (interpolated, 1024+2 positions)
├── Encoder x 12
│   ├── Self-attention (12 heads, 768-d)
│   ├── Feed-forward (768 → 3072 → 768, GELU)
│   └── Residual + LayerNorm
└── Classification head (768 → 527 logits, mean of CLS + distill)

• Source/confidence: ASTForAudioClassification source and pinned checkpoint config (verified).

Validation and support evidence

Baseline

• origin/main: 38767add6f91c7b10b6394fae3af6f437e02effd
• winml version: 0.2.0
• Optimum probe: VENDOR-ONLY — audio-classification and feature-extraction registered by Optimum vendor for audio-spectrogram-transformer; winml adds nothing.
• Build: PASS (22.5s, 557 ONNX nodes, 100% tagged)
• Perf: PASS (367ms avg, 2.72 samples/s, RAM +540MB)
• Eval: CLI-BLOCKED (audio-classification not in winml eval supported tasks)
• Goal floor: L1 (baseline builds and perfs)
• Starting recipe: auto-config output identical to shipped recipe (no delta)

Goal

• Effort: L0 (recipe-only — Optimum vendor-native, auto-config correct, no code change)
• Goal ceiling: L2 (numeric vs PyTorch — CPU-reachable; L3 CLI-BLOCKED)
• Outcome: L0 (recipe + report)
• Success definition: L0 = build PASS; L1 = perf runs without crash; L2 = cosine similarity >= 0.999 and max abs diff < 1e-3 vs PyTorch

Outcome

• Highest reached: L2 PASS on both (cpu, cpu, fp32) and (cpu, cpu, fp16)
• Coverage: full (all target tuples passed or CLI-BLOCKED)
• Deferred tuples: none
• Shipped recipes:
  • examples/recipes/MIT_ast-finetuned-audioset-10-10-0.4593/cpu/cpu/audio-classification_fp32_config.json
  • examples/recipes/MIT_ast-finetuned-audioset-10-10-0.4593/cpu/cpu/audio-classification_fp16_config.json

Per-EP/device/precision results

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM delta	Task metric
L0 (build)	CPUExecutionProvider / cpu	fp32	PASS	—	—	—	—	—
L0 (build)	CPUExecutionProvider / cpu	fp16	PASS	—	—	—	—	—
L1 (perf)	CPUExecutionProvider / cpu	fp32	PASS	438.76 ms	433.28 ms	2.28 samples/s	+538.4 MB	—
L1 (perf)	CPUExecutionProvider / cpu	fp16	PASS	409.30 ms	411.60 ms	2.44 samples/s	+539.8 MB	—
L2 (numeric)	CPUExecutionProvider / cpu	fp32	PASS	—	—	—	—	cosine=1.00000000, max_abs=1.03e-05
L2 (numeric)	CPUExecutionProvider / cpu	fp16	PASS	—	—	—	—	cosine=1.00000000, max_abs=1.03e-05
L3 (eval)	CPUExecutionProvider / cpu	fp32	CLI-BLOCKED	—	—	—	—	audio-classification not in eval task list
L3 (eval)	CPUExecutionProvider / cpu	fp16	CLI-BLOCKED	—	—	—	—	same as above

Delta

Recipe is identical to winml config auto-config output — no field overrides. Filed for verified cpu/cpu coverage claim that the out-of-the-box auto-build does not formally establish. examples/recipes/README.md remains untouched.

Analyze summary — component level and op level

ANALYZE-UNAVAILABLE: winml analyze exited with error "No runtime rule parquet files were found" — rule parquet files are not present in this installation. This is an environment gap, not a model issue. Static rule analysis was not possible on this host.

Reproduce commands

# L0 — build fp32
winml build -m MIT/ast-finetuned-audioset-10-10-0.4593 -c examples/recipes/MIT_ast-finetuned-audioset-10-10-0.4593/cpu/cpu/audio-classification_fp32_config.json -o $OUT/fp32 --ep cpu --device cpu --rebuild

# L0 — build fp16
winml build -m MIT/ast-finetuned-audioset-10-10-0.4593 -c examples/recipes/MIT_ast-finetuned-audioset-10-10-0.4593/cpu/cpu/audio-classification_fp16_config.json -o $OUT/fp16 --ep cpu --device cpu --rebuild

# L1 — perf fp32
winml perf -m $OUT/fp32/model.onnx --ep cpu --device cpu

# L1 — perf fp16
winml perf -m $OUT/fp16/model.onnx --ep cpu --device cpu

# L2 — numeric comparison vs PyTorch (Python, seed=42)
python -c "
import numpy as np, torch, onnxruntime as ort
from transformers import ASTForAudioClassification
model = ASTForAudioClassification.from_pretrained('MIT/ast-finetuned-audioset-10-10-0.4593'); model.eval()
np.random.seed(42); x = np.random.uniform(-1, 1, (1, 1024, 128)).astype(np.float32)
pt = model(torch.from_numpy(x)).logits.numpy()
for tag, path in [('fp32','$OUT/fp32/model.onnx'), ('fp16','$OUT/fp16/model.onnx')]:
    ox = ort.InferenceSession(path, providers=['CPUExecutionProvider']).run(None, {'input_values': x})[0]
    cos = np.dot(pt.flatten(), ox.flatten()) / (np.linalg.norm(pt) * np.linalg.norm(ox))
    print(f'{tag}: cosine={cos:.8f}, max_abs={np.max(np.abs(pt - ox)):.2e}')
"