Skip to content

Fix CTC model class resolution by passing model_type to TasksManager#1154

Open
codykk wants to merge 2 commits into
mainfrom
yongyue/fix-ctc-model-class-resolution
Open

Fix CTC model class resolution by passing model_type to TasksManager#1154
codykk wants to merge 2 commits into
mainfrom
yongyue/fix-ctc-model-class-resolution

Conversation

@codykk

@codykk codykk commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Pass model_type to Optimum's TasksManager.get_model_class_for_task in resolve_task() Stage 2, fixing model class resolution for all CTC-based ASR architectures (Wav2Vec2ForCTC, HubertForCTC, WavLMForCTC, etc.). One-line generalized fix — no per-model hardcoding. Add regression recipe for jonatasgrosman/wav2vec2-large-xlsr-53-russian (cpu/fp32, cpu/fp16).

Effort: L1-light | Goal: L1 (perf) | Outcome: L1

Model metadata

  • What it does: Wav2Vec2-Large-XLSR-53 fine-tuned on Russian Common Voice for CTC-based automatic speech recognition. Converts 16kHz raw audio waveforms to Russian text transcriptions.
  • Primary user stories: Russian speech-to-text transcription for voice applications
  • Supported tasks: automatic-speech-recognition (CTC decoding)
  • Architecture:
    Wav2Vec2ForCTC
    ├── Wav2Vec2FeatureEncoder (7 CNN layers: raw audio → features)
    ├── Wav2Vec2FeatureProjection (LayerNorm + Linear → hidden_size=1024)
    ├── Wav2Vec2Encoder (24× transformer layers, 16 heads)
    └── Linear CTC head (1024 → 39 Cyrillic vocab)

Validation and support evidence

Baseline (main): Build FAIL — Unrecognized configuration class Wav2Vec2Config for this kind of AutoModel: AutoModelForSpeechSeq2Seq

Root cause: TasksManager.get_model_class_for_task("automatic-speech-recognition") without model_type defaults to AutoModelForSpeechSeq2Seq; CTC models need AutoModelForCTC.

Goal ceiling: L1 (perf) — L2/L3 CLI-BLOCKED (automatic-speech-recognition not in winml eval supported tasks)

Outcome: L1 — generalized code fix + regression recipes

EP Device Precision Build Perf
cpu cpu fp32 ✅ PASS 125.6ms avg, 7.96 samples/sec, +1268MB RAM
cpu cpu fp16 ✅ PASS 140.5ms avg, 7.12 samples/sec, +1276MB RAM

Delta: resolution.py line 647 — added model_type=model_type_norm or None. When model_type_norm is available (from config.model_type), Optimum looks up the correct class per architecture family instead of defaulting to the first registered class. Empty string falls back to None (original behavior preserved).

Reproduce:

winml build -m jonatasgrosman/wav2vec2-large-xlsr-53-russian -o output/ --ep cpu --device cpu
winml perf -m output/model.onnx --ep cpu --device cpu

@codykk
codykk marked this pull request as ready for review July 21, 2026 03:43
@codykk
codykk requested a review from a team as a code owner July 21, 2026 03:43
@ssss141414 ssss141414 added the model-scale-by-skill Model support PR created or maintained by the adding-model-support skill label Jul 21, 2026
Comment thread src/winml/modelkit/loader/resolution.py
@codykk
codykk force-pushed the yongyue/fix-ctc-model-class-resolution branch from 9d42f4d to 0643c4d Compare July 22, 2026 06:12
Yong Yue added 2 commits July 22, 2026 15:56
When a task like automatic-speech-recognition maps to multiple model
classes (AutoModelForSpeechSeq2Seq, AutoModelForCTC), Optimum defaults
to the first one without model_type context. This caused CTC-based ASR
models (e.g. Wav2Vec2ForCTC) to fail with an incorrect class. Passing
model_type_norm enables Optimum to select the correct class per
architecture family.

Add regression recipe for jonatasgrosman/wav2vec2-large-xlsr-53-russian
(cpu/fp32 and cpu/fp16).
The USER_TASK branch called get_model_class_for_task without model_type,
causing models like Wav2Vec2 (CTC) to resolve to AutoModelForSpeechSeq2Seq
instead of AutoModelForCTC when task is explicit but model_class is not.

Add regression test for explicit-task resolution without model-class override.
@codykk
codykk force-pushed the yongyue/fix-ctc-model-class-resolution branch from 0643c4d to 8fe87bf Compare July 22, 2026 07:56
try:
resolved = TasksManager.get_model_class_for_task(normalized, framework="pt")
resolved = TasksManager.get_model_class_for_task(
normalized, framework="pt", model_type=model_type_norm or None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please pass the raw model_type to TasksManager here rather than model_type_norm. The normalization is needed for WinML's internal mapping keys, but Optimum registers some model types with underscores. For example, speech_to_text is a supported Transformers model type; converting it to speech-to-text prevents Optimum from matching AutoModelForSpeechSeq2Seq, so explicit ASR resolution raises instead of resolving the class. The Stage 2 call added by this PR has the same issue. Please keep model_type_norm for _get_custom_model_class, pass model_type to both TasksManager calls, and add an underscore-containing regression case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model-scale-by-skill Model support PR created or maintained by the adding-model-support skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants