Add Qwen3.5-35B-A3B contrib model

[jimburtoft]

Description

Add Qwen3.5-35B-A3B (hybrid DeltaNet + GQA + MoE) as a contrib model. This is a novel architecture with 30 linear recurrent DeltaNet layers and 10 standard GQA attention layers, all with 256-expert sparse MoE. Implements custom NKI v2 kernels for the DeltaNet gated delta rule recurrence, padding-aware state management, sigmoid-gated shared experts, partial RoPE, and attention output gates. Achieves 100% token match against CPU reference and 5.5x higher TKG throughput than a 4x L4 GPU setup.

Model Information

Model Name: Qwen3.5-35B-A3B
Model Architecture: Hybrid decoder-only transformer (30 DeltaNet linear recurrent layers + 10 GQA attention layers + 256-expert sparse MoE on all 40 layers)
Purpose: Text generation (language backbone of a Vision-Language model)

Checklist

Please ensure your PR includes the following items. Refer to the contrib/CONTRIBUTING.md for detailed guidelines.

Required Components

• Accuracy Test (ex. test/integration/test_model.py)
  • At least one integration test that validates model accuracy
  • Uses logit validation or equivalent accuracy verification
  • Test can compile and run the model on Neuron
• README.md with the following sections:
  • Usage Example: Clear code example showing how to use the model
  • Compatibility Matrix: Table showing tested Neuron SDK versions and instance types (Trn1/Trn2/Inf2)
  • Example Checkpoints: Links to compatible model checkpoints (e.g., HuggingFace Hub)
  • Testing Instructions: Command to run the test suite for the model
• Source Code (src/)
  • Modeling code following NxD Inference patterns
  • Properly structured in the contrib folder hierarchy

Optional Components

• Unit Tests (CPU or Neuron-based)
  • Tests for individual modeling components
  • Located in test/unit/ directory

Folder Structure

Confirm your contribution follows this structure:
/contrib/models/Qwen3.5-35B-A3B/
README.md
/src
init.py
modeling_qwen35_moe.py
nki_deltanet.py
/test
init.py
/unit
init.py
/integration
init.py
test_model.py

Testing

How did you test this change?
Tested on trn2.3xlarge with Neuron SDK 2.28 (PyTorch 2.9). Model was compiled, loaded, and run through multiple test prompts. Token generation was validated against CPU reference (HuggingFace transformers) achieving 100% top-1 token match across 3 test prompts. Benchmark measurements taken for both short context (128 tokens) and long context (2048 tokens) sequences.
Test Results:

Test	Status	Result
Smoke Test	PASS	Model loads successfully
Generation ("Paris")	PASS	CTE top-1 = "Paris" (score 17.88)
Coherence	PASS	Multi-prompt coherent generation
Token Match vs CPU	PASS	3/3 = 100% token match
TTFT (seq_len=128)	1,051 ms
TKG throughput	54.3 tok/s (18.4 ms/tok)

Compatibility

Tested with:

• Neuron SDK Version(s): 2.28
• Instance Type(s): trn2.3xlarge (TP=4, LNC=2)
• PyTorch Version: 2.9
• Python Version: 3.10

Additional Information

Key implementation details:

• NKI DeltaNet kernel: Custom NKI v2 kernels (nki_deltanet.py) for the gated delta rule recurrence, processing one (batch, head) pair per kernel call with sequential token iteration over a 128x128 state matrix in SBUF.
• State carry-over: DeltaNet recurrent state and conv1d state are carried between context encoding (CTE) and token generation (TKG) via nn.Parameter buffers with input_output_aliases through a custom Qwen35DecoderModelInstance.
• Padding-aware recurrence: Right-padded inputs have their decay factor g zeroed so padding tokens preserve (rather than decay) the recurrent state.
• Sigmoid-gated shared expert: Wraps NxDI's SharedExperts with a sigmoid gate, since Qwen3.5's shared expert gating differs from NxDI's default additive behavior.
• Known SDK issues: MoE forward_blockwise produces incorrect output on trn2 SDK 2.28 (workaround: large block_size). NKI flash attention limited to head_dim<=128 (workaround: attn_kernel_enabled=False).

Related Issues

N/A

vLLM Integration

• This model/feature is intended for use with vLLM
• Documentation includes vLLM registration instructions

---

By submitting this PR, I confirm that:

• I have read and followed the contributing guidelines
• This is a community contribution and may have limited testing compared to officially-supported models
• The code follows best practices and is well-documented
• All required components listed above are included