Conversation
…OTICE The LICENSE file still carried the Apache 2.0 template placeholders (Copyright [yyyy] [name of copyright owner]), so the notice was never actually applied to this work. Fill it in, add the NOTICE file Apache 2.0 expects for the copyright line, and point the four READMEs at both.
license: name the copyright holder and add NOTICE
Qwen3ForCausalLM is the one Qwen shape colibri could not run: qwen36 and qwen38 both key off model_type and both are MoE, so a plain dense Qwen3-8B had no engine and no converter path. This adds the third member of the family rather than widening either sibling -- the two share almost nothing below the tokenizer. c/qwen3.c is the engine: GQA 32/8 over head_dim 128, q_norm/k_norm, full RoPE, SwiGLU 12288, untied lm_head. Its RMSNorm is the plain `x * rsqrt(...) * w` form, NOT Qwen3.6's `(1.0 + w)` -- the two are a silent-garbage swap, so the difference is commented where it is implemented. rope_theta is read from both the flat key and transformers 5.x's `rope_parameters` nest, and the engine refuses a container that carries neither instead of defaulting to 1e4 (a 100x error against Qwen3's 1e6, which degrades gradually enough with position to look like a bad sample). c/tools/convert_qwen3_dense.py writes the container: int4-gs64 for the matmul weights per quant.h's nibble+8 convention, f16 for embeddings, lm_head and norms. `--selftest` checks the pack/unpack round trip on its own, without a checkpoint. The family is dense, so it declares no planner geometry: planner_geometry() sizes an expert cache and refuses a zero-expert model by design, and a fabricated expert count would answer a question this model does not ask. planner_unsupported_reason says so to the caller. Validation is make qwen3-tiny-check: a generated 4-layer fixture, f16 token-exact against the torch reference, int4 container asserted to run. CI adds the same under ASan/UBSan plus the no-rope_theta refusal.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Dense Qwen3 (
model_type: qwen3/Qwen3ForCausalLM) is not covered by any existing engine —qwen36/qwen38dispatch on model_type and are both MoE. This adds the missing family end to end:c/qwen3.c— engine: GQA 32Q/8KV, head_dim 128, q_norm/k_norm, full RoPE, SwiGLU 12288, untied lm_head, plain RMSNorm (not qwen36's1.0+w).Q3_MAXTlowers ctx only.c/tools/convert_qwen3_dense.py— converter (--model/--out/--bits,--selftest). Matmul weights int4-grouped gs64 (nibble+8, low nibble = even element, scales[O, ceil(I/64)]); embeddings/lm_head/norms f16.c/tools/make_qwen3_tiny.py— 4-layer fixture + torch reference generator for the oracle.c/family_registry.py—qwen3descriptor withplanner_geometry=None(dense → no expert cache to size).docs/qwen3.md,docs/docs.json, README), CI/release registry-coverage contracts, segment-conformance fixtures + manifest.Container format
Directory of safetensors shards +
config.json+qwen3_meta.json(+ tokenizer).rope_thetais read from the flat key and the transformers-5.xrope_parametersnest; a container carrying neither is refused, not defaulted (both previously silently defaulted to 1e6 and broke multi-token attention).Evidence
Multi-token attention divergence (S>1) was bisected to an RoPE theta default + a stride-math typo; both fixed, oracle now exact at every step.