Add metile.compile(model): one call, structural detection, verified before it is kept - #33
Merged
Merged
Conversation
…efore it is kept
The integration worked but nobody could reach it. Accelerating a model meant importing from
metile.integrations.mlx_lm, choosing among eleven keyword flags, and knowing which combination
was safe for the architecture in hand. This is the Liger-shaped entry point:
import metile
print(metile.compile(model))
Two changes underneath it, both from failures this project has had rather than from taste.
Architectures are now matched by structure as well as by name. The patcher gated on a list of
module and class names, which only ever covers what someone remembered to add: Qwen3.5, Qwen3.6
and Qwen3-VL were all excluded by it, and their equivalence tests reported skips that read like
passes. A class carrying gate_proj, up_proj and down_proj is now a candidate whether or not it
has been seen, which is what makes an unlisted model work today.
Structure alone is not enough to act on, so compile verifies. A class can have the parts of a
gated MLP and still scale the product or use a different activation, presenting identically. So
it runs the model before and after, compares decode-step logits, and keeps only what reproduces
MLX. Decode steps rather than prefill, because attention only engages at query length one and a
prefill comparison reports agreement while never running the kernel. When the full set disagrees
it bisects per feature instead of reverting everything, which is what keeps three of four
features on Llama-3.2-1B rather than none.
Comparison is exact by default and the report says what that costs. Llama-3.2-1B's quantized_mlp
moves a logit by 0.035 against a magnitude near 20, 2.3e-3 relative -- a summation-order
difference where meTile measured as the more accurate side, 4.10 against MLX's 18.05 versus a
float32 reference. Declining it by default is the conservative call, and the report names the
feature, the absolute and relative size, and that 5e-3 tolerance would keep it, so the trade is
visible before it is taken rather than discovered later.
The report is falsy when nothing was replaced. That is the point of it. The dangerous outcome
here is not a crash but a silent no-op, and sixteen tests in this repository reported "skipped"
for models meTile was not touching for as long as it took someone to read the skip list.
Verified on three real checkpoints: Qwen2.5-1.5B and Qwen3.5-4B take all four features with
exactly matching logits, and Llama-3.2-1B takes three with the fourth declined and explained.
693 pass. Lint and vulture clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The integration worked but nobody could reach it. Accelerating a model meant importing from
metile.integrations.mlx_lm, choosing among eleven keyword flags, and knowing which combination was safefor the architecture in hand. This is the Liger-shaped entry point:
Structural detection is what makes unlisted models work today
The patcher gated on a list of module and class names, which only ever covers what someone remembered to
add — Qwen3.5, Qwen3.6 and Qwen3-VL were all excluded by it, and their equivalence tests reported skips
that read like passes. A class carrying
gate_proj/up_proj/down_projis now a candidate whether ornot it has been seen.
gemma2.MLP, never listed, is admitted;llama.Attentionis not.Verification is what makes that safe
Structure is a weaker claim than a name: a class can have the parts of a gated MLP and still scale the
product or use a different activation, presenting identically. So
compileruns the model before andafter, compares decode-step logits, and keeps only what reproduces MLX.
Decode steps rather than prefill, because attention only engages at query length one — a prefill
comparison reports agreement while never running the kernel. When the full set disagrees it bisects per
feature rather than reverting everything, which is what keeps three of four features on Llama-3.2-1B
instead of none:
Exact by default, and the report says what that costs
That 0.0352 is a summation-order difference where meTile measured as the more accurate side — 4.10
against MLX's 18.05 versus a float32 reference. Declining it by default is the conservative call, and the
report names the feature, the absolute and relative size, and that
tolerance=5e-3would keep it. Thetrade is visible before it's taken rather than discovered later.
The report is falsy when nothing was replaced
That's the point of it. The dangerous outcome here is not a crash but a silent no-op, and sixteen tests in
this repository reported "skipped" for models meTile wasn't touching for as long as it took someone to read
the skip list.
if not metile.compile(model)is now a real check.Verified on three real checkpoints: Qwen2.5-1.5B and Qwen3.5-4B take all four features with exactly
matching logits; Llama-3.2-1B takes three with the fourth declined and explained.
693 pass. Lint and vulture clean.
🤖 Generated with Claude Code