Skip to content

Add metile.compile(model): one call, structural detection, verified before it is kept - #33

Merged
AndreSlavescu merged 1 commit into
mainfrom
metile-compile-api
Jul 31, 2026
Merged

Add metile.compile(model): one call, structural detection, verified before it is kept#33
AndreSlavescu merged 1 commit into
mainfrom
metile-compile-api

Conversation

@AndreSlavescu

Copy link
Copy Markdown
Owner

The integration worked but nobody could reach it. Accelerating a model meant importing from
metile.integrations.mlx_lm, choosing among eleven keyword flags, and knowing which combination was safe
for the architecture in hand. This is the Liger-shaped entry point:

import metile
print(metile.compile(model))
meTile on qwen2
  accelerating: attention, rms_norm, graph_fusion, quantized_mlp
  surfaces replaced: mlp, input_layernorm, post_attention_layernorm, block
  verification: logits match MLX exactly

Structural detection is what makes unlisted models work today

The patcher gated on a list of module and class names, which only ever covers what someone remembered to
add — Qwen3.5, Qwen3.6 and Qwen3-VL were all excluded by it, and their equivalence tests reported skips
that read like passes
. A class carrying gate_proj/up_proj/down_proj is now a candidate whether or
not it has been seen. gemma2.MLP, never listed, is admitted; llama.Attention is not.

Verification is what makes that safe

Structure is a weaker claim than a name: a class can have the parts of a gated MLP and still scale the
product or use a different activation, presenting identically. So compile runs the model before and
after, compares decode-step logits, and keeps only what reproduces MLX.

Decode steps rather than prefill, because attention only engages at query length one — a prefill
comparison reports agreement while never running the kernel. When the full set disagrees it bisects per
feature
rather than reverting everything, which is what keeps three of four features on Llama-3.2-1B
instead of none:

meTile on llama
  accelerating: attention, rms_norm, graph_fusion
  verification: logits match MLX exactly
  declined quantized_mlp: changed the logits by 0.0352, 2.3e-03 relative
                          -- reduction-order scale, raise tolerance to keep it

Exact by default, and the report says what that costs

That 0.0352 is a summation-order difference where meTile measured as the more accurate side — 4.10
against MLX's 18.05 versus a float32 reference. Declining it by default is the conservative call, and the
report names the feature, the absolute and relative size, and that tolerance=5e-3 would keep it. The
trade is visible before it's taken rather than discovered later.

The report is falsy when nothing was replaced

That's the point of it. The dangerous outcome here is not a crash but a silent no-op, and sixteen tests in
this repository reported "skipped" for models meTile wasn't touching for as long as it took someone to read
the skip list. if not metile.compile(model) is now a real check.

Verified on three real checkpoints: Qwen2.5-1.5B and Qwen3.5-4B take all four features with exactly
matching logits; Llama-3.2-1B takes three with the fourth declined and explained.

693 pass. Lint and vulture clean.

🤖 Generated with Claude Code

…efore it is kept

The integration worked but nobody could reach it. Accelerating a model meant importing from
metile.integrations.mlx_lm, choosing among eleven keyword flags, and knowing which combination
was safe for the architecture in hand. This is the Liger-shaped entry point:

    import metile
    print(metile.compile(model))

Two changes underneath it, both from failures this project has had rather than from taste.

Architectures are now matched by structure as well as by name. The patcher gated on a list of
module and class names, which only ever covers what someone remembered to add: Qwen3.5, Qwen3.6
and Qwen3-VL were all excluded by it, and their equivalence tests reported skips that read like
passes. A class carrying gate_proj, up_proj and down_proj is now a candidate whether or not it
has been seen, which is what makes an unlisted model work today.

Structure alone is not enough to act on, so compile verifies. A class can have the parts of a
gated MLP and still scale the product or use a different activation, presenting identically. So
it runs the model before and after, compares decode-step logits, and keeps only what reproduces
MLX. Decode steps rather than prefill, because attention only engages at query length one and a
prefill comparison reports agreement while never running the kernel. When the full set disagrees
it bisects per feature instead of reverting everything, which is what keeps three of four
features on Llama-3.2-1B rather than none.

Comparison is exact by default and the report says what that costs. Llama-3.2-1B's quantized_mlp
moves a logit by 0.035 against a magnitude near 20, 2.3e-3 relative -- a summation-order
difference where meTile measured as the more accurate side, 4.10 against MLX's 18.05 versus a
float32 reference. Declining it by default is the conservative call, and the report names the
feature, the absolute and relative size, and that 5e-3 tolerance would keep it, so the trade is
visible before it is taken rather than discovered later.

The report is falsy when nothing was replaced. That is the point of it. The dangerous outcome
here is not a crash but a silent no-op, and sixteen tests in this repository reported "skipped"
for models meTile was not touching for as long as it took someone to read the skip list.

Verified on three real checkpoints: Qwen2.5-1.5B and Qwen3.5-4B take all four features with
exactly matching logits, and Llama-3.2-1B takes three with the fourth declined and explained.

693 pass. Lint and vulture clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@AndreSlavescu
AndreSlavescu merged commit 2815e70 into main Jul 31, 2026
4 checks passed
@AndreSlavescu
AndreSlavescu deleted the metile-compile-api branch July 31, 2026 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant