Repository navigation
Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B - #1835
BlivionIaG wants to merge 57 commits into
Conversation
51cc5e2 to
f57cc5d
Compare
|
Thanks @BlivionIaG. We reviewed this independently. We'd take the generic, opt-in
On scope, our decision: the engine ships the mechanism only. Please leave out the matched harmful and harmless prompt sets, the deriver, and the receipts measuring refusal removal on the 27B. Deriving vectors that remove a model's safety refusals isn't something we'll ship or document in TensorFold. A user can supply their own direction files. |
Review changes from ashhart#1835: - the app's settings panel, IPC, preload and locale packs move to feature/glp-steering-app, so app changes go through the app's review - glp.read refuses a file that names one layer twice (direction.5 and direction.05 both parse as 5), with a test - the serve refuses a layer id past the checkpoint's num_hidden_layers at startup, before the model loads - the GLP container is read by the engine's own GGUF reader; its writer stays in glp.zig, so there is one GGUF reader - the deriver, the matched prompt sets, the refusal-measurement gate and their receipt are removed; a vector file is supplied by the operator
|
All three changes are in, plus the scope cut.
The branch is merged with |
|
CI caught one thing the local suites did not: after merging main, Fixed in |
|
Thanks @BlivionIaG. The 18:50 push answers every 17:58 ask:
Off by default is main (
It's 38 commits behind main, so a rebase with these would be ideal. |
|
CI is green on Two things came out of the first full run:
|
|
Thanks @BlivionIaG: the rebase applies on main and lean is back to 424 (the served helpers moved verbatim). Three asks are still open: (1) |
|
All three are in,
|
|
End-to-end run on an M4 Pro, macOS, native Metal server built from The served checkpoint's
Serving with the published 49-layer vector from
So the projection changes the served tokens on a benign prompt, and the fused path runs. Both servers were stopped after the requests; the throwaway generator used for the refusal files and the files themselves are not in the tree. |
|
Thanks @BlivionIaG. An independent re-review of One rebase is needed. On current main, 42 of the 43 files apply. The only conflict is the Once it's rebased, we'll run the Mac cells on an M5 Ultra and land it on a clean result:
|
fd03885 to
b61f6fe
Compare
Review changes from ashhart#1835: - the app's settings panel, IPC, preload and locale packs move to feature/glp-steering-app, so app changes go through the app's review - glp.read refuses a file that names one layer twice (direction.5 and direction.05 both parse as 5), with a test - the serve refuses a layer id past the checkpoint's num_hidden_layers at startup, before the model loads - the GLP container is read by the engine's own GGUF reader; its writer stays in glp.zig, so there is one GGUF reader - the deriver, the matched prompt sets, the refusal-measurement gate and their receipt are removed; a vector file is supplied by the operator
|
Rebased onto The conflict was
|
|
All 11 checks pass on |
|
Thanks @BlivionIaG. Two of the four Mac checks are in, on an M5 Ultra at your rebased head:
Still owed, and these need your vector files:
If you post those two receipts from your Mac, this lands on them. |
Mac receipts on
|
| prompt | off 1 | on 1 | off 2 | on 2 | off mean | on mean | delta |
|---|---|---|---|---|---|---|---|
| fibonacci-raw | 13.9855 | 13.9951 | 13.8591 | 14.0267 | 13.9223 | 14.0109 | +0.64% |
| gpu-chat-no-think | 13.9507 | 13.8885 | 13.9249 | 13.9714 | 13.9378 | 13.9300 | -0.06% |
Both prompts are inside 1% of off; the spread is the run-to-run noise, not a steering cost.
Four refusals by name
1. Nested config past depth. /tmp/tf-glp-nested/config.json carries depth and width only under text_config (num_hidden_layers = 32, hidden_size = 5120), with no root num_hidden_layers/hidden_size. A 63-layer vector:
$ tensorfold-native serve /tmp/tf-glp-nested --glp .../Qwen3.8-27B-abliterated-cyber-GLP-63-L1-63-a1.gguf --context 1024
tensorfold: --glp: the vector steers layer 32 of a model with 32 layers: its deepest layer is 31
error: GlpRefused
Exit 1 in 0.19 s, before the model is touched.
2. Add-mode NaN. A one-layer add file with direction.10[0] = NaN:
$ tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit --glp /tmp/glp-nan-add.gguf --context 1024
tensorfold serve: error: argument --glp: layer 10's direction holds a value that is not finite
Exit 2 in 0.012 s, at argument parsing.
3. --glp with no file.
$ tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit --glp
tensorfold serve: error: argument --glp: expected one argument
Exit 2 in 0.010 s.
4. CUDA build given --glp. Cross-compiled from the same tree for aarch64-linux-gnu (zig build native -Dtarget=aarch64-linux-gnu -Doptimize=ReleaseSafe) and run in a Debian bookworm container; the Mac has no NVIDIA GPU. The model directory held the checkpoint's config.json and tokenizer files (no weights), and the vector was the same 49-layer file:
$ /tf serve /run/model --glp /run/vec/vector.gguf --context 1024
tensorfold: --glp: only the Metal engine steers: this server runs the CUDA engine
error: GlpRefused
Exit 1. The engine-open path refuses before any weights load; the text/tokenizer load runs alongside it, as startup.both always does. A refusal on a real CUDA host is still the one thing not exercised here.
Recipe: exact commands
git clone https://github.com/BlivionIaG/TensorFold.git
cd TensorFold
git checkout efd421d9615956bd3b414ab87e1305c59270c39f
# Zig 0.17.0 must be on PATH.
zig build native # Metal server
zig build tf-glp-check # tf-glp-checkOff/on decode (fresh server per run; only --glp/--glp-alpha differ):
V49=$(ls ~/.cache/huggingface/hub/models--msuiche--Qwen3.8-27B-abliterated-cyber-GLP-49/snapshots/*/Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf)
# off
./zig-out/native/bin/tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit \
--context 2048 --port 8123 --host 127.0.0.1 --name bench --no-update-check
# on
./zig-out/native/bin/tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit \
--glp "$V49" --glp-alpha 1.0 \
--context 2048 --port 8123 --host 127.0.0.1 --name bench --no-update-check
# for each server, once it is listening:
python3 tools/bench_openai.py http://127.0.0.1:8123 Qwen3.8-27B-MLX-4bit \
--tokens 64 --reps 5 --temperatures 0 --output /tmp/bench.jsonNested-config fixture:
import glob, json, os, shutil
snap = glob.glob(os.path.expanduser(
"~/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots/*/"))[0]
out = "/tmp/tf-glp-nested"
shutil.rmtree(out, ignore_errors=True)
os.makedirs(out)
for name in os.listdir(snap):
os.symlink(os.path.realpath(os.path.join(snap, name)), os.path.join(out, name))
d = json.load(open(os.path.join(snap, "config.json")))
d.pop("num_hidden_layers", None)
d.pop("hidden_size", None)
tc = d.setdefault("text_config", {})
tc["num_hidden_layers"] = 32
tc["hidden_size"] = 5120
json.dump(d, open(os.path.join(out, "config.json"), "w"), indent=2)Throwaway NaN-vector generator (zig/src/core/tmp_glp_bad.zig, deleted after the run):
const std = @import("std");
const glp = @import("glp.zig");
const gguf = @import("glp_container.zig");
const width = 5120;
pub fn main(init: std.process.Init) !void {
const a = init.arena.allocator();
const args = try init.minimal.args.toSlice(a);
if (args.len != 3) return error.Usage;
const kind = args[1];
const out_path = args[2];
var dir = try a.alloc(f32, width);
for (dir) |*v| v.* = 0;
dir[0] = 1;
if (std.mem.eql(u8, kind, "deep")) {
const layers = [_]glp.Layer{.{ .id = 64, .dir = dir }};
const bytes = try glp.write(a, .{
.mode = .project,
.alpha = 1,
.hook_point = glp.residual_site,
.derived_at = null,
.base_model = "TensorFold/Qwen3.8-27B-MLX-4bit",
.base_revision = "",
.method = "test",
.contrast = "",
.layers = &layers,
});
try std.Io.Dir.cwd().writeFile(init.io, .{ .sub_path = out_path, .data = bytes });
return;
}
if (std.mem.eql(u8, kind, "nan-add")) {
var raw = try a.alloc(u8, width * 4);
std.mem.writeInt(u32, raw[0..4], @bitCast(std.math.nan(f32)), .little);
@memset(raw[4..], 0);
const entries = [_]gguf.Entry{
.{ .key = "general.architecture", .value = .{ .string = "controlvector" } },
.{ .key = "glp.spec_version", .value = .{ .u32v = 1 } },
.{ .key = "glp.mode", .value = .{ .string = "add" } },
.{ .key = "glp.rank", .value = .{ .u32v = 1 } },
.{ .key = "glp.hook_point", .value = .{ .string = glp.residual_site } },
.{ .key = "glp.alpha_default", .value = .{ .f32v = 1 } },
.{ .key = "glp.base_model", .value = .{ .string = "TensorFold/Qwen3.8-27B-MLX-4bit" } },
};
const dims = [_]u64{width};
const tensors = [_]gguf.OutTensor{.{ .name = "direction.10", .dims = &dims, .kind = gguf.f32_type, .data = raw }};
const bytes = try gguf.write(a, &entries, &tensors);
try std.Io.Dir.cwd().writeFile(init.io, .{ .sub_path = out_path, .data = bytes });
return;
}
return error.UnknownKind;
}zig run zig/src/core/tmp_glp_bad.zig -- nan-add /tmp/glp-nan-add.gguf
rm -f zig/src/core/tmp_glp_bad.zigCUDA build and container run:
zig build native -Dtarget=aarch64-linux-gnu -Doptimize=ReleaseSafe
mkdir -p "$HOME/tf-glp-cuda/run/model" "$HOME/tf-glp-cuda/run/vec"
cp zig-out/native/bin/tensorfold-native "$HOME/tf-glp-cuda/tensorfold-native-cuda-aarch64"
# rebuild the Metal binary afterwards, since the cross-build writes the same path:
zig build native
# populate run/model with the checkpoint's config.json and tokenizer files (no weights),
# and run/vec with the 49-layer vector above.
python3 - <<'PY'
import glob, os, shutil
home = os.path.expanduser("~")
snap = glob.glob(home + "/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots/*/")[0]
model = home + "/tf-glp-cuda/run/model"
for name in ("config.json", "generation_config.json", "tokenizer.json", "tokenizer_config.json",
"vocab.json", "chat_template.jinja", "preprocessor_config.json",
"processor_config.json", "model.safetensors.index.json"):
src = os.path.join(snap, name)
if os.path.exists(src):
shutil.copy(os.path.realpath(src), os.path.join(model, name))
vec = glob.glob(home + "/.cache/huggingface/hub/models--msuiche--Qwen3.8-27B-abliterated-cyber-GLP-49/snapshots/*/Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf")[0]
shutil.copy(os.path.realpath(vec), home + "/tf-glp-cuda/run/vec/vector.gguf")
PY
podman run --rm --platform linux/arm64 \
-v "$HOME/tf-glp-cuda/tensorfold-native-cuda-aarch64:/tf:ro" \
-v "$HOME/tf-glp-cuda/run:/run:ro" \
docker.io/library/debian:bookworm-slim \
/tf serve /run/model --glp /run/vec/vector.gguf --context 1024 --port 8193|
Thanks @BlivionIaG for both receipts. On-mode decode is within 1% of off, and all four refusals stop by name before load. A few things before it lands:
We'll check the CUDA refusal ourselves on a GB10. |
efd421d to
b721c88
Compare
Review changes from ashhart#1835: - the app's settings panel, IPC, preload and locale packs move to feature/glp-steering-app, so app changes go through the app's review - glp.read refuses a file that names one layer twice (direction.5 and direction.05 both parse as 5), with a test - the serve refuses a layer id past the checkpoint's num_hidden_layers at startup, before the model loads - the GLP container is read by the engine's own GGUF reader; its writer stays in glp.zig, so there is one GGUF reader - the deriver, the matched prompt sets, the refusal-measurement gate and their receipt are removed; a vector file is supplied by the operator
|
Rebased onto
|
|
|
Review changes from ashhart#1835: - the app's settings panel, IPC, preload and locale packs move to feature/glp-steering-app, so app changes go through the app's review - glp.read refuses a file that names one layer twice (direction.5 and direction.05 both parse as 5), with a test - the serve refuses a layer id past the checkpoint's num_hidden_layers at startup, before the model loads - the GLP container is read by the engine's own GGUF reader; its writer stays in glp.zig, so there is one GGUF reader - the deriver, the matched prompt sets, the refusal-measurement gate and their receipt are removed; a vector file is supplied by the operator
The receipt said the device gates run the kernel on the GPU and compare it with float64. Now it says what they reported: the steering kernel passes gates 1 to 4 in both modes it serves, the two controls are red as planted, and the limits gate passes with the launch's own declared threads and threadgroup memory.
…lags with it The control vector's two flags pushed the serve flag table's file over the rule's 600 lines, and it was 596 before they arrived, so the next flag on main would have done the same. The shape of a flag is now a module of its own, which lets the control vector's flags live in the module that owns the control vector instead of in the table. The table keeps every other flag where it was, and the parser and ``capabilities`` see the same list as before. The file is 588 lines, and the lean rule reports no finding over the base.
…t CI did The receipt still carried the counts from before the merge with main: the number of new Zig tests, the step and test totals of an older build, the old pinned base, and a claim that every check was green. The numbers are now the ones this head produces: 66 new Zig tests, 552 steps succeeded and 161 of 162 tests passed, and the lean count against the base the branch merged, b30d5b9. The CI line says what happened. The format check, the code, the tests and the tools are green on Linux, Windows and macOS. The app's macOS packaging job is cancelled by the runner pool, four attempts across two runs, and the base tree's own runs are cancelled the same way in the same window, so that line no longer claims it is green. The job's four steps were run here under the job's own pins instead, and they pass.
…ow gated The steering kernel replaced the norm kernel, so the vector costs one dispatch per steered layer and not two. What was left was measured: against the same build and the same server without a vector, the cost is under a percent of a decoded token and under half a percent of a prefill, which is under what this machine resolves between runs. The first reading looked like three percent, but a second baseline run moved by six, so the honest reading is that the cost is below the noise. Two things in the kernel were still worth fixing. The add mode does not use the dot, so the kernel now skips it: the reduction the project mode needs is not run, and the mode matches its spec, which declares no sums for it. The branch is on a uniform value, the same way the kernel already returns early on the rows past the end. The direction is read once per row and kept in registers, so the dot and the projection read the same stretch of it instead of reading it twice. The device suite now gates the fused kernel's normalised row against the family's own plain norm kernel: the row the fused kernel projected, normalised by the plain kernel, has to be the same row, element for element. The gates compared the projected row; nothing compared the values the model consumes, so a wrong sum of squares would have passed unnoticed. A fault planted in the normalisation alone is caught, and the suite says so.
Settings -> AI & Models -> Engine advanced gains a toggle, a file picker and a strength. The engine's own two flags carry the choice at its next start, so the app does not invent a way to apply a vector: it passes ``--glp <file>`` and ``--glp-alpha <alpha>`` and the engine says what it is holding when it starts. The path comes only from the OS picker. The renderer can turn the vector on and set its strength, and it cannot name a file: the picker is the one place a path enters the app, the same rule the model-folder picker already follows. The audit log records the state and the strength and never the path, because the file may sit in a directory the operator keeps private. The app only offers the control when the engine in use can take it. It reads ``capabilities --json`` and looks for ``--glp`` among the serve flags; when the engine does not offer it, the panel says so rather than failing at the next start. The packaged app pins its engine by sha256 and the pinned release does not carry the flag, so a shipped build says the engine cannot take one, and the panel stays inert until the engine is re-pinned. The dev path, which takes an engine built from this branch, works today. The change and the model restart happen together, so a refused start leaves the old choice in place. The file, the strength and the state are kept in ``engine/preferences.json``, so the choice survives a restart of the app. A field the caller leaves out keeps its current value, so changing the strength does not switch the vector off and switching it off does not reset the strength. The panel says plainly what a vector does: it steers the model, so the model answers differently, and it is a steering direction rather than a safety setting. It is off until it is turned on. The new strings are registered and translated into all twelve locales, the language index is rebuilt, the typecheck is clean, and the app's own suite is 856 of 856. The packaged macOS build carries the panel and passes the repository's smoke test.
…y in force Choosing a file while a vector was in force wrote the new path to the saved preferences and left the running engine holding the old one, so the panel and the engine named different files until the next start. The pick now goes through the same apply-and-restart path as the other changes, so the engine and the setting never disagree, and a refused start leaves the old file in place. A file chosen while no vector is in force is still only remembered, and still does not restart the model.
The branch was rebased onto the upstream main at 8abd450, 166 commits past the tree it merged before. The engine is upstream's 1.1.8 now, the test suite grew with it, and the lean baseline moved with it. The receipt says 1.1.8, 584 steps and 8214 of 8259 tests, and 425 lean problems against 425 on the base the branch sits on. The measurements did not move. The published vector on the derivation set still leaves 19 of 30 harmful prompts refused and 6 of 30 benign ones, with 60 of 60 coherent replies, which is what the earlier tree gave as well.
The upstream tree grew by four lines since this branch merged it, and with the control vector's own three the API's file crossed the lean rule's 600 lines. The rule compares against the base branch, so the file that was 591 lines upstream and 602 here read as a problem this branch had added, when it had only inherited the crossing. The control vector's shape moves into the module that owns the container, and the shapes a request carries move into a module of their own, the way the vision and speech types already sit beside the API. Every name stays where it was: the API re-exports them, so no caller moved. The file is 565 lines, and the lean rule reports no finding over the base.
The test totals are the ones a cold build of the rebased tree produces, and the lean count is the one after the engine API's split: 424 problems on this branch against 425 on the base it sits on. Co-Authored-By: BlivionIaG <15323295+BlivionIaG@users.noreply.github.com>
The server compares the checkpoint a vector names with the one it serves: the served model name has to be the vector's model name or start with it. The same weights re-released under another name were refused with it, although the directions are exactly right for them. tf-glp-check rewrite now takes --repo and --revision, so a file can be re-stamped for that name. The directions stay the same bytes and the digest over them does not move; the test asserts both, and that the binding moves with the name.
The GLP container format, its reference implementation and its validator are Matt Suiche's (the weightless plugin, MIT). The module now says so where a reader of the source meets it first. The receipt now names the base this branch sits on (7b65d60) and the lean count against it (424 against 424, zero new).
The limits pass picked the source for a launch by its label, and the steering launch's label starts with "qwen27 ", so the family branch took it and handed the glue source to the compiler: the kernel was never built, and the gate reported the launch as missing from the library. The same line also overwrote the source of the four "metal pair " launches upstream owns, so their pipelines went missing the same way. The 27B's source now comes from one place, by what the label says, and the steering kernel is specialized at the 27B's width as the engine builds it. On this GPU: 2 kernels pass gates 1-4, 2 controls red as planted, 0 limit problems.
The module header's credit made the file's doc two comment lines, and the new helper's doc ran past the column limit: two findings the base branch does not have. The credit stays, in one line, and the helper's doc says the same thing shorter. Lean: 424 problems, 424 on the base, 0 new.
The family's attention gained a split argument in the upstream range this branch was rebased onto: the kernel that takes the head count at run time is the one a split run uses. The producer's own call still had the old arity, so the dev-tools build stopped on it. The producer runs the model on one Mac, where the reduce is null, so the call passes the same rule the family's own round does.
The receipt still carried the engine version, the test counts and the gate transcript of the tree before the rebase, and it did not say what the rebased producer writes: the same directions, the same digest. It now carries the engine's own version, the rebased suite's counts, the gate's transcript as it runs today, the re-derivation's digest and the measurement on the same weights under their other name.
The entry sat in two released sections, the same line twice: once under 1.1.9 and once under 1.1.7, and in neither of them the release it describes. It is one entry under Unreleased now, and it says whose format it implements.
The suite grew with the upstream commits the branch now sits on: 605 steps and 9348 tests, 55 of them skipped.
Review changes from ashhart#1835: - the app's settings panel, IPC, preload and locale packs move to feature/glp-steering-app, so app changes go through the app's review - glp.read refuses a file that names one layer twice (direction.5 and direction.05 both parse as 5), with a test - the serve refuses a layer id past the checkpoint's num_hidden_layers at startup, before the model loads - the GLP container is read by the engine's own GGUF reader; its writer stays in glp.zig, so there is one GGUF reader - the deriver, the matched prompt sets, the refusal-measurement gate and their receipt are removed; a vector file is supplied by the operator
Zig places a source file in one module; engine_api reaches glp.zig, and core reaches gguf.zig, so glp importing the engine's reader made gguf.zig a member of both modules and every target that links them failed to compile. The GLP container's reader is now glp_container.zig, named for its scope as the review allowed. It reads and writes the GGUF v3 subset a vector needs; the engine's gguf.zig stays the engine's reader. A repeated layer id is still refused by glp.read.
CI's lean gate counted three new problems: two comment lines and a file past 600 lines. The steering helpers (fit check, plan, startup line) move to qwen27_steer_served.zig, qwen27_served.zig returns to 521 lines, and the two comments are one line each. lean_check: 424 problems, 424 on the base, 0 new.
… reader named The startup depth check reads num_hidden_layers from text_config, falling back to the config root, with a nested-config test. read() refuses a direction holding a NaN or an infinity in either mode, with a test over both. The changelog and the guide name core/glp_container.zig, the reader the file actually goes through.
The publication sample and contrast strings name no refusal or harmful set.
check-release-paths compiles the contract module with bf16_round as its own module; importing the file by path put the same file in two modules. Import the module instead.
The maintainer's CHANGELOG credits people by GitHub handle. Name the format as the weightless project's, with its author's handle, in the CHANGELOG line and the container reader's header. The GLP reader's header follows the same wording.
metal_main.zig was 605 lines after the rebase, past the 600-line rule. The suite now takes its entries and runs its normalisation check in metal_qwen27_steer.zig, so metal_main.zig carries three lines for it, as the attention rows do. The check prepares and launches its own rows at each served shape: since the gates free a kernel's buffers once they are done with it, reading the rows runAll left behind used released buffers, and the contract run segfaulted.
ed8c3e1 to
405e53f
Compare
|
Rebased onto current main (
Local, Apple M4 Pro:
Every commit has the noreply author and committer. CI is running. |
Fixes #1834
What this is
The engine can serve a model with a GLP control vector applied: one direction per layer,
projected out of (or added to) the residual stream after each steered block.
The file is GGUF v3 and follows the GLP standard (
glp.spec_version,glp.mode,glp.rank,glp.hook_point,glp.alpha_default,glp.base_model,glp.base_revision,glp.layer_ids_zero_based,glp.content_sha256, plusgeneral.*publication metadata);llama.cpp's additive control vector is read as it stands.
tf-glp-checkshows a file as this readersees it and writes it back out.
The GLP container format, its reference implementation and its validator are Matt Suiche's: the
format as implemented by the
weightlessplugin (MIT).This work implements that format on the native engine rather than redefining it, and the reference
implementation is not vendored:
tools/zig/glp_parity.pytakes aweightlesscheckout and reads itas a package, so a file this engine accepts is checked against the implementation the format comes
from. The layer ids, the metadata, every direction's bytes and the digest all agree.
This pull request ships the mechanism only. It carries no derived vectors, no prompt sets, no
derivation tool and no refusal measurements: a vector file is supplied by the operator.
What it does to the model
For
projectmode at strengthalpha, every steered layer computesNothing is written back to the checkpoint. The intervention lives in the activation path, so it is
turned on and off per server run and its strength is a runtime argument.
What the server refuses
A file the server cannot apply is refused before the model is loaded, and the server does not start:
checkpoint is not the one served, or whose content digest does not cover its own vectors;
direction.5anddirection.05both parse as 5);num_hidden_layers, read at startup so the model never loads;The checkpoint is compared by name, because that is what the file carries. A file can be re-stamped
for another name with
tf-glp-check rewrite --repo ID --revision REV; the directions stay the samebytes and only the name moves, so that is the operator's claim and not one the reader verifies.
The kernel
zig/kernels/metal/qwen27/norm_steer.metal. It replaces the norm kernel rather than adding adispatch, so the vector costs a pass the model already makes.
zig/src/core/contract/steer.zig, and the family's own unit test there holds the kernel'sarithmetic inside the bound the spec declares.
zig/src/core/contract/steer_records.zig: 320 threads and 80 threadgroup bytes, against M1 to M5and sm_80, sm_86, sm_89, sm_120 and sm_121.
What the projection costs: measured against the same build and the same server without a vector, on
an M4 Pro with
Qwen3.8-27B-MLX-4bit, under one percent of a decoded token and under half a percentof a prefill — below what the machine resolves between runs.
Review changes
#1951, so app changes go through the app's
review. This PR has no
app/file.glp.readrefuses a file that names one layer twice, with a test.glp_startupreadsnum_hidden_layersfrom the checkpoint's configand refuses a vector whose deepest layer is past it, before the model loads.
zig/src/core/glp_container.zigreads and writesthe GGUF v3 subset a vector needs. It cannot import the engine's
gguf.zig:engine_apireachesglp.zigandcorereachesgguf.zig, and Zig puts a file in one module, so the merge made everytarget that links both fail to compile. Renaming is the review's stated alternative.
measurement receipt are removed.
Verification
zig build test-glp: 57 tests pass (container, publication metadata, repeated layers, refusals).zig build test-glp-serve: 25 tests pass (flags, refusals, depth, the startup line).upstream/mainat16d95ccec; the merge conflicts were incontract/registry.zigandnative/qwen27_served.zig, resolved by keeping both sides' additions.zig build tf-qwen27-served-checkcompiles the server path that linksengine_apiandcore.Not covered
Metal-only for now.
zig build test-kernel-contract, and the device gates ran on an M4 Pro.