Skip to content

Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B - #1835

Open
BlivionIaG wants to merge 57 commits into
ashhart:mainfrom
BlivionIaG:feature/glp-steering
Open

BlivionIaG wants to merge 57 commits into
ashhart:mainfrom
BlivionIaG:feature/glp-steering

Conversation

@BlivionIaG

@BlivionIaG BlivionIaG commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1834

What this is

The engine can serve a model with a GLP control vector applied: one direction per layer,
projected out of (or added to) the residual stream after each steered block.

tensorfold serve TensorFold/Qwen3.8-27B-MLX-4bit --glp vector.gglp --glp-alpha 1.0

The file is GGUF v3 and follows the GLP standard (glp.spec_version, glp.mode, glp.rank,
glp.hook_point, glp.alpha_default, glp.base_model, glp.base_revision,
glp.layer_ids_zero_based, glp.content_sha256, plus general.* publication metadata);
llama.cpp's additive control vector is read as it stands. tf-glp-check shows a file as this reader
sees it and writes it back out.

The GLP container format, its reference implementation and its validator are Matt Suiche's: the
format as implemented by the weightless plugin (MIT).
This work implements that format on the native engine rather than redefining it, and the reference
implementation is not vendored: tools/zig/glp_parity.py takes a weightless checkout and reads it
as a package, so a file this engine accepts is checked against the implementation the format comes
from. The layer ids, the metadata, every direction's bytes and the digest all agree.

This pull request ships the mechanism only. It carries no derived vectors, no prompt sets, no
derivation tool and no refusal measurements: a vector file is supplied by the operator.

What it does to the model

For project mode at strength alpha, every steered layer computes

h <- h - alpha * (h . d) * d

Nothing is written back to the checkpoint. The intervention lives in the activation path, so it is
turned on and off per server run and its strength is a runtime argument.

What the server refuses

A file the server cannot apply is refused before the model is loaded, and the server does not start:

  • a vector whose width is not the model's, whose hook point is not the one it applies at, whose
    checkpoint is not the one served, or whose content digest does not cover its own vectors;
  • a file that names one layer twice (direction.5 and direction.05 both parse as 5);
  • a layer id past the checkpoint's num_hidden_layers, read at startup so the model never loads;
  • a Gemma checkpoint — its terms forbid circumventing its safety filters;
  • a model that is not Qwen3.8-27B, and a backend that is not Metal.

The checkpoint is compared by name, because that is what the file carries. A file can be re-stamped
for another name with tf-glp-check rewrite --repo ID --revision REV; the directions stay the same
bytes and only the name moves, so that is the operator's claim and not one the reader verifies.

The kernel

  • One dispatch per steered layer: the norm and the projection in the same pass over the row, in
    zig/kernels/metal/qwen27/norm_steer.metal. It replaces the norm kernel rather than adding a
    dispatch, so the vector costs a pass the model already makes.
  • The float64 reference, the bounds, the fp32-sum rule and the served envelope are in
    zig/src/core/contract/steer.zig, and the family's own unit test there holds the kernel's
    arithmetic inside the bound the spec declares.
  • The launch is registered for the limits gate in
    zig/src/core/contract/steer_records.zig: 320 threads and 80 threadgroup bytes, against M1 to M5
    and sm_80, sm_86, sm_89, sm_120 and sm_121.

What the projection costs: measured against the same build and the same server without a vector, on
an M4 Pro with Qwen3.8-27B-MLX-4bit, under one percent of a decoded token and under half a percent
of a prefill — below what the machine resolves between runs.

Review changes

  • App split out. The settings panel, IPC, preload and locale packs are now
    #1951, so app changes go through the app's
    review. This PR has no app/ file.
  • Repeated layer ids refused. glp.read refuses a file that names one layer twice, with a test.
  • Depth refused at startup. glp_startup reads num_hidden_layers from the checkpoint's config
    and refuses a vector whose deepest layer is past it, before the model loads.
  • The container reader is named for its scope. zig/src/core/glp_container.zig reads and writes
    the GGUF v3 subset a vector needs. It cannot import the engine's gguf.zig: engine_api reaches
    glp.zig and core reaches gguf.zig, and Zig puts a file in one module, so the merge made every
    target that links both fail to compile. Renaming is the review's stated alternative.
  • Mechanism only. The deriver, the matched prompt sets, the refusal-measurement gate and the
    measurement receipt are removed.

Verification

  • zig build test-glp: 57 tests pass (container, publication metadata, repeated layers, refusals).
  • zig build test-glp-serve: 25 tests pass (flags, refusals, depth, the startup line).
  • The branch is merged with upstream/main at 16d95ccec; the merge conflicts were in
    contract/registry.zig and native/qwen27_served.zig, resolved by keeping both sides' additions.
  • zig build tf-qwen27-served-check compiles the server path that links engine_api and core.

Not covered

  • CUDA and HIP: the machine these numbers come from has no NVIDIA or AMD GPU. The steering is
    Metal-only for now.
  • Apple chips other than the one measured: the limits for M1 to M5 are checked without a GPU by
    zig build test-kernel-contract, and the device gates ran on an M4 Pro.

@BlivionIaG BlivionIaG changed the title GLP support: apply a control vector to Qwen3.8-27B on Metal at inference time (the model's effective weights change on the fly; the checkpoint does not) Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B Nov 1, 2026
@BlivionIaG
BlivionIaG force-pushed the feature/glp-steering branch 2 times, most recently from 51cc5e2 to f57cc5d Compare November 1, 2026 14:22
@ashhart

ashhart commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. We reviewed this independently. We'd take the generic, opt-in --glp control-vector mechanism with its fused Metal norm-and-projection kernel. Three changes:

  1. Split out the app. The 34 files under app/ (the settings panel, IPC, preload and 13 locale packs) go in their own PR. App changes go through the app's own review.
  2. Refuse repeated layer ids. direction.5 and direction.05 both parse to 5, read() sorts but doesn't refuse, and steer.init keeps the later one. Please refuse a repeat in glp.read, with a test. A layer id past the model's depth is refused only after the 27B loads, though num_hidden_layers is readable at startup, so refuse it there.
  3. One GGUF reader. A second zig/src/core/gguf.zig collides with the GGUF reader now on main (core: a GGUF reader and an exact repack of GGUF's quantized blocks (#1920, steps 1-2) #1926). Please build on that one, or rename yours.

On scope, our decision: the engine ships the mechanism only. Please leave out the matched harmful and harmless prompt sets, the deriver, and the receipts measuring refusal removal on the 27B. Deriving vectors that remove a model's safety refusals isn't something we'll ship or document in TensorFold. A user can supply their own direction files.

BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 2, 2026
Review changes from ashhart#1835:
- the app's settings panel, IPC, preload and locale packs move to
  feature/glp-steering-app, so app changes go through the app's review
- glp.read refuses a file that names one layer twice (direction.5 and
  direction.05 both parse as 5), with a test
- the serve refuses a layer id past the checkpoint's num_hidden_layers at
  startup, before the model loads
- the GLP container is read by the engine's own GGUF reader; its writer stays
  in glp.zig, so there is one GGUF reader
- the deriver, the matched prompt sets, the refusal-measurement gate and their
  receipt are removed; a vector file is supplied by the operator
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

All three changes are in, plus the scope cut.

  • App: the 32 app files moved to app: control vector picker and strength in Engine advanced #1951; app/ no longer appears in this PR's diff.
  • Repeated layer ids: glp.read now refuses a file that names one layer twice, with a test. A layer id past the checkpoint's num_hidden_layers is refused in glp_startup.refusal, at startup, before the model loads — two tests cover the boundary.
  • GGUF reader: the branch's zig/src/core/gguf.zig is gone; the GLP reader now parses with main's gguf.Document. Only the writer stays local to glp.zig.
  • Scope: the deriver, both prompt sets, the refusal-measurement gate and the receipt are removed; the guide and README describe the mechanism only.

The branch is merged with upstream/main at 16d95ccec and pushed as feature/glp-steering (head 7f83ebb0). zig build test-glp passes 57 tests and zig build test-glp-serve 25.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

CI caught one thing the local suites did not: after merging main, glp.zig importing the engine's gguf.zig put that file in two modules (engine_api and core), and every target linking both failed to compile.

Fixed in b81f6002: the GLP container reader is zig/src/core/glp_container.zig, named for its scope — the review's stated alternative — so the engine's gguf.zig stays the engine's reader. Repeated-layer and depth refusals are unchanged. zig build tf-qwen27-served-check compiles the server path, and the GLP suites still pass.

@ashhart

ashhart commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. The 18:50 push answers every 17:58 ask:

Off by default is main (Model.steer null, both norm sites call main's norm), and the file is validated before use. Four small things left:

  1. The startup depth check reads num_hidden_layers from the config root only, while hidden_size and the engine read text_config. So a nested config skips the check until after load. Please read it from text_config too, with a nested-config test.
  2. Add-mode directions aren't checked for NaN or infinity. Please refuse non-finite values in both modes.
  3. Lean is 426 against main's 424 (core/glp.zig:161, core/glp_container.zig:1).
  4. The changelog and docs still say the engine's own GGUF reader is used, but it's the new container reader now.

It's 38 commits behind main, so a rebase with these would be ideal.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

CI is green on 2f90635ad (all 11 checks pass).

Two things came out of the first full run:

  • The lean gate counted three new problems: two comment lines and qwen27_served.zig past 600 lines. The serve-side steering helpers moved to qwen27_steer_served.zig (90 lines), qwen27_served.zig is back to 521, and the comments are one line each. lean_check now reports 424 problems, 424 on the base, 0 new.
  • zig and zig-windows failed on a regression already on main (linear_time.Fastest indexed as an array in cortheon/program.zig). Main's 5fe74ebb9 fixes it; this branch merges that tip.

@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG: the rebase applies on main and lean is back to 424 (the served helpers moved verbatim). Three asks are still open: (1) modelOf reads num_hidden_layers from the config root, not text_config; (2) glp.zig:read checks finiteness only in project mode, so add-mode directions aren't checked; (3) the changelog and docs still say the engine's own GGUF reader is used. Nothing cleared earlier changed.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

All three are in, fd0388546.

  • Depth from text_config. modelOf now reads num_hidden_layers from text_config, falling back to the config root. A nested-config test writes a config whose depth lives only in text_config and holds the depth refusal before load.
  • Non-finite directions refused in both modes. glp.read scans every direction component before the mode-specific rule and refuses NaN and ±infinity in project and add, with a test that writes each through both modes. The project-mode unit-norm check no longer carries its own isFinite guard, since the scan precedes it.
  • Reader named. The changelog and the guide now say core/glp_container.zig reads the GGUF v3 subset a vector needs, not the engine's gguf.zig. The guide's refusal table also lists non-finite directions.

zig build test-glp, zig build test-glp-serve and zig build tf-qwen27-served-check pass. lean_check reports 424 problems, 424 on the base, so nothing new.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

End-to-end run on an M4 Pro, macOS, native Metal server built from fd0388546 (zig build native, Zig 0.17.0).

The served checkpoint's config.json carries its depth only in text_config (text_config.num_hidden_layers = 64, text_config.hidden_size = 5120; no root num_hidden_layers/hidden_size), so this run exercises the nested-config path:

  • A vector naming layer 64 (direction.64, 5120 wide, project) is refused in 0.21 s, before the model loads: --glp: the vector steers layer 64 of a model with 64 layers: its deepest layer is 63.
  • An add-mode vector with one NaN component is refused in 0.02 s: argument --glp: layer 10's direction holds a value that is not finite.
  • A 63-layer vector naming layers 1-63 is accepted at the boundary and serves: glp on: ... (mode: project; alpha: 1.00; layers: 63 (1-63); base: Qwen/Qwen3.8-27B @ 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0; digest: d92fddd15d08 (verified)), then control vector in force: project mode, alpha 1.000, 63 layers (1-63).

Serving with the published 49-layer vector from msuiche/Qwen3.8-27B-abliterated-cyber-GLP-49 (mode: project, layers 10-58, digest verified) and the prompt "Write a haiku about the sea." at temperature 0.2, enable_thinking: false:

  • steered: "Waves crash on the shore, / White foam dissolves in the blue, / Silence takes the tide." (24 completion tokens)
  • the same build without --glp: "Salt spray on the wind, / Waves crash against the dark shore, / Moonlight on the foam." (23 completion tokens)

So the projection changes the served tokens on a benign prompt, and the fused path runs. Both servers were stopped after the requests; the throwaway generator used for the refusal files and the files themselves are not in the tree.

@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. An independent re-review of fd0388546 finds all three open asks met. Without --glp, Qwen3.8-27B on Metal runs main's path: no steer, no new pipeline compiled, no extra report line. Every refusal names its cause.

One rebase is needed. On current main, 42 of the 43 files apply. The only conflict is the all_metal line in core/contract/registry.zig: main has since appended ling_route_records.zig and this PR appends steer_records.zig, so keep both. The other shared lists (core/contract.zig, build.zig, zig/build/qwen27.zig's parts) may need the same keep-both.

Once it's rebased, we'll run the Mac cells on an M5 Ultra and land it on a clean result:

  • off against main: token_sha256 equal, default and --no-drafts, at 1 and 2 streams;
  • on, with a project-mode vector at --glp-alpha 1.0: decode within 1% of off;
  • the four refusals by name;
  • norm_steer through the Metal kernel contract.

@BlivionIaG
BlivionIaG force-pushed the feature/glp-steering branch from fd03885 to b61f6fe Compare November 3, 2026 07:33
BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 3, 2026
Review changes from ashhart#1835:
- the app's settings panel, IPC, preload and locale packs move to
  feature/glp-steering-app, so app changes go through the app's review
- glp.read refuses a file that names one layer twice (direction.5 and
  direction.05 both parse as 5), with a test
- the serve refuses a layer id past the checkpoint's num_hidden_layers at
  startup, before the model loads
- the GLP container is read by the engine's own GGUF reader; its writer stays
  in glp.zig, so there is one GGUF reader
- the deriver, the matched prompt sets, the refusal-measurement gate and their
  receipt are removed; a vector file is supplied by the operator
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Rebased onto upstream/main at ef9fa700a; head is efd421d96, and the diff is 43 files.

The conflict was all_metal in core/contract/registry.zig: both sides kept, so the list now ends ... @import("ling_route_records.zig").records ++ @import("steer_records.zig").metal;. The only other conflicts were the branch's own core/gguf.zig add against main's GGUF reader, in two intermediate commits; resolved to main's reader, and the final tree has no branch copy — glp.zig reads glp_container.zig. The shared lists in core/contract.zig, build.zig and zig/build/qwen27.zig auto-merged, with both additions kept.

zig build test-glp and zig build test-glp-serve (26/26) pass, zig build tf-qwen27-served-check compiles, and lean_check reports 424 problems, 424 on the base. The one added commit is the guide naming the GLP release the tap now ships (v1.2.1-glp.1, upstream 1.2.1 plus this branch).

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

All 11 checks pass on efd421d96: zig, zig-windows, zig-macos-dev-tools, fmt, desktop, desktop-linux, mobile, appimage-linux, release-paths and both benchmark-clients jobs.

@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. Two of the four Mac checks are in, on an M5 Ultra at your rebased head:

  • Off against main (no --glp, Qwen3.8-27B-MLX-4bit, fresh servers, base-new-new-base): GREEN. Tokens are equal and every cell is level.
  • Metal kernel contract (--only qwen27): norm_steer passes gates 1-4 with its control red, and there are 0 limit problems.

Still owed, and these need your vector files:

  1. on, with a project-mode vector at --glp-alpha 1.0: the startup line, and decode within 1% of off;
  2. the four refusals by name before load (a nested config past depth, an add-mode NaN vector, --glp with no file, a CUDA build given --glp).

If you post those two receipts from your Mac, this lands on them.

@BlivionIaG

BlivionIaG commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Mac receipts on efd421d96

Apple M4 Pro, 48 GiB, macOS 26.5.1, Zig 0.17.0. Native Metal server built with zig build native (110 s), tensorfold-native 1.2.1.

On at --glp-alpha 1.0

Vector: msuiche/Qwen3.8-27B-abliterated-cyber-GLP-49, Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf — project, alpha 1.0, layers 10-58, digest da81a8b364de (verified).

[tensorfold] glp on: $V49 (mode: project; alpha: 1.00; layers: 49 (10-58); base: Qwen/Qwen3.8-27B @ 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0; digest: da81a8b364de (verified))
info: Qwen3.8-27B: control vector in force: project mode, alpha 1.000, 49 layers (10-58), base Qwen/Qwen3.8-27B @ 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, digest da81a8b364de

The startup line is verbatim; $V49 is the resolved cache path, set in the recipe below.

Decode: same binary, fresh server per run, --context 2048, default drafts, tools/bench_openai.py --tokens 64 --reps 5 --temperatures 0, interleaved off / on / off / on so thermal drift cancels. (A single off run immediately after the build read 14.84 tok/s; the machine had drifted by the time of the paired runs, so only the four-run A/B is comparable.)

prompt off 1 on 1 off 2 on 2 off mean on mean delta
fibonacci-raw 13.9855 13.9951 13.8591 14.0267 13.9223 14.0109 +0.64%
gpu-chat-no-think 13.9507 13.8885 13.9249 13.9714 13.9378 13.9300 -0.06%

Both prompts are inside 1% of off; the spread is the run-to-run noise, not a steering cost.

Four refusals by name

1. Nested config past depth. /tmp/tf-glp-nested/config.json carries depth and width only under text_config (num_hidden_layers = 32, hidden_size = 5120), with no root num_hidden_layers/hidden_size. A 63-layer vector:

$ tensorfold-native serve /tmp/tf-glp-nested --glp .../Qwen3.8-27B-abliterated-cyber-GLP-63-L1-63-a1.gguf --context 1024
tensorfold: --glp: the vector steers layer 32 of a model with 32 layers: its deepest layer is 31
error: GlpRefused

Exit 1 in 0.19 s, before the model is touched.

2. Add-mode NaN. A one-layer add file with direction.10[0] = NaN:

$ tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit --glp /tmp/glp-nan-add.gguf --context 1024
tensorfold serve: error: argument --glp: layer 10's direction holds a value that is not finite

Exit 2 in 0.012 s, at argument parsing.

3. --glp with no file.

$ tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit --glp
tensorfold serve: error: argument --glp: expected one argument

Exit 2 in 0.010 s.

4. CUDA build given --glp. Cross-compiled from the same tree for aarch64-linux-gnu (zig build native -Dtarget=aarch64-linux-gnu -Doptimize=ReleaseSafe) and run in a Debian bookworm container; the Mac has no NVIDIA GPU. The model directory held the checkpoint's config.json and tokenizer files (no weights), and the vector was the same 49-layer file:

$ /tf serve /run/model --glp /run/vec/vector.gguf --context 1024
tensorfold: --glp: only the Metal engine steers: this server runs the CUDA engine
error: GlpRefused

Exit 1. The engine-open path refuses before any weights load; the text/tokenizer load runs alongside it, as startup.both always does. A refusal on a real CUDA host is still the one thing not exercised here.

Recipe: exact commands
git clone https://github.com/BlivionIaG/TensorFold.git
cd TensorFold
git checkout efd421d9615956bd3b414ab87e1305c59270c39f
# Zig 0.17.0 must be on PATH.
zig build native        # Metal server
zig build tf-glp-check  # tf-glp-check

Off/on decode (fresh server per run; only --glp/--glp-alpha differ):

V49=$(ls ~/.cache/huggingface/hub/models--msuiche--Qwen3.8-27B-abliterated-cyber-GLP-49/snapshots/*/Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf)

# off
./zig-out/native/bin/tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit \
  --context 2048 --port 8123 --host 127.0.0.1 --name bench --no-update-check

# on
./zig-out/native/bin/tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit \
  --glp "$V49" --glp-alpha 1.0 \
  --context 2048 --port 8123 --host 127.0.0.1 --name bench --no-update-check

# for each server, once it is listening:
python3 tools/bench_openai.py http://127.0.0.1:8123 Qwen3.8-27B-MLX-4bit \
  --tokens 64 --reps 5 --temperatures 0 --output /tmp/bench.json

Nested-config fixture:

import glob, json, os, shutil
snap = glob.glob(os.path.expanduser(
    "~/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots/*/"))[0]
out = "/tmp/tf-glp-nested"
shutil.rmtree(out, ignore_errors=True)
os.makedirs(out)
for name in os.listdir(snap):
    os.symlink(os.path.realpath(os.path.join(snap, name)), os.path.join(out, name))
d = json.load(open(os.path.join(snap, "config.json")))
d.pop("num_hidden_layers", None)
d.pop("hidden_size", None)
tc = d.setdefault("text_config", {})
tc["num_hidden_layers"] = 32
tc["hidden_size"] = 5120
json.dump(d, open(os.path.join(out, "config.json"), "w"), indent=2)

Throwaway NaN-vector generator (zig/src/core/tmp_glp_bad.zig, deleted after the run):

const std = @import("std");
const glp = @import("glp.zig");
const gguf = @import("glp_container.zig");

const width = 5120;

pub fn main(init: std.process.Init) !void {
    const a = init.arena.allocator();
    const args = try init.minimal.args.toSlice(a);
    if (args.len != 3) return error.Usage;
    const kind = args[1];
    const out_path = args[2];

    var dir = try a.alloc(f32, width);
    for (dir) |*v| v.* = 0;
    dir[0] = 1;

    if (std.mem.eql(u8, kind, "deep")) {
        const layers = [_]glp.Layer{.{ .id = 64, .dir = dir }};
        const bytes = try glp.write(a, .{
            .mode = .project,
            .alpha = 1,
            .hook_point = glp.residual_site,
            .derived_at = null,
            .base_model = "TensorFold/Qwen3.8-27B-MLX-4bit",
            .base_revision = "",
            .method = "test",
            .contrast = "",
            .layers = &layers,
        });
        try std.Io.Dir.cwd().writeFile(init.io, .{ .sub_path = out_path, .data = bytes });
        return;
    }
    if (std.mem.eql(u8, kind, "nan-add")) {
        var raw = try a.alloc(u8, width * 4);
        std.mem.writeInt(u32, raw[0..4], @bitCast(std.math.nan(f32)), .little);
        @memset(raw[4..], 0);
        const entries = [_]gguf.Entry{
            .{ .key = "general.architecture", .value = .{ .string = "controlvector" } },
            .{ .key = "glp.spec_version", .value = .{ .u32v = 1 } },
            .{ .key = "glp.mode", .value = .{ .string = "add" } },
            .{ .key = "glp.rank", .value = .{ .u32v = 1 } },
            .{ .key = "glp.hook_point", .value = .{ .string = glp.residual_site } },
            .{ .key = "glp.alpha_default", .value = .{ .f32v = 1 } },
            .{ .key = "glp.base_model", .value = .{ .string = "TensorFold/Qwen3.8-27B-MLX-4bit" } },
        };
        const dims = [_]u64{width};
        const tensors = [_]gguf.OutTensor{.{ .name = "direction.10", .dims = &dims, .kind = gguf.f32_type, .data = raw }};
        const bytes = try gguf.write(a, &entries, &tensors);
        try std.Io.Dir.cwd().writeFile(init.io, .{ .sub_path = out_path, .data = bytes });
        return;
    }
    return error.UnknownKind;
}
zig run zig/src/core/tmp_glp_bad.zig -- nan-add /tmp/glp-nan-add.gguf
rm -f zig/src/core/tmp_glp_bad.zig

CUDA build and container run:

zig build native -Dtarget=aarch64-linux-gnu -Doptimize=ReleaseSafe
mkdir -p "$HOME/tf-glp-cuda/run/model" "$HOME/tf-glp-cuda/run/vec"
cp zig-out/native/bin/tensorfold-native "$HOME/tf-glp-cuda/tensorfold-native-cuda-aarch64"
# rebuild the Metal binary afterwards, since the cross-build writes the same path:
zig build native

# populate run/model with the checkpoint's config.json and tokenizer files (no weights),
# and run/vec with the 49-layer vector above.
python3 - <<'PY'
import glob, os, shutil
home = os.path.expanduser("~")
snap = glob.glob(home + "/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots/*/")[0]
model = home + "/tf-glp-cuda/run/model"
for name in ("config.json", "generation_config.json", "tokenizer.json", "tokenizer_config.json",
             "vocab.json", "chat_template.jinja", "preprocessor_config.json",
             "processor_config.json", "model.safetensors.index.json"):
    src = os.path.join(snap, name)
    if os.path.exists(src):
        shutil.copy(os.path.realpath(src), os.path.join(model, name))
vec = glob.glob(home + "/.cache/huggingface/hub/models--msuiche--Qwen3.8-27B-abliterated-cyber-GLP-49/snapshots/*/Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf")[0]
shutil.copy(os.path.realpath(vec), home + "/tf-glp-cuda/run/vec/vector.gguf")
PY

podman run --rm --platform linux/arm64 \
  -v "$HOME/tf-glp-cuda/tensorfold-native-cuda-aarch64:/tf:ro" \
  -v "$HOME/tf-glp-cuda/run:/run:ro" \
  docker.io/library/debian:bookworm-slim \
  /tf serve /run/model --glp /run/vec/vector.gguf --context 1024 --port 8193

@ashhart

ashhart commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG for both receipts. On-mode decode is within 1% of off, and all four refusals stop by name before load. A few things before it lands:

  1. It conflicts with main again, so please rebase.
  2. Please remove the "The engine that steers" section from docs/control-vectors.md. Once this is merged, --glp ships in TensorFold itself, so the docs shouldn't point to another tap or release.
  3. In glp_publication_test.zig, please use a neutral contrast string in the sample metadata, in line with the scope we agreed.

We'll check the CUDA refusal ourselves on a GB10.

@BlivionIaG
BlivionIaG force-pushed the feature/glp-steering branch from efd421d to b721c88 Compare November 4, 2026 07:15
BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 4, 2026
Review changes from ashhart#1835:
- the app's settings panel, IPC, preload and locale packs move to
  feature/glp-steering-app, so app changes go through the app's review
- glp.read refuses a file that names one layer twice (direction.5 and
  direction.05 both parse as 5), with a test
- the serve refuses a layer id past the checkpoint's num_hidden_layers at
  startup, before the model loads
- the GLP container is read by the engine's own GGUF reader; its writer stays
  in glp.zig, so there is one GGUF reader
- the deriver, the matched prompt sets, the refusal-measurement gate and their
  receipt are removed; a vector file is supplied by the operator
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Rebased onto 8db1ee1a3; head is b721c88ae.

  • Removed the external-build section from docs/control-vectors.md. The guide now documents only the in-tree --glp surface and points at no tap or release.
  • Neutralized the sample metadata: the publication sample no longer names refusal or harm, and the contrast sample reads matched_prompts_unpaired.
  • Resolved the rebase conflicts: metal_main.zig keeps the attention and steering imports; engine_api.zig keeps upstream's engine_request_controls.zig, so the branch no longer duplicates those request shapes.

zig build test-glp, zig build test-glp-serve and zig build tf-qwen27-served-check pass locally. CI is running on the new head. The CUDA refusal is yours to check on the GB10.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

release-paths caught one rebase follow-on: the contract imported bf16_round.zig by path, and the release build also gives that file its own module. Fixed in 9c90e0956 by importing the module. Head is now 9c90e0956; test-glp, test-glp-serve, tf-qwen27-served-check and check-release-paths pass locally.

BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 4, 2026
Review changes from ashhart#1835:
- the app's settings panel, IPC, preload and locale packs move to
  feature/glp-steering-app, so app changes go through the app's review
- glp.read refuses a file that names one layer twice (direction.5 and
  direction.05 both parse as 5), with a test
- the serve refuses a layer id past the checkpoint's num_hidden_layers at
  startup, before the model loads
- the GLP container is read by the engine's own GGUF reader; its writer stays
  in glp.zig, so there is one GGUF reader
- the deriver, the matched prompt sets, the refusal-measurement gate and their
  receipt are removed; a vector file is supplied by the operator
The receipt said the device gates run the kernel on the GPU and compare it
with float64. Now it says what they reported: the steering kernel passes
gates 1 to 4 in both modes it serves, the two controls are red as planted,
and the limits gate passes with the launch's own declared threads and
threadgroup memory.
…lags with it

The control vector's two flags pushed the serve flag table's file over the
rule's 600 lines, and it was 596 before they arrived, so the next flag on
main would have done the same.

The shape of a flag is now a module of its own, which lets the control
vector's flags live in the module that owns the control vector instead of in
the table. The table keeps every other flag where it was, and the parser
and ``capabilities`` see the same list as before.

The file is 588 lines, and the lean rule reports no finding over the base.
…t CI did

The receipt still carried the counts from before the merge with main: the
number of new Zig tests, the step and test totals of an older build, the old
pinned base, and a claim that every check was green.

The numbers are now the ones this head produces: 66 new Zig tests, 552 steps
succeeded and 161 of 162 tests passed, and the lean count against the base the
branch merged, b30d5b9.

The CI line says what happened. The format check, the code, the tests and the
tools are green on Linux, Windows and macOS. The app's macOS packaging job is
cancelled by the runner pool, four attempts across two runs, and the base
tree's own runs are cancelled the same way in the same window, so that line no
longer claims it is green. The job's four steps were run here under the job's
own pins instead, and they pass.
…ow gated

The steering kernel replaced the norm kernel, so the vector costs one dispatch
per steered layer and not two. What was left was measured: against the same
build and the same server without a vector, the cost is under a percent of a
decoded token and under half a percent of a prefill, which is under what this
machine resolves between runs. The first reading looked like three percent,
but a second baseline run moved by six, so the honest reading is that the cost
is below the noise.

Two things in the kernel were still worth fixing.

The add mode does not use the dot, so the kernel now skips it: the reduction
the project mode needs is not run, and the mode matches its spec, which
declares no sums for it. The branch is on a uniform value, the same way the
kernel already returns early on the rows past the end.

The direction is read once per row and kept in registers, so the dot and the
projection read the same stretch of it instead of reading it twice.

The device suite now gates the fused kernel's normalised row against the
family's own plain norm kernel: the row the fused kernel projected, normalised
by the plain kernel, has to be the same row, element for element. The gates
compared the projected row; nothing compared the values the model consumes, so
a wrong sum of squares would have passed unnoticed. A fault planted in the
normalisation alone is caught, and the suite says so.
Settings -> AI & Models -> Engine advanced gains a toggle, a file picker and a
strength. The engine's own two flags carry the choice at its next start, so the
app does not invent a way to apply a vector: it passes ``--glp <file>`` and
``--glp-alpha <alpha>`` and the engine says what it is holding when it starts.

The path comes only from the OS picker. The renderer can turn the vector on and
set its strength, and it cannot name a file: the picker is the one place a path
enters the app, the same rule the model-folder picker already follows. The
audit log records the state and the strength and never the path, because the
file may sit in a directory the operator keeps private.

The app only offers the control when the engine in use can take it. It reads
``capabilities --json`` and looks for ``--glp`` among the serve flags; when the
engine does not offer it, the panel says so rather than failing at the next
start. The packaged app pins its engine by sha256 and the pinned release does
not carry the flag, so a shipped build says the engine cannot take one, and the
panel stays inert until the engine is re-pinned. The dev path, which takes an
engine built from this branch, works today.

The change and the model restart happen together, so a refused start leaves the
old choice in place. The file, the strength and the state are kept in
``engine/preferences.json``, so the choice survives a restart of the app. A
field the caller leaves out keeps its current value, so changing the strength
does not switch the vector off and switching it off does not reset the
strength.

The panel says plainly what a vector does: it steers the model, so the model
answers differently, and it is a steering direction rather than a safety
setting. It is off until it is turned on.

The new strings are registered and translated into all twelve locales, the
language index is rebuilt, the typecheck is clean, and the app's own suite is
856 of 856. The packaged macOS build carries the panel and passes the
repository's smoke test.
…y in force

Choosing a file while a vector was in force wrote the new path to the saved
preferences and left the running engine holding the old one, so the panel and
the engine named different files until the next start.

The pick now goes through the same apply-and-restart path as the other changes,
so the engine and the setting never disagree, and a refused start leaves the
old file in place. A file chosen while no vector is in force is still only
remembered, and still does not restart the model.
The branch was rebased onto the upstream main at 8abd450, 166 commits past
the tree it merged before. The engine is upstream's 1.1.8 now, the test suite
grew with it, and the lean baseline moved with it.

The receipt says 1.1.8, 584 steps and 8214 of 8259 tests, and 425 lean
problems against 425 on the base the branch sits on.

The measurements did not move. The published vector on the derivation set
still leaves 19 of 30 harmful prompts refused and 6 of 30 benign ones, with 60
of 60 coherent replies, which is what the earlier tree gave as well.
The upstream tree grew by four lines since this branch merged it, and with the
control vector's own three the API's file crossed the lean rule's 600 lines.
The rule compares against the base branch, so the file that was 591 lines
upstream and 602 here read as a problem this branch had added, when it had
only inherited the crossing.

The control vector's shape moves into the module that owns the container, and
the shapes a request carries move into a module of their own, the way the
vision and speech types already sit beside the API. Every name stays where it
was: the API re-exports them, so no caller moved.

The file is 565 lines, and the lean rule reports no finding over the base.
The test totals are the ones a cold build of the rebased tree produces, and
the lean count is the one after the engine API's split: 424 problems on this
branch against 425 on the base it sits on.

Co-Authored-By: BlivionIaG <15323295+BlivionIaG@users.noreply.github.com>
The server compares the checkpoint a vector names with the one it serves: the
served model name has to be the vector's model name or start with it. The
same weights re-released under another name were refused with it, although
the directions are exactly right for them.

tf-glp-check rewrite now takes --repo and --revision, so a file can be
re-stamped for that name. The directions stay the same bytes and the digest
over them does not move; the test asserts both, and that the binding moves
with the name.
The GLP container format, its reference implementation and its validator are
Matt Suiche's (the weightless plugin, MIT). The module now says so where a
reader of the source meets it first.

The receipt now names the base this branch sits on (7b65d60) and the lean
count against it (424 against 424, zero new).
The limits pass picked the source for a launch by its label, and the
steering launch's label starts with "qwen27 ", so the family branch took
it and handed the glue source to the compiler: the kernel was never
built, and the gate reported the launch as missing from the library.

The same line also overwrote the source of the four "metal pair " launches
upstream owns, so their pipelines went missing the same way.

The 27B's source now comes from one place, by what the label says, and
the steering kernel is specialized at the 27B's width as the engine
builds it. On this GPU: 2 kernels pass gates 1-4, 2 controls red as
planted, 0 limit problems.
The module header's credit made the file's doc two comment lines, and the new helper's doc ran past the column limit: two findings the base branch does not have. The credit stays, in one line, and the helper's doc says the same thing shorter.

Lean: 424 problems, 424 on the base, 0 new.
The family's attention gained a split argument in the upstream range this
branch was rebased onto: the kernel that takes the head count at run time
is the one a split run uses. The producer's own call still had the old
arity, so the dev-tools build stopped on it.

The producer runs the model on one Mac, where the reduce is null, so the
call passes the same rule the family's own round does.
The receipt still carried the engine version, the test counts and the gate
transcript of the tree before the rebase, and it did not say what the
rebased producer writes: the same directions, the same digest.

It now carries the engine's own version, the rebased suite's counts, the
gate's transcript as it runs today, the re-derivation's digest and the
measurement on the same weights under their other name.
The entry sat in two released sections, the same line twice: once under
1.1.9 and once under 1.1.7, and in neither of them the release it
describes. It is one entry under Unreleased now, and it says whose format
it implements.
The suite grew with the upstream commits the branch now sits on: 605
steps and 9348 tests, 55 of them skipped.
Review changes from ashhart#1835:
- the app's settings panel, IPC, preload and locale packs move to
  feature/glp-steering-app, so app changes go through the app's review
- glp.read refuses a file that names one layer twice (direction.5 and
  direction.05 both parse as 5), with a test
- the serve refuses a layer id past the checkpoint's num_hidden_layers at
  startup, before the model loads
- the GLP container is read by the engine's own GGUF reader; its writer stays
  in glp.zig, so there is one GGUF reader
- the deriver, the matched prompt sets, the refusal-measurement gate and their
  receipt are removed; a vector file is supplied by the operator
Zig places a source file in one module; engine_api reaches glp.zig, and core
reaches gguf.zig, so glp importing the engine's reader made gguf.zig a member
of both modules and every target that links them failed to compile.

The GLP container's reader is now glp_container.zig, named for its scope as
the review allowed. It reads and writes the GGUF v3 subset a vector needs;
the engine's gguf.zig stays the engine's reader. A repeated layer id is still
refused by glp.read.
CI's lean gate counted three new problems: two comment lines and a file past
600 lines. The steering helpers (fit check, plan, startup line) move to
qwen27_steer_served.zig, qwen27_served.zig returns to 521 lines, and the two
comments are one line each. lean_check: 424 problems, 424 on the base, 0 new.
… reader named

The startup depth check reads num_hidden_layers from text_config, falling back to the config root, with a nested-config test. read() refuses a direction holding a NaN or an infinity in either mode, with a test over both. The changelog and the guide name core/glp_container.zig, the reader the file actually goes through.
The publication sample and contrast strings name no refusal or harmful set.
check-release-paths compiles the contract module with bf16_round as its own module; importing the file by path put the same file in two modules. Import the module instead.
The maintainer's CHANGELOG credits people by GitHub handle. Name the format as the weightless project's, with its author's handle, in the CHANGELOG line and the container reader's header. The GLP reader's header follows the same wording.
metal_main.zig was 605 lines after the rebase, past the 600-line rule.
The suite now takes its entries and runs its normalisation check in
metal_qwen27_steer.zig, so metal_main.zig carries three lines for it,
as the attention rows do.

The check prepares and launches its own rows at each served shape:
since the gates free a kernel's buffers once they are done with it,
reading the rows runAll left behind used released buffers, and the
contract run segfaulted.
@BlivionIaG
BlivionIaG force-pushed the feature/glp-steering branch from ed8c3e1 to 405e53f Compare November 5, 2026 21:34
@BlivionIaG

BlivionIaG commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (34ea8cb7d), since the branch had started to conflict; head is 405e53f09.

  • Conflicts: build.zig's test list (both kept: main's stage.zig and the GLP steps) and the CHANGELOG.
  • CHANGELOG: the control-vector line was under 1.2.1, which has shipped. It now sits under Unreleased with (#1835, fixes #1834).
  • Code: no source changes beyond the rebase.

Local, Apple M4 Pro:

  • zig build test exits 0.
  • zig build test-glp test-glp-serve test-glp-steer test-steer check-release-paths tf-qwen27-served-check test-kernel-contract tf-kernel-contract native passes 173/173 steps and 187/187 tests.
  • tf-kernel-contract --only "qwen27 steer": 2 of 2 pass gates 1-4, both controls red, PASS normalised.
  • lean_check: 425 problems, 425 on main, 0 new.
  • Since main changed the 27B's Metal scheduling (91e4120b2), I also ran a server with the published 49-layer vector (project, alpha 1.0). It starts with the control vector in force line and answers.

Every commit has the noreply author and committer. CI is running.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GLP control vector support for Qwen3.8-27B on Metal

2 participants