From 8fc0ba26331f5d0f91b734ae353da3624fdfbecb Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Fri, 18 Sep 2026 12:50:36 +1000 Subject: [PATCH 1/3] docs(binaries): archive the payload, not just the binary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BINARIES.md has said since 2026-08-30 that "a binary is not a rollback on its own" — the MLX library is dlopen'd from build/lib/ollama and the binary does not carry it. The "Adding one" procedure never acted on that: it archived the binary alone. So every row in the table was a one-way door in practice. The payload for each archived build existed in exactly one place, the checkout's build/lib/ollama, and was destroyed by the next build that moved a pin. That became concrete today. Staging 0.34.0-maxusai-8a7ba949 on :11435 meant overwriting the only copy of the 0.33.2 payload; rolling back would have been a ~25 minute cold rebuild at 2b95b4a5, itself possible only because Xcode's separately-downloadable Metal toolchain happened to get installed this morning. Both pairings are now archived — payload-0.33.2-maxusai-2b95b4a5/ and payload-0.34.0-maxusai-8a7ba949/ beside the binaries — so rolling between them is a copy. The procedure is updated to archive both halves, and to archive the OUTGOING pairing before overwriting it. Also adds the 0.34.0-maxusai-8a7ba949 row. It is recorded as built and staged, NOT serving: preflight PASS 19/12 against the mlx-metal-0-34-0 profile (#306), and #301's whitespace bound confirmed behaviourally on real documents across the MLX pin move, but the :11435 kickstart has not run. The 0.33.2 row loses "current" and says what it actually is. Co-Authored-By: Claude Opus 5 --- docs/maxusai/vision-suite/BINARIES.md | 34 ++++++++++++++++++++------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/docs/maxusai/vision-suite/BINARIES.md b/docs/maxusai/vision-suite/BINARIES.md index 4d82a8816d1..6d6686f4983 100644 --- a/docs/maxusai/vision-suite/BINARIES.md +++ b/docs/maxusai/vision-suite/BINARIES.md @@ -18,7 +18,8 @@ A host can be rebuilt from it; a binary sitting only in `/tmp` cannot. | `0.32.14-maxusai-9594f81e` | [`9594f81e`](https://github.com/MaxusAI/ollama/commit/9594f81e) | llama.cpp **b10434** | v0.32.14 pin | `711d4ad126773ddf…` | the `mlx-metal-0-32-14` preflight baseline | | `0.32.14-maxusai-c82b0464` | [`c82b0464`](https://github.com/MaxusAI/ollama/commit/c82b0464) | llama.cpp **b10434** | v0.32.14 pin | `03f9f9289dbaba1b…` | last pre-0.33.0 deploy on :11435 (2026-08-22 → 27); preflight PASS 2026-08-22; rollback target (Go half) for the 0.33.0 swap | | `0.33.0-maxusai-21cfe88e` | [`21cfe88e`](https://github.com/MaxusAI/ollama/commit/21cfe88e) | llama.cpp **b10488** | 27fec909 pin | `6ab35025981be587…` | provenance for the `mlx-metal-0-33-0` profile measurement and the 27fec909 golden recalibration; rollback target (Go half) for the 0.33.2 swap | -| `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | current; the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin | +| `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | serving on :11435 until the 0.34.0 swap; the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin | +| `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **Built and staged, NOT yet serving** — the `:11435` swap is pending its kickstart | Full checksums: @@ -28,6 +29,7 @@ d807360e94e0e17ac346df9bef198b6a182ef2f47bff78a0e772f6d1d67bad72 ~/.ollama/bina 03f9f9289dbaba1bba2a5826a22e1aa85525e96c7e78942bf41828ff90908a71 ~/.ollama/binaries/ollama-0.32.14-maxusai-c82b0464 6ab35025981be587ff0a73f0b7ae007b608300defb46f176065c8f3f52d78139 ~/.ollama/binaries/ollama-0.33.0-maxusai-21cfe88e cd6033d6e72a430e0b96170adb3d7408c908edfdae392654faeaae46f31e9ee2 ~/.ollama/binaries/ollama-0.33.2-maxusai-2b95b4a5 +867ee5d6063d8f77be6d8278c51d18f49892f00aa80c38c51630227aa9f68a57 ~/.ollama/binaries/ollama-0.34.0-maxusai-8a7ba949 ``` ## What b10353 is the provenance for @@ -59,10 +61,19 @@ been rebuilt at MLX `c793734`, reported `MLX version=0.32.1-37-gc793734` — old Go, new MLX, a pairing nothing was ever measured on. `mlx_payload_pin` failed it against the profile's `27fec909`, which is what that check is for. -So "rollback target" below means **the Go half**. A true rollback restores the -payload too: check out the commit and rebuild (`CLEAN_DEPS=1`, below) so -`build/lib/ollama` holds that pin's MLX again, or verify what you actually got by -reading the engine-init line before trusting the restored server. +So "rollback target" meant **the Go half** only, and for the rows above +`0.33.2` it still does: a true rollback to those needs a checkout and rebuild +(`CLEAN_DEPS=1`, below) so `build/lib/ollama` holds that pin's MLX again. + +**From 2026-09-18 the payload is archived too.** `~/.ollama/binaries/` now holds +`payload-0.33.2-maxusai-2b95b4a5/` and `payload-0.34.0-maxusai-8a7ba949/` beside +the binaries, so rolling between those two is a copy rather than a ~25 minute cold +rebuild — which on this host also depends on Xcode's separately-downloadable Metal +toolchain being installed. Archiving only the binary made every earlier row a +one-way door in practice; see "Adding one". + +Either way, verify what you actually got by reading the engine-init line before +trusting a restored server. ## Rebuilding one @@ -85,12 +96,19 @@ gates on. ## Adding one -Archive the binary whenever a payload pin moves, before the new build replaces -it on the host: +Archive **both halves** whenever a payload pin moves, before the new build +replaces them on the host. The binary alone is not enough — it does not carry +its MLX (above), so a binary-only archive cannot restore a working pairing: ```sh -cp -p /tmp/ollama-vs ~/.ollama/binaries/ollama-$(OLLAMA_HOST=127.0.0.1:1 /tmp/ollama-vs --version 2>&1 \ +VER=$(OLLAMA_HOST=127.0.0.1:1 /tmp/ollama-vs --version 2>&1 \ | sed -n 's/^Warning: client version is //p') +cp -p /tmp/ollama-vs ~/.ollama/binaries/ollama-$VER +mkdir -p ~/.ollama/binaries/payload-$VER +cp -R build/lib/ollama/. ~/.ollama/binaries/payload-$VER/ ``` +Archive the OUTGOING pairing the same way before overwriting it, because +`build/lib/ollama` is the only copy of the payload the running server resolves. + Then add a row here with its sha256 and what it is the provenance for. From 9fc22fb5a026f3cc120e7203453d0dbfcc657259 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Fri, 18 Sep 2026 13:19:26 +1000 Subject: [PATCH 2/3] docs(binaries): 8a7ba949 is now serving on :11435 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The kickstart ran at 13:17. Verified three ways rather than by version alone: :11435 reports 0.34.0-maxusai-8a7ba949 on a new pid, a live MLX load emitted "MLX version"=0.32.2-61-gd9add9d, and the dlopen path resolved /opt/github/MaxusAI/ollama/build/lib/ollama — so the Go and MLX halves are the pairing that was measured, not a skew. The 0.33.2 row becomes the rollback target and now says what is new about it: both halves are archived, so that rollback is a copy. Co-Authored-By: Claude Opus 5 --- docs/maxusai/vision-suite/BINARIES.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/maxusai/vision-suite/BINARIES.md b/docs/maxusai/vision-suite/BINARIES.md index 6d6686f4983..ca1ef6fef7a 100644 --- a/docs/maxusai/vision-suite/BINARIES.md +++ b/docs/maxusai/vision-suite/BINARIES.md @@ -18,8 +18,8 @@ A host can be rebuilt from it; a binary sitting only in `/tmp` cannot. | `0.32.14-maxusai-9594f81e` | [`9594f81e`](https://github.com/MaxusAI/ollama/commit/9594f81e) | llama.cpp **b10434** | v0.32.14 pin | `711d4ad126773ddf…` | the `mlx-metal-0-32-14` preflight baseline | | `0.32.14-maxusai-c82b0464` | [`c82b0464`](https://github.com/MaxusAI/ollama/commit/c82b0464) | llama.cpp **b10434** | v0.32.14 pin | `03f9f9289dbaba1b…` | last pre-0.33.0 deploy on :11435 (2026-08-22 → 27); preflight PASS 2026-08-22; rollback target (Go half) for the 0.33.0 swap | | `0.33.0-maxusai-21cfe88e` | [`21cfe88e`](https://github.com/MaxusAI/ollama/commit/21cfe88e) | llama.cpp **b10488** | 27fec909 pin | `6ab35025981be587…` | provenance for the `mlx-metal-0-33-0` profile measurement and the 27fec909 golden recalibration; rollback target (Go half) for the 0.33.2 swap | -| `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | serving on :11435 until the 0.34.0 swap; the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin | -| `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **Built and staged, NOT yet serving** — the `:11435` swap is pending its kickstart | +| `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | served :11435 2026-08-30 → 09-18; rollback target (BOTH halves archived); the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin | +| `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **current** — deployed to `:11435` 2026-09-18 13:17, verified by version, a live MLX load reporting `0.32.2-61-gd9add9d`, and the dlopen path resolving the repo payload | Full checksums: From 8fa7034ceb93f409da17b872601e51c69e63ff30 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Fri, 18 Sep 2026 14:13:07 +1000 Subject: [PATCH 3/3] docs(binaries): record golden parity at the d9add9d1 pin The 0.33.2 row carries "goldens NOT recalibrated at this MLX pin" as a caution. It turns out they did not need to be. Golden parity passes on this build for 31b, 26b and 12b, with max sampled element deltas of 0.109, 0.051 and 0.063. The 12b golden is the striking one: generated 2026-08-07 and never recalibrated, it is three MLX pins behind (27fec909, c793734e, d9add9d1) and still matches to four decimal places on mean, std and norm_mean. The 31b and 26b goldens were recalibrated once at 27fec909 (d53d33a57) and have held through both moves since. So the vision encoder path has been numerically stable across every pin move this fork has made, and this is the first row that can say its goldens were checked rather than assumed. Co-Authored-By: Claude Opus 5 --- docs/maxusai/vision-suite/BINARIES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/maxusai/vision-suite/BINARIES.md b/docs/maxusai/vision-suite/BINARIES.md index ca1ef6fef7a..13fe0d15fdc 100644 --- a/docs/maxusai/vision-suite/BINARIES.md +++ b/docs/maxusai/vision-suite/BINARIES.md @@ -19,7 +19,7 @@ A host can be rebuilt from it; a binary sitting only in `/tmp` cannot. | `0.32.14-maxusai-c82b0464` | [`c82b0464`](https://github.com/MaxusAI/ollama/commit/c82b0464) | llama.cpp **b10434** | v0.32.14 pin | `03f9f9289dbaba1b…` | last pre-0.33.0 deploy on :11435 (2026-08-22 → 27); preflight PASS 2026-08-22; rollback target (Go half) for the 0.33.0 swap | | `0.33.0-maxusai-21cfe88e` | [`21cfe88e`](https://github.com/MaxusAI/ollama/commit/21cfe88e) | llama.cpp **b10488** | 27fec909 pin | `6ab35025981be587…` | provenance for the `mlx-metal-0-33-0` profile measurement and the 27fec909 golden recalibration; rollback target (Go half) for the 0.33.2 swap | | `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | served :11435 2026-08-30 → 09-18; rollback target (BOTH halves archived); the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin | -| `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **current** — deployed to `:11435` 2026-09-18 13:17, verified by version, a live MLX load reporting `0.32.2-61-gd9add9d`, and the dlopen path resolving the repo payload | +| `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **current** — deployed to `:11435` 2026-09-18 13:17, verified by version, a live MLX load reporting `0.32.2-61-gd9add9d`, and the dlopen path resolving the repo payload. Native gates pass: `x/mlxrunner` and all subpackages, and vision golden parity on 31b/26b/12b (max element delta 0.109 / 0.051 / 0.063) — the first build whose goldens were CHECKED at its MLX pin | Full checksums: