From 8cf73f4248250ff5dd3abc33ed9e81745cb22fce Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Wed, 14 Oct 2026 13:03:47 +1000 Subject: [PATCH 1/3] gate: 0.40.2-dynres.1 promoted on gfx1151, the payload moves to b11351 ollama-rocm moved from 0.35.0-dynres-0-g043f441 to 0.40.2-dynres.1-0-g3960244 at 12:47 on 2026-10-14, after the #477 campaign, on the maintainer's word. Post-deploy preflight on production: PASS=27 SKIP=5, equal to the release image's run in every row but the image tag; the rocm7 matrix row is now production's own run. The decision records v0.40.2's store migration (46 GB of /opt at the deploy, no switch; v0.40.3 pauses it), the rollback, and the clause outcomes. Co-Authored-By: Claude Opus 5.5 --- README.md | 11 +- docs/maxusai/amd-upgrade-gate.md | 92 ++- ...02-dynres1-prod-g3960244.host-profile.json | 102 ++++ ...ight-rocm7-0402-dynres1-prod-g3960244.json | 575 ++++++++++++++++++ 4 files changed, 770 insertions(+), 10 deletions(-) create mode 100644 docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0402-dynres1-prod-g3960244.host-profile.json create mode 100644 docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0402-dynres1-prod-g3960244.json diff --git a/README.md b/README.md index 95df4c04a03..e381525b3b5 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ > |---|---|---|---| > | Apple Silicon, mlx-metal (`:11435`) | `0.40.2-dynres.1-17-g75c6d35`, the same build as the CUDA host's: main `75c6d350`, a Go-only build on the point tag's payload | 2026-10-14 10:13 | [#486](https://github.com/MaxusAI/ollama/pull/486), this record | > | CUDA host, ai-server (`:11497`) | `0.40.2-dynres.1-17-g75c6d35`: the v0.40.3 fold ([#488](https://github.com/MaxusAI/ollama/pull/488)) on the point tag plus [#487](https://github.com/MaxusAI/ollama/pull/487), a Go-only build of main `75c6d350` (image `main-75c6d350`) | 2026-10-14 07:53 | [#487](https://github.com/MaxusAI/ollama/pull/487), [#488](https://github.com/MaxusAI/ollama/pull/488), [#492](https://github.com/MaxusAI/ollama/pull/492) | -> | AMD gfx1151 | `v0.35.0-dynres`; the `v0.40.2-dynres.1` release build passes `rocm7-0-40-2`, and its deploy is scheduled after the [#477](https://github.com/MaxusAI/ollama/pull/477) campaign | 2026-10-01 | [#485](https://github.com/MaxusAI/ollama/pull/485) | +> | AMD gfx1151 (`:11434`) | `0.40.2-dynres.1-0-g3960244`, a build at the point tag (image `0.40.2-dynres.1-rocm724-main-39602449`); preflight passes `rocm7-0-40-2` on the release image and on production | 2026-10-14 12:47 | [#485](https://github.com/MaxusAI/ollama/pull/485), [the 2026-10-14 decision](docs/maxusai/amd-upgrade-gate.md#decision-2026-10-14--0402-dynres1-promoted-the-payload-moves-to-b11351) | > | H100 (CUDA) | `0.35.0-dynres-15-ga657392`, a build of `main` past the v0.35.0 tag | 2026-10-03 | [v0.35.0 record](docs/maxusai/tasks/upstream-sync-0.35.0.md) | > > The CUDA host serves [#487](https://github.com/MaxusAI/ollama/pull/487), merged after the point tag: without it, an MLX @@ -78,14 +78,16 @@ For v0.40.2-dynres.1 the table below is the verbatim output of release_matrix.py --version 0.40.2-dynres.1 \ runs/preflight-mlx-metal-0403-prod-g75c6d35.json \ - runs/preflight-rocm7-0402-dynres1-release-g3960244.json \ + runs/preflight-rocm7-0402-dynres1-prod-g3960244.json \ runs/preflight-cuda-0403-prod-g75c6d35.json \ runs/preflight-mlx-cuda-0403-prod-g75c6d35.json The runs are listed by name because runs/ also holds #487's A/B runs and canaries, which are not release runs. The CUDA and Apple Silicon hosts both serve 0.40.2-dynres.1-17-g75c6d35 (main 75c6d350: the v0.40.3 fold on the point tag plus #487, Go only); their cuda, mlx-cuda and mlx-metal rows are production's own runs on that build. The cuda row is the idle re-run (preflight-cuda-0403-prod-g75c6d35.json); the first post-deploy run, -contended.json, failed gemma4's ladder and - pinned budget by a constant +573 while a client's gemma4 q8 shared the GPU and capped image_max_tokens to n_ubatch. --> + pinned budget by a constant +573 while a client's gemma4 q8 shared the GPU and capped image_max_tokens to n_ubatch. + gfx1151 serves 0.40.2-dynres.1-0-g3960244 since 2026-10-14; its rocm7 row is production's own run on that build, + equal to the release image's run (-release-g3960244.json) in every row but the image tag. --> | surface | Build identity | MLX build pin | Toolchain pin | M5 tensor path | Image size ladder | Aspect ladder | Pinned image budget | thinking on/off | Output quality | fp16 overflow canary | Runner isolation | measured on | |---|---|---|---|---|---|---|---|---|---|---|---|---| @@ -99,8 +101,7 @@ Generated by `release_matrix.py` from recorded preflight runs. A surface with no run for this release reads *not run* — absence is shown, never assumed green. A group is reported at its weakest check, so one skipped probe does not read as a pass. -`cuda` and `mlx-metal` here are the deployed serving surfaces. `rocm7` is the release build, measured on a bench -container; production runs `0.35.0` until its scheduled deploy. `apple-silicon-mlx` is the +`cuda`, `mlx-metal` and `rocm7` here are the deployed serving surfaces, each measured on production itself. `apple-silicon-mlx` is the deprecated alias for `mlx-metal` and will disappear from the generator with it. diff --git a/docs/maxusai/amd-upgrade-gate.md b/docs/maxusai/amd-upgrade-gate.md index a26b24baefb..7906097f70b 100644 --- a/docs/maxusai/amd-upgrade-gate.md +++ b/docs/maxusai/amd-upgrade-gate.md @@ -23,13 +23,13 @@ back to `0.32.1-gemma4budget-85ebcb79`. | | | |---|---| -| Deployed image | `maxusai-ollama:0.35.0-rocm724-main-043f441a` (promoted 2026-10-01 21:38) | -| Deployed version | `0.35.0-dynres-0-g043f441`, a build of `main` at the `v0.35.0-dynres` tag (ADR 0032); see the [2026-10-01 decision](#decision-2026-10-01--0350-promoted-a-go-only-fold) | +| Deployed image | `maxusai-ollama:0.40.2-dynres.1-rocm724-main-39602449` (promoted 2026-10-14 12:47) | +| Deployed version | `0.40.2-dynres.1-0-g3960244`, a build of `main` at the `v0.40.2-dynres.1` point tag (ADR 0032); see the [2026-10-14 decision](#decision-2026-10-14--0402-dynres1-promoted-the-payload-moves-to-b11351) | | Build type | `Dockerfile.rocm` through `scripts/build_rocm.sh` (`ROCM_TOOLCHAIN=rocm7 AMDGPU_TARGETS=gfx1151`), on `rocm/dev-ubuntu-24.04:7.2.4-complete` ([ADR 0042](adr/0042-rocm-images-build-on-ubuntu-rocm-images.md)), as every production image since 0.34.3. It is gfx1151 only. Compat **001 + 002 + 004 + 005 + 801 + 802 + 805 + 903 + 908**: 908 reverts the device half of `ce8caa6e6`, a CUDA flash-attention tiling, and changes no gfx1151 kernel. 805 ([ADR 0041](adr/0041-gfx1151-vision-prefill-is-bandwidth-bound.md)) is a diagnostic switch, `GGML_CUDA_FATTN_FORCE_TILE`, inert unless set; on gfx1151 it adds 64 bytes of host code and no GPU code. 802, the LM-graph node meter, is inert unless `OLLAMA_LM_NODE_STATS` is set | | KV cache | **f16** since 2026-09-26 07:23 ([ADR 0005](adr/0005-per-model-kv-cache-type.md)). From the 2026-08-08 cutover until then it was `q8_0` by accident; see the [2026-09-26 decision](#decision-2026-09-26--productions-kv-cache-back-to-f16) | -| Think+format flow | **two-pass** (`OLLAMA_FORMAT_TWO_PASS=1`) since 2026-09-28, the maintainer's choice at the 0.34.4 deploy, kept at the 0.35.0 deploy. The default is the single pass; see the [2026-09-28 decision](#decision-2026-09-28--0344-promoted-with-the-two-pass-flow) | -| Payload | **b11081** (`161755f29`) since 2026-09-28, unchanged by 0.35.0; b10969 (`391fac164`) for 0.34.2 and 0.34.3. Compat 906 stays **retired**: upstream ships its own HIP `prop.integrated` revert | -| Previous image | `maxusai-ollama:0.34.4-rocm724-main-b43ee8e3` (`0.34.4-dynres-0-gb43ee8e`, b11081 + 908) — **retained for rollback** as the stopped container `ollama-rocm-0.34.4-b43ee8e3`, which runs f16 and two-pass. Before it, `0.34.3-rocm724-main-650f8fda` (b10969; the stopped container `ollama-rocm-0.34.3-650f8fda`, f16), `0.34.2-rocm724-main-f67b1aef` (the stopped container `ollama-rocm-0.34.2-f67b1aef`, recreated with f16 on 2026-09-28), `0.34.1-rocm724-main-16649e8c` (b10864 + 906) and `0.32.1-rocm-dynres-5d5b7a72` (b9888) | +| Think+format flow | **two-pass** (`OLLAMA_FORMAT_TWO_PASS=1`) since 2026-09-28, the maintainer's choice at the 0.34.4 deploy, kept at the 0.35.0 and 0.40.2-dynres.1 deploys. The default is the single pass; see the [2026-09-28 decision](#decision-2026-09-28--0344-promoted-with-the-two-pass-flow) | +| Payload | **b11351** (`631109b34`) since 2026-10-14; b11081 (`161755f29`) from 2026-09-28 for 0.34.4 and 0.35.0; b10969 (`391fac164`) for 0.34.2 and 0.34.3. Compat 906 stays **retired**: upstream ships its own HIP `prop.integrated` revert | +| Previous image | `maxusai-ollama:0.35.0-rocm724-main-043f441a` (`0.35.0-dynres-0-g043f441`, b11081) — **retained for rollback** as the stopped container `ollama-rocm-0.35.0-043f441a`, which runs f16 and two-pass. Before it, `0.34.4-rocm724-main-b43ee8e3` (b11081 + 908; the stopped container `ollama-rocm-0.34.4-b43ee8e3`, f16 and two-pass), `0.34.3-rocm724-main-650f8fda` (b10969; the stopped container `ollama-rocm-0.34.3-650f8fda`, f16), `0.34.2-rocm724-main-f67b1aef` (the stopped container `ollama-rocm-0.34.2-f67b1aef`, recreated with f16 on 2026-09-28), `0.34.1-rocm724-main-16649e8c` (b10864 + 906) and `0.32.1-rocm-dynres-5d5b7a72` (b9888) | | Superseded pin | `0.32.1-dynres-296eb020` recorded here until 2026-09-19; the host was in fact running `5d5b7a72`, so this row had drifted from the host it describes | | Blocked target | `0.32.5-gemma4budget-4259c191` (built, verified, **rolled back** 2026-07-31) — never unblocked; superseded, not cleared | | Host | Ryzen AI Max+ 395 / Radeon 8060S, **gfx1151**, ROCm, Linux | @@ -735,6 +735,88 @@ the 0.34.4 fold image's. So this promotion neither adds a loop nor removes one. **How the host was confirmed idle.** As for 0.34.4: no model was loaded, and there was no POST request in the two minutes before the swap. `deploy-0350.sh` (in the run directory) refuses to swap otherwise. +## Decision 2026-10-14 — 0.40.2-dynres.1 promoted, the payload moves to b11351 + +**Outcome: promoted.** `ollama-rocm` moved from `0.35.0-dynres-0-g043f441` to `0.40.2-dynres.1-0-g3960244` at 12:47 on +2026-10-14. The maintainer asked for it ("deploy the release image to production") and chose to run it after the #477 +campaign. It was down for about one second. +- **The payload moved.** b11081 → **b11351** (`631109b34`), upstream v0.40.2's pin + ([upstream-sync-0.40.2.md](tasks/upstream-sync-0.40.2.md)). The compat set has the same nine patches. 903 is now + the #29953 backport with its y-tile guard (#462), and 906 stays retired. +- **The Go side moved.** It carries upstream v0.35.0 → v0.40.2 and the point tag's fixes. One of them is #483: the + converted nemotron child (`nemotron_h_moe` with a `nemotron_v2_vl` projector) gets its image budget flags again (#480). +- **The settings are unchanged:** an f16 KV cache, flash attention on, the two-pass flow and two slots. +- **The tag.** This host cut `v0.40.2-dynres.1` on `39602449`, on the maintainer's word (ADR 0032). + +**v0.40.2 rewrites the model store.** On a legacy GGUF's first load, `compatmigrate` writes a converted +`llamacpp` child into the store, in the background. That is a second full copy of the weights. +- **There is no switch for it in v0.40.2.** +- **The only guard is a space check before each conversion.** A model is skipped when free space is below + 1.25 × its size + 512 MiB. Conversions that start together do not count each other's space. +- **On this host it used 46 GB at the deploy.** `/opt` fell from 106 GB to 60 GB free in four minutes, for three + conversions: + - the smoke test's `qwen2.5vl:3b-q4_K_M`; + - `gemma4:31b-it-q4_K_M` and `nemotron3:33b-q4_K_M`, warmed for the post-deploy preflight. These two ran + concurrently, taking 64 s and 78 s. +- **The store holds 804 GB of blobs,** so far more can convert than `/opt` can hold. Free space is logged every 10 + minutes. +- **Upstream v0.40.3 pauses the migration** (#18908, folded in #488). A gfx1151 image of that fold passed + `rocm7-0-40-2` with no migration line. It is the remedy if `/opt` runs low, and it is the maintainer's call. + +**The compose `.env` still names 0.34.4's image.** Do not redeploy through compose until its image line names +`0.40.2-dynres.1-rocm724-main-39602449` (MaxusAI/ollama-deployments). + +Rollback: + +1. Stop the new container and rename it aside. +2. Rename the retained 0.35.0 container back to `ollama-rocm`. +3. Restore its restart policy and start it. + +``` +docker stop ollama-rocm && docker rename ollama-rocm ollama-rocm-0.40.2-rolledback && +docker rename ollama-rocm-0.35.0-043f441a ollama-rocm && +docker update --restart unless-stopped ollama-rocm && docker start ollama-rocm +``` + +The retained container runs f16 and two-pass, and has `--restart no`. + +**A rollback does not undo the conversions.** v0.40.2's converted children stay in the store, and 0.35.0 predates them. +That has not been tested here: load one converted model on 0.35.0 before trusting a rollback with it. + +**What the evidence covers.** The promoted image is `maxusai-ollama:0.40.2-dynres.1-0-g3960244-rocm7-gfx1151`, a build +of `main` at the point tag, tagged in production's naming. +- **Payload structure.** Against the #477 campaign's image (`0.40.2-dynres-1-g9e80e78`), `payload_diff.sh` finds + 1863 = 1863 entries, no SONAME or symlink change, and 96/96 gfx1151 rocBLAS kernels. +- **Gate 5, preflight.** The profile is `rocm7-0-40-2` (#481, #485). With `--quality`, both runs are **PASS=27 SKIP=5**: + - the release image, in a bench container with production's environment, on a hard-linked clone of production's + store (`runs/preflight-rocm7-0402-dynres1-release-g3960244.json`, #485); + - the promoted container itself (`runs/preflight-rocm7-0402-dynres1-prod-g3960244.json`). + + The two runs agree on every row except the image tag's name. On production, all 12 model loads show + `--cache-type-k f16 --cache-type-v f16 --flash-attn on`. nemotron's `payload_proof` logs custom bounds on both + min and max, which is #483 at work. +- **Gate 6, campaigns (#477).** Nine GGUF models ran in both think modes, on the tag build, in production's environment + (`bench-runs/vision-campaign-2026-10-13-rocm-0402.json`, on `bench/0402-rocm`). + - **Think off, across hosts.** CUDA and Metal agree with gfx1151: `prompt_eval_count` is identical on every row, and + OCRBench reads 170–172 of 200 on every arm. No pass/fail check differs between hosts once warm-state flips are + captured cold (#477). + - **Think on, against this host's 0.35.0.** gemma4:31b and qwen3.8 reproduce 0.35.0's finishes and contract counts. + gemma4:26b leaves 1 test unfinished instead of 3. qwen3.6 and nemotron3 move as much as two 0.35.0 repeats move + between themselves. + +### Clause outcomes + +| clause | outcome | +|---|---| +| 1–2 | **As 2026-09-19.** Both overridden on evidence; nothing has changed upstream. | +| 3. `--direct-io` | **Met by outcome, not re-run as an A/B.** The payload moved, so the 2026-09-28 dio-on/dio-off A/B (0/54 blocks differ) no longer carries over by identity. Every load used `--load-mode dio`: all 35 launches in #477's campaign and all 16 on the promoted container. Their outputs pass gate 5 and agree with CUDA's and Metal's in gate 6. A dio-off A/B on b11351 is open. | +| 4. Vision A/B, ≥6 consecutive rows, 0 degenerate | **PASSED.** In #477's think-off cell, `qwen3.6:35b-a3b-q4_k_m` (`qwen35moe`) finishes all 27 tests with `stop` and valid JSON, and none is degenerate. Its prompt token counts match CUDA's and Metal's on every row. The run had a cold server per model. Across the nine models, the only rows that did not finish are glimmer's two `token repeat limit` stops. CUDA scores the same two rows 0. | +| 5. `make proof` | **Waived — still does not exist.** | + +**How the host was confirmed idle.** As for 0.35.0: no model was loaded, and there was no POST request in the two +minutes before the swap. `deploy-04021.sh` refuses to swap otherwise. The swap ran unattended from `after-campaign.sh`, +once the campaign had finished and its store clone had been removed. + ## Decision 2026-09-21 — 0.34.2 promoted, ROCm 10.0.0 declined on measurement **Outcome: promoted.** `ollama-rocm` moved from `0.34.1-dynres-16649e8c` to diff --git a/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0402-dynres1-prod-g3960244.host-profile.json b/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0402-dynres1-prod-g3960244.host-profile.json new file mode 100644 index 00000000000..3d1f91eb565 --- /dev/null +++ b/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0402-dynres1-prod-g3960244.host-profile.json @@ -0,0 +1,102 @@ +{ + "schema": "host-profile/1", + "collected_at": "2026-10-14T02:50:27Z", + "label": "amd-server", + "label_host": "amd-server", + "label_surface": null, + "machine": { + "vendor": "GMKtec", + "model": "NucBox_EVO-X2", + "model_id": null, + "virtualized": false, + "hypervisor": null + }, + "cloud": null, + "os": { + "family": "linux", + "name": "Ubuntu", + "version": "24.04", + "pretty_name": "Ubuntu 24.04.5 LTS", + "kernel": "6.14.0-1018-oem", + "arch": "x86_64" + }, + "cpu": { + "vendor": "amd", + "model": "AMD RYZEN AI MAX+ 395 w/ Radeon 8060S", + "logical_cpus": 32, + "sockets": 1, + "cores": 16, + "threads_per_core": 2, + "performance_cores": null, + "efficiency_cores": null, + "numa_nodes": 1, + "l3_cache_mib": 64.0, + "max_mhz": 5187.0, + "flags": { + "avx2": true, + "avx512f": true, + "avx512_vnni": true, + "avx512_bf16": true, + "avx512_fp16": false, + "amx_tile": false, + "amx_bf16": false, + "amx_int8": false + } + }, + "memory_total_gib": 31.0, + "gpu_count": 1, + "gpus": [ + { + "vendor": "amd", + "name": "Strix Halo [Radeon Graphics / Radeon 8050S Graphics / Radeon 8060S Graphics]", + "pci_name": "Strix Halo [Radeon Graphics / Radeon 8050S Graphics / Radeon 8060S Graphics]", + "pci_bus_id": "0000:c6:00.0", + "pci_vendor_id": "0x1002", + "pci_device_id": "0x1586", + "driver": "amdgpu", + "memory_total_mib": 98304, + "memory_gtt_mib": 12288, + "unified_memory": null, + "compute_capability": null, + "gfx_target": "gfx1151", + "gpu_cores": null, + "metal_family": null, + "vbios": null, + "power_limit_w": null, + "power_max_limit_w": null, + "max_core_clock_mhz": null, + "max_memory_clock_mhz": null, + "ecc_enabled": null, + "mig_enabled": null, + "persistence_enabled": null, + "pcie_link_gen": 4, + "pcie_link_width": 16, + "pcie_device_max_gen": 4, + "pcie_device_max_width": 16, + "pcie_upstream_max_gen": 4, + "pcie_upstream_max_width": 16, + "pcie_gen_now": 4, + "pcie_width_now": 16, + "numa_node": null, + "local_cpus": "0-31", + "index": 0 + } + ], + "filesystems": [], + "software": { + "nvidia_driver_version": null, + "nvidia_kernel_module": null, + "cuda_driver_version": null, + "cuda_toolkit_version": null, + "cudnn_version": null, + "nccl_version": null, + "rocm_version": "7.2.1", + "amdgpu_driver_version": null, + "docker_version": "29.4.0", + "nvidia_container_toolkit_version": null, + "dcgm_version": null, + "ollama_version": "0.40.2-dynres.1-0-g3960244", + "go_version": "1.27.1", + "python_version": "3.12.3" + } +} diff --git a/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0402-dynres1-prod-g3960244.json b/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0402-dynres1-prod-g3960244.json new file mode 100644 index 00000000000..151b458668c --- /dev/null +++ b/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0402-dynres1-prod-g3960244.json @@ -0,0 +1,575 @@ +{ + "meta": { + "host": "http://127.0.0.1:11434", + "platform": "rocm7", + "image_tag": "maxusai-ollama:0.40.2-dynres.1-rocm724-main-39602449", + "started_utc": "20261014T025029", + "expectations_file": "expectations.toml", + "schema_version": 1, + "version": "0.40.2-dynres.1-0-g3960244", + "profile": "rocm7-0-40-2", + "patchset": [ + "001", + "002", + "004", + "005", + "801", + "802", + "805", + "903", + "908" + ], + "container": "ollama-rocm", + "llama_cpp_build": "631109b34" + }, + "results": [ + { + "check": "version", + "arch": null, + "status": "PASS", + "summary": "0.40.2-dynres.1-0-g3960244 matches profile rocm7-0-40-2", + "expected": "^0\\.40\\.[23]-dynres(\\.\\d+)?-\\d+-g[0-9a-f]{7,40}$", + "actual": "0.40.2-dynres.1-0-g3960244" + }, + { + "check": "payload_pin", + "arch": null, + "status": "PASS", + "summary": "llama.cpp build 631109b34 matches the measured payload (docker exec)", + "expected": "631109b34", + "actual": "631109b34" + }, + { + "check": "toolchain_pin", + "arch": null, + "status": "PASS", + "summary": "GPU toolchain is rocm-7.2.4", + "expected": "rocm-7.2.4", + "actual": "rocm-7.2.4" + }, + { + "check": "image_tag", + "arch": null, + "status": "PASS", + "summary": "ollama-rocm runs maxusai-ollama:0.40.2-dynres.1-rocm724-main-39602449", + "expected": "maxusai-ollama:0.40.2-dynres.1-rocm724-main-39602449", + "actual": "maxusai-ollama:0.40.2-dynres.1-rocm724-main-39602449" + }, + { + "check": "go_patch_marker", + "arch": null, + "status": "PASS", + "summary": "--image-max-tokens present in binary (1)", + "expected": 1, + "actual": 1 + }, + { + "check": "metal_tensor_host", + "arch": null, + "status": "SKIP", + "summary": "does not apply on rocm7" + }, + { + "check": "metal_tensor_runtime", + "arch": null, + "status": "SKIP", + "summary": "does not apply on rocm7" + }, + { + "check": "poison_probe", + "arch": null, + "status": "PASS", + "summary": "1.06x-ceiling trigger decodes healthily (52 chars, done_reason='stop'); slot clean after; no node-level check (meter off \u2014 set OLLAMA_CLIP_NODE_STATS=ffn_down to enable)", + "actual": "The image is a black and white checkerboard pattern." + }, + { + "check": "text_baseline", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 20; text-only is +1 and MUST NOT be used (B8)", + "actual": 19, + "calibration": { + "one_a": 285, + "one_b": 285, + "both": 551 + } + }, + { + "check": "token_ladder", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 266, + "512x288": 266, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 3270 + }, + "actual": { + "256x144": 266, + "512x288": 266, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 3270 + }, + "rows": [ + { + "size": "256x144", + "expected": 266, + "actual": 266, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 266, + "actual": 266, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 3270, + "actual": 3270, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "budget logged as custom on max/min (min 256*32^2=262144, max 3328*32^2=3407872)", + "expected": "min 256*32^2=262144, max 3328*32^2=3407872", + "actual": { + "min": "262144 (custom value)", + "max": "3407872 (custom value)" + } + }, + { + "check": "aspect_ladder", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "valid JSON after thinking (840 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 296, + "thinking_chars": 2758, + "eval_count": 840 + } + }, + { + "check": "extraction_quality", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "json_valid=1.00 label_recall=1.00 qty_price_exact=1.00", + "expected": { + "min_json_valid": 1.0, + "min_label_recall": 0.7, + "min_qty_price_exact": 0.7 + }, + "actual": { + "json_valid": 1.0, + "label_recall": 1.0, + "qty_price_exact": 1.0 + }, + "scores_file": "scores_preflight-rocm7-nemotron_h_omni.json" + }, + { + "check": "pinned_image_token_budget", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "pinned 3328 -> 3270 (ceiling 3328, +2 markers)", + "expected": 3270, + "actual": 3270, + "arms": [ + { + "arm": "pinned", + "pin": 3328, + "expected": 3270, + "actual": 3270, + "ok": true + }, + { + "arm": "ceiling_invariant", + "expected": "grid <= 3328", + "actual": 3268, + "ok": true + }, + { + "arm": "unpinned_control", + "expected": 2306, + "actual": 2306, + "ok": true + } + ] + }, + { + "check": "text_baseline", + "arch": "gemma4", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19, + "calibration": { + "one_a": 1121, + "one_b": 1121, + "both": 2223 + } + }, + { + "check": "token_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4", + "status": "PASS", + "summary": "budget logged as custom on max/min (min 70*48^2=161280, max 1120*48^2=2580480)", + "expected": "min 70*48^2=161280, max 1120*48^2=2580480", + "actual": { + "min": "161280 (custom value)", + "max": "2580480 (custom value)" + } + }, + { + "check": "aspect_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4", + "status": "PASS", + "summary": "valid JSON after thinking (128 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 168, + "thinking_chars": 368, + "eval_count": 128 + } + }, + { + "check": "extraction_quality", + "arch": "gemma4", + "status": "PASS", + "summary": "json_valid=1.00 label_recall=1.00 qty_price_exact=1.00", + "expected": { + "min_json_valid": 1.0, + "min_label_recall": 0.7, + "min_qty_price_exact": 0.7 + }, + "actual": { + "json_valid": 1.0, + "label_recall": 1.0, + "qty_price_exact": 1.0 + }, + "scores_file": "scores_preflight-rocm7-gemma4.json" + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4", + "status": "PASS", + "summary": "pinned 560 -> 529 (ceiling 1120, +2 markers)", + "expected": 529, + "actual": 529, + "arms": [ + { + "arm": "pinned", + "pin": 560, + "expected": 529, + "actual": 529, + "ok": true + }, + { + "arm": "ceiling_invariant", + "expected": "grid <= 1120", + "actual": 527, + "ok": true + }, + { + "arm": "unpinned_control", + "expected": 1102, + "actual": 1102, + "ok": true + } + ] + }, + { + "check": "text_baseline", + "arch": "qwen35", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13, + "calibration": { + "one_a": 1047, + "one_b": 1047, + "both": 2081 + } + }, + { + "check": "token_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1034, + "512x288": 1034, + "1024x576": 1034, + "2048x1152": 2306, + "3072x1728": 4082 + }, + "actual": { + "256x144": 1034, + "512x288": 1034, + "1024x576": 1034, + "2048x1152": 2306, + "3072x1728": 4082 + }, + "rows": [ + { + "size": "256x144", + "expected": 1034, + "actual": 1034, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1034, + "actual": 1034, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1034, + "actual": 1034, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 4082, + "actual": 4082, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35", + "status": "PASS", + "summary": "budget logged as custom on min (min 1024*32^2=1048576, max 4096*32^2=4194304)", + "expected": "min 1024*32^2=1048576, max 4096*32^2=4194304", + "actual": { + "min": "1048576 (custom value)", + "max": "4194304" + } + }, + { + "check": "aspect_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1038, + "1920x480": 1026, + "768x768": 1026 + }, + "actual": { + "1024x768": 1038, + "1920x480": 1026, + "768x768": 1026 + } + }, + { + "check": "think_format", + "arch": "qwen35", + "status": "PASS", + "summary": "valid JSON after thinking (306 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 396, + "thinking_chars": 888, + "eval_count": 306 + } + }, + { + "check": "extraction_quality", + "arch": "qwen35", + "status": "PASS", + "summary": "json_valid=1.00 label_recall=1.00 qty_price_exact=1.00", + "expected": { + "min_json_valid": 1.0, + "min_label_recall": 0.7, + "min_qty_price_exact": 0.7 + }, + "actual": { + "json_valid": 1.0, + "label_recall": 1.0, + "qty_price_exact": 1.0 + }, + "scores_file": "scores_preflight-rocm7-qwen35.json" + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35", + "status": "SKIP", + "summary": "pinned budget does not apply to this arch: image_min_tokens/image_max_tokens are arch-gated to gemma4 and nemotron_h_omni; qwen35's visionServerArgs branch ignores them." + }, + { + "check": "metal_tensor_payload", + "arch": null, + "status": "SKIP", + "summary": "does not apply on rocm7" + }, + { + "check": "mlx_payload_pin", + "arch": null, + "status": "SKIP", + "summary": "profile records no mlx_build to assert against", + "diagnosis": "Add mlx_build (the MLX_VERSION commit this profile was measured on) so an MLX bump fails loudly instead of inheriting stale expectations. Platforms with no MLX payload leave it unset and keep this skip." + }, + { + "check": "endpoint_exclusive", + "arch": null, + "status": "PASS", + "summary": "no contention detected (worst queue wait 0.0s)", + "expected": "queue wait <= 10.0s", + "actual": "0.0s", + "models_loaded": [ + "qwen3.8:27b-q4_K_M", + "gemma4:31b-it-q4_K_M" + ] + } + ], + "summary": { + "PASS": 27, + "SKIP": 5 + } +} \ No newline at end of file From c86080fde65d7539e9bcb4a93be42bcd16e7cc31 Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Wed, 14 Oct 2026 13:08:45 +1000 Subject: [PATCH 2/3] gate: glimmer's two unfinished rows fail differently on CUDA, not by the repeat limit Co-Authored-By: Claude Opus 5.5 --- docs/maxusai/amd-upgrade-gate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/maxusai/amd-upgrade-gate.md b/docs/maxusai/amd-upgrade-gate.md index 7906097f70b..7ad9bec3b76 100644 --- a/docs/maxusai/amd-upgrade-gate.md +++ b/docs/maxusai/amd-upgrade-gate.md @@ -810,7 +810,7 @@ of `main` at the point tag, tagged in production's naming. |---|---| | 1–2 | **As 2026-09-19.** Both overridden on evidence; nothing has changed upstream. | | 3. `--direct-io` | **Met by outcome, not re-run as an A/B.** The payload moved, so the 2026-09-28 dio-on/dio-off A/B (0/54 blocks differ) no longer carries over by identity. Every load used `--load-mode dio`: all 35 launches in #477's campaign and all 16 on the promoted container. Their outputs pass gate 5 and agree with CUDA's and Metal's in gate 6. A dio-off A/B on b11351 is open. | -| 4. Vision A/B, ≥6 consecutive rows, 0 degenerate | **PASSED.** In #477's think-off cell, `qwen3.6:35b-a3b-q4_k_m` (`qwen35moe`) finishes all 27 tests with `stop` and valid JSON, and none is degenerate. Its prompt token counts match CUDA's and Metal's on every row. The run had a cold server per model. Across the nine models, the only rows that did not finish are glimmer's two `token repeat limit` stops. CUDA scores the same two rows 0. | +| 4. Vision A/B, ≥6 consecutive rows, 0 degenerate | **PASSED.** In #477's think-off cell, `qwen3.6:35b-a3b-q4_k_m` (`qwen35moe`) finishes all 27 tests with `stop` and valid JSON, and none is degenerate. Its prompt token counts match CUDA's and Metal's on every row. The run had a cold server per model. Across the nine models, the only rows that did not finish are glimmer's two `token repeat limit` stops (`bboxm_free_noanc_pos`, `bbox_contract_box2d_1img`). CUDA does not loop on them: there both end with `stop` and score 0, 0 of 6 labels found. The score is the same and the way it fails is not. | | 5. `make proof` | **Waived — still does not exist.** | **How the host was confirmed idle.** As for 0.35.0: no model was loaded, and there was no POST request in the two From c04c249343ebbbcf61d68e4d26289d9b99c49cdb Mon Sep 17 00:00:00 2001 From: Glenn Neuber Date: Wed, 14 Oct 2026 13:59:18 +1000 Subject: [PATCH 3/3] gate: the v0.40.3 fold replaces 0.40.2-dynres.1 on gfx1151, pausing the store migration At 13:51 ollama-rocm moved to 0.40.2-dynres.1-6-g9f3baaf (#488's head; Go equal to main 4d3387af3; native libraries byte-identical to the release's), on the maintainer's word. A legacy GGUF now loads without a migration line, and /opt held at 59 GB. Post-deploy preflight: PASS=27 SKIP=5, equal to the point tag's production run in every measured value. The rocm7 matrix row is this run. Co-Authored-By: Claude Opus 5.5 --- README.md | 11 +- docs/maxusai/amd-upgrade-gate.md | 47 +- ...rocm7-0403-prod-g9f3baaf.host-profile.json | 102 ++++ .../preflight-rocm7-0403-prod-g9f3baaf.json | 576 ++++++++++++++++++ 4 files changed, 724 insertions(+), 12 deletions(-) create mode 100644 docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0403-prod-g9f3baaf.host-profile.json create mode 100644 docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0403-prod-g9f3baaf.json diff --git a/README.md b/README.md index e381525b3b5..7797bcf1438 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ > |---|---|---|---| > | Apple Silicon, mlx-metal (`:11435`) | `0.40.2-dynres.1-17-g75c6d35`, the same build as the CUDA host's: main `75c6d350`, a Go-only build on the point tag's payload | 2026-10-14 10:13 | [#486](https://github.com/MaxusAI/ollama/pull/486), this record | > | CUDA host, ai-server (`:11497`) | `0.40.2-dynres.1-17-g75c6d35`: the v0.40.3 fold ([#488](https://github.com/MaxusAI/ollama/pull/488)) on the point tag plus [#487](https://github.com/MaxusAI/ollama/pull/487), a Go-only build of main `75c6d350` (image `main-75c6d350`) | 2026-10-14 07:53 | [#487](https://github.com/MaxusAI/ollama/pull/487), [#488](https://github.com/MaxusAI/ollama/pull/488), [#492](https://github.com/MaxusAI/ollama/pull/492) | -> | AMD gfx1151 (`:11434`) | `0.40.2-dynres.1-0-g3960244`, a build at the point tag (image `0.40.2-dynres.1-rocm724-main-39602449`); preflight passes `rocm7-0-40-2` on the release image and on production | 2026-10-14 12:47 | [#485](https://github.com/MaxusAI/ollama/pull/485), [the 2026-10-14 decision](docs/maxusai/amd-upgrade-gate.md#decision-2026-10-14--0402-dynres1-promoted-the-payload-moves-to-b11351) | +> | AMD gfx1151 (`:11434`) | `0.40.2-dynres.1-6-g9f3baaf`: the v0.40.3 fold ([#488](https://github.com/MaxusAI/ollama/pull/488)), a build of `9f3baafce`. Its Go equals main's, and it pauses v0.40.2's store migration. The point tag's build served 12:47–13:51. Preflight passes `rocm7-0-40-2` on production | 2026-10-14 13:51 | [#485](https://github.com/MaxusAI/ollama/pull/485), [the 2026-10-14 decision](docs/maxusai/amd-upgrade-gate.md#decision-2026-10-14--0402-dynres1-promoted-the-payload-moves-to-b11351) | > | H100 (CUDA) | `0.35.0-dynres-15-ga657392`, a build of `main` past the v0.35.0 tag | 2026-10-03 | [v0.35.0 record](docs/maxusai/tasks/upstream-sync-0.35.0.md) | > > The CUDA host serves [#487](https://github.com/MaxusAI/ollama/pull/487), merged after the point tag: without it, an MLX @@ -78,7 +78,7 @@ For v0.40.2-dynres.1 the table below is the verbatim output of release_matrix.py --version 0.40.2-dynres.1 \ runs/preflight-mlx-metal-0403-prod-g75c6d35.json \ - runs/preflight-rocm7-0402-dynres1-prod-g3960244.json \ + runs/preflight-rocm7-0403-prod-g9f3baaf.json \ runs/preflight-cuda-0403-prod-g75c6d35.json \ runs/preflight-mlx-cuda-0403-prod-g75c6d35.json The runs are listed by name because runs/ also holds #487's A/B runs and canaries, which are not release runs. @@ -86,8 +86,9 @@ point tag plus #487, Go only); their cuda, mlx-cuda and mlx-metal rows are production's own runs on that build. The cuda row is the idle re-run (preflight-cuda-0403-prod-g75c6d35.json); the first post-deploy run, -contended.json, failed gemma4's ladder and pinned budget by a constant +573 while a client's gemma4 q8 shared the GPU and capped image_max_tokens to n_ubatch. - gfx1151 serves 0.40.2-dynres.1-0-g3960244 since 2026-10-14; its rocm7 row is production's own run on that build, - equal to the release image's run (-release-g3960244.json) in every row but the image tag. --> + gfx1151 serves 0.40.2-dynres.1-6-g9f3baaf (the #488 head, Go equal to main) since 2026-10-14 13:51; its rocm7 row + is production's own run on that build, equal in every measured value to the point tag's runs on production + (-prod-g3960244.json) and on the release image (-release-g3960244.json). --> | surface | Build identity | MLX build pin | Toolchain pin | M5 tensor path | Image size ladder | Aspect ladder | Pinned image budget | thinking on/off | Output quality | fp16 overflow canary | Runner isolation | measured on | |---|---|---|---|---|---|---|---|---|---|---|---|---| @@ -95,7 +96,7 @@ | **mlx-cuda** | skipped | green | skipped | n/a | green | skipped | skipped | skipped | not run | skipped | green | `0.40.2-dynres.1-17-g75c6d35` | | **mlx-metal** | skipped | green | skipped | green | green | skipped | skipped | green | skipped | skipped | green | `0.40.2-dynres.1-17-g75c6d35` | | **apple-silicon-mlx** | not run | not run | not run | not run | not run | not run | not run | not run | not run | not run | not run | — | -| **rocm7** | green | skipped | green | n/a | green | green | green | green | green | green | green | `0.40.2-dynres.1-0-g3960244` | +| **rocm7** | green | skipped | green | n/a | green | green | green | green | green | green | green | `0.40.2-dynres.1-6-g9f3baaf` | | **rocm10** | not run | not run | not run | not run | not run | not run | not run | not run | not run | not run | not run | — | | **cpu** | not run | not run | not run | not run | not run | not run | not run | not run | not run | not run | not run | — | diff --git a/docs/maxusai/amd-upgrade-gate.md b/docs/maxusai/amd-upgrade-gate.md index 7ad9bec3b76..bd85d0b40ac 100644 --- a/docs/maxusai/amd-upgrade-gate.md +++ b/docs/maxusai/amd-upgrade-gate.md @@ -23,13 +23,13 @@ back to `0.32.1-gemma4budget-85ebcb79`. | | | |---|---| -| Deployed image | `maxusai-ollama:0.40.2-dynres.1-rocm724-main-39602449` (promoted 2026-10-14 12:47) | -| Deployed version | `0.40.2-dynres.1-0-g3960244`, a build of `main` at the `v0.40.2-dynres.1` point tag (ADR 0032); see the [2026-10-14 decision](#decision-2026-10-14--0402-dynres1-promoted-the-payload-moves-to-b11351) | +| Deployed image | `maxusai-ollama:0.40.2-dynres.1-6-rocm724-main-9f3baafc` (promoted 2026-10-14 13:51), the v0.40.3 fold | +| Deployed version | `0.40.2-dynres.1-6-g9f3baaf`, a build of #488's head `9f3baafce`, the v0.40.3 fold. Its Go is identical to `main` at `4d3387af3`. It replaced `0.40.2-dynres.1-0-g3960244` after 64 minutes, because v0.40.3 pauses the store migration; see the [2026-10-14 decision](#decision-2026-10-14--0402-dynres1-promoted-the-payload-moves-to-b11351) | | Build type | `Dockerfile.rocm` through `scripts/build_rocm.sh` (`ROCM_TOOLCHAIN=rocm7 AMDGPU_TARGETS=gfx1151`), on `rocm/dev-ubuntu-24.04:7.2.4-complete` ([ADR 0042](adr/0042-rocm-images-build-on-ubuntu-rocm-images.md)), as every production image since 0.34.3. It is gfx1151 only. Compat **001 + 002 + 004 + 005 + 801 + 802 + 805 + 903 + 908**: 908 reverts the device half of `ce8caa6e6`, a CUDA flash-attention tiling, and changes no gfx1151 kernel. 805 ([ADR 0041](adr/0041-gfx1151-vision-prefill-is-bandwidth-bound.md)) is a diagnostic switch, `GGML_CUDA_FATTN_FORCE_TILE`, inert unless set; on gfx1151 it adds 64 bytes of host code and no GPU code. 802, the LM-graph node meter, is inert unless `OLLAMA_LM_NODE_STATS` is set | | KV cache | **f16** since 2026-09-26 07:23 ([ADR 0005](adr/0005-per-model-kv-cache-type.md)). From the 2026-08-08 cutover until then it was `q8_0` by accident; see the [2026-09-26 decision](#decision-2026-09-26--productions-kv-cache-back-to-f16) | | Think+format flow | **two-pass** (`OLLAMA_FORMAT_TWO_PASS=1`) since 2026-09-28, the maintainer's choice at the 0.34.4 deploy, kept at the 0.35.0 and 0.40.2-dynres.1 deploys. The default is the single pass; see the [2026-09-28 decision](#decision-2026-09-28--0344-promoted-with-the-two-pass-flow) | | Payload | **b11351** (`631109b34`) since 2026-10-14; b11081 (`161755f29`) from 2026-09-28 for 0.34.4 and 0.35.0; b10969 (`391fac164`) for 0.34.2 and 0.34.3. Compat 906 stays **retired**: upstream ships its own HIP `prop.integrated` revert | -| Previous image | `maxusai-ollama:0.35.0-rocm724-main-043f441a` (`0.35.0-dynres-0-g043f441`, b11081) — **retained for rollback** as the stopped container `ollama-rocm-0.35.0-043f441a`, which runs f16 and two-pass. Before it, `0.34.4-rocm724-main-b43ee8e3` (b11081 + 908; the stopped container `ollama-rocm-0.34.4-b43ee8e3`, f16 and two-pass), `0.34.3-rocm724-main-650f8fda` (b10969; the stopped container `ollama-rocm-0.34.3-650f8fda`, f16), `0.34.2-rocm724-main-f67b1aef` (the stopped container `ollama-rocm-0.34.2-f67b1aef`, recreated with f16 on 2026-09-28), `0.34.1-rocm724-main-16649e8c` (b10864 + 906) and `0.32.1-rocm-dynres-5d5b7a72` (b9888) | +| Previous image | `maxusai-ollama:0.40.2-dynres.1-rocm724-main-39602449` (`0.40.2-dynres.1-0-g3960244`, b11351) — **retained for rollback** as the stopped container `ollama-rocm-0.40.2-dynres.1-39602449`. Before it, `0.35.0-rocm724-main-043f441a` (b11081; the stopped container `ollama-rocm-0.35.0-043f441a`, f16 and two-pass), `0.34.4-rocm724-main-b43ee8e3` (b11081 + 908; the stopped container `ollama-rocm-0.34.4-b43ee8e3`, f16 and two-pass), `0.34.3-rocm724-main-650f8fda` (b10969; the stopped container `ollama-rocm-0.34.3-650f8fda`, f16), `0.34.2-rocm724-main-f67b1aef` (the stopped container `ollama-rocm-0.34.2-f67b1aef`, recreated with f16 on 2026-09-28), `0.34.1-rocm724-main-16649e8c` (b10864 + 906) and `0.32.1-rocm-dynres-5d5b7a72` (b9888) | | Superseded pin | `0.32.1-dynres-296eb020` recorded here until 2026-09-19; the host was in fact running `5d5b7a72`, so this row had drifted from the host it describes | | Blocked target | `0.32.5-gemma4budget-4259c191` (built, verified, **rolled back** 2026-07-31) — never unblocked; superseded, not cleared | | Host | Ryzen AI Max+ 395 / Radeon 8060S, **gfx1151**, ROCm, Linux | @@ -760,11 +760,10 @@ campaign. It was down for about one second. concurrently, taking 64 s and 78 s. - **The store holds 804 GB of blobs,** so far more can convert than `/opt` can hold. Free space is logged every 10 minutes. -- **Upstream v0.40.3 pauses the migration** (#18908, folded in #488). A gfx1151 image of that fold passed - `rocm7-0-40-2` with no migration line. It is the remedy if `/opt` runs low, and it is the maintainer's call. +- **Upstream v0.40.3 pauses the migration** (#18908, folded in #488). Production moved to that fold at 13:51, below. -**The compose `.env` still names 0.34.4's image.** Do not redeploy through compose until its image line names -`0.40.2-dynres.1-rocm724-main-39602449` (MaxusAI/ollama-deployments). +**The compose `.env` still names 0.34.4's image.** Do not redeploy through compose until its image line names the +deployed image (MaxusAI/ollama-deployments); since 13:51 that is the fold's, below. Rollback: @@ -804,6 +803,40 @@ of `main` at the point tag, tagged in production's naming. gemma4:26b leaves 1 test unfinished instead of 3. qwen3.6 and nemotron3 move as much as two 0.35.0 repeats move between themselves. +### The same day: the v0.40.3 fold replaces it, and the migration stops + +At 13:51 `ollama-rocm` moved again, to **`0.40.2-dynres.1-6-g9f3baaf`** +(`maxusai-ollama:0.40.2-dynres.1-6-rocm724-main-9f3baafc`). The maintainer asked for it ("deploy the v0.40.3 fold +image"). It was down for about one second, after the same idle check. +- **Why.** Upstream v0.40.3 pauses the compat migration (#18908). With the migration paused, a legacy GGUF loads as it + is, and no second copy is written. +- **The build.** This is a build of #488's head `9f3baafce`, gfx1151's gate-5 image for the fold. + - Its Go equals `main` at `4d3387af3`: the later commits change no `.go` file. That is the Go the CUDA and Apple + Silicon hosts serve. + - Its 19 native libraries are byte-identical to the `v0.40.2-dynres.1` release's, so the payload stays b11351. +- **The smoke.** `qwen2.5vl:3b-q4_K_M`, converted under 0.40.2, answered. So did `qwen2.5vl:3b-q8_0`, a legacy GGUF that + 0.40.2 would have converted. The log shows no migration line. + - Free space on `/opt` stayed at 59 GB through the swap, the smoke and the preflight. + - The conversions 0.40.2 made stay in the store. +- **Preflight (gate 5) on the promoted container:** **PASS=27 SKIP=5** + (`runs/preflight-rocm7-0403-prod-g9f3baaf.json`). + - It equals the 0.40.2-dynres.1 production run in every measured value. Only the `version` and `image_tag` rows + differ. + - All 13 model loads show `--cache-type-k f16 --cache-type-v f16 --flash-attn on --load-mode dio`. +- **Gate 6 carries over.** The fold changes no native code, and its Go changes are the migration pause and upstream's + v0.40.3 fixes (#488). #488's gfx1151 preflight matched the release run on every row. + +Rollback, to 0.40.2-dynres.1, which resumes the migration: + +``` +docker stop ollama-rocm && docker rename ollama-rocm ollama-rocm-0403fold-rolledback && +docker rename ollama-rocm-0.40.2-dynres.1-39602449 ollama-rocm && +docker update --restart unless-stopped ollama-rocm && docker start ollama-rocm +``` + +The 0.35.0 container stays as the further fallback. **The compose `.env` still names 0.34.4's image.** Its image +line should now name `0.40.2-dynres.1-6-rocm724-main-9f3baafc`. + ### Clause outcomes | clause | outcome | diff --git a/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0403-prod-g9f3baaf.host-profile.json b/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0403-prod-g9f3baaf.host-profile.json new file mode 100644 index 00000000000..0a40ccb40bc --- /dev/null +++ b/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0403-prod-g9f3baaf.host-profile.json @@ -0,0 +1,102 @@ +{ + "schema": "host-profile/1", + "collected_at": "2026-10-14T03:52:06Z", + "label": "amd-server", + "label_host": "amd-server", + "label_surface": null, + "machine": { + "vendor": "GMKtec", + "model": "NucBox_EVO-X2", + "model_id": null, + "virtualized": false, + "hypervisor": null + }, + "cloud": null, + "os": { + "family": "linux", + "name": "Ubuntu", + "version": "24.04", + "pretty_name": "Ubuntu 24.04.5 LTS", + "kernel": "6.14.0-1018-oem", + "arch": "x86_64" + }, + "cpu": { + "vendor": "amd", + "model": "AMD RYZEN AI MAX+ 395 w/ Radeon 8060S", + "logical_cpus": 32, + "sockets": 1, + "cores": 16, + "threads_per_core": 2, + "performance_cores": null, + "efficiency_cores": null, + "numa_nodes": 1, + "l3_cache_mib": 64.0, + "max_mhz": 5187.0, + "flags": { + "avx2": true, + "avx512f": true, + "avx512_vnni": true, + "avx512_bf16": true, + "avx512_fp16": false, + "amx_tile": false, + "amx_bf16": false, + "amx_int8": false + } + }, + "memory_total_gib": 31.0, + "gpu_count": 1, + "gpus": [ + { + "vendor": "amd", + "name": "Strix Halo [Radeon Graphics / Radeon 8050S Graphics / Radeon 8060S Graphics]", + "pci_name": "Strix Halo [Radeon Graphics / Radeon 8050S Graphics / Radeon 8060S Graphics]", + "pci_bus_id": "0000:c6:00.0", + "pci_vendor_id": "0x1002", + "pci_device_id": "0x1586", + "driver": "amdgpu", + "memory_total_mib": 98304, + "memory_gtt_mib": 12288, + "unified_memory": null, + "compute_capability": null, + "gfx_target": "gfx1151", + "gpu_cores": null, + "metal_family": null, + "vbios": null, + "power_limit_w": null, + "power_max_limit_w": null, + "max_core_clock_mhz": null, + "max_memory_clock_mhz": null, + "ecc_enabled": null, + "mig_enabled": null, + "persistence_enabled": null, + "pcie_link_gen": 4, + "pcie_link_width": 16, + "pcie_device_max_gen": 4, + "pcie_device_max_width": 16, + "pcie_upstream_max_gen": 4, + "pcie_upstream_max_width": 16, + "pcie_gen_now": 4, + "pcie_width_now": 16, + "numa_node": null, + "local_cpus": "0-31", + "index": 0 + } + ], + "filesystems": [], + "software": { + "nvidia_driver_version": null, + "nvidia_kernel_module": null, + "cuda_driver_version": null, + "cuda_toolkit_version": null, + "cudnn_version": null, + "nccl_version": null, + "rocm_version": "7.2.1", + "amdgpu_driver_version": null, + "docker_version": "29.4.0", + "nvidia_container_toolkit_version": null, + "dcgm_version": null, + "ollama_version": "0.40.2-dynres.1-6-g9f3baaf", + "go_version": "1.27.1", + "python_version": "3.12.3" + } +} diff --git a/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0403-prod-g9f3baaf.json b/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0403-prod-g9f3baaf.json new file mode 100644 index 00000000000..2dac03372d7 --- /dev/null +++ b/docs/maxusai/vision-suite/preflight/runs/preflight-rocm7-0403-prod-g9f3baaf.json @@ -0,0 +1,576 @@ +{ + "meta": { + "host": "http://127.0.0.1:11434", + "platform": "rocm7", + "image_tag": "maxusai-ollama:0.40.2-dynres.1-6-rocm724-main-9f3baafc", + "started_utc": "20261014T035207", + "expectations_file": "expectations.toml", + "schema_version": 1, + "version": "0.40.2-dynres.1-6-g9f3baaf", + "profile": "rocm7-0-40-2", + "patchset": [ + "001", + "002", + "004", + "005", + "801", + "802", + "805", + "903", + "908" + ], + "container": "ollama-rocm", + "llama_cpp_build": "631109b34" + }, + "results": [ + { + "check": "version", + "arch": null, + "status": "PASS", + "summary": "0.40.2-dynres.1-6-g9f3baaf matches profile rocm7-0-40-2", + "expected": "^0\\.40\\.[23]-dynres(\\.\\d+)?-\\d+-g[0-9a-f]{7,40}$", + "actual": "0.40.2-dynres.1-6-g9f3baaf" + }, + { + "check": "payload_pin", + "arch": null, + "status": "PASS", + "summary": "llama.cpp build 631109b34 matches the measured payload (docker exec)", + "expected": "631109b34", + "actual": "631109b34" + }, + { + "check": "toolchain_pin", + "arch": null, + "status": "PASS", + "summary": "GPU toolchain is rocm-7.2.4", + "expected": "rocm-7.2.4", + "actual": "rocm-7.2.4" + }, + { + "check": "image_tag", + "arch": null, + "status": "PASS", + "summary": "ollama-rocm runs maxusai-ollama:0.40.2-dynres.1-6-rocm724-main-9f3baafc", + "expected": "maxusai-ollama:0.40.2-dynres.1-6-rocm724-main-9f3baafc", + "actual": "maxusai-ollama:0.40.2-dynres.1-6-rocm724-main-9f3baafc" + }, + { + "check": "go_patch_marker", + "arch": null, + "status": "PASS", + "summary": "--image-max-tokens present in binary (1)", + "expected": 1, + "actual": 1 + }, + { + "check": "metal_tensor_host", + "arch": null, + "status": "SKIP", + "summary": "does not apply on rocm7" + }, + { + "check": "metal_tensor_runtime", + "arch": null, + "status": "SKIP", + "summary": "does not apply on rocm7" + }, + { + "check": "poison_probe", + "arch": null, + "status": "PASS", + "summary": "1.06x-ceiling trigger decodes healthily (52 chars, done_reason='stop'); slot clean after; no node-level check (meter off \u2014 set OLLAMA_CLIP_NODE_STATS=ffn_down to enable)", + "actual": "The image is a black and white checkerboard pattern." + }, + { + "check": "text_baseline", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 20; text-only is +1 and MUST NOT be used (B8)", + "actual": 19, + "calibration": { + "one_a": 285, + "one_b": 285, + "both": 551 + } + }, + { + "check": "token_ladder", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 266, + "512x288": 266, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 3270 + }, + "actual": { + "256x144": 266, + "512x288": 266, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 3270 + }, + "rows": [ + { + "size": "256x144", + "expected": 266, + "actual": 266, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 266, + "actual": 266, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 3270, + "actual": 3270, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "budget logged as custom on max/min (min 256*32^2=262144, max 3328*32^2=3407872)", + "expected": "min 256*32^2=262144, max 3328*32^2=3407872", + "actual": { + "min": "262144 (custom value)", + "max": "3407872 (custom value)" + } + }, + { + "check": "aspect_ladder", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "valid JSON after thinking (840 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 296, + "thinking_chars": 2758, + "eval_count": 840 + } + }, + { + "check": "extraction_quality", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "json_valid=1.00 label_recall=1.00 qty_price_exact=1.00", + "expected": { + "min_json_valid": 1.0, + "min_label_recall": 0.7, + "min_qty_price_exact": 0.7 + }, + "actual": { + "json_valid": 1.0, + "label_recall": 1.0, + "qty_price_exact": 1.0 + }, + "scores_file": "scores_preflight-rocm7-nemotron_h_omni.json" + }, + { + "check": "pinned_image_token_budget", + "arch": "nemotron_h_omni", + "status": "PASS", + "summary": "pinned 3328 -> 3270 (ceiling 3328, +2 markers)", + "expected": 3270, + "actual": 3270, + "arms": [ + { + "arm": "pinned", + "pin": 3328, + "expected": 3270, + "actual": 3270, + "ok": true + }, + { + "arm": "ceiling_invariant", + "expected": "grid <= 3328", + "actual": 3268, + "ok": true + }, + { + "arm": "unpinned_control", + "expected": 2306, + "actual": 2306, + "ok": true + } + ] + }, + { + "check": "text_baseline", + "arch": "gemma4", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19, + "calibration": { + "one_a": 1121, + "one_b": 1121, + "both": 2223 + } + }, + { + "check": "token_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4", + "status": "PASS", + "summary": "budget logged as custom on max/min (min 70*48^2=161280, max 1120*48^2=2580480)", + "expected": "min 70*48^2=161280, max 1120*48^2=2580480", + "actual": { + "min": "161280 (custom value)", + "max": "2580480 (custom value)" + } + }, + { + "check": "aspect_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4", + "status": "PASS", + "summary": "valid JSON after thinking (128 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 168, + "thinking_chars": 368, + "eval_count": 128 + } + }, + { + "check": "extraction_quality", + "arch": "gemma4", + "status": "PASS", + "summary": "json_valid=1.00 label_recall=1.00 qty_price_exact=1.00", + "expected": { + "min_json_valid": 1.0, + "min_label_recall": 0.7, + "min_qty_price_exact": 0.7 + }, + "actual": { + "json_valid": 1.0, + "label_recall": 1.0, + "qty_price_exact": 1.0 + }, + "scores_file": "scores_preflight-rocm7-gemma4.json" + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4", + "status": "PASS", + "summary": "pinned 560 -> 529 (ceiling 1120, +2 markers)", + "expected": 529, + "actual": 529, + "arms": [ + { + "arm": "pinned", + "pin": 560, + "expected": 529, + "actual": 529, + "ok": true + }, + { + "arm": "ceiling_invariant", + "expected": "grid <= 1120", + "actual": 527, + "ok": true + }, + { + "arm": "unpinned_control", + "expected": 1102, + "actual": 1102, + "ok": true + } + ] + }, + { + "check": "text_baseline", + "arch": "qwen35", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13, + "calibration": { + "one_a": 1047, + "one_b": 1047, + "both": 2081 + } + }, + { + "check": "token_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1034, + "512x288": 1034, + "1024x576": 1034, + "2048x1152": 2306, + "3072x1728": 4082 + }, + "actual": { + "256x144": 1034, + "512x288": 1034, + "1024x576": 1034, + "2048x1152": 2306, + "3072x1728": 4082 + }, + "rows": [ + { + "size": "256x144", + "expected": 1034, + "actual": 1034, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1034, + "actual": 1034, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1034, + "actual": 1034, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 4082, + "actual": 4082, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35", + "status": "PASS", + "summary": "budget logged as custom on min (min 1024*32^2=1048576, max 4096*32^2=4194304)", + "expected": "min 1024*32^2=1048576, max 4096*32^2=4194304", + "actual": { + "min": "1048576 (custom value)", + "max": "4194304" + } + }, + { + "check": "aspect_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1038, + "1920x480": 1026, + "768x768": 1026 + }, + "actual": { + "1024x768": 1038, + "1920x480": 1026, + "768x768": 1026 + } + }, + { + "check": "think_format", + "arch": "qwen35", + "status": "PASS", + "summary": "valid JSON after thinking (306 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 396, + "thinking_chars": 888, + "eval_count": 306 + } + }, + { + "check": "extraction_quality", + "arch": "qwen35", + "status": "PASS", + "summary": "json_valid=1.00 label_recall=1.00 qty_price_exact=1.00", + "expected": { + "min_json_valid": 1.0, + "min_label_recall": 0.7, + "min_qty_price_exact": 0.7 + }, + "actual": { + "json_valid": 1.0, + "label_recall": 1.0, + "qty_price_exact": 1.0 + }, + "scores_file": "scores_preflight-rocm7-qwen35.json" + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35", + "status": "SKIP", + "summary": "pinned budget does not apply to this arch: image_min_tokens/image_max_tokens are arch-gated to gemma4 and nemotron_h_omni; qwen35's visionServerArgs branch ignores them." + }, + { + "check": "metal_tensor_payload", + "arch": null, + "status": "SKIP", + "summary": "does not apply on rocm7" + }, + { + "check": "mlx_payload_pin", + "arch": null, + "status": "SKIP", + "summary": "profile records no mlx_build to assert against", + "diagnosis": "Add mlx_build (the MLX_VERSION commit this profile was measured on) so an MLX bump fails loudly instead of inheriting stale expectations. Platforms with no MLX payload leave it unset and keep this skip." + }, + { + "check": "endpoint_exclusive", + "arch": null, + "status": "PASS", + "summary": "no contention detected (worst queue wait 0.0s)", + "expected": "queue wait <= 10.0s", + "actual": "0.0s", + "models_loaded": [ + "qwen3.8:27b-q4_K_M", + "gemma4:31b-it-q4_K_M", + "nemotron3:33b-q4_K_M" + ] + } + ], + "summary": { + "PASS": 27, + "SKIP": 5 + } +} \ No newline at end of file