diff --git a/docs/maxusai/vision-suite/BINARIES.md b/docs/maxusai/vision-suite/BINARIES.md index 8538658c079..74055af3931 100644 --- a/docs/maxusai/vision-suite/BINARIES.md +++ b/docs/maxusai/vision-suite/BINARIES.md @@ -21,7 +21,9 @@ A host can be rebuilt from it; a binary sitting only in `/tmp` cannot. | `0.33.2-maxusai-2b95b4a5` | [`2b95b4a5`](https://github.com/MaxusAI/ollama/commit/2b95b4a5) | llama.cpp **b10630** | c793734e pin | `cd6033d6e72a430e…` | served :11435 2026-08-30 → 09-18; rollback target (BOTH halves archived); the v0.33.2 fold (#232, ADR 0033 xgrammar). Provenance for the `mlx-metal-0-33-2` profile measurement; preflight PASS 2026-08-30, goldens NOT recalibrated at this MLX pin. **Rolling back to it reinstates the pre-#3912 `fp_qmm_t` defect** on the gemma4 26b/31b vision `down_proj` (K = 4304): OCR-neutral on OCRBench, arithmetically wrong (ADR 0037) | | `0.34.0-maxusai-8a7ba949` | [`8a7ba949`](https://github.com/MaxusAI/ollama/commit/8a7ba949) | llama.cpp **b10864** | d9add9d1 pin | `867ee5d6063d8f77…` | the v0.34.0 + v0.34.1 folds (#297, #302). The same build as `0.34.1-dynres-0-g8a7ba94` (ADR 0032, 2026-09-19 amendment): built 49 minutes before the fold tag, under the retired native stamp. Provenance for the `mlx-metal-0-34-0` profile (#306); preflight PASS 2026-09-18 (19/12); whitespace bound (#301) confirmed behaviourally on real documents across the MLX pin move. **rollback target** (both halves archived) — served `:11435` from 2026-09-18 13:17 to 2026-09-28 14:07, verified by version, a live MLX load reporting `0.32.2-61-gd9add9d`, and the dlopen path resolving the repo payload. Native gates pass: `mlxrunner` and all subpackages, and vision golden parity on 31b/26b/12b (max element delta 0.109 / 0.051 / 0.063) — the first build whose goldens were CHECKED at its MLX pin. **Promoted 2026-09-19** with the MLX #3912 kernel fix kept (ADR 0037): OCRBench v1, all 1000 items, `31b-nvfp4` 835/1000 against 0.33.2's 833/1000, exact McNemar p = 0.80 (`bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0340-vs-0332.json`) | | `0.34.4-dynres-0-gb43ee8e` | [`b43ee8e3`](https://github.com/MaxusAI/ollama/commit/b43ee8e3) | llama.cpp **b11081** | 59d600b5 pin | `f343fea9ea7bc66e…` | the `v0.34.4-dynres` tag (#375), both halves built at the tag in a detached worktree. Not a profile's provenance: `mlx-metal-0-34-4` (#384) was measured on the fold's build, which has the same pins. Served `:11435` 2026-09-28 14:07 → 2026-10-01 23:45, and the rollback target for `0.35.0-dynres-0-g043f441`, which runs on this payload. Deployed with `OLLAMA_FORMAT_TWO_PASS=1` (ADR 0045) and `OLLAMA_KV_CACHE_TYPE=f16` (ADR 0043) added to the launchd environment, verified by version, the startup config, a live MLX load reporting `0.32.2-65-g59d600b`, and llama-server resolving the repo payload. Preflight PASS 23/12 on a `:11437` stage and again on production itself (`preflight/runs/preflight-mlx-metal-0344-prod-gb43ee8e.json`). Native gates pass: `go test ./mlxrunner/... -p 1 -count=1` with `OLLAMA_VISION_E2E=1`, 459 tests and 347 subtests passed, 0 failed, 1 skipped (`TestGlobalScaleSurvivesStorageBitExactly`), in the 18 of 24 packages that have tests; vision golden parity per model, each named by its manifest digest in the Apple Silicon host's store (ADR 0038): 31b-nvfp4 `637cc0ff1570` and 26b-nvfp4 `c8656f50f0a6`, both the 4-bit-tower artifacts, and 12b-nvfp4 `117d0d84cf2a`, at max sampled element delta 0.0898 / 0.0508 / 0.0625, and 26b-mlx-bf16 `fa29b1495bbb` at 0.0625. `go test ./mlx/... -p 1 -count=1`: 59 tests and 38 subtests passed, 0 failed, 1 skipped (`TestControlsDoNotStopWorker/replay_skip`), with `TestMulGatherQMMGlobalScale` passing on Metal (open item 4 of the [v0.34.4 fold record](../tasks/upstream-sync-0.34.4.md#open-items)). A think+format request on production drafts in pass one only, and its first GGUF load shows `--cache-type-k f16 --cache-type-v f16` and an f16 KV allocation. On 2026-09-29 the archived pairing, on a scratch server in production's environment, gave the fold's think-off answers byte for byte (189 of 189) and scored OCRBench v1 835/1000 against 0.34.0's 835/1000, exact McNemar p = 1.000 (`bench-runs/ocrbench-v1-1000-gemma4-31b-nvfp4-0344-vs-0340.json`). | -| `0.35.0-dynres-0-g043f441` | [`043f441a`](https://github.com/MaxusAI/ollama/commit/043f441a) | llama.cpp **b11081** | 59d600b5 pin | `7ea4fd966cc100f0…` | the `v0.35.0-dynres` tag (#427), a Go-only build at the tag in a detached worktree (`go build` with `build-macos.sh`'s flags and stamp). It runs on `0.34.4-dynres-0-gb43ee8e`'s payload, unchanged, so it has no payload archive of its own: the fold moves no native input. **current** — deployed to `:11435` 2026-10-01 23:45, binary only; the launchd plist is unchanged. Verified by version, the startup config, a live MLX load reporting `0.32.2-65-g59d600b` and a two-pass smoke. Preflight PASS 23/16 on a `:11437` stage and again on production itself (`preflight/runs/preflight-mlx-metal-0350-release-g043f441.json`, `-prod-g043f441.json`), resolved by `mlx-metal-0-34-4`, which now admits 0.35.0 (#437). Native gates pass: `go test ./mlxrunner/... -p 1 -count=1` with `OLLAMA_VISION_E2E=1`, 806 passed and 1 skipped. Rollback: the 0.34.4 binary; the payload is the same | +| `0.35.0-dynres-0-g043f441` | [`043f441a`](https://github.com/MaxusAI/ollama/commit/043f441a) | llama.cpp **b11081** | 59d600b5 pin | `7ea4fd966cc100f0…` | the `v0.35.0-dynres` tag (#427), a Go-only build at the tag in a detached worktree (`go build` with `build-macos.sh`'s flags and stamp). It runs on `0.34.4-dynres-0-gb43ee8e`'s payload, unchanged, so it has no payload archive of its own: the fold moves no native input. served `:11435` 2026-10-01 23:45 → 2026-10-11 09:29, binary only; the launchd plist is unchanged. Verified by version, the startup config, a live MLX load reporting `0.32.2-65-g59d600b` and a two-pass smoke. Preflight PASS 23/16 on a `:11437` stage and again on production itself (`preflight/runs/preflight-mlx-metal-0350-release-g043f441.json`, `-prod-g043f441.json`), resolved by `mlx-metal-0-34-4`, which now admits 0.35.0 (#437). Native gates pass: `go test ./mlxrunner/... -p 1 -count=1` with `OLLAMA_VISION_E2E=1`, 806 passed and 1 skipped. Rollback: the 0.34.4 binary; the payload is the same | +| `0.40.2-dynres-0-g5623192` | [`5623192c`](https://github.com/MaxusAI/ollama/commit/5623192c) | llama.cpp **b11351** (`631109b34`) | a59cc231 pin | `85597909da3be5ae…` | the `v0.40.2-dynres` tag (#460), both halves built at the tag in a detached worktree; payload archived as `payload-0.40.2-dynres-0-g5623192`. Served `:11435` 2026-10-11 09:29 → 2026-10-13 22:01. Preflight PASS 23/16 on `mlx-metal-0-40-2`, stage and production. **Rollback target** for `0.40.2-dynres.1-0-g3960244`: restore this binary and its payload, same plist | +| `0.40.2-dynres.1-0-g3960244` | [`39602449`](https://github.com/MaxusAI/ollama/commit/39602449) | llama.cpp **b11351** (`631109b34`) | a59cc231 pin | `023b4082a42240cf…` | the `v0.40.2-dynres.1` point tag, both halves built at the tag in a detached worktree; payload archived as `payload-0.40.2-dynres.1-0-g3960244`. Go-only on Metal: the tag's one native change (#462's 903) is `ggml-cuda/mmq.cu`; 14 of 19 libraries are byte-identical to 0.40.2's and 5 differ only by the build path in `__FILE__` strings. **current** — deployed to `:11435` 2026-10-13 22:01 by a stopped swap of both halves; the launchd plist is unchanged. Native gate `go test ./mlx/... ./mlxrunner/... -p 1` with `OLLAMA_VISION_E2E=1`: 1296 passed, 0 failed, 8 skipped. Preflight PASS 23/16 on `mlx-metal-0-40-2`, stage and production (`preflight/runs/preflight-mlx-metal-0402-dynres1-release-g3960244.json`, `-prod-g3960244.json`), plus the deploy smoke (two-pass MLX and converted GGUF, GGUF vision, System One 400) | Full checksums: @@ -34,6 +36,8 @@ cd6033d6e72a430e0b96170adb3d7408c908edfdae392654faeaae46f31e9ee2 ~/.ollama/bina 867ee5d6063d8f77be6d8278c51d18f49892f00aa80c38c51630227aa9f68a57 ~/.ollama/binaries/ollama-0.34.0-maxusai-8a7ba949 f343fea9ea7bc66e7c85fd47500fb2b6e5cc45aacc2f0dad49f7adac06940d83 ~/.ollama/binaries/ollama-0.34.4-dynres-0-gb43ee8e 7ea4fd966cc100f085e7c5d1e7e68096e538fb989af39e03a7e0204116eae249 ~/.ollama/binaries/ollama-0.35.0-dynres-0-g043f441 +85597909da3be5ae8f3d4a203c80959bd9c1743ce834fa3c69b1de4f343ee434 ~/.ollama/binaries/ollama-0.40.2-dynres-0-g5623192 +023b4082a42240cf237f5f77e078d3bad1f99b0c573592fe46915a875d446401 ~/.ollama/binaries/ollama-0.40.2-dynres.1-0-g3960244 ``` ## What b10353 is the provenance for diff --git a/docs/maxusai/vision-suite/preflight/expectations.toml b/docs/maxusai/vision-suite/preflight/expectations.toml index bb8ff15ef62..09dc4e89250 100644 --- a/docs/maxusai/vision-suite/preflight/expectations.toml +++ b/docs/maxusai/vision-suite/preflight/expectations.toml @@ -779,9 +779,11 @@ platform = "mlx-metal" # (631109b34), with MLX-C unchanged at ebc88f10. A new profile by the ADR 0011 rule # that created every predecessor: a payload move gets its own profile even when # the numbers turn out identical, because "identical" is a measurement. -version_pattern = '^0\.40\.2-dynres-\d+-g[0-9a-f]{7,40}$' -# Admits the v0.40.2-dynres tag's lineage and nothing else: not -dirty, not a .N -# point tag, not the legacy 0.34.x-maxusai- shape. The fold's interim builds +version_pattern = '^0\.40\.2-dynres(\.\d+)?-\d+-g[0-9a-f]{7,40}$' +# Admits the v0.40.2-dynres tag's lineage and its .N point tags (v0.40.2-dynres.1 +# moves Go only on Metal; its one native change, #462's 903, is in ggml-cuda/mmq.cu, +# which this build does not compile), and nothing else: not -dirty, not the legacy +# 0.34.x-maxusai- shape. The fold's interim builds # stamp 0.35.0-dynres-N (mlx-metal-0-34-4's lineage, whose pins they fail); the # Metal host validated the interim build with a campaign copy of this file whose # only change pinned this pattern to that one stamp (#460). @@ -819,6 +821,19 @@ APFS clone of production's (#460). The four arches below are MLX safetensors and are not converted; the clone does not change what they load. aspect_ladder and the think_format probes are carried from 0-34-4 unchanged and run live on every preflight. + +THE v0.40.2-dynres.1 RELEASE BUILD PASSES AND IS DEPLOYED, 2026-10-13: +0.40.2-dynres.1-0-g3960244, built at the point tag in a detached worktree, VERDICT +PASS, PASS=23 SKIP=16 on a :11437 stage against a cloned store +(runs/preflight-mlx-metal-0402-dynres1-release-g3960244.json), and again on +production :11435 after the stopped swap (-prod-g3960244.json). 36 of 39 rows +match the v0.40.2 release run byte for byte; the other three are the stamp, the +stage path and gemma4_unified think_format's token count (242 stage, 739 prod), +which repeats as 496/260/260 on production and 242/260/261 on the stage: the +first request after a load drafts to a different depth, then both settle at 260. +The payload is the v0.40.2 one rebuilt: 14 of 19 libraries byte-identical, and +the other 5 differ only by the build worktree's path in __FILE__ strings (same +xgrammar 82505d0d source, same MLX a59cc231 and llama.cpp 631109b34 pins). """ [profiles.metal-0-32-14] diff --git a/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0402-dynres1-prod-g3960244.json b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0402-dynres1-prod-g3960244.json new file mode 100644 index 00000000000..0c9efdf462e --- /dev/null +++ b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0402-dynres1-prod-g3960244.json @@ -0,0 +1,594 @@ +{ + "meta": { + "host": "http://127.0.0.1:11435", + "platform": "mlx-metal", + "image_tag": null, + "started_utc": "20261013T120212", + "expectations_file": "expectations.toml", + "schema_version": 1, + "version": "0.40.2-dynres.1-0-g3960244", + "profile": "mlx-metal-0-40-2", + "patchset": [], + "container": null, + "llama_cpp_build": "631109b34", + "mlx_build": "0.32.3-53-ga59cc23" + }, + "results": [ + { + "check": "version", + "arch": null, + "status": "PASS", + "summary": "0.40.2-dynres.1-0-g3960244 matches profile mlx-metal-0-40-2", + "expected": "^0\\.40\\.2-dynres(\\.\\d+)?-\\d+-g[0-9a-f]{7,40}$", + "actual": "0.40.2-dynres.1-0-g3960244" + }, + { + "check": "payload_pin", + "arch": null, + "status": "PASS", + "summary": "llama.cpp build 631109b34 matches the measured payload (resolved from the executable listening on :11435)", + "expected": "631109b34", + "actual": "631109b34" + }, + { + "check": "toolchain_pin", + "arch": null, + "status": "SKIP", + "summary": "profile records no toolchain_build to assert against", + "diagnosis": "Add toolchain_build (e.g. \"rocm-7.2.4\" or \"cuda-13.0\") so a ROCm/CUDA bump under an unchanged payload fails loudly instead of inheriting these expectations. Platforms with no such toolchain (metal, cpu) leave it unset and keep this skip; the MLX library is pinned separately by mlx_build." + }, + { + "check": "image_tag", + "arch": null, + "status": "SKIP", + "summary": "no container resolved (remote host?)" + }, + { + "check": "go_patch_marker", + "arch": null, + "status": "SKIP", + "summary": "no container resolved; cannot grep the binary", + "expected": 1 + }, + { + "check": "metal_tensor_host", + "arch": null, + "status": "PASS", + "summary": "Apple M5 Max has_tensor=true (supports_metal4_family=true, name_allowlisted=true, dummy_kernel_compiles=true), under the server's environment", + "expected": true, + "actual": true, + "probe": { + "device": "Apple M5 Max", + "supports_metal4_family": true, + "name_allowlisted": true, + "dummy_kernel_compiles": true, + "pipeline_ms": 6.6, + "has_tensor": true + } + }, + { + "check": "metal_tensor_runtime", + "arch": null, + "status": "PASS", + "summary": "no tensor-API fallback since this server started", + "expected": "no discovery fallback", + "actual": "clean discovery" + }, + { + "check": "poison_probe", + "arch": null, + "status": "SKIP", + "summary": "no poison-probe expectation recorded for this profile" + }, + { + "check": "text_baseline", + "arch": "gemma4", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19 + }, + { + "check": "token_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the MLX runner emits none, on either backend) and are recorded as budgets_observed = false. Nothing here is waiting on a container; the block says how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4", + "status": "PASS", + "summary": "valid JSON after thinking (138 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 184, + "thinking_chars": 373, + "eval_count": 138 + } + }, + { + "check": "extraction_quality", + "arch": "gemma4", + "status": "SKIP", + "summary": "no quality thresholds recorded for this arch" + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19 + }, + { + "check": "token_ladder", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the MLX runner emits none, on either backend) and are recorded as budgets_observed = false. Nothing here is waiting on a container; the block says how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "valid JSON after thinking (739 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 245, + "thinking_chars": 2677, + "eval_count": 739 + } + }, + { + "check": "extraction_quality", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "no quality thresholds recorded for this arch" + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "qwen35moe", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13 + }, + { + "check": "token_ladder", + "arch": "qwen35moe", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "actual": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "rows": [ + { + "size": "256x144", + "expected": 68, + "actual": 68, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 146, + "actual": 146, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 5186, + "actual": 5186, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the MLX runner emits none, on either backend) and are recorded as budgets_observed = false. Nothing here is waiting on a container; the block says how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "qwen35moe", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "qwen35moe", + "status": "PASS", + "summary": "valid JSON after thinking (889 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 347, + "thinking_chars": 3328, + "eval_count": 889 + } + }, + { + "check": "extraction_quality", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "no quality thresholds recorded for this arch" + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "qwen35", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13 + }, + { + "check": "token_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "actual": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "rows": [ + { + "size": "256x144", + "expected": 68, + "actual": 68, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 146, + "actual": 146, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 5186, + "actual": 5186, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the MLX runner emits none, on either backend) and are recorded as budgets_observed = false. Nothing here is waiting on a container; the block says how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "qwen35", + "status": "PASS", + "summary": "valid JSON after thinking (267 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 310, + "thinking_chars": 779, + "eval_count": 267 + } + }, + { + "check": "extraction_quality", + "arch": "qwen35", + "status": "SKIP", + "summary": "no quality thresholds recorded for this arch" + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "metal_tensor_payload", + "arch": null, + "status": "PASS", + "summary": "42 GGML_METAL_HAS_TENSOR strings in /opt/github/MaxusAI/ollama/build/lib/ollama/llama-server (resolved from the executable listening on :11435)", + "expected": true, + "actual": 42, + "path": "/opt/github/MaxusAI/ollama/build/lib/ollama/llama-server" + }, + { + "check": "mlx_payload_pin", + "arch": null, + "status": "PASS", + "summary": "MLX build a59cc23 matches the measured payload (0.32.3-53-ga59cc23), from the runner's engine-init line", + "expected": "a59cc23190f1e26fabcba8693607bb4c34b2ac2b", + "actual": "0.32.3-53-ga59cc23" + }, + { + "check": "endpoint_exclusive", + "arch": null, + "status": "PASS", + "summary": "no contention detected (worst queue wait 0.0s)", + "expected": "queue wait <= 10.0s", + "actual": "0.0s", + "models_loaded": [ + "qwen3.8:27b-nvfp4", + "qwen3.6:35b-a3b-nvfp4", + "gemma4:12b-nvfp4" + ] + } + ], + "summary": { + "PASS": 23, + "SKIP": 16 + } +} \ No newline at end of file diff --git a/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0402-dynres1-release-g3960244.host-profile.json b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0402-dynres1-release-g3960244.host-profile.json new file mode 100644 index 00000000000..0bd7cac57f5 --- /dev/null +++ b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0402-dynres1-release-g3960244.host-profile.json @@ -0,0 +1,98 @@ +{ + "schema": "host-profile/1", + "collected_at": "2026-10-13T11:57:17Z", + "label": "macbook-pro-m5-max-128GB/mlx-metal", + "label_host": "macbook-pro-m5-max-128GB", + "label_surface": "mlx-metal", + "machine": { + "vendor": "Apple", + "model": "MacBook Pro", + "model_id": "Mac17,7", + "virtualized": false, + "hypervisor": null + }, + "cloud": null, + "os": { + "family": "darwin", + "name": "macOS", + "version": "26.6.2", + "pretty_name": "macOS 26.6.2", + "kernel": "25.6.0", + "arch": "arm64" + }, + "cpu": { + "vendor": "apple", + "model": "Apple M5 Max", + "logical_cpus": 18, + "sockets": 1, + "cores": 18, + "threads_per_core": 1, + "performance_cores": 6, + "efficiency_cores": 12, + "numa_nodes": null, + "l3_cache_mib": null, + "max_mhz": null, + "flags": { + "bf16": true, + "i8mm": true, + "sme": true, + "sme2": true + } + }, + "memory_total_gib": 128.0, + "gpu_count": 1, + "gpus": [ + { + "vendor": "apple", + "name": "Apple M5 Max", + "pci_name": null, + "pci_bus_id": null, + "pci_vendor_id": null, + "pci_device_id": null, + "driver": null, + "memory_total_mib": null, + "memory_gtt_mib": null, + "unified_memory": true, + "compute_capability": null, + "gfx_target": null, + "gpu_cores": 40, + "metal_family": "metal4", + "vbios": null, + "power_limit_w": null, + "power_max_limit_w": null, + "max_core_clock_mhz": null, + "max_memory_clock_mhz": null, + "ecc_enabled": null, + "mig_enabled": null, + "persistence_enabled": null, + "pcie_link_gen": null, + "pcie_link_width": null, + "pcie_device_max_gen": null, + "pcie_device_max_width": null, + "pcie_upstream_max_gen": null, + "pcie_upstream_max_width": null, + "pcie_gen_now": null, + "pcie_width_now": null, + "numa_node": null, + "local_cpus": null, + "index": 0 + } + ], + "filesystems": [], + "software": { + "nvidia_driver_version": null, + "nvidia_kernel_module": null, + "cuda_driver_version": null, + "cuda_toolkit_version": null, + "cudnn_version": null, + "nccl_version": null, + "rocm_version": null, + "amdgpu_driver_version": null, + "docker_version": "29.8.2", + "nvidia_container_toolkit_version": null, + "dcgm_version": null, + "ollama_version": "0.40.2-dynres.1-0-g3960244", + "go_version": "1.27.2", + "python_version": "3.14.8" + } +} diff --git a/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0402-dynres1-release-g3960244.json b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0402-dynres1-release-g3960244.json new file mode 100644 index 00000000000..f041f3a79cf --- /dev/null +++ b/docs/maxusai/vision-suite/preflight/runs/preflight-mlx-metal-0402-dynres1-release-g3960244.json @@ -0,0 +1,592 @@ +{ + "meta": { + "host": "http://127.0.0.1:11437", + "platform": "mlx-metal", + "image_tag": null, + "started_utc": "20261013T115717", + "expectations_file": "expectations.toml", + "schema_version": 1, + "version": "0.40.2-dynres.1-0-g3960244", + "profile": "mlx-metal-0-40-2", + "patchset": [], + "container": null, + "llama_cpp_build": "631109b34", + "mlx_build": "0.32.3-53-ga59cc23" + }, + "results": [ + { + "check": "version", + "arch": null, + "status": "PASS", + "summary": "0.40.2-dynres.1-0-g3960244 matches profile mlx-metal-0-40-2", + "expected": "^0\\.40\\.2-dynres(\\.\\d+)?-\\d+-g[0-9a-f]{7,40}$", + "actual": "0.40.2-dynres.1-0-g3960244" + }, + { + "check": "payload_pin", + "arch": null, + "status": "PASS", + "summary": "llama.cpp build 631109b34 matches the measured payload (resolved from the executable listening on :11437)", + "expected": "631109b34", + "actual": "631109b34" + }, + { + "check": "toolchain_pin", + "arch": null, + "status": "SKIP", + "summary": "profile records no toolchain_build to assert against", + "diagnosis": "Add toolchain_build (e.g. \"rocm-7.2.4\" or \"cuda-13.0\") so a ROCm/CUDA bump under an unchanged payload fails loudly instead of inheriting these expectations. Platforms with no such toolchain (metal, cpu) leave it unset and keep this skip; the MLX library is pinned separately by mlx_build." + }, + { + "check": "image_tag", + "arch": null, + "status": "SKIP", + "summary": "no container resolved (remote host?)" + }, + { + "check": "go_patch_marker", + "arch": null, + "status": "SKIP", + "summary": "no container resolved; cannot grep the binary", + "expected": 1 + }, + { + "check": "metal_tensor_host", + "arch": null, + "status": "PASS", + "summary": "Apple M5 Max has_tensor=true (supports_metal4_family=true, name_allowlisted=true, dummy_kernel_compiles=true), under the server's environment", + "expected": true, + "actual": true, + "probe": { + "device": "Apple M5 Max", + "supports_metal4_family": true, + "name_allowlisted": true, + "dummy_kernel_compiles": true, + "pipeline_ms": 3.5, + "has_tensor": true + } + }, + { + "check": "metal_tensor_runtime", + "arch": null, + "status": "PASS", + "summary": "no tensor-API fallback since this server started", + "expected": "no discovery fallback", + "actual": "clean discovery" + }, + { + "check": "poison_probe", + "arch": null, + "status": "SKIP", + "summary": "no poison-probe expectation recorded for this profile" + }, + { + "check": "text_baseline", + "arch": "gemma4", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19 + }, + { + "check": "token_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the MLX runner emits none, on either backend) and are recorded as budgets_observed = false. Nothing here is waiting on a container; the block says how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "gemma4", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4", + "status": "PASS", + "summary": "valid JSON after thinking (138 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 184, + "thinking_chars": 373, + "eval_count": 138 + } + }, + { + "check": "extraction_quality", + "arch": "gemma4", + "status": "SKIP", + "summary": "no quality thresholds recorded for this arch" + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "in-image text prefix = 19 (calibrated on 256x144+512x288); text-only = 19", + "actual": 19 + }, + { + "check": "token_ladder", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "actual": { + "256x144": 1102, + "512x288": 1102, + "1024x576": 1102, + "2048x1152": 1102, + "3072x1728": 1102 + }, + "rows": [ + { + "size": "256x144", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 1102, + "actual": 1102, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "flat", + "expected_shape": "flat" + }, + { + "check": "payload_proof", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the MLX runner emits none, on either backend) and are recorded as budgets_observed = false. Nothing here is waiting on a container; the block says how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + }, + "actual": { + "1024x768": 1066, + "1920x480": 1058, + "768x768": 1091 + } + }, + { + "check": "think_format", + "arch": "gemma4_unified", + "status": "PASS", + "summary": "valid JSON after thinking (242 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 273, + "thinking_chars": 803, + "eval_count": 242 + } + }, + { + "check": "extraction_quality", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "no quality thresholds recorded for this arch" + }, + { + "check": "pinned_image_token_budget", + "arch": "gemma4_unified", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "qwen35moe", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13 + }, + { + "check": "token_ladder", + "arch": "qwen35moe", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "actual": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "rows": [ + { + "size": "256x144", + "expected": 68, + "actual": 68, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 146, + "actual": 146, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 5186, + "actual": 5186, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the MLX runner emits none, on either backend) and are recorded as budgets_observed = false. Nothing here is waiting on a container; the block says how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "qwen35moe", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "qwen35moe", + "status": "PASS", + "summary": "valid JSON after thinking (889 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 347, + "thinking_chars": 3328, + "eval_count": 889 + } + }, + { + "check": "extraction_quality", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "no quality thresholds recorded for this arch" + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35moe", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "text_baseline", + "arch": "qwen35", + "status": "PASS", + "summary": "in-image text prefix = 13 (calibrated on 256x144+512x288); text-only = 13", + "actual": 13 + }, + { + "check": "token_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "5/5 geometries within +/-2", + "expected": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "actual": { + "256x144": 68, + "512x288": 146, + "1024x576": 578, + "2048x1152": 2306, + "3072x1728": 5186 + }, + "rows": [ + { + "size": "256x144", + "expected": 68, + "actual": 68, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "512x288", + "expected": 146, + "actual": 146, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "1024x576", + "expected": 578, + "actual": 578, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "2048x1152", + "expected": 2306, + "actual": 2306, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + }, + { + "size": "3072x1728", + "expected": 5186, + "actual": 5186, + "delta": 0, + "ok": true, + "queue_wait_s": 0.0 + } + ], + "shape": "scaling", + "expected_shape": "dynamic" + }, + { + "check": "payload_proof", + "arch": "qwen35", + "status": "SKIP", + "summary": "budget/pixel values are not observable on this platform", + "diagnosis": "This block's budgets were established without a load log (the MLX runner emits none, on either backend) and are recorded as budgets_observed = false. Nothing here is waiting on a container; the block says how the values were established." + }, + { + "check": "aspect_ladder", + "arch": "qwen35", + "status": "PASS", + "summary": "3/3 aspect ratios within +/-2", + "expected": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + }, + "actual": { + "1024x768": 770, + "1920x480": 902, + "768x768": 578 + } + }, + { + "check": "think_format", + "arch": "qwen35", + "status": "PASS", + "summary": "valid JSON after thinking (268 tokens)", + "expected": "non-empty JSON response with think:true + format:json", + "actual": { + "response_chars": 310, + "thinking_chars": 779, + "eval_count": 268 + } + }, + { + "check": "extraction_quality", + "arch": "qwen35", + "status": "SKIP", + "summary": "no quality thresholds recorded for this arch" + }, + { + "check": "pinned_image_token_budget", + "arch": "qwen35", + "status": "SKIP", + "summary": "no pinned-budget expectation recorded for this arch", + "diagnosis": "Omitted deliberately: the 005 defect was found on nemotron and this arch's probe has not been measured. See README.md to add one." + }, + { + "check": "metal_tensor_payload", + "arch": null, + "status": "PASS", + "summary": "42 GGML_METAL_HAS_TENSOR strings in ~/.ollama/binaries/stage-0.40.2-dynres.1-0-g3960244/lib/ollama/llama-server (resolved from the executable listening on :11437)", + "expected": true, + "actual": 42, + "path": "~/.ollama/binaries/stage-0.40.2-dynres.1-0-g3960244/lib/ollama/llama-server" + }, + { + "check": "mlx_payload_pin", + "arch": null, + "status": "PASS", + "summary": "MLX build a59cc23 matches the measured payload (0.32.3-53-ga59cc23), from the runner's engine-init line", + "expected": "a59cc23190f1e26fabcba8693607bb4c34b2ac2b", + "actual": "0.32.3-53-ga59cc23" + }, + { + "check": "endpoint_exclusive", + "arch": null, + "status": "PASS", + "summary": "no contention detected (worst queue wait 0.0s)", + "expected": "queue wait <= 10.0s", + "actual": "0.0s", + "models_loaded": [ + "qwen3.8:27b-nvfp4" + ] + } + ], + "summary": { + "PASS": 23, + "SKIP": 16 + } +} \ No newline at end of file diff --git a/docs/maxusai/vision-suite/preflight/test_verdicts.py b/docs/maxusai/vision-suite/preflight/test_verdicts.py index 9b03981975e..d0dbb7ab37b 100644 --- a/docs/maxusai/vision-suite/preflight/test_verdicts.py +++ b/docs/maxusai/vision-suite/preflight/test_verdicts.py @@ -1902,7 +1902,8 @@ class TestMlxMetalProfileForTheV0402Fold(unittest.TestCase): """ADR 0011 on Apple Silicon. The v0.40.2 fold (#460) moves BOTH native inputs: MLX 59d600b5 -> a59cc231 and llama.cpp b11081 -> b11351. So it gets its own profile, mlx-metal-0-40-2, rather than a widened 0-34-4. - The new profile admits the v0.40.2-dynres tag's lineage only. 0-34-4 keeps production's 0.35.0 build and must not + The new profile admits the v0.40.2-dynres tag's lineage and its .N point tags (v0.40.2-dynres.1 moves Go only on + Metal; its one native change is in ggml-cuda), and nothing else. 0-34-4 keeps production's 0.35.0 build and must not admit 0.40.2: that payload is not its own. The fold's interim builds stamp 0.35.0-dynres-N, which 0-34-4 admits and whose pins they fail; they were validated with a campaign copy of this file, not by widening either pattern. """ @@ -1911,13 +1912,14 @@ class TestMlxMetalProfileForTheV0402Fold(unittest.TestCase): ADMIT = ( "0.40.2-dynres-0-gabcdef0", # the fold's tag stamp "0.40.2-dynres-5-g0123456", # a build after that tag + "0.40.2-dynres.1-0-gabcdef0", # the v0.40.2-dynres.1 point tag (Go-only on Metal) ) REJECT = ( "0.35.0-dynres-0-g043f441", # production's build: mlx-metal-0-34-4's "0.35.0-dynres-27-g948ef3a", # the fold's interim stamp: not admitted by widening "0.40.1-dynres-0-gabcdef0", # an upstream release this fold passed through "0.40.2-dynres-0-gabcdef0-dirty", # a dirty tree describes nothing - "0.40.2-dynres.1-0-gabcdef0", # a point tag is a new deploy: widen deliberately + "0.40.2-dynres.1-0-gabcdef0-dirty", # nor does a dirty point-tag tree ) @classmethod @@ -1942,8 +1944,9 @@ def test_the_previous_profile_does_not_claim_v0402(self): def test_the_tag_stamp_resolves_to_this_profile_through_preflight(self): import preflight - pid, _ = preflight.resolve_profile(self.exp, "mlx-metal", "0.40.2-dynres-0-gabcdef0") - self.assertEqual(pid, self.PROFILE) + for stamp in ("0.40.2-dynres-0-gabcdef0", "0.40.2-dynres.1-0-gabcdef0"): + pid, _ = preflight.resolve_profile(self.exp, "mlx-metal", stamp) + self.assertEqual(pid, self.PROFILE, stamp) def test_the_pins_name_the_fold_payload(self): prof = self.exp["profiles"][self.PROFILE]