ClefFlash: clef-flash, every question of a request read in one pass (KitClefDecider, kind jointDecision, images on /v1/systemone) by john-rocky · Pull Request #66 · john-rocky/coreai-kit

john-rocky · 2026-10-03T16:18:26Z

Adds clef-flash (Cloudflare, Apache-2.0, from Qwen3.5-9B) to the kit: the catalog entry clef-flash under a new kind, jointDecision; KitClefDecider (Sources/CoreAIKit/ClefFlash/); and an images field on /v1/systemone, so systemone serve --model clef-flash answers questions about one image.

The model. A state (text, JSON or one image) and typed questions (noul, choice, score) go in, and every option of every question comes back with a probability. Nothing is generated. A joint schema head reads the decoder's hidden states for all the questions at once; each question is a softmax over its own options.

The host. The model zoo's apps/ClefFlash library, ported with every numeric path unchanged. The request is rendered as the author's encode_record(). The image is resized by Pillow's own integer bicubic to a fixed grid (g448 by default, or g256) and read by that grid's tower. The decoder runs without a vocabulary head. The head graph then gives each option its logit, reading the lm_head table on the host. Mac only. systemone serve and the MCP server load it through the new SystemOne.backend(catalog:); decide-cli routes on the kind. Any other model answers a request carrying images with a 422 that names the model.

Catalog. mlboydaisuke/clef-flash-CoreAI @ 220ac149, the repo's main. systemone models lists it at 18.2 GB: the decoder, the head and the lm_head table. A tower downloads when an image asks for its grid. The new assets field names the head, the table and the towers, and format: jointHead says how the entry is read.

Shipped kits. Kits 0.1.0–0.7.3 decode jointDecision as unknown and leave the entry out of available(_:). Every list in those kits goes through available(_:), so none of them shows clef-flash. TypedDecisions(catalog: "clef-flash") refuses it at its kind check, before any download. With kind: decision, 0.5.0–0.7.3 would list it, and loading it there would download the decoder for a TypedDecisions that cannot read it. Measured on all twelve tags: each tag's own catalog code (ModelCatalog.swift and the files it needs), compiled with a small main, decodes all 73 entries of this branch's catalog.json. It reports clef-flash as unknown, in neither available() nor available(.decision).

Gates (M4 Max, macOS 27.0 26A428, Xcode 27.0 RC). decide-cli parity (Release) on the zoo's fixture, with the model downloaded from the catalog pin, against the author's fp32 read-out:

set	runs	ids and spans = author	argmax (top-2 margin > 0.02)	near-ties agreeing	max \|Δp\|
fixture	200	200/200	352/352	3/3	0.0122
held out	40	40/40	186/186	2/2	0.0082

Against the zoo's own Swift run of the same files, the logits, probabilities, hidden-state digests and responses are bit-equal on all 240 runs. The fixture's 14 native runs (the processor's own grid) are skipped: no tower here has that grid.

The parity ran before the kind change, which touches no numeric path. After the change:

KIT_CLEFFLASH_GATE=1 swift test --filter ClefFlash: 12/12 fixture runs bit-equal to the zoo's Swift run, each with the same hidden-state digest as before the change.
systemone serve --model clef-flash: conformance/check.py 25/25. The fixture's receipt sent in images returns the zoo Swift run's probabilities to the wire's four decimals, 14/14 as a data URL and 14/14 as a file path.
The whole suite: XCTest 100 (23 skipped, 0 failures), swift-testing 276 passed.
systemone models (built-in catalog) and decide-cli --list-models list clef-flash as jointDecision.

Not in this PR. No release and no pin bump: no existing catalog pin moves, and Package.resolved is unchanged. The int8mix decoder in the same Hub repo stays out of the catalog. No iOS variant. CoreAI.decide and CoreAI.systemOne load through TypedDecisions(catalog:), so they refuse clef-flash by name before any download; the op route is not added. Use KitClefDecider or systemone serve.

Check it yourself. The test and the server each download the model from the catalog pin unless it is cached (18.2 GB).

# The fixture's images: the model zoo's conversion/clef_flash/make_fixtures.py --fetch-photos
KIT_CLEFFLASH_GATE=1 KIT_CLEFFLASH_IMAGES=<its fixtures/images> swift test --filter ClefFlash

swift build -c release --product systemone
.build/release/systemone serve --model clef-flash --port 8091 &
python3 Examples/Decide/conformance/check.py http://127.0.0.1:8091 --timeout 600

Docs. CHANGELOG (Unreleased / Added), README (the Decide line, 73 entries), AGENTS.md (73 entries), docs/SYSTEM_ONE.md (the images field and a new section, "Every question at once: clef-flash").

… image on /v1/systemone Cloudflare's clef-flash (Apache-2.0, from Qwen3.5-9B) through the new KitClefDecider: every question of a request read in one pass by its joint schema head, the state as text or JSON and at most one image. The host is the model zoo's apps/ClefFlash (5ef2247), ported into Sources/CoreAIKit/ClefFlash/ with names prefixed Clef* and every numeric path unchanged; its Float16 files sit behind the kit's Intel-Mac guard. - catalog.json and the built-in literal: clef-flash, kind decision with the new format jointHead (Decision.Format gains the case), a new `assets` field naming the head, the lm_head table and a tower per grid; macOS only, 18,178 MB as measured. TypedDecisions refuses a jointHead entry before it downloads anything. - /v1/systemone: optional `images` (one image: a data URL, base64, or a file path, which SystemOneServer reads only on the loopback address) and `grid`. DecisionBackend.readsImages and imageRefusal: a backend that reads no images answers with a 422 that names it. systemone serve/ask/models, decide-cli serve and the MCP server load through SystemOne.backend(catalog:). - decide-cli: ask and serve with --model clef-flash; parity on the zoo's clef-flash fixture, with --reference comparing the zoo's Swift run bit for bit. - ClefFlashTests: the entry, the wire, the request in the author's form; opt-in, twelve fixture runs bit for bit against the zoo's Swift run. Gate (M4 Max, macOS 27.0 26A428, the catalog pin downloaded by ModelStore, sha256 = the pin's SHA256SUMS): decide-cli parity on 240 fixture runs PASS (max |dp| 0.0122 / 0.0082 against the author's fp32), logits, probabilities, hidden rows and responses bit-equal to the zoo's Swift run on 240/240; ClefFlashTests 12/12 bit-equal.

CHANGELOG (Unreleased / Added) and docs/SYSTEM_ONE.md: what clef-flash is and how it is read, the `images` / `grid` fields of the wire, the option limit, and the gate numbers measured on this branch (decide-cli parity on 240 fixture runs, bit-equal to the zoo's Swift run; conformance 22/22 and an image request through systemone serve).

…ts it nowhere `kind: decision` put clef-flash in every shipped kit's `systemone models` (0.6.0-0.7.3) and in any list of `available(.decision)`, where loading it downloads the 15.9 GB decoder for a `TypedDecisions` that cannot read it. A new kind decodes as `unknown` on every tag 0.1.0-0.7.3: out of `available(_:)`, and `TypedDecisions(catalog:)` refuses it at its kind check before any download. - catalog.json and the built-in literal: `kind: jointDecision`; `format: jointHead` and `assets` stay. - `CatalogEntry.Kind.jointDecision`. `KitClefDecider.supports` and `init(catalog:)` expect it; `SystemOne.backend(catalog:)` and decide-cli route on it; `decide-cli --list-models` lists it. `TypedDecisions` keeps its format refusal as a second guard. - ClefFlashTests: `TypedDecisions(catalog: "clef-flash")` throws `catalogKindMismatch` naming the model, with nothing downloaded; a `decision` entry that says `jointHead` is neither loader's. - CHANGELOG, docs/SYSTEM_ONE.md, README: the kind; the fixture's 14 `native` runs and the 13 the zoo's gate ran, as the zoo card counts them. M4 Max, macOS 27.0 26A428: `KIT_CLEFFLASH_GATE=1 swift test --filter ClefFlash` 12/12 runs bit-equal to the zoo's Swift run, hidden-state sha256 equal to the previous commit's run on all 12; the whole suite passes (XCTest 100, 23 skipped; swift-testing 276); `systemone serve --model clef-flash` passes conformance/check.py 25/25 and answers the receipt in `images` 14/14 as a data URL and as a path; `systemone models` (built-in catalog) and `decide-cli --list-models` show clef-flash as jointDecision. Each tag's own ModelCatalog.swift, compiled against this catalog.json, decodes clef-flash as unknown.

…the MCP server through SystemOne.backend(catalog:), decide-cli ask and serve through its kind check

…d at its option markers (KitD1OmniDecider, kind omniDecision, audio on /v1/systemone) (#68) Adds Liquid AI's d1-omni-600M (LFM Open License v1.0) to the kit: the catalog entry `d1-omni-600m` under a new kind, `omniDecision`; `KitD1OmniDecider` (`Sources/CoreAIKit/D1Omni/`); `audio` on `/v1/systemone`; and the routes that load it, so `systemone serve --model d1-omni-600m` answers requests about a text, an image or a voice clip. **The model.** A state (text or JSON), images or one 16 kHz clip, and typed questions (noul, choice, score) go in. Every option of every question comes back with a probability; nothing is generated. Each question is its own row in the publisher's form, with a `<|mask|>` marker before each option. One forward pass of a static decision graph scores every position, and each option is read at its marker. A text row is divided by the publisher's temperature for its type and option count, then softmaxed in float32. The checkpoint is an LFM2.5-Encoder-350M trunk with a decision head, a SigLIP2 vision tower and a FastConformer audio tower. **The host.** The model zoo's `apps/D1Omni` library (f9e0e09), ported with every numeric path unchanged. Across its twelve files only names (prefixed `D1`; the zoo's `D1Omni` is `D1OmniPipeline`), access, `@available(macOS 27, iOS 27, *)` and a provenance line differ (`port_d1.py`, in the lane). One change for speed: a request's state is tokenized once for all its questions, since swift-transformers takes 0.4 s on a 3.4k-token state. The decision graph has seven lengths (64 to 4,096 positions); a row runs at the smallest that holds it. A row past 4,096 positions is refused with a 422. Images go through the vision graph once per crop, a clip through the audio graph of its 5 / 10 / 20 / 30 s bucket, and their rows go in front of each question's ids. `KitD1OmniDecider` is a `DecisionBackend` that reads images and audio; `readout(requestJSON:images:audio:)` gives each row, its graph length, the marker logits and the response in the publisher's form. **The wire.** `/v1/systemone` gains `audio`: one 16 kHz mono 16-bit WAV as a data URL, base64, or a file path (`SystemOneServer` reads a path only on 127.0.0.1). With `audio`, `state` may be left out: the model then reads the `{}` its audio questions were trained on. Every other backend refuses a request with `audio` by name (422), through a new `DecisionBackend.audioRefusal` beside `imageRefusal`. For d1-omni, `images` is the existing field (#66) and `grid` is refused. The MCP `decide` tool's schema gains `audio`. **Catalog.** `mlboydaisuke/d1-omni-600M-CoreAI` @ `914184e5`, the repo's main. `variants.macos.path = "macos"` and `variants.ios.path = "ios"`, 6,458 MB each (`measure-catalog-sizes.py`; the same `.aimodel`s). `format: markerScores`. The driver downloads the platform folder's twelve bundles at first use, each as its own download at `<variant>/<folder>`. The store takes such a path as given, so an iPhone reads the JIT `ios/` graphs. It never takes the repo's `ios-h19p/` AOT set, which has no L4096 (its compile exceeds the phone's memory limit). A graph loads the first time a row or a medium needs it. **Shipped kits.** The bundle's metadata says `decision.head = "encoder"`, but its rows, temperatures and media prefixes are not laya's. With `kind: decision`, a shipped kit would list it and download 6.5 GB for a `TypedDecisions` that cannot read it. Kits 0.1.0–0.7.3 decode any kind they do not know as `unknown` and leave the entry out of `available(_:)`: each tag's `ModelCatalog.swift` has the `?? .unknown` decode and the filter (read, not compiled for this kind; #67 compiled the same tags for `rowDecision`). `TypedDecisions(catalog: "d1-omni-600m")` refuses it at its kind check, before any download. **Tests** (M4 Max, macOS 27.0 26A428, Xcode 27.0 RC; the kit resolves swift-transformers 1.3.3, the version the zoo gated). - Without files (CI): the catalog entry, the kind refusals before a download, the wire's `audio` (forms, refusals, a clip without a state, the loopback rule), the request in the publisher's form, the option order, and the readout of the fixture's 38 smoke rows from the zoo Swift host's marker logits: the probabilities bit for bit, and all 14 responses byte-equal to the publisher's `response()` text (host.py under Python 3.12, `json.dumps(ensure_ascii=False)`). - With the repo's `tokenizer/` and `reference/` at the pin: the zoo's 360 public fixture rows rebuilt by the kit, ids, markers and graph length equal to the publisher's, 360 of 360. - With the bundles (`KIT_D1_OMNI_GATE=1`, or a store that holds them): 14 requests through `KitD1OmniDecider(catalog:)`, 20 text rows at 64 to 4,096 positions, 3 images (one tiled) and 3 clips (10, 20 and 30 s buckets). All 38 rows give the zoo Swift host's marker logits and probabilities bit for bit, and all 14 responses equal the publisher's text. An image (`"state": ""`) and a clip (no state) sent again over the wire as base64 give the same probabilities. Only the six lengths those rows need are loaded. This ran on the zoo's staged copy of the repo, checked file by file against the Hub at the pin (sizes, LFS sha256, git blob ids). - The whole suite: XCTest 100 (23 skipped, 0 failures), swift-testing 297 passed. **Not in this PR.** No release and no pin bump; no other catalog entry moves, and `Package.resolved` is unchanged. The kit has not run on a phone, and the phone's memory with several decision lengths loaded in one process was not measured. The zoo's gate app ran these graphs on an iPhone 18 Pro, one combination of graphs per process. Speed was not measured through the kit; the numbers in `docs/SYSTEM_ONE.md` are the zoo's. `CoreAI.decide` and `CoreAI.systemOne` load through `TypedDecisions(catalog:)`, so they refuse d1-omni by its kind; use `KitD1OmniDecider` or `systemone serve`. Open: the store can download one folder as its own bundle (the driver already does, per folder), so downloading a length the first time a row needs it means moving those calls from init to first use. That is the next round. **On merge.** The live catalog gains the entry, and a kit released before this one shows it nowhere. Until the merge, a build of this branch finds it by id in its built-in catalog. **Check it yourself.** The gate downloads `macos/` (6.5 GB) and `reference/` (9.3 MB) from the catalog pin unless they are cached. ```bash swift test --filter D1Omni # without files KIT_D1_OMNI_GATE=1 swift test --filter D1Omni # + the 360 rows and the 14 requests through the graphs swift build -c release --product systemone .build/release/systemone serve --model d1-omni-600m --port 8091 & curl -s http://127.0.0.1:8091/v1/systemone -H 'Content-Type: application/json' \ -d '{"audio": "/path/to/clip.wav", "questions": {"refund": {"type": "noul", "instructions": "Does the caller want a refund?"}}}' ``` **Docs.** CHANGELOG (Unreleased / Added), README (the Decide line, 76 entries), AGENTS.md (76 entries), `docs/SYSTEM_ONE.md` (a section "Text, images or audio: d1-omni", a row in the options table, `audio` in the request paragraph, `Sources/CoreAIKit/D1Omni/` in "Where the pieces are"). Merge is the lane supervisor's call.

john-rocky added 4 commits October 4, 2026 00:33

CHANGELOG: clef-flash, the loaders as they are — systemone serve and …

be77f6b

…the MCP server through SystemOne.backend(catalog:), decide-cli ask and serve through its kind check

john-rocky merged commit 5a29498 into main Oct 3, 2026
2 checks passed

john-rocky mentioned this pull request Oct 8, 2026

d1-omni-600M: a text, images or a clip, each question its own row read at its option markers (KitD1OmniDecider, kind omniDecision, audio on /v1/systemone) #68

Merged

Navigation Menu

Repository navigation

ClefFlash: clef-flash, every question of a request read in one pass (KitClefDecider, kind jointDecision, images on /v1/systemone) - #66

ClefFlash: clef-flash, every question of a request read in one pass (KitClefDecider, kind jointDecision, images on /v1/systemone)#66
john-rocky merged 4 commits into
mainfrom
clef-flash

john-rocky commented Oct 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

john-rocky commented Oct 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant