d1-omni-600M: a text, images or a clip, each question its own row read at its option markers (KitD1OmniDecider, kind omniDecision, audio on /v1/systemone)

[john-rocky]

Adds Liquid AI's d1-omni-600M (LFM Open License v1.0) to the kit: the catalog entry d1-omni-600m under a new kind, omniDecision; KitD1OmniDecider (Sources/CoreAIKit/D1Omni/); audio on /v1/systemone; and the routes that load it, so systemone serve --model d1-omni-600m answers requests about a text, an image or a voice clip.

The model. A state (text or JSON), images or one 16 kHz clip, and typed questions (noul, choice, score) go in. Every option of every question comes back with a probability; nothing is generated. Each question is its own row in the publisher's form, with a <|mask|> marker before each option. One forward pass of a static decision graph scores every position, and each option is read at its marker. A text row is divided by the publisher's temperature for its type and option count, then softmaxed in float32. The checkpoint is an LFM2.5-Encoder-350M trunk with a decision head, a SigLIP2 vision tower and a FastConformer audio tower.

The host. The model zoo's apps/D1Omni library (f9e0e09), ported with every numeric path unchanged. Across its twelve files only names (prefixed D1; the zoo's D1Omni is D1OmniPipeline), access, @available(macOS 27, iOS 27, *) and a provenance line differ (port_d1.py, in the lane). One change for speed: a request's state is tokenized once for all its questions, since swift-transformers takes 0.4 s on a 3.4k-token state. The decision graph has seven lengths (64 to 4,096 positions); a row runs at the smallest that holds it. A row past 4,096 positions is refused with a 422. Images go through the vision graph once per crop, a clip through the audio graph of its 5 / 10 / 20 / 30 s bucket, and their rows go in front of each question's ids. KitD1OmniDecider is a DecisionBackend that reads images and audio; readout(requestJSON:images:audio:) gives each row, its graph length, the marker logits and the response in the publisher's form.

The wire. /v1/systemone gains audio: one 16 kHz mono 16-bit WAV as a data URL, base64, or a file path (SystemOneServer reads a path only on 127.0.0.1). With audio, state may be left out: the model then reads the {} its audio questions were trained on. Every other backend refuses a request with audio by name (422), through a new DecisionBackend.audioRefusal beside imageRefusal. For d1-omni, images is the existing field (#66) and grid is refused. The MCP decide tool's schema gains audio.

Catalog. mlboydaisuke/d1-omni-600M-CoreAI @ 914184e5, the repo's main. variants.macos.path = "macos" and variants.ios.path = "ios", 6,458 MB each (measure-catalog-sizes.py; the same .aimodels). format: markerScores. The driver downloads the platform folder's twelve bundles at first use, each as its own download at <variant>/<folder>. The store takes such a path as given, so an iPhone reads the JIT ios/ graphs. It never takes the repo's ios-h19p/ AOT set, which has no L4096 (its compile exceeds the phone's memory limit). A graph loads the first time a row or a medium needs it.

Shipped kits. The bundle's metadata says decision.head = "encoder", but its rows, temperatures and media prefixes are not laya's. With kind: decision, a shipped kit would list it and download 6.5 GB for a TypedDecisions that cannot read it. Kits 0.1.0–0.7.3 decode any kind they do not know as unknown and leave the entry out of available(_:): each tag's ModelCatalog.swift has the ?? .unknown decode and the filter (read, not compiled for this kind; #67 compiled the same tags for rowDecision). TypedDecisions(catalog: "d1-omni-600m") refuses it at its kind check, before any download.

Tests (M4 Max, macOS 27.0 26A428, Xcode 27.0 RC; the kit resolves swift-transformers 1.3.3, the version the zoo gated).

• Without files (CI): the catalog entry, the kind refusals before a download, the wire's audio (forms, refusals, a clip without a state, the loopback rule), the request in the publisher's form, the option order, and the readout of the fixture's 38 smoke rows from the zoo Swift host's marker logits: the probabilities bit for bit, and all 14 responses byte-equal to the publisher's response() text (host.py under Python 3.12, json.dumps(ensure_ascii=False)).
• With the repo's tokenizer/ and reference/ at the pin: the zoo's 360 public fixture rows rebuilt by the kit, ids, markers and graph length equal to the publisher's, 360 of 360.
• With the bundles (KIT_D1_OMNI_GATE=1, or a store that holds them): 14 requests through KitD1OmniDecider(catalog:), 20 text rows at 64 to 4,096 positions, 3 images (one tiled) and 3 clips (10, 20 and 30 s buckets). All 38 rows give the zoo Swift host's marker logits and probabilities bit for bit, and all 14 responses equal the publisher's text. An image ("state": "") and a clip (no state) sent again over the wire as base64 give the same probabilities. Only the six lengths those rows need are loaded. This ran on the zoo's staged copy of the repo, checked file by file against the Hub at the pin (sizes, LFS sha256, git blob ids).
• The whole suite: XCTest 100 (23 skipped, 0 failures), swift-testing 297 passed.

Not in this PR. No release and no pin bump; no other catalog entry moves, and Package.resolved is unchanged. The kit has not run on a phone, and the phone's memory with several decision lengths loaded in one process was not measured. The zoo's gate app ran these graphs on an iPhone 18 Pro, one combination of graphs per process. Speed was not measured through the kit; the numbers in docs/SYSTEM_ONE.md are the zoo's. CoreAI.decide and CoreAI.systemOne load through TypedDecisions(catalog:), so they refuse d1-omni by its kind; use KitD1OmniDecider or systemone serve. Open: the store can download one folder as its own bundle (the driver already does, per folder), so downloading a length the first time a row needs it means moving those calls from init to first use. That is the next round.

On merge. The live catalog gains the entry, and a kit released before this one shows it nowhere. Until the merge, a build of this branch finds it by id in its built-in catalog.

Check it yourself. The gate downloads macos/ (6.5 GB) and reference/ (9.3 MB) from the catalog pin unless they are cached.

swift test --filter D1Omni                                   # without files
KIT_D1_OMNI_GATE=1 swift test --filter D1Omni                # + the 360 rows and the 14 requests through the graphs

swift build -c release --product systemone
.build/release/systemone serve --model d1-omni-600m --port 8091 &
curl -s http://127.0.0.1:8091/v1/systemone -H 'Content-Type: application/json' \
  -d '{"audio": "/path/to/clip.wav", "questions": {"refund": {"type": "noul", "instructions": "Does the caller want a refund?"}}}'

Docs. CHANGELOG (Unreleased / Added), README (the Decide line, 76 entries), AGENTS.md (76 entries), docs/SYSTEM_ONE.md (a section "Text, images or audio: d1-omni", a row in the options table, audio in the request paragraph, Sources/CoreAIKit/D1Omni/ in "Where the pieces are").

Merge is the lane supervisor's call.