Skip to content

Upstream llama.cpp and validate PS5 model workflows - #43

Merged
quatrixone merged 204 commits into
mainfrom
codex/five-improvements-plan
Nov 11, 2026
Merged

quatrixone merged 204 commits into
mainfrom
codex/five-improvements-plan

Conversation

@quatrixone

@quatrixone quatrixone commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Updates the shared Vulkan runtime to upstream llama.cpp cb44fdb / GGML cd6d2b7, including PS5 and Ternary Bonsai compatibility. It also incorporates the latest main UI-kit source fix.

Validation:

  • make test: 138 passed, 2 skipped; host integration suite passed. make lint passed.
  • Built and deployed to test title PPSA99023; it stayed alive through the requests.
  • All 25 installed curated model presets passed generic or workflow-specific console checks. Cover produced ABC/event artifacts; HTDemucs produced all five stems. Both inpainting presets and SDXL Canny ControlNet passed with real image inputs; ControlNet used the catalog default of 25 steps. Real-ESRGAN 2x and 4x passed.
  • Remote Play welcome/workspace screenshot and generated workflow outputs were visually reviewed.

Limit: two optional, non-catalog local GGUFs (Gemma 4 26B and Qwen3.8 27B) exceed available console Vulkan memory. This PR does not claim arbitrary local GGUF compatibility.

Evidence: console validation, per-model checks. Release version is 01.007.000.

Skystar9999 and others added 30 commits October 16, 2026 21:19
ACE-Step takes lyrics as a separate request option (options["lyrics"])
from the style caption. ProsperoAI never set it, so every generation
came out instrumental no matter what the prompt contained.

- store lyrics in the runtime settings (base64 line in the cfg so
  multi-line text survives the line-oriented parser), max 4096 bytes
- expose audio_lyrics on POST/GET /api/settings/runtime
- forward options.lyrics to the audio.cpp request in the ACE-Step
  adapter when lyrics are set
- HTTP round-trip/persistence tests
Wires the audio.cpp yue2 family (already present in the pinned
886076fc) into the PS5 runtime:

- inference/adapters/media/yue2_ps5.cpp: Vulkan adapter mirroring the
  ACE-Step path (session reuse guard, allocator hygiene between
  requests). Forwards the prompt as the style option and
  audio_lyrics as lyrics; cot=off; length is derived by the model
  from the lyric structure, so audio_duration_sec is not forwarded.
- registry: dispatch models whose id contains 'yue2' and release on
  media release.
- AUDIOCPP_MODELS gains yue2; build script compiles the new adapter.
- Preset yue2-3b-q8-0 (ngquocvinh/YuE2-3B-GGUF @9711e46e): q8_0 main
  + f16 VAE + required sidecars, matching the audio.cpp default
  packaged layout under the preset root.

Stacked on feature/audio-lyrics (uses settings.audio_lyrics).
Weights are CC-BY-NC-4.0.
audio.cpp vendors a ggml fork whose unary GGML_UNARY_OP_ROUND_BF16 and
helper ggml_round_bf16 do not exist in the llama.cpp ggml ProsperoAI
pins, so the yue2 sources fail to compile in the shared build.

Declare the helper for the audio.cpp subproject via a -include header
and define it as an f32 -> bf16 -> f32 ggml_cast round trip, which is
the same round-to-nearest-even result as the fork's kernel. Link the
definition into the native eboot objects and the media test.

The router host test gains no-op ps5_yue2_* stubs like the other media
adapters.
test_minicpm5 pins the first five preset ids; the YuE2 entry now lands
at the end of presets.json instead of after ACE-Step.

The -include path moves to a configure_file copy in the build tree so
an absent header fails configure instead of failing deep inside the
cJSON compile, and no longer depends on the checkout layout.
CMAKE_CURRENT_LIST_DIR is tools/vulkan-runtime; the header lives one
level up in tools/. Verified locally with a cmake -P probe that the
copy resolves and the content hash matches.
The narrow audio.cpp checkout omits causal_decoder.cpp and
causal_decoder_runtime.cpp for size; YuE2's AR runtime links against
CausalDecoderRuntime, so the eboot link failed with four undefined
symbols. Restore both entries in the model-dependency style already
used for ACE-Step. Verified locally against the pinned CMakeLists:
each entry reappears exactly once, in dependency order.
audio.cpp lowered the YuE2 graph arena defaults to 32 MiB each, which
fits short generations only; longer AR streams fail deterministically
in the NAR/VAE decode stages with 'generation failed'. Set explicit
session arenas (prefill 256, decode 64, NAR 1024, VAE 512 MiB) sized
for songs up to ~5 minutes on PS5 memory budgets.
Build with NAR 1024/Vae 512 MiB arenas crashed the whole app during NAR
stage on a base PS5 (Phat) — total arena + weights exceeded the game
heap. Reduced to NAR 512 / prefill 128 / VAE 64 / decode 64 (~760 MB
total), which leaves margin over the 32 MiB default that only fit
~1-minute songs.
Full-length songs fail inside the engine at ~190 s but the app only
reports a generic 'generation failed'. Return the engine's last error
text (same pattern as the chatterbox adapter) so the HTTP response
names the failing stage.
quatrixone and others added 11 commits November 3, 2026 08:29
The PS5SD presets install checkpoints under names the adapter never looked
for: it had three filenames hard-coded, so eight of the ten could not be
loaded at all. Resolve the checkpoint, VAE, ControlNet and FLUX.2 parts from
the preset table instead.

The stages then had to reach the GPU. The default backend was CPU with three
components assigned to Vulkan, which left the ControlNet on a few hundred MiB
of libc flexible memory: a 512x512 guide took 705 s, against 105 s once every
stage runs on Vulkan, as the PS5SD reference build does. For the same reason
an SDXL VAE decode could not allocate its multi-GiB workspace, and FLUX.2
Klein's text encoder failed on 172 MiB. Large VAE decodes are tiled on the
device, and weights keep the type they ship with: forcing F16 dates from when
the only image model was an fp16 diffusers folder, and it doubled the resident
size of every Q8_0 preset.

Masks and ControlNet guides reach the engine as decoded, so an RGBA mask could
not broadcast against the init tensor and an RGBA guide tripped
GGML_ASSERT(a->ne[2] == b->ne[2]), which aborted the title. Hand them over with
the channel count the engine expects.

/api/tags built its reply in a 2 KiB buffer with gpt_runtime_model_id() called
twice in one snprintf. The accessor returns a thread_local string that the
second call rewrites, so names came out as stray bytes, and the catalog was
truncated mid-token: eight of 28 models, including every audio entry, never
reached the client. Build the reply with the JSON writer already in the file.

Add the Real-ESRGAN upscaler as an installable preset, with a new preset kind
so it is offered for download without appearing as a selectable model, and let
each image request choose whether to upscale, defaulting to the Settings
toggle.

Verified on PPSA99023: nine of ten presets generate, image-to-image on all six
that declare it, inpainting on both, ControlNet, and a 512x512 run upscaled to
2048x2048 in 15 s. FLUX.2 Klein stays blocked; its diffusion workspace does not
fit this console's Vulkan budget.
window.confirm is not available in the console browser: it returns undefined,
so every action behind it silently did nothing. That blocked deleting archived
media, deleting an installed model, starting a download and cancelling one.
Ask with a dialog, the way the conversation delete already did.

The Media filters offered only "All media" and "Images", so generated audio
could not be listed on its own; the filter now has the three states the page
describes. Conversation actions kept a rule from the layout where Delete was
an icon in the row, which squeezed the menu's "Delete" into a 36x36 box at
22px; the rule is gone.

The image workspace no longer carries its own Models entry, since the same
page is one click away in the main navigation, and its destinations now use
the underlined tabs the audio sections use, as does the main navigation.

The upscale toggle sits with the other per-request image controls and is sent
with every request: omitting it would fall back to the Settings toggle, so
clearing the box could not turn an upscale off.
The console had no way to see what it generated: the native UI browsed the
audio library only, and pictures were not reachable at all. Add a Media tab
that lists generated images, audio and voice files, newest first. The library
scan now takes the names it accepts rather than assuming the audio ones.

The header tabs were compared by number in 29 places, which made inserting one
risky, so they are named. The workspace modes read Text, Image, Audio, TTS in
that order, and the picker applies a mode when Cross confirms it rather than
on every move, leaving Circle to back out.

The conversation list showed everything regardless of the workspace mode:
session_category_ was never assigned, so the filter was always "all". It now
follows the mode, so Text lists text conversations and Image lists image ones.
A generation started from a conversation is written to a folder named after it,
so a gallery that read only the top of work/image would have missed exactly the
pictures the interface produces. The scan now looks one level down and keeps
each entry's path relative to the root. The audio picker keeps its flat view,
which its own read and save paths depend on.
Deleting from Media only ever worked for an archived copy stored in a folder
named after its conversation. Everything a generation writes straight into
work/image or work/audio -- most of the gallery -- came back 409 with an error
about archived copies, so the button looked broken.

media_path_allowed already confines the path to the work and sessions trees,
and the delete still requires exactly one conversation to reference the file,
so the folder-name rule only narrowed which of those could be removed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FLUX.2 Klein returned structured noise: the adapter set prediction to FLOW_PRED,
which turns off the library's own detection and builds a DiscreteFlowDenoiser.
FLUX wants FLUX_FLOW_PRED and the flow shift that comes with it. Prediction,
VAE format and scheduler are now left on auto, where the library reads the
family from the tensors -- the split-file layout is shared by FLUX.2, FLUX.3 and
Z-Image, which do not share a denoiser, so it cannot stand in for the family.

Sampling then failed to allocate: naming a params backend disables auto-fit and
pins every weight, and the model manager can only evict a tensor whose
parameters live somewhere other than its compute backend, so it had nothing to
release. Auto-fit on its own keeps everything resident too, because the heap
looks empty until the ~1.9 GiB attention workspace is reserved; a 6 GiB cap
moves the 2.4 GiB text encoder to disk, where it belongs -- it is read once,
before the first step.

The same layout flag also made every split-file preset require the FLUX.2 Klein
edit strategy, so a plain text-to-image prompt -- which carries no edit strategy
at all -- was rejected before it reached the model. That rule now binds only
when the request actually supplies an input image.

Verified on PPSA99023: every installed image preset generates, FLUX.2 Klein in
186 s at 1024x1024, and SDXL and the ControlNet guide are 20% faster than with
the pinned placement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pinned stable-diffusion.cpp (b13a482 in tools/deps.json) has no
VERSION_FLUX3, so both FLUX.3 Image Klein presets fail version detection at
load time with "get sd version from file failed". Offering a model that cannot
run is worse than not offering it; bringing them back means moving the pin
forward, which also moves the ggml ABI the text runtime shares.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Image upscaling was on or off, and off meant 4x. The web Studio and Settings
now pick 0, 2 or 4, as does the native Settings choice, and the request field
and sd_upscale setting carry the factor.

Real-ESRGAN x4 produces only its own scale -- the upscale_factor argument is
ignored by the library -- so 2x is that pass box-averaged back down, which
keeps the restored detail at half the pixels. A boolean upscale field and a
stored sd_upscale=1 still mean 4x, so existing clients and settings files are
unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The once-a-second Vulkan memory poll took the runtime lock, which generation
holds for its whole run, and one worker serves every job -- so for four
minutes of a song no other action could start and the interface looked dead.
The trace shows it plainly: "refresh Vulkan memory: 243.01 s".

A memory reading is worth no wait at all, so the poll now tries the lock and
keeps the previous figures when the runtime is busy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
state.md still described an older checkout and branch. It now records what was
verified on PPSA99023, the FLUX.3 dependency blocker, and what the next agent
should pick up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@quatrixone

Copy link
Copy Markdown
Owner Author

Image runtime round: FLUX fixed, Media delete fixed, UI freeze fixed

Five commits, all verified on test title PPSA99023 unless noted. state.md on this branch
carries the full handoff; this is the short version.

Fixed

  • FLUX.2 Klein produced noise (58fdabc). The adapter forced prediction = FLOW_PRED, which
    turns off the library's own family detection and builds a DiscreteFlowDenoiser. FLUX needs
    FLUX_FLOW_PRED and the flow shift that comes with it. Prediction, VAE format and scheduler are
    now left on auto, where the library reads the family from the tensors.
  • FLUX.2 Klein then could not allocate (same commit). Naming a params backend disables
    auto-fit and pins every weight, and the model manager can only evict a tensor whose parameters
    live somewhere other than its compute backend — so it had nothing to release and sampling died
    on the first double block with 220 MB of an 11.7 GiB heap free. Auto-fit alone was not enough;
    max_vram = "6" for the split-file presets moves the 2.4 GiB text encoder to disk, where it
    belongs. Result: 1024x1024 in 186 s, and SDXL and the ControlNet guide are ~20% faster than with
    the pinned placement.
  • Media delete returned 409 for most of the gallery (e41bdd8). Deleting only ever worked for
    an archived copy stored in a folder named after its conversation; everything written straight
    into work/image or work/audio was refused.
  • The interface froze during generation (0a1135d). The once-a-second Vulkan memory poll took
    the runtime lock that generation holds for its whole run, and one worker serves every job, so no
    other action could start. The trace shows it: refresh Vulkan memory: 243.01 s after an
    ACE-Step run. The regression test was confirmed to fail without the fix.
  • Upscaling is a factor now (fcc9865). Off / 2x / 4x in the web Studio, web Settings and
    native Settings. Real-ESRGAN x4 only produces its own scale — the library ignores its
    upscale_factor argument — so 2x is that pass box-averaged back down. A boolean upscale and a
    stored sd_upscale=1 still mean 4x. Not yet exercised on the console.

Removed

flux3-image-klein-q4-k and -q6-k (9aa01bf). The pinned stable-diffusion.cpp (b13a482 in
tools/deps.json) has no VERSION_FLUX3, so both fail version detection at load with
get sd version from file failed. Bringing them back means moving the dependency pin forward,
which also moves the ggml ABI the text runtime shares — a dependency upgrade, not a bugfix, and
it needs a decision before anyone starts it.

Verified on console

Ten image presets generate at their defaults: sd-turbo 29 s, sdxs 8 s, realistic-vision 34 s,
SDXL 94 s, raehoshi 166 s, ilustmix 111 s, SDXL inpainting 134 s, SD1.5 inpainting 33 s,
SDXL canny 134 s, FLUX.2 Klein 186 s.

Host checks: make test 134 OK (1 skipped), make lint clean, PS5 app build OK.

Next

  1. Generate once with flux2-klein-9b-q4-0 and z-image-turbo-q4-k — both finished downloading
    this session, neither has been run. Z-Image shares the split-file layout but not the denoiser,
    so it is the likeliest place for a wrong family inference to show; max_vram = "6" applies to
    it too and may be the wrong budget for a 9B model. Look at the PNG — a run that reports
    done can still be noise, which is exactly how the FLUX bug hid.
  2. Exercise the upscale factor on the console at 2x and 4x and check the output dimensions.
    halve_image is new and has no direct test.
  3. Click Delete in the web Media page against the running title; the fix has host coverage but has
    not gone through a browser.
  4. Not started: Enhance and prompt templates for the Image Studio, mirroring enhance(field) and
    TPL_BUILTIN in assets/web/music.js.
  5. Not started: the web TTS voice picker. The server round-trips kokoro_voice correctly and the
    unit test passes, so the fault is client-side — most likely applyRuntimeSettings re-running
    populateWorkspaceVoicePicker and discarding the selection.
  6. The native UI was never seen in this work (P5_SESSION unset, no screenshots). The Media tab
    layout and the new three-way upscaler choice are visually unverified.

🤖 Generated with Claude Code

quatrixone and others added 12 commits November 3, 2026 21:54
The Vulkan memory job took the runtime lock, which generation holds for its
whole run, so the panel kept showing the model from before the generation --
and, with one worker serving every job, nothing else could start either.

Neither call needs that lock: the router takes its own short lock for the model
identity, and the heap query reads the device directly. Dropping it reports the
model that is running and leaves the interface responsive.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Studio staged each preset's trained size, so SDXL opened at 1024x1024 and
took 94 s a render. It now starts every model at 512x512, which the console
renders in a fraction of that and the upscaler takes to 2048. The rest of each
preset's defaults -- steps, guidance, sampler, scheduler -- are untouched, and
a size typed in the Studio still wins.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/api/models/memory took the runtime lock for two reads that do not need it, so
the panel stalled for the whole of every generation -- precisely when a caller
wants to know which model is resident. Same cause and same fix as the native
readout; the regression test uses the harness's existing runtime-busy hook and
was confirmed to hang without it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No weight budget fits every split-file preset. 6 GiB carried FLUX.2 Klein 4B
and 9B on a fresh heap, but Z-Image failed by 175 MB when it followed the 9B
and found the heap down to 10.88 of 11.7 GiB with its largest block at 7.6 --
the previous model's leavings and fragmentation, which no fixed number can
anticipate.

These presets now take the placement the library documents for exactly this
case: compute on Vulkan0, parameters on disk, each segment read as it is needed
and released after use. Because the parameters then live off the compute
backend, the model manager can also evict them, which it could not do while
they were resident. It costs load time, not correctness.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Auto-fit keeps a module's weights on the device while they fit and sends only
the overflow to disk, but it reads the heap as nearly empty and so keeps
everything resident, leaving nothing for the sampling pass. It was given a
fixed 6 GiB budget, which carried FLUX.2 Klein 4B and 9B on a fresh heap and
then failed Z-Image by 175 MB when it followed the 9B and found the heap down
to 10.88 of 11.7 GiB. No constant can track what the previous model left
behind.

The budget is now the free heap less the room the sampling pass needs beside
the weights: 6 GiB works with 11.33 GiB free and fails with 10.88 GiB free, so
the reserve is a little over 5 GiB.

Naming "disk" outright was tried and is worse: it turns auto-fit off, the whole
graph becomes one segment, and Z-Image asked for 4.88 GiB at once where 36
segments needed 572 MB.

Verified on PPSA99023: Z-Image Turbo renders 1024x1024 in 374 s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The curated presets staged their trained sizes, so SDXL opened at 1024x1024 and
took 94 s a render where 512x512 takes a fraction of that and the upscaler
takes it further when asked. Every image preset's default is now 512x512, in
the catalog that the web Studio reads and in the console's own fallback, which
had kept 768 for the SD3 and FLUX families.

The catalog is the single source of truth again: the Studio's hardcoded 512 is
gone, so a future change to a preset's default takes effect in the web. Steps,
guidance, sampler and scheduler stay each model's own, and a size typed in the
Studio or set in Settings still wins.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two reasons an entry would not delete, both hit on the console.

An archived generation is usually referenced twice by the same message: the
text names the file and the conversation records it as that message's output.
Counting those separately tripped the "referenced more than once" guard, which
refused nearly every archived generation. A message that does both is one
reference, and deleting now clears the text as well as the file, rather than
leaving a path to nothing behind.

An entry whose file had gone missing behind our back could never be removed:
std::remove failed, so the listing kept it for good. A file that is already
gone is the state the caller asked for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Studio had a smaller heading, its own narrower grid, and a static "Model
settings" card where Music Studio puts the Library. It now shares the heading,
layout and panel styling, the model defaults moved under the model picker where
they belong, and the right column is a library of generated images with Edit,
Guide and Delete on each. A finished generation lands there immediately, so
seeing it no longer means a trip to the Gallery page.

Media also lists the work folders now, not only what conversations reference.
Music Studio writes straight to the audio folder and never saves a
conversation, so its tracks were missing from Media entirely. Entries with no
conversation can be deleted too -- there is simply nothing to rewrite -- and
the Gallery drops its "From:" button for them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Upscaling was a setting the console read on every render, which is a slow
default for a feature wanted on one picture in ten. It is now an action: the
web gallery and the Studio library carry Upscale 2x and 4x on each image, and
POST /api/images/upscale runs Real-ESRGAN over a stored file with no diffusion
model loaded at all. Console renders no longer upscale, and the native Settings
entry says where the upscaler lives instead of offering a switch.

The upscaler lookup and the pass itself are shared with the generate path
rather than written twice, and the standalone call releases the diffusion
context first, as that path already did.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SDXL Inpainting and the SDXL Canny guide rendered once and then failed their
second request on the spot -- "unet segment 1/1 failed during allocated
capacity check" in 0.14 s -- with the first render's weights still resident.

Keeping the model loaded is what takes a second render from 62 s to 9 s, so it
is given up only when the heap can no longer hold a pass beside it. Both
presets now render twice, 77/75 s and 75/22 s, and the reload fires once, at
1127 MiB free.

Giving every image model the weight budget instead was tried and is much
worse: the reserve is measured at 1024x1024 and far too generous for a model
that fits, so it sent SDXL Inpainting's weights to disk and took one render
from 77 s to 1643 s. The budget stays with the split-file presets that need it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@quatrixone quatrixone changed the title Release candidate: stability, Music Studio, Qwen-Image 2.1 Upstream llama.cpp runtime and PS5 Image Studio validation Nov 11, 2026
@quatrixone quatrixone changed the title Upstream llama.cpp runtime and PS5 Image Studio validation Upstream llama.cpp and validate PS5 model workflows Nov 11, 2026
@quatrixone
quatrixone marked this pull request as ready for review November 11, 2026 23:48
@quatrixone
quatrixone merged commit 3d6c122 into main Nov 11, 2026
3 checks passed
@quatrixone
quatrixone deleted the codex/five-improvements-plan branch November 11, 2026 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants