Repository navigation
Upstream llama.cpp and validate PS5 model workflows - #43
Conversation
ACE-Step takes lyrics as a separate request option (options["lyrics"]) from the style caption. ProsperoAI never set it, so every generation came out instrumental no matter what the prompt contained. - store lyrics in the runtime settings (base64 line in the cfg so multi-line text survives the line-oriented parser), max 4096 bytes - expose audio_lyrics on POST/GET /api/settings/runtime - forward options.lyrics to the audio.cpp request in the ACE-Step adapter when lyrics are set - HTTP round-trip/persistence tests
Wires the audio.cpp yue2 family (already present in the pinned 886076fc) into the PS5 runtime: - inference/adapters/media/yue2_ps5.cpp: Vulkan adapter mirroring the ACE-Step path (session reuse guard, allocator hygiene between requests). Forwards the prompt as the style option and audio_lyrics as lyrics; cot=off; length is derived by the model from the lyric structure, so audio_duration_sec is not forwarded. - registry: dispatch models whose id contains 'yue2' and release on media release. - AUDIOCPP_MODELS gains yue2; build script compiles the new adapter. - Preset yue2-3b-q8-0 (ngquocvinh/YuE2-3B-GGUF @9711e46e): q8_0 main + f16 VAE + required sidecars, matching the audio.cpp default packaged layout under the preset root. Stacked on feature/audio-lyrics (uses settings.audio_lyrics). Weights are CC-BY-NC-4.0.
audio.cpp vendors a ggml fork whose unary GGML_UNARY_OP_ROUND_BF16 and helper ggml_round_bf16 do not exist in the llama.cpp ggml ProsperoAI pins, so the yue2 sources fail to compile in the shared build. Declare the helper for the audio.cpp subproject via a -include header and define it as an f32 -> bf16 -> f32 ggml_cast round trip, which is the same round-to-nearest-even result as the fork's kernel. Link the definition into the native eboot objects and the media test. The router host test gains no-op ps5_yue2_* stubs like the other media adapters.
test_minicpm5 pins the first five preset ids; the YuE2 entry now lands at the end of presets.json instead of after ACE-Step. The -include path moves to a configure_file copy in the build tree so an absent header fails configure instead of failing deep inside the cJSON compile, and no longer depends on the checkout layout.
CMAKE_CURRENT_LIST_DIR is tools/vulkan-runtime; the header lives one level up in tools/. Verified locally with a cmake -P probe that the copy resolves and the content hash matches.
The narrow audio.cpp checkout omits causal_decoder.cpp and causal_decoder_runtime.cpp for size; YuE2's AR runtime links against CausalDecoderRuntime, so the eboot link failed with four undefined symbols. Restore both entries in the model-dependency style already used for ACE-Step. Verified locally against the pinned CMakeLists: each entry reappears exactly once, in dependency order.
audio.cpp lowered the YuE2 graph arena defaults to 32 MiB each, which fits short generations only; longer AR streams fail deterministically in the NAR/VAE decode stages with 'generation failed'. Set explicit session arenas (prefill 256, decode 64, NAR 1024, VAE 512 MiB) sized for songs up to ~5 minutes on PS5 memory budgets.
Build with NAR 1024/Vae 512 MiB arenas crashed the whole app during NAR stage on a base PS5 (Phat) — total arena + weights exceeded the game heap. Reduced to NAR 512 / prefill 128 / VAE 64 / decode 64 (~760 MB total), which leaves margin over the 32 MiB default that only fit ~1-minute songs.
Full-length songs fail inside the engine at ~190 s but the app only reports a generic 'generation failed'. Return the engine's last error text (same pattern as the chatterbox adapter) so the HTTP response names the failing stage.
Amp-Thread-ID: https://ampcode.com/threads/T-01a1865c-314c-7281-a224-dad719485944 Co-authored-by: Amp <amp@ampcode.com>
The PS5SD presets install checkpoints under names the adapter never looked for: it had three filenames hard-coded, so eight of the ten could not be loaded at all. Resolve the checkpoint, VAE, ControlNet and FLUX.2 parts from the preset table instead. The stages then had to reach the GPU. The default backend was CPU with three components assigned to Vulkan, which left the ControlNet on a few hundred MiB of libc flexible memory: a 512x512 guide took 705 s, against 105 s once every stage runs on Vulkan, as the PS5SD reference build does. For the same reason an SDXL VAE decode could not allocate its multi-GiB workspace, and FLUX.2 Klein's text encoder failed on 172 MiB. Large VAE decodes are tiled on the device, and weights keep the type they ship with: forcing F16 dates from when the only image model was an fp16 diffusers folder, and it doubled the resident size of every Q8_0 preset. Masks and ControlNet guides reach the engine as decoded, so an RGBA mask could not broadcast against the init tensor and an RGBA guide tripped GGML_ASSERT(a->ne[2] == b->ne[2]), which aborted the title. Hand them over with the channel count the engine expects. /api/tags built its reply in a 2 KiB buffer with gpt_runtime_model_id() called twice in one snprintf. The accessor returns a thread_local string that the second call rewrites, so names came out as stray bytes, and the catalog was truncated mid-token: eight of 28 models, including every audio entry, never reached the client. Build the reply with the JSON writer already in the file. Add the Real-ESRGAN upscaler as an installable preset, with a new preset kind so it is offered for download without appearing as a selectable model, and let each image request choose whether to upscale, defaulting to the Settings toggle. Verified on PPSA99023: nine of ten presets generate, image-to-image on all six that declare it, inpainting on both, ControlNet, and a 512x512 run upscaled to 2048x2048 in 15 s. FLUX.2 Klein stays blocked; its diffusion workspace does not fit this console's Vulkan budget.
window.confirm is not available in the console browser: it returns undefined, so every action behind it silently did nothing. That blocked deleting archived media, deleting an installed model, starting a download and cancelling one. Ask with a dialog, the way the conversation delete already did. The Media filters offered only "All media" and "Images", so generated audio could not be listed on its own; the filter now has the three states the page describes. Conversation actions kept a rule from the layout where Delete was an icon in the row, which squeezed the menu's "Delete" into a 36x36 box at 22px; the rule is gone. The image workspace no longer carries its own Models entry, since the same page is one click away in the main navigation, and its destinations now use the underlined tabs the audio sections use, as does the main navigation. The upscale toggle sits with the other per-request image controls and is sent with every request: omitting it would fall back to the Settings toggle, so clearing the box could not turn an upscale off.
The console had no way to see what it generated: the native UI browsed the audio library only, and pictures were not reachable at all. Add a Media tab that lists generated images, audio and voice files, newest first. The library scan now takes the names it accepts rather than assuming the audio ones. The header tabs were compared by number in 29 places, which made inserting one risky, so they are named. The workspace modes read Text, Image, Audio, TTS in that order, and the picker applies a mode when Cross confirms it rather than on every move, leaving Circle to back out. The conversation list showed everything regardless of the workspace mode: session_category_ was never assigned, so the filter was always "all". It now follows the mode, so Text lists text conversations and Image lists image ones.
A generation started from a conversation is written to a folder named after it, so a gallery that read only the top of work/image would have missed exactly the pictures the interface produces. The scan now looks one level down and keeps each entry's path relative to the root. The audio picker keeps its flat view, which its own read and save paths depend on.
Deleting from Media only ever worked for an archived copy stored in a folder named after its conversation. Everything a generation writes straight into work/image or work/audio -- most of the gallery -- came back 409 with an error about archived copies, so the button looked broken. media_path_allowed already confines the path to the work and sessions trees, and the delete still requires exactly one conversation to reference the file, so the folder-name rule only narrowed which of those could be removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
FLUX.2 Klein returned structured noise: the adapter set prediction to FLOW_PRED, which turns off the library's own detection and builds a DiscreteFlowDenoiser. FLUX wants FLUX_FLOW_PRED and the flow shift that comes with it. Prediction, VAE format and scheduler are now left on auto, where the library reads the family from the tensors -- the split-file layout is shared by FLUX.2, FLUX.3 and Z-Image, which do not share a denoiser, so it cannot stand in for the family. Sampling then failed to allocate: naming a params backend disables auto-fit and pins every weight, and the model manager can only evict a tensor whose parameters live somewhere other than its compute backend, so it had nothing to release. Auto-fit on its own keeps everything resident too, because the heap looks empty until the ~1.9 GiB attention workspace is reserved; a 6 GiB cap moves the 2.4 GiB text encoder to disk, where it belongs -- it is read once, before the first step. The same layout flag also made every split-file preset require the FLUX.2 Klein edit strategy, so a plain text-to-image prompt -- which carries no edit strategy at all -- was rejected before it reached the model. That rule now binds only when the request actually supplies an input image. Verified on PPSA99023: every installed image preset generates, FLUX.2 Klein in 186 s at 1024x1024, and SDXL and the ControlNet guide are 20% faster than with the pinned placement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pinned stable-diffusion.cpp (b13a482 in tools/deps.json) has no VERSION_FLUX3, so both FLUX.3 Image Klein presets fail version detection at load time with "get sd version from file failed". Offering a model that cannot run is worse than not offering it; bringing them back means moving the pin forward, which also moves the ggml ABI the text runtime shares. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Image upscaling was on or off, and off meant 4x. The web Studio and Settings now pick 0, 2 or 4, as does the native Settings choice, and the request field and sd_upscale setting carry the factor. Real-ESRGAN x4 produces only its own scale -- the upscale_factor argument is ignored by the library -- so 2x is that pass box-averaged back down, which keeps the restored detail at half the pixels. A boolean upscale field and a stored sd_upscale=1 still mean 4x, so existing clients and settings files are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The once-a-second Vulkan memory poll took the runtime lock, which generation holds for its whole run, and one worker serves every job -- so for four minutes of a song no other action could start and the interface looked dead. The trace shows it plainly: "refresh Vulkan memory: 243.01 s". A memory reading is worth no wait at all, so the poll now tries the lock and keeps the previous figures when the runtime is busy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
state.md still described an older checkout and branch. It now records what was verified on PPSA99023, the FLUX.3 dependency blocker, and what the next agent should pick up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Image runtime round: FLUX fixed, Media delete fixed, UI freeze fixedFive commits, all verified on test title Fixed
Removed
Verified on consoleTen image presets generate at their defaults: sd-turbo 29 s, sdxs 8 s, realistic-vision 34 s, Host checks: Next
🤖 Generated with Claude Code |
The Vulkan memory job took the runtime lock, which generation holds for its whole run, so the panel kept showing the model from before the generation -- and, with one worker serving every job, nothing else could start either. Neither call needs that lock: the router takes its own short lock for the model identity, and the heap query reads the device directly. Dropping it reports the model that is running and leaves the interface responsive. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Studio staged each preset's trained size, so SDXL opened at 1024x1024 and took 94 s a render. It now starts every model at 512x512, which the console renders in a fraction of that and the upscaler takes to 2048. The rest of each preset's defaults -- steps, guidance, sampler, scheduler -- are untouched, and a size typed in the Studio still wins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/api/models/memory took the runtime lock for two reads that do not need it, so the panel stalled for the whole of every generation -- precisely when a caller wants to know which model is resident. Same cause and same fix as the native readout; the regression test uses the harness's existing runtime-busy hook and was confirmed to hang without it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
No weight budget fits every split-file preset. 6 GiB carried FLUX.2 Klein 4B and 9B on a fresh heap, but Z-Image failed by 175 MB when it followed the 9B and found the heap down to 10.88 of 11.7 GiB with its largest block at 7.6 -- the previous model's leavings and fragmentation, which no fixed number can anticipate. These presets now take the placement the library documents for exactly this case: compute on Vulkan0, parameters on disk, each segment read as it is needed and released after use. Because the parameters then live off the compute backend, the model manager can also evict them, which it could not do while they were resident. It costs load time, not correctness. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Auto-fit keeps a module's weights on the device while they fit and sends only the overflow to disk, but it reads the heap as nearly empty and so keeps everything resident, leaving nothing for the sampling pass. It was given a fixed 6 GiB budget, which carried FLUX.2 Klein 4B and 9B on a fresh heap and then failed Z-Image by 175 MB when it followed the 9B and found the heap down to 10.88 of 11.7 GiB. No constant can track what the previous model left behind. The budget is now the free heap less the room the sampling pass needs beside the weights: 6 GiB works with 11.33 GiB free and fails with 10.88 GiB free, so the reserve is a little over 5 GiB. Naming "disk" outright was tried and is worse: it turns auto-fit off, the whole graph becomes one segment, and Z-Image asked for 4.88 GiB at once where 36 segments needed 572 MB. Verified on PPSA99023: Z-Image Turbo renders 1024x1024 in 374 s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The curated presets staged their trained sizes, so SDXL opened at 1024x1024 and took 94 s a render where 512x512 takes a fraction of that and the upscaler takes it further when asked. Every image preset's default is now 512x512, in the catalog that the web Studio reads and in the console's own fallback, which had kept 768 for the SD3 and FLUX families. The catalog is the single source of truth again: the Studio's hardcoded 512 is gone, so a future change to a preset's default takes effect in the web. Steps, guidance, sampler and scheduler stay each model's own, and a size typed in the Studio or set in Settings still wins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two reasons an entry would not delete, both hit on the console. An archived generation is usually referenced twice by the same message: the text names the file and the conversation records it as that message's output. Counting those separately tripped the "referenced more than once" guard, which refused nearly every archived generation. A message that does both is one reference, and deleting now clears the text as well as the file, rather than leaving a path to nothing behind. An entry whose file had gone missing behind our back could never be removed: std::remove failed, so the listing kept it for good. A file that is already gone is the state the caller asked for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Studio had a smaller heading, its own narrower grid, and a static "Model settings" card where Music Studio puts the Library. It now shares the heading, layout and panel styling, the model defaults moved under the model picker where they belong, and the right column is a library of generated images with Edit, Guide and Delete on each. A finished generation lands there immediately, so seeing it no longer means a trip to the Gallery page. Media also lists the work folders now, not only what conversations reference. Music Studio writes straight to the audio folder and never saves a conversation, so its tracks were missing from Media entirely. Entries with no conversation can be deleted too -- there is simply nothing to rewrite -- and the Gallery drops its "From:" button for them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Upscaling was a setting the console read on every render, which is a slow default for a feature wanted on one picture in ten. It is now an action: the web gallery and the Studio library carry Upscale 2x and 4x on each image, and POST /api/images/upscale runs Real-ESRGAN over a stored file with no diffusion model loaded at all. Console renders no longer upscale, and the native Settings entry says where the upscaler lives instead of offering a switch. The upscaler lookup and the pass itself are shared with the generate path rather than written twice, and the standalone call releases the diffusion context first, as that path already did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SDXL Inpainting and the SDXL Canny guide rendered once and then failed their second request on the spot -- "unet segment 1/1 failed during allocated capacity check" in 0.14 s -- with the first render's weights still resident. Keeping the model loaded is what takes a second render from 62 s to 9 s, so it is given up only when the heap can no longer hold a pass beside it. Both presets now render twice, 77/75 s and 75/22 s, and the reload fires once, at 1127 MiB free. Giving every image model the weight budget instead was tried and is much worse: the reserve is measured at 1024x1024 and far too generous for a model that fits, so it sent SDXL Inpainting's weights to disk and took one render from 77 s to 1643 s. The budget stays with the split-file presets that need it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Updates the shared Vulkan runtime to upstream llama.cpp
cb44fdb/ GGMLcd6d2b7, including PS5 and Ternary Bonsai compatibility. It also incorporates the latestmainUI-kit source fix.Validation:
make test: 138 passed, 2 skipped; host integration suite passed.make lintpassed.PPSA99023; it stayed alive through the requests.Limit: two optional, non-catalog local GGUFs (Gemma 4 26B and Qwen3.8 27B) exceed available console Vulkan memory. This PR does not claim arbitrary local GGUF compatibility.
Evidence: console validation, per-model checks. Release version is
01.007.000.