Skip to content

All-backends server fat jars as GitHub Release assets + model-cache overhaul + llama.cpp b9894#301

Merged
bernardladenthin merged 17 commits into
mainfrom
claude/java-llama-fat-jars-classifiers-kyb4wg
Jul 7, 2026
Merged

All-backends server fat jars as GitHub Release assets + model-cache overhaul + llama.cpp b9894#301
bernardladenthin merged 17 commits into
mainfrom
claude/java-llama-fat-jars-classifiers-kyb4wg

Conversation

@bernardladenthin

@bernardladenthin bernardladenthin commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

  • Per-OS all-backends server fat jars as GitHub Release assets (never Maven Central). Every pipeline run assembles llama-<version>-all-<os>-<arch>-jar-with-dependencies.jar for linux-x86-64 (CUDA 13, ROCm, SYCL fp16/fp32, Vulkan, OpenVINO), linux-aarch64 (Vulkan), windows-x86-64 (CUDA 13, ROCm, SYCL, Vulkan, OpenCL, OpenVINO) and windows-aarch64 (OpenCL/Adreno), plus the default CPU fat jar — each with a .sha256 file. The release jobs attach them to the tag GitHub Release and the rolling snapshot pre-release; the Central deploy invocations are untouched. Users run the embedded server with one download: java -jar llama-<version>-all-linux-x86-64-jar-with-dependencies.jar -m model.gguf --port 8080.

    • LlamaLoader gains manifest-driven backend selection: each combined jar carries every GPU backend of its OS/arch under net/ladenthin/llama/<OS>/<ARCH>/<backend>/ plus a jllama-backends.txt priority manifest; the first backend whose native library loads wins (a missing vendor runtime fails cleanly → next backend), with automatic fallback to the default CPU natives. New system property net.ladenthin.llama.backend forces one backend (fail-loud) or default/cpu. Jars without a manifest behave exactly as before (zero behavior change for all existing artifacts). Covered by 15 new pure-Java LlamaLoaderTest cases plus the fixture-driven BackendManifestLoadTest, which drives initialize() end-to-end (extraction, extra files, clash skip, forced/default overrides, success path via two trivial committed ELF dummies) — SonarCloud coverage on new code: 94%.
    • .github/package-fatjars.sh enumerates the classifier set from llama/pom.xml (source of truth), cross-checks it in both directions against the built classifier jars, and fails loud on unparseable classifiers, unranked backends, missing native trees, or zip-update corruption — a future classifier cannot be silently skipped. New smoke-fatjar-linux/smoke-fatjar-windows jobs run the combined jar via real java -jar on GPU-less runners (health poll + chat completion + backend-selection log assertion) and gate publishing together with package-fatjars. Both smokes are green in CI.

  • Model-cache overhaul (single source of truth + 3-OS verification gate). The Windows smoke exposed that the Windows-side model cache entry had been re-saved empty after an eviction and that validate-models.bat printed its ERROR without failing the step — the Windows Java test jobs had been silently self-skipping every model-backed test. Fixes:

    • New .github/models.csv (filename,url per row) is the single source of truth; download-models replaces its ten hardcoded curl steps with one manifest-driven loop and is the only writer of the cache. The cache key is gguf-models-<hash of models.csv>, so editing the manifest automatically creates a fresh complete entry (no manual cache deletion), and enableCrossOsArchive: true makes the one ubuntu-built entry the same entry macOS and Windows restore.
    • New verify-model-cache matrix job (ubuntu / macOS / Windows) proves the entry is restorable (actions/cache/restore + fail-on-cache-miss) and complete (validate gate) on every OS before any model-consuming job starts; all ten consumer jobs use the restore-only action (a consumer can never re-save an empty/partial entry) and need: verify-model-cache.
    • validate-models.{sh,bat} read their required list from the manifest; the .bat quoting bug that swallowed the exit /b 1 is fixed, so an invalid cache now fails loud on Windows too. Verified in CI: Windows validated all ten models for the first time and its model-backed test suites are executing again.

  • llama.cpp b9888 → b9894 (two bumps: b9886→b9888, b9888→b9894). Both ranges verified in the sandbox: all eight priority API headers byte-identical, all eight patches/ apply cleanly in filename order on clean checkouts (incl. 0008 despite an unrelated server-models.cpp change in the b9894 range); the b9894 bulk is GPU-backend kernel code inside upstream-compiled TUs (CUDA convert refactor, OpenCL flash-attention kernels, Metal ops, Vulkan guard).

  • Housekeeping surfaced along the way: corrected stale CLAUDE.md claims about split Windows DLLs (the build ships one monolithic jllama.dll since BUILD_SHARED_LIBS OFF; verified against the published 5.0.5 jars); fixed a small leak where the extracted macOS ggml-metal.metal was never cleaned from the temp dir; fixed 5 SpotBugs findings in the new loader code (3 code fixes + 2 documented exclusions following existing precedent); REUSE/SPDX headers for all new files; README gains the "Standalone server fat jars (GitHub Releases)" download table and the net.ladenthin.llama.backend system-property row.

Test plan

  • Affected unit / integration tests pass locally (LlamaLoaderTest 42/42, BackendManifestLoadTest 7/7, full suite 1401 tests / 0 failures; javadoc jar clean; package-fatjars.sh verified against synthetic fixture jars incl. both fail-loud directions; validate-models.sh manifest parsing verified incl. missing-file and bad-magic failures)
  • CI is green on this branch (run 28816316040: 59/59 jobs — all native builds incl. full-arch CUDA, all Java test suites with real models on every OS, verify-model-cache on 3 OSes, package-fatjars, both fat-jar smokes, SonarCloud quality gate passed with 94% coverage on new code; the b9894 bump run is in progress re-validating the same graph)
  • Docs / CHANGELOG updated where applicable (README fat-jar section + system-properties row; CLAUDE.md sections for the fat-jar mechanism and the model-cache design; docs/history/llama-cpp-breaking-changes.md rows for both bump ranges)

Related issues / PRs

Checklist

  • I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md
  • My commits follow Conventional Commits
  • No security-sensitive changes (if there are, I have notified the maintainer privately per SECURITY.md)

🤖 Generated with Claude Code

https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq

claude added 8 commits July 6, 2026 14:19
@claude
Correct stale Windows 3-DLL docs; clean up extracted ggml-metal.metal
ad30f78 
Investigation for the per-classifier fat-jar work surfaced that CLAUDE.md
still claimed the Windows build ships three co-located DLLs
(jllama.dll + llama.dll + ggml.dll) that the loader must resolve
together. That has been stale since BUILD_SHARED_LIBS was forced OFF:
llama.cpp and ggml are statically linked, so every platform produces a
single monolithic jllama library (verified against the published 5.0.5
jars on Maven Central, which contain only jllama.dll per Windows arch).
Loading any jar via java -jar therefore works on Windows today; the
suspected missing-sibling-DLL extraction bug does not exist.

The one true gap the audit found: initialize() extracts ggml-metal.metal
on macOS, but cleanup()'s shouldCleanPath() never matched the "ggml"
prefix, so a stale extracted copy accumulated in the temp dir forever.
shouldCleanPath() now also matches "ggml"; covered by two new
LlamaLoaderTest cases (27 total, all passing).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
LlamaLoader: backend-manifest selection for multi-backend fat jars
c90a061 
Groundwork for per-OS "all backends" server fat jars distributed as
GitHub Release assets: one jar carrying the default CPU natives plus
every GPU backend for its OS/arch in backend subdirectories
(net/ladenthin/llama/<os>/<arch>/<backend>/), described by a
jllama-backends.txt manifest in priority order.

Loader behavior:
- Without a manifest (every existing artifact) nothing changes; the
  backend block is skipped entirely.
- With a manifest, each listed backend subdirectory is tried in order;
  the first whose library loads wins (a missing vendor runtime fails
  the load cleanly via UnsatisfiedLinkError and the next backend is
  tried), falling back to the default CPU library after the list.
- A manifest line may name extra files (e.g. a bundled OpenCL ICD
  loader) that are extracted and loaded before the backend's main
  library; a backend whose extra module is already resident from a
  previously failed attempt is skipped to avoid by-name import
  cross-wiring.
- New system property net.ladenthin.llama.backend forces one backend
  (fail-loud, no fallback) or skips all backends via default/cpu.
- Backends extract into per-backend temp subdirectories
  (jllama-backend-<name>/, same file names across backends); cleanup()
  now deletes directories recursively.

Pure parsing/selection logic is package-private static and covered by
15 new LlamaLoaderTest cases (42 total, all passing; javadoc clean).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Add package-fatjars.sh: per-OS all-backends fat jar assembly
aff68f5 
Assembles the multi-backend server fat jars for GitHub Release
distribution: for each OS/arch with GPU classifiers (linux-x86-64,
linux-aarch64, windows-x86-64, windows-aarch64) the default
jar-with-dependencies is copied and every backend's native tree is
added under net/ladenthin/llama/<OS>/<ARCH>/<backend>/ plus a
jllama-backends.txt priority manifest that LlamaLoader consumes at
runtime. The default CPU fat jar is copied to the output as-is; each
output jar gets a .sha256 file.

Enumeration is driven by the <classifier> set in llama/pom.xml
(source of truth), cross-checked in both directions against the
classifier jars actually built, with fail-loud handling for
unparseable classifiers, unranked backends, missing native trees, and
zip-update corruption (entry list, Main-Class, sample byte-compare).
Excluded by design: msvc-windows (redundant CPU variant) and
opencl-android-aarch64 (no java -jar on Android). Sibling files like
openvino-windows' bundled OpenCL.dll are recorded as manifest extras.

Verified locally against synthetic fixture jars mirroring the real
16-classifier set: 4 combined jars + default assembled with correct
manifests; missing-jar and unknown-jar cases fail loud; a LlamaLoader
probe run against the fixture combined jar exercised the full
manifest -> per-backend extraction -> load-failure fallback chain and
the forced/default override paths end-to-end.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
CI: add package-fatjars job assembling the all-backends fat jars
fa4130d 
Runs after `package` on every pipeline (PR/main/tag) so the assembly
mechanism is validated long before a release: downloads the llama-jars
artifact and runs .github/package-fatjars.sh, uploading the 4 combined
per-OS fat jars + the default CPU fat jar + sha256 files as the
`llama-fatjars` artifact (compression off - jars are deflated;
retention 7 days - the release jobs consume it within the same run).
Central deployment is untouched (deploy runs without `assembly`).

Also refreshes the stale Build JARs comment (the fat-jar Main-Class is
ServerLauncher, and the fat jar now seeds the GitHub-Release assets).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
CI: smoke-test the all-backends fat jars on GPU-less Linux + Windows
47fd99d 
Two new jobs (needs: package-fatjars + download-models) launch the
all-<os>-x86-64 fat jar via a real `java -jar` with the cached
AMD-Llama-135m draft model (--chat-template chatml; the base code
model has no embedded template), poll GET /health to 200, POST
/v1/chat/completions and assert choices[0].message, and require the
loader's backend-selection log line - pinning that the manifest was
read and the per-backend extraction + clean-load-failure fallback
chain executed end-to-end on runners without any GPU vendor runtime.
Server logs are uploaded on failure.

package-fatjars additionally uploads the two smoke jars as single-jar
artifacts so the smoke jobs don't download the multi-GB full set.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
CI: attach all-backends fat jars to GitHub releases; gate publishing
ac645ab 
- github-release-signed and github-snapshot additionally download the
  llama-fatjars artifact into their asset directory, so the 4 combined
  per-OS fat jars, the default CPU fat jar, and the sha256 files land
  on the tag release and the rolling snapshot pre-release next to the
  signed Maven artifacts. GitHub download assets only - the Central
  deploy invocations are untouched (no assembly profile there).
- publish-snapshot and publish-release now also need package-fatjars +
  both smoke jobs (fail-loud release gating: a broken fat-jar assembly
  or a failing java -jar smoke blocks any publish).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Docs: all-backends fat jars (README download table + CLAUDE.md wiring)
403cd7d 
README: new "Standalone server fat jars (GitHub Releases)" section
after the classifier table - per-OS download table, java -jar usage,
backend auto-selection semantics with CPU fallback, the
net.ladenthin.llama.backend override (also added to the System
Properties Reference), sha256 files, and the not-on-Central policy.

CLAUDE.md: new section documenting the three-piece mechanism
(package-fatjars.sh fail-loud enumeration/assembly, LlamaLoader
manifest-driven backend selection, publish.yml smoke + release-asset
wiring) plus the zero-devices trade-off and its escape hatch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Set executable bit on package-fatjars.sh
440c2ce 
Matches the other .github/*.sh scripts; CI invokes it via bash, so
this is consistency, not a functional fix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Add REUSE/SPDX headers to the new fat-jar scripts
304173d 
reuse lint is green again (371/371 files with copyright + license).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Fix the 5 SpotBugs findings in the new LlamaLoader backend code
63c3e56 
Sonar "Build and analyze" failed spotbugs-check on PR #301. Real fixes:
hoist the repeated getNativeResourcePath() call out of the backend loop
(PRMC), narrow cleanPath's catch to IOException|RuntimeException (REC),
and build the backend temp dir as a Path instead of File->toPath (UAC).
Documented exclusions following existing precedent: BackendEntry keeps
its List parameter because extra files are ORDER-SIGNIFICANT (OCP, same
rationale as SessionCheckpoint), and the diagnostic triedPaths list has
no knowable size to preallocate (PSC, same rationale as
OpenAiServerCli.parseChatTemplateKwargs).

Verified locally: spotbugs:check reports 0 bugs; LlamaLoaderTest 42/42.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Upgrade llama.cpp from b9886 to b9888
ae70f61 
Tiny upstream range (2 files, ~2.4 KiB, 1 commit): CUDA flash attention
extends K-type validation to V-types (#24403, upstream-compiled CUDA TU
only) plus an upstream router python test addition (not compiled here).

Verified in the sandbox: all eight priority API headers and every
patch-target file (common/arg.{h,cpp}, server-context/-common/-models/
-http/-stream/server.cpp, tools/tts/tts.cpp, tests/test-arg-parser.cpp)
are byte-identical across the range, and all eight patches
(0001-0008) apply cleanly in filename order onto a clean b9888
checkout. Full build + ctest confirmed by the CI pipeline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Cover the loader backend chain with fixture-driven tests (Sonar gate)
42f9880 
SonarCloud's quality gate failed PR #301 with 38.8% coverage on new
code (required >= 80%): the backend-manifest runtime paths only ran
with a real multi-backend jar. New BackendManifestLoadTest drives
LlamaLoader.initialize() end-to-end against two committed fixture
trees, selected via the existing osinfo.architecture override so no
real native library or manifest is ever visible to other tests:

- backendtest/ (unloadable fakes): every failure branch - manifest
  read, per-backend temp-dir extraction, missing extra file, missing
  backend dir, clean load-failure fallback to the default path, the
  forced-backend fail-loud error (known + unknown name), the
  default/cpu skip, and the no-manifest legacy path.
- backendtest-ok/ (two trivial real x86-64 ELF dummies built from a
  one-line C TU): the success path ("using native backend"), the
  resident-extra bookkeeping, and the same-extra clash skip.

Remaining uncovered new lines are the unreachable IOException catches
(~9 lines), putting new-code coverage around 90%. REUSE.toml annotates
the fixture trees (the .so placeholders cannot carry inline headers).
Full suite green: 1401 tests, 0 failures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Smoke jobs: fall back to downloading the draft model on cache miss
78053bf 
The Windows fat-jar smoke failed in run 28805360584 because the
Windows-side gguf-models-v1 cache entry is a poisoned EMPTY archive
(343 bytes; cache entries are per-OS and the Windows one was evidently
evicted and re-saved empty by a later job's post step). The smoke
script's hard Test-Path check was the first thing to catch it -
validate-models.bat had printed "ERROR: Model not found" for all ten
models yet the step passed (exit code not propagated under the pwsh
step shell), so the Windows Java test jobs have been silently
self-skipping their model-backed tests.

The smoke jobs need exactly one ~80 MB model, so they now ensure it
themselves: keep the cache restore (free when warm) and download the
draft model directly when absent, using download-models' own curl
pattern. This removes the fragile validate-models dependency from the
smoke path on both OSes. The underlying repo-wide issues (poisoned
Windows cache entry, validate-models.bat exit propagation) are
reported separately - they predate and exceed this PR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Model cache: single writer + restore-only consumers + working .bat gate
69dbf4d 
Implements the intended model-cache design end-to-end: download-models
is the ONLY place that downloads and the only writer of the cache;
every consumer restores read-only and the validate gate fails loud on
an invalid cache. Three changes:

1. download-models' cache step sets enableCrossOsArchive: true, so the
   one ubuntu-built entry is the same entry macOS AND Windows restore.
   Without it entries are versioned per-OS: the Windows-side entry was
   unreachable from the ubuntu writer and was found re-saved EMPTY
   (343 B) after an eviction in run 28805360584.
2. All 10 consumer cache steps (test-java-*, langchain4j integration,
   Android emulator, both fat-jar smokes) switch to the restore-only
   actions/cache/restore action with the matching cross-OS flag - a
   consumer running on a cache miss can never re-save an empty/partial
   entry under the immutable key again.
3. validate-models.bat: the MODELS list is now unquoted. The previous
   set "MODELS=..." "..." form embedded literal quotes, which made the
   for-loop see the whole list as ONE token; `if not exist` parsed it
   as path-plus-command, so only a fragment was ever checked and the
   exit /b 1 never fired - the ERROR printed but the step passed,
   letting the empty Windows cache sail through the gate while every
   model-backed Windows test silently self-skipped.

Note: the old per-OS cache entries under gguf-models-v1 are obsolete
(the cross-OS flag changes the entry version, so they can never match
again); deleting them is cleanup, not a prerequisite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Model manifest as single truth + 3-OS cache verification gate
d278b5e 
Introduces .github/models.csv (filename,url per row) as the single
source of truth for the CI model set:

- download-models replaces its ten hardcoded curl steps with one
  manifest-driven loop; the *_URL workflow env vars are gone (the
  *_NAME vars stay as consumer-side -D property wiring).
- The cache key is now gguf-models-<hash of models.csv>: editing the
  manifest automatically creates a fresh, complete cache entry - the
  manual "delete the immutable cache entry when the model set changes"
  runbook is obsolete (that stale-entry hazard is likely how the
  Windows cache first went bad).
- validate-models.sh and validate-models.bat read their required list
  from the same manifest, so the gates can never drift from what
  download-models actually fetches. Verified locally (.sh): all 10
  manifest models parse and validate; a missing file and a corrupted
  magic both exit 1.
- New verify-model-cache matrix job (ubuntu / macOS / Windows) runs
  right after download-models: restore-only with fail-on-cache-miss
  plus the full validate gate, proving the single cross-OS entry is
  restorable AND complete on every OS. All ten model-consuming jobs
  now `need: verify-model-cache` instead of download-models, so
  nothing model-backed starts before the cache is proven working.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@claude
Upgrade llama.cpp from b9888 to b9894
328244f 
26 files, ~340 KiB upstream range — nearly all GPU-backend kernel code
inside upstream-compiled TUs (large CUDA dequantize/convert refactor,
new/extended OpenCL flash-attention kernels with Adreno tuning, new
Metal ops, a Vulkan SET_ROWS f16 guard). CPU-side additions are
AIX/PowerPC core detection in common.cpp (irrelevant to our targets)
and a router download-thread join deadlock fix in server-models.cpp.

Verified in the sandbox: all eight priority API headers and all other
patch-target files are byte-identical across the range, and all eight
patches (0001-0008) apply cleanly in filename order onto a clean b9894
checkout (0008's server-models.cpp changed in a different function).
Full build + ctest confirmed by the CI pipeline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6QLUwyT2vDGK2Z4MNzQXq
@bernardladenthin bernardladenthin changed the title Per-OS all-backends server fat jars as GitHub Release assets All-backends server fat jars as GitHub Release assets + model-cache overhaul + llama.cpp b9894 Jul 7, 2026
@sonarqubecloud

sonarqubecloud Bot commented Jul 7, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
8 New issues
0 Accepted issues

Measures
0 Security Hotspots
94.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@bernardladenthin bernardladenthin merged commit fd6529d into main Jul 7, 2026
13 of 71 checks passed
@bernardladenthin bernardladenthin deleted the claude/java-llama-fat-jars-classifiers-kyb4wg branch July 7, 2026 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bernardladenthin @claude