Skip to content

package: every shipped program must resolve its NEEDED libraries the way a user runs it - #556

Merged
bong-water-water-bong merged 3 commits into
mainfrom
buildsecrecy-pkg-needed
Oct 25, 2026
Merged

bong-water-water-bong merged 3 commits into
mainfrom
buildsecrecy-pkg-needed

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

package: every shipped program must resolve its NEEDED libraries the way a user runs it

The 2026-10-23 rc shipped hrx/llama/bin/llama-perplexity without libllama-perplexity-impl.so: the tool
never started, and anything that used it failed at launch rather than at build time. Two things in the stage
made that possible, and both are fixed here.

  • The implementation library is built. cmake/hrx.cmake's HRX build targeted llama-server llama-bench
    only, so the impl library for llama-perplexity did not exist to be packed. llama-perplexity joins the
    target list, so the next package carries both halves.
  • The stage now refuses to ship a program it cannot load. pack_prune_unloadable() drops a program whose
    NEEDED libraries do not resolve from the package and the system, with a line saying so, instead of shipping
    a binary that fails at first use; pack_artifact_hygiene() reports any that remain, so a package cannot
    quietly pass with one.

The resolve check runs the way a user launches the binary: plain ldd with no LD_LIBRARY_PATH, so only
the ELF's own RUNPATH and the system are consulted. A library path inherited from the build shell resolves
against the build tree and would hide exactly this defect.

Proof on a copy of the rc (no program run):

# as shipped, on this box: it resolves *through a RUNPATH into the build cache*, which no user has
libllama-perplexity-impl.so => /home/bcloud/.cache/hw-gate/build-685489b587/bin/libllama-perplexity-impl.so
RUNPATH starts at: /home/bcloud/.cache/hw-gate/build-685489b587/bin

# with that RUNPATH removed — the shape the next package ships in, $ORIGIN only
libllama-perplexity-impl.so => not found
libllama-common.so.0 => not found

# the gate catches it
1bit/hrx/llama/bin/llama-perplexity: libllama-perplexity-impl.so
1bit/hrx/llama/bin/llama-perplexity: libllama-common.so.0   (…)

# and the stage drops the program rather than shipping it broken
  drop 1bit/hrx/llama/bin/llama-perplexity: needs a library the package does not carry
  1 program(s) dropped as unloadable
  → gate on the pruned tree: clean

This is for the next package, not the rc: the rc keeps the defect it was cut with.

🤖 Generated with Claude Code

…way a user runs it

The 2026-10-23 rc shipped `hrx/llama/bin/llama-perplexity` without `libllama-perplexity-impl.so`: the tool
never started, and anything that used it failed at launch rather than at build time. Two things in the stage
made that possible, and both are fixed here.

- **The implementation library is built.** `cmake/hrx.cmake`'s HRX build targeted `llama-server llama-bench`
  only, so the impl library for `llama-perplexity` did not exist to be packed. `llama-perplexity` joins the
  target list, so the next package carries both halves.
- **The stage now refuses to ship a program it cannot load.** `pack_prune_unloadable()` drops a program whose
  NEEDED libraries do not resolve from the package and the system, with a line saying so, instead of shipping
  a binary that fails at first use; `pack_artifact_hygiene()` reports any that remain, so a package cannot
  quietly pass with one.

The resolve check runs the way a user launches the binary: **plain `ldd` with no `LD_LIBRARY_PATH`**, so only
the ELF's own RUNPATH and the system are consulted. A library path inherited from the build shell resolves
against the build tree and would hide exactly this defect.

Proof on a copy of the rc (no program run):

```
# as shipped, on this box: it resolves *through a RUNPATH into the build cache*, which no user has
libllama-perplexity-impl.so => /home/bcloud/.cache/hw-gate/build-685489b587/bin/libllama-perplexity-impl.so
RUNPATH starts at: /home/bcloud/.cache/hw-gate/build-685489b587/bin

# with that RUNPATH removed — the shape the next package ships in, $ORIGIN only
libllama-perplexity-impl.so => not found
libllama-common.so.0 => not found

# the gate catches it
1bit/hrx/llama/bin/llama-perplexity: libllama-perplexity-impl.so
1bit/hrx/llama/bin/llama-perplexity: libllama-common.so.0   (…)

# and the stage drops the program rather than shipping it broken
  drop 1bit/hrx/llama/bin/llama-perplexity: needs a library the package does not carry
  1 program(s) dropped as unloadable
  → gate on the pruned tree: clean
```

This is for the next package, not the rc: the rc keeps the defect it was cut with.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 7ae1c10)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Redundant Function Definition

The function pack_require_provenance is defined twice in scripts/pack-elf.sh. This will cause a shell parsing error and should be removed.

pack_require_provenance() {  # <stage> <build>
    local stage=$1 build=$2 f rel bad=0
    while IFS= read -r -d '' f; do
        pack_is_elf "$f" || continue
        rel=${f#"$stage"/}
        if [ ! -f "$build/$rel" ]; then
            echo "  $rel is not in the build being packaged"
            bad=1
        elif ! cmp -s "$f" "$build/$rel"; then
            echo "  $rel differs from the build being packaged"
            bad=1
        fi
    done < <(find "$stage" -type f -print0)
    return $bad
}
Potential Silent Failure in Tool Validation

The pack_require_tools function calls pack_unresolved_needed and uses grep -F to match the full path. If a program's path contains special regex characters, it could lead to incorrect filtering or missed errors. This could silently allow unloadable programs to be included in the package.

  ./1bit serve -m model.gguf --port 8000      (--device auto: HRX0, else the CPU)
$( [ -d "$stage/npu/kernels" ] && printf '  ./1bit serve -m <Qwen3 NPU model dir> --port 8000   (the 1bit NPU engine, XDNA 2 NPU)\n  ./1bit serve -m model.gguf --device npu --port 8000  (a GGUF repacked for the NPU: needs python3\n                                                       with numpy and gguf, and the Q4NX converter\n                                                       in ONEBIT_Q4NX_CONVERTER or ~/np-model-generic)\n' )

Ubuntu 26.04 with the Lemonade PPA's lemond: sudo ./install-lemonade-ppa.sh

… one cannot load

The 2026-10-23 rc shipped `hrx/llama/bin/llama-perplexity` with RUNPATH entries into
`/home/bcloud/.cache/hw-gate/build-685489b587/…` and without `libllama-perplexity-impl.so`: the binary came from
a hardware-gate build, not from the build being packaged, so it carries that build's paths and is missing the
part that build never produced. It failed at first use, and the KLD rows that used it failed with it.

Four checks, each catching one step of that:

- **`pack_require_provenance <stage> <build>`** — every ELF in the stage must exist at the same relative path
  under the build being packaged and be byte-identical to it. A tool copied in from another tree fails here,
  before anything else looks at it.
- **`pack_require_tools <stage> <rel> <program…>`** — the programs the release promises (`llama-server`,
  `llama-bench`, `llama-perplexity`, the three ds4 binaries) must be in the package and must resolve; otherwise
  the package stops instead of quietly losing a tool.
- **`pack_unresolved_needed <stage>`** — plain `ldd`, **no** `LD_LIBRARY_PATH`, so only the ELF's own RUNPATH
  and the system count, which is how a user launches it.
- **`pack_artifact_hygiene`** — additionally fails on an absolute RUNPATH outside the toolchain root (that is
  what points into a build machine), and `pack_prune_unloadable` drops any incidental ELF that cannot load.

And the tool now exists in the package build: `cmake/hrx.cmake` builds `llama-perplexity` alongside
`llama-server` and `llama-bench`, so its implementation library is produced and ships beside it with `$ORIGIN`.

Measured against the rc (copy, no program run):

```
# provenance, with the rc build tree as the build being packaged
  hrx/llama/bin/llama-perplexity is not in the build being packaged        ← the defect, named
# the promised tools, once the build-machine RUNPATH is gone (the next package's shape)
  libllama-perplexity-impl.so => not found
  1bit/hrx/llama/bin/llama-perplexity: libllama-perplexity-impl.so        ← gate
  drop 1bit/hrx/llama/bin/llama-perplexity: needs a library the package does not carry
```

This is for the next package; the rc keeps the defect it was cut with.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Precise cause and the checks for it, after the freeze/reboot — re-verified on both boxes before trusting any of it.

Cause: hrx/llama/bin/llama-perplexity in the rc came from a hardware-gate build, not from the build being
packaged. Evidence on the rc as shipped: RUNPATH /home/bcloud/.cache/hw-gate/build-685489b587/bin:… (that
build's own bin + libhrx dirs), no libllama-perplexity-impl.so in the package, while llama-server and
llama-bench carry $ORIGIN and do have their impl libraries. Nothing in the stage objected.

Now (this PR, second commit):

  • pack_require_provenance <stage> <build>: every staged ELF must exist at the same relative path under the
    build being packaged and be byte-identical — a foreign copy fails here first. Against the rc it names exactly
    hrx/llama/bin/llama-perplexity is not in the build being packaged.
  • pack_require_tools: the promised programs must be present and must resolve, or the package stops.
  • pack_unresolved_needed: plain ldd, no LD_LIBRARY_PATH (the user's view). With the build-machine
    RUNPATH removed — the next package's shape — it reports libllama-perplexity-impl.so => not found, the gate
    reports it, and pack_prune_unloadable drops the program instead of shipping it broken.
  • cmake/hrx.cmake builds llama-perplexity beside llama-server/llama-bench, so the impl library comes
    from the package build and ships with $ORIGIN.

Correction to my earlier note: the ds4 binaries are in the rc package (three ~25 MB binaries, RUNPATH
$ORIGIN:/opt/rocm-therock/…). My earlier "missing from the package" lines were a path bug in my own check —
it used the package root instead of <pkg>/1bit. Fixed in the harness; there is no ds4 packaging defect.

One caveat worth knowing: pack_require_provenance compares bytes, so replaying it against a historical rc
and a build tree that has since been rebuilt also lists those rebuild differences (e.g. libloomc.so.0). At
packaging time the build is the same tree, so it only reports genuine foreign files.

@bong-water-water-bong
bong-water-water-bong marked this pull request as ready for review October 24, 2026 14:00
@github-actions

Copy link
Copy Markdown

Preparing review...

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit e6f2333

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 7ae1c10

@bong-water-water-bong
bong-water-water-bong merged commit e7d64b8 into main Oct 25, 2026
8 of 9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the buildsecrecy-pkg-needed branch October 25, 2026 05:07
bong-water-water-bong added a commit that referenced this pull request Oct 25, 2026
#563)

After #556 the package stage required llama-perplexity but never staged it, and the byte-for-byte provenance check ran after localization rewrote RUNPATHs so it could never pass. Stage the binary and its impl library; compare provenance before pack_localize.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bong-water-water-bong pushed a commit that referenced this pull request Oct 25, 2026
… comment

Rebased onto main (0d0c846) and reviewed against the current packaging, image and release check
(#556, #563, #569):

- docs/releases.md conflicted on the pins table and the NPU paragraph: the branch put the private
  repositories' names back into a public file ("gpu-kernels (private, HIP)", "npu-kernels (private,
  signed)", "npu-runtime (private)"), which main had scrubbed. Resolved keeping main's wording
  ("private kernel add-on (HIP)", "private NPU kernel set (signed)", "private NPU runtime") and the
  new shape.
- scripts/weekly.sh package_checks: the set's manifest is now derived from `rel` once, instead of
  rebuilding the path with `${f%/npu/kernels/*}` plus `cut` (which can leave a double slash). Checked
  that the derived path resolves for <pkg>/npu/kernels/<set>/<id>/layer1.elf.
- os/mini/init.sh: comment wrapped.

Not marked ready: the package stage stages nothing against the pins in this tree (see the PR comment).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant