From 01cd19c808f0974b335b164100d6238b2b9ae1a8 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sat, 24 Oct 2026 03:57:24 -0300 Subject: [PATCH 1/3] npu: kernel sets now ship with a signed manifest instead of sealing The kernel sets carry a signed manifest: every file in a set is listed with its sha256 and the engine, which holds only the release public key, refuses a file the manifest does not cover. Packaging, the image, the release check and the launcher follow the same shape: a set without its manifest, a file that does not match it, and anything unlisted under npu/kernels is a failure; nothing else about the NPU lane changes. Co-Authored-By: Claude Fable 5.1 --- config/npu-kernels.config | 4 +-- docs/npu.md | 12 ++++--- docs/releases.md | 29 ++++++++--------- os/README.md | 7 +++-- os/mini/init.sh | 4 +-- os/mini/mkimage.sh | 33 ++++++++++++++------ os/mini/release-check.sh | 38 ++++++++++++++++------ scripts/lemond-onebit.sh | 6 ++-- scripts/package-linux.sh | 16 +++++----- scripts/weekly.sh | 66 ++++++++++++++++++++++++--------------- 10 files changed, 134 insertions(+), 81 deletions(-) diff --git a/config/npu-kernels.config b/config/npu-kernels.config index 213cb927..bc0486f0 100644 --- a/config/npu-kernels.config +++ b/config/npu-kernels.config @@ -15,8 +15,8 @@ # # The private NPU kernels the release builds into the NPU lane (docs/npu.md; docs/releases.md). # The kernels are private: their source is not part of this repository and is never published with -# the release. They ship only as compiled, encrypted artifacts that `1bit` verifies and decrypts -# in memory at load. scripts/fetch-private.sh checks out exactly this commit of this repository +# the release. They ship as compiled artifacts with a signed manifest per set, which `1bit` +# verifies at load. scripts/fetch-private.sh checks out exactly this commit of this repository # (with the release machine's GitHub credentials); the build passes the checkout as # ONEBIT_NPU_PRIVATE, and the release stops if it cannot be fetched. # diff --git a/docs/npu.md b/docs/npu.md index b21f7d46..e462776d 100644 --- a/docs/npu.md +++ b/docs/npu.md @@ -26,10 +26,12 @@ package serves GGUF models on HRX, not NPU models ([releases.md](releases.md#the The NPU runtime and the NPU kernels are proprietary and closed source ([releases.md](releases.md#private-components)). They ship only as compiled binaries: the runtime -is compiled into `1bit`, and the kernels ship as compiled, encrypted artifacts that `1bit` verifies -and decrypts in memory at load (a wrong key fails cleanly, and nothing phones home). Their source, -build and method are not in this repository or in any release. This page covers the public parts: -the model format, how `1bit serve` runs an NPU model, and how to build. +is compiled into `1bit`, and each kernel set ships with a signed manifest that lists every file it +contains. `1bit` holds only the release public key and verifies a kernel against its manifest at +load; a modified file, a file the manifest does not list, or a set signed by another key is refused +before anything reaches the device, and nothing phones home. Their source, build and method are not +in this repository or in any release. This page covers the public parts: the model format, how +`1bit serve` runs an NPU model, and how to build. ## A model directory @@ -137,7 +139,7 @@ The model directory holds the weights only; the kernels ship inside the package. ### Dense Qwen3, checked on the NPU -All three dense Qwen3 models run on the NPU from a sealed kernel set, and each one passes the +All three dense Qwen3 models run on the NPU from a signed kernel set, and each one passes the hardware gate's `npu` phase ([hw-gate.md](hw-gate.md)): greedy decoding token for token against an independent fp64 reference over a short prompt and teacher-forced steps, the 64-position build, the prompt prefill, and byte-identical output when the run is repeated, with the phase's speed diff --git a/docs/releases.md b/docs/releases.md index b9a6cbf3..1efc2546 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -42,7 +42,7 @@ Release sha and pins (the submodules this candidate is built from): | hrx-system | `82804197bcbe` | | Lemonade (fork) | `27abcc24da99` | | private kernel add-on (HIP) | `744bd713d968` | -| private NPU kernel set | `17ed5be7a28a` | +| private NPU kernel set (signed) | `17ed5be7a28a` | | private NPU runtime | `ce9fa59adf95` | | Laya | source `6f0496f797dc`, model `7b928d828b7b` | | amdxdna (out-of-tree NPU driver) | `526c608197` | @@ -108,9 +108,9 @@ context (`#466`), allocation safety bounds the program cache and poisons a devic (`#486`), and the ZAYA1-8B NaN re-check closed on **120 fresh loads, 0 NaN** at the current pins (`#433`). -**NPU (sealed).** The gate gained an NPU phase and the packages carry **only the sealed kernel -sets** — a plaintext artifact fails the packaging check, and the launcher refuses an unsealed one -(`#503`, `#505`, `#509`, `#512`). The 0.6B gate phase measures **90.33 tok/s**; the OS image proves +**NPU (signed).** The gate gained an NPU phase and the packages carry **only signed kernel sets** — +every shipped file is listed in the set's signed manifest, a set that does not verify fails the +packaging check, and the engine refuses an unlisted kernel file (`#503`, `#505`, `#509`, `#512`). The 0.6B gate phase measures **90.33 tok/s**; the OS image proves `amd_pmf` sensors and the NPU from the booted image (`#517`, `#523`). The runtime's public header and prebuilt-library build are in the tree (`#467`, `#479`, `#469`). @@ -121,7 +121,7 @@ the Ubuntu 26.04 PPA's `lemond` runs our HRX server unchanged (`#483`). Synced t one request in role, `lemonade bench ... chat-short` — and 3 NPU entries (fork `#19`, `#20`, `#23`); chat-short spans 43.4 (Qwen3-8B) to 330.7 tok/s (Qwen3-0.6B), 0 faults in 16 holds. -**1bit OS.** The image carries the sealed NPU sets and the new Lemonade (`#512`, `#523`), the UKI is +**1bit OS.** The image carries the signed NPU sets and the new Lemonade (`#512`, `#523`), the UKI is signed with the enrolled MOK so a stick boots with Secure Boot on (`#513`), the USB4 v2 / Thunderbolt stack is loaded (`#514`), and `amd_pmf` sensors come up from the booted image (`#517`). The install step is not in this candidate (branch `1bit/os-install`). @@ -163,17 +163,18 @@ The builds target Strix Halo: the llama.cpp backends are built with `GGML_NATIVE The release is the HRX runtime with our HIP kernels and the NPU engine, all inside Lemonade. The release carries the NPU runtime, compiled into `1bit`, and the `onebit` recipe's `npu` backend. -The NPU kernels are closed too: they ship as sealed artifacts that `1bit` verifies and opens in -memory at load, failing cleanly on a -wrong key with no network callback of any kind; a lifted kernel file is ciphertext and does not run -([npu.md](npu.md)). The release package carries them, and serves dense Qwen3 (0.6B, 1.7B, 4B) +The NPU kernels are closed too: they ship as compiled artifacts with a signed manifest per set +(`ONEBIT_NPU_PRIVATE`), and `1bit` verifies each kernel against that manifest at load, failing +cleanly on a modified or unlisted file with no network callback of any kind ([npu.md](npu.md)). The +release package carries them, and serves dense Qwen3 (0.6B, 1.7B, 4B) on the NPU. -`scripts/weekly.sh` packs them by default (`WEEKLY_NPU_KERNELS=0` builds without): the sealed -kernel sets go beside `1bit` in `npu/kernels`, the build loads sealed kernels only -(`ONEBIT_NPU_RELEASE`), the `lemond-onebit` launcher sets `ONEBIT_KCRYPT_REQUIRE_SEALED=1`, and the package checks refuse a plaintext kernel, -an ELF for anything but the host, a symbol table in `1bit`, or a private source name in it. The -release machine holds the kernel key in `ONEBIT_NPU_KEY_DIR` (default `~/.config/1bit-npu-key`). +`scripts/weekly.sh` packs them by default (`WEEKLY_NPU_KERNELS=0` builds without): each signed +kernel set goes beside `1bit` in `npu/kernels`, the build loads signed sets only +(`ONEBIT_NPU_RELEASE`), the `lemond-onebit` launcher sets `ONEBIT_KMANIFEST_REQUIRED=1`, and the +package checks refuse a kernel the manifest does not list, a set without its manifest, an ELF for +anything but the host, a symbol table in `1bit`, or a private source name in it. The release machine +holds the signing key in `ONEBIT_NPU_KEY_DIR` (default `~/.config/1bit-npu-key`). ### Private components diff --git a/os/README.md b/os/README.md index be2f337e..a1f48c2c 100644 --- a/os/README.md +++ b/os/README.md @@ -169,10 +169,11 @@ install to disk. Every check prints its own PASS/FAIL/SKIP line, a transcript la It checks the boot mode (LIVE or INSTALLED), the kernel and the Secure Boot state, the GPU and NPU devices, that Lemonade answers with its API key and refuses without it, that `/api/v1/models` lists -the onebit entries the build shipped (22 on HRX, plus the NPU entries whose sealed kernel set is on +the onebit entries the build shipped (22 on HRX, plus the NPU entries whose signed kernel set is on the data partition) and the stick's own GGUF, one chat per backend the machine can run, one -`lemonade bench` chat-short row on the default model, that every file under `/data/npu/kernels` is a -sealed artifact, that `LICENSE-BINARY`/`NOTICE-BINARY` are present, and the image's tag + sha256. +`lemonade bench` chat-short row on the default model, that every file under `/data/npu/kernels` is +listed in its set's signed manifest and matches its sha256, that `LICENSE-BINARY`/`NOTICE-BINARY` +are present, and the image's tag + sha256. The rows that need the hardware - `gpu`, `npu`, `models`, `chat-hrx`, `chat-npu` - report `SKIP(reason)` in a VM. diff --git a/os/mini/init.sh b/os/mini/init.sh index 257909b9..6460d590 100644 --- a/os/mini/init.sh +++ b/os/mini/init.sh @@ -94,12 +94,12 @@ else say "[engine] no /data/1bit-llama: '1bit serve' has no HRX llama-server on this boot" fi -# The sealed NPU kernels live on the stick (they used to be inside the RAM image, which made it too big +# The NPU kernels (each with its signed manifest) live on the stick (they used to be inside the RAM image, which made it too big # to load): link them to the path the engine looks for, beside itself. if [ -d /data/npu/kernels ] && [ -d /home ]; then mkdir -p /home/bcloud/wt/npupkg-build/npu 2>/dev/null ln -sfn /data/npu/kernels /home/bcloud/wt/npupkg-build/npu/kernels 2>/dev/null - say "[kernels] $(ls /data/npu/kernels | wc -l) sealed NPU sets linked beside the engine" + say "[kernels] $(ls /data/npu/kernels | wc -l) signed NPU sets linked beside the engine" fi n=0 diff --git a/os/mini/mkimage.sh b/os/mini/mkimage.sh index 902d98fc..b092e496 100755 --- a/os/mini/mkimage.sh +++ b/os/mini/mkimage.sh @@ -220,7 +220,8 @@ bin=$build/onebit; [ -x "$bin" ] || bin=$build/1bit [ -x "$bin" ] || { echo "no engine binary in $build (build with -DONEBIT_NPU=ON -DONEBIT_HRX=ON)"; exit 1; } copy_elf "$bin" ln -s "$bin" "$root/bin/1bit" -# The sealed NPU kernel sets travel BESIDE the engine binary (docs/releases.md, "The NPU engine"): +# The NPU kernel sets travel BESIDE the engine binary (docs/releases.md, "The NPU engine"), each +# with the signed manifest the engine verifies it against at load: # in a package that is /npu/kernels next to /1bit, and here the binary keeps its build # path, so the sets go to the same place relative to it. Without them the image ships the NPU # route and nothing it can load - the route then refuses rather than falling back. @@ -562,12 +563,13 @@ for m in ${MODELS:-}; do done data_mb=${DATA_MB:-$(( $(du -sm "$out/data.dir" | cut -f1) + 1024 ))} # the contents plus 1 GiB rm -f "$out/data.part"; if [ -d "$build/npu/kernels" ]; then - # The sealed sets go on the STICK'S DATA PARTITION, not inside the RAM image. They are ~124 MB and + # The signed sets go on the STICK'S DATA PARTITION, not inside the RAM image. They are ~124 MB and # were making the initramfs 103 MB compressed, which is what grub was failing to load on real # hardware (the machine's own 37 MB initrd loads fine). init.sh links them to the engine's # expected path at boot, so the engine still finds npu/kernels beside itself. - # Only the artifact surface goes on the stick: /layout.json and /layer1_*/layer1.elf, the - # allowlist scripts/package-linux.sh packs. A set directory also holds layer1.work/ (plaintext + # Only the artifact surface goes on the stick: /layout.json, /layer1_*/layer1.elf, + # manifest and manifest.sig - the allowlist scripts/package-linux.sh packs. A set directory also + # holds layer1.work/ (plaintext # MLIR, object files, intermediate ELFs) because that is where it was built, so copying the tree # would put the kernel method on the stick; the guard that caught that used `find ... | head -3`, # which exits 141 (SIGPIPE) under `set -o pipefail` and aborted the whole image instead. @@ -576,19 +578,32 @@ rm -f "$out/data.part"; if [ -d "$build/npu/kernels" ]; then setname=$(basename "$set") [ -f "$set/layout.json" ] || continue mkdir -p "$out/data.dir/npu/kernels/$setname" - cp -a "$set/layout.json" "$out/data.dir/npu/kernels/$setname/" + cp -a "$set/layout.json" "$set/manifest" "$set/manifest.sig" "$out/data.dir/npu/kernels/$setname/" for layer in "$set"/layer1_*/; do [ -f "$layer/layer1.elf" ] || continue mkdir -p "$out/data.dir/npu/kernels/$setname/$(basename "$layer")" cp -a "$layer/layer1.elf" "$out/data.dir/npu/kernels/$setname/$(basename "$layer")/" done done - sealed=$(find "$out/data.dir/npu/kernels" -name layer1.elf -exec \ - sh -c 'head -c 8 "$1" | grep -q 1BPKCRYP' _ {} \; -print | wc -l) total=$(find "$out/data.dir/npu/kernels" -name layer1.elf | wc -l) [ "$total" -gt 0 ] || { echo "npu/kernels in $build carries no layer1.elf"; exit 1; } - [ "$sealed" = "$total" ] || { echo "npu/kernels: $sealed of $total kernels are sealed (plaintext would leak the method)"; exit 1; } - echo " npu/kernels: $(ls "$out/data.dir/npu/kernels" | wc -l) sealed sets on the data partition ($sealed kernels)" + # Each set must carry the manifest it is verified against, and every kernel it ships must be + # listed in it with the same sha256: a set that does not verify never reaches a stick. + bad=0 + for set in "$out/data.dir/npu/kernels"/*/; do + [ -f "$set/layout.json" ] || continue + [ -s "$set/manifest" ] && [ "$(stat -c%s "$set/manifest.sig" 2>/dev/null || echo 0)" = 64 ] || + { echo "npu/kernels: $(basename "$set") has no signed manifest"; bad=1; continue; } + for elf in "$set"/layer1_*/layer1.elf; do + [ -f "$elf" ] || continue + rel=${elf#"$set"} + want=$(awk -v p="$rel" '$1=="file" && $4==p {print $2}' "$set/manifest") + [ -n "$want" ] && [ "$want" = "$(sha256sum "$elf" | cut -d' ' -f1)" ] || + { echo "npu/kernels: $(basename "$set")/$rel does not match its manifest"; bad=1; } + done + done + [ "$bad" = 0 ] || exit 1 + echo " npu/kernels: $(ls "$out/data.dir/npu/kernels" | wc -l) signed sets on the data partition ($total kernels)" fi # The engine's own llama-server goes on the DATA PARTITION, not into the RAM image. This point in the # script is AFTER the initramfs was packed (above), so what used to happen here - copying llama-server, diff --git a/os/mini/release-check.sh b/os/mini/release-check.sh index cde3ad95..baf3606a 100755 --- a/os/mini/release-check.sh +++ b/os/mini/release-check.sh @@ -24,8 +24,8 @@ # Rows that cannot PASS without the hardware are marked SKIP in a VM (Lemonade hides its onebit # entries on a machine with no AMD GPU or NPU, and there is no HRX or NPU device to run them on): # gpu, npu, models, chat-hrx, chat-npu. Everything else - mode, kernel, Secure Boot, Lemonade up -# with its key, the stick's GGUF answering on vulkan, `lemonade bench`, the sealed NPU sets, the -# licences and the image identity - is checked in the VM as well. +# with its key, the stick's GGUF answering on vulkan, `lemonade bench`, the signed NPU kernel sets, +# the licences and the image identity - is checked in the VM as well. set -u LOG=/data/release-check.log @@ -304,20 +304,38 @@ else fi fi -# ── sealed NPU sets: the method never ships in the clear ──────────────────────────────────────── +# ── NPU kernel sets: every file verifies against the set's signed manifest ───────────────────── +# The signature is checked by the engine at load (it holds only the release public key and refuses +# a file its set's manifest does not cover), so this row proves the bytes on the stick are the ones +# that were signed and that nothing unlisted rides along. SETS=$(find /data/npu/kernels -name layout.json 2>/dev/null | wc -l) if [ "$SETS" = 0 ]; then - fail sealed "no NPU kernel set on the data partition (find /data/npu/kernels -name layout.json)" + fail npu-kernels "no NPU kernel set on the data partition (find /data/npu/kernels -name layout.json)" else BAD=0 - for f in $(find /data/npu/kernels -type f 2>/dev/null); do - [ "$(basename "$f")" = layout.json ] && continue - [ "$(head -c 8 "$f" 2>/dev/null)" = 1BPKCRYP ] && continue - BAD=$((BAD + 1)); say " not sealed: $f" + for set in $(find /data/npu/kernels -mindepth 1 -maxdepth 1 -type d 2>/dev/null); do + [ -f "$set/layout.json" ] || continue + if [ ! -s "$set/manifest" ] || [ "$(stat -c%s "$set/manifest.sig" 2>/dev/null || echo 0)" != 64 ]; then + BAD=$((BAD + 1)); say " no signed manifest in $set"; continue + fi + for f in "$set"/layout.json "$set"/layer1_*/layer1.elf; do + [ -f "$f" ] || continue + rel=${f#"$set"/} + want=$(awk -v p="$rel" '$1=="file" && $4==p {print $2}' "$set/manifest" 2>/dev/null) + if [ -z "$want" ]; then BAD=$((BAD + 1)); say " not in the manifest: $f" + elif [ "$want" != "$(sha256sum "$f" 2>/dev/null | cut -d' ' -f1)" ]; then + BAD=$((BAD + 1)); say " does not match the manifest: $f"; fi + done + for f in $(find "$set" -type f 2>/dev/null); do + rel=${f#"$set"/} + case "$rel" in layout.json|manifest|manifest.sig) continue ;; esac + [ -n "$(awk -v p="$rel" '$1=="file" && $4==p {print $2}' "$set/manifest" 2>/dev/null)" ] || + { BAD=$((BAD + 1)); say " unlisted file: $f"; } + done done [ "$(find /data/npu/kernels -name '*.mlir' -o -name 'layer1.work' 2>/dev/null | wc -l)" = 0 ] || { BAD=$((BAD + 1)); say " plaintext build tree on the data partition"; } - if [ "$BAD" = 0 ]; then pass sealed "$SETS sealed sets, $PLAIN_TOTAL kernels, none in the clear" - else fail sealed "$BAD file(s) under /data/npu/kernels are not sealed artifacts"; fi + if [ "$BAD" = 0 ]; then pass npu-kernels "$SETS signed sets, $PLAIN_TOTAL kernels, every file matches its manifest" + else fail npu-kernels "$BAD problem(s) under /data/npu/kernels"; fi fi # ── licences and image identity ──────────────────────────────────────────────────────────────── diff --git a/scripts/lemond-onebit.sh b/scripts/lemond-onebit.sh index 626bd4a6..a5129239 100755 --- a/scripts/lemond-onebit.sh +++ b/scripts/lemond-onebit.sh @@ -24,8 +24,8 @@ set -euo pipefail here=$(cd "$(dirname "$(readlink -f "$0")")" && pwd) export LEMONADE_ONEBIT_BIN=${LEMONADE_ONEBIT_BIN:-$here/1bit/1bit} -# the NPU engine loads sealed kernels only, always: no opt-out (the release 1bit refuses plain -# kernel sets on its own too) -export ONEBIT_KCRYPT_REQUIRE_SEALED=1 +# the NPU engine loads signed kernel sets only, always: no opt-out (the release 1bit refuses an +# unlisted kernel file on its own too) +export ONEBIT_KMANIFEST_REQUIRED=1 export PATH=$here/bin:$PATH exec "$here/bin/lemond" "$@" diff --git a/scripts/package-linux.sh b/scripts/package-linux.sh index 55a2d3ed..91e8aa2a 100755 --- a/scripts/package-linux.sh +++ b/scripts/package-linux.sh @@ -55,26 +55,26 @@ cp -a "$build/1bit" "$stage/" put hrx/llama/bin llama-server llama-bench llama-perplexity put ds4/hip ds4-server ds4 ds4-bench -# the sealed NPU kernel sets (scripts/weekly.sh, WEEKLY_NPU_KERNELS): encrypted artifacts, packed -# beside 1bit in npu/kernels, where it looks for them -# Only the artifact surface is shipped: /layout.json and /layer1_*/layer1.elf. The set -# directory also holds layer1.work/ (plaintext MLIR, object files, intermediate ELFs) because that is -# where it was built; copying the tree would ship the kernel method and feed those static ELFs to -# patchelf. package_checks asserts this allowlist on the tarball. +# the NPU kernel sets (scripts/weekly.sh, WEEKLY_NPU_KERNELS): packed beside 1bit in npu/kernels, +# where it looks for them, each with the signed manifest it is verified against at load. +# Only the artifact surface is shipped: /layout.json, /layer1_*/layer1.elf, manifest and +# manifest.sig. The set directory also holds layer1.work/ (plaintext MLIR, object files, +# intermediate ELFs) because that is where it was built; copying the tree would ship the kernel +# method and feed those static ELFs to patchelf. package_checks asserts this allowlist on the tarball. if [ -d "$build/npu/kernels" ]; then mkdir -p "$stage/npu/kernels" for set in "$build"/npu/kernels/*/; do setname=$(basename "$set") # not $name: that is the package name tar uses below [ -f "$set/layout.json" ] || continue mkdir -p "$stage/npu/kernels/$setname" - cp -a "$set/layout.json" "$stage/npu/kernels/$setname/" + cp -a "$set/layout.json" "$set/manifest" "$set/manifest.sig" "$stage/npu/kernels/$setname/" for layer in "$set"/layer1_*/; do [ -f "$layer/layer1.elf" ] || continue mkdir -p "$stage/npu/kernels/$setname/$(basename "$layer")" cp -a "$layer/layer1.elf" "$stage/npu/kernels/$setname/$(basename "$layer")/" done done - echo " npu/kernels: $(ls "$stage/npu/kernels" | wc -l) sets, $(find "$stage/npu/kernels" -name layer1.elf | wc -l) sealed kernels" + echo " npu/kernels: $(ls "$stage/npu/kernels" | wc -l) sets, $(find "$stage/npu/kernels" -name layer1.elf | wc -l) kernels, each with its signed manifest" fi # The NPU lane's GGUF path runs public helper scripts (the repack, its declared-scale guard and the diff --git a/scripts/weekly.sh b/scripts/weekly.sh index 775a6bac..a5266070 100755 --- a/scripts/weekly.sh +++ b/scripts/weekly.sh @@ -123,18 +123,18 @@ build() { [ -d "$HOME/models/laya-pinned" ] && extra+=(-DONEBIT_LAYA_ROOT="$HOME/models/laya-pinned") # shellcheck disable=SC2206 extra+=(${WEEKLY_CMAKE_ARGS:-}) - # The NPU kernels (docs/releases.md, "The NPU engine"): sealed, packed beside 1bit in - # npu/kernels. On by default (the npu gate phase passed on Strix Halo; WEEKLY_NPU_KERNELS=0 - # builds a package without them). The private checkout does the work through its one entry - # point (key install, sealing, the per-commit kernel cache); this script names nothing in it. - # ONEBIT_NPU_RELEASE makes the built 1bit load sealed kernels only. + # The NPU kernels (docs/releases.md, "The NPU engine"): each set ships with a signed manifest, + # packed beside 1bit in npu/kernels. On by default (the npu gate phase passed on Strix Halo; + # WEEKLY_NPU_KERNELS=0 builds a package without them). The private checkout does the work + # through its one entry point (key install, signing, the per-commit kernel cache); this script + # names nothing in it. ONEBIT_NPU_RELEASE makes the built 1bit load signed sets only. if [ "${WEEKLY_NPU_KERNELS:-1}" = 1 ]; then "$SRC/scripts/fetch-private.sh" "$SRC/config/npu-kernels.config" "$W/npu-kernels" > /dev/null 2>> "$LOGS/private.log" || { cat "$LOGS/private.log"; return 1; } guard "$W/npu-kernels/addons/release-kernels.sh" "$BUILD/npu/kernels" "${ONEBIT_NPU_KEY_DIR:-$HOME/.config/1bit-npu-key}" \ > "$LOGS/npu-kernels.log" 2>&1 || { tail -n 5 "$LOGS/npu-kernels.log"; return 1; } extra+=(-DONEBIT_NPU_PRIVATE="$W/npu-kernels" -DONEBIT_NPU_RELEASE=ON) - say "NPU kernels $(git -C "$W/npu-kernels" rev-parse --short HEAD): $(ls "$BUILD/npu/kernels" | wc -l) sealed sets" + say "NPU kernels $(git -C "$W/npu-kernels" rev-parse --short HEAD): $(ls "$BUILD/npu/kernels" | wc -l) signed sets" fi cmake -S "$SRC" -B "$BUILD" -G Ninja -DCMAKE_BUILD_TYPE=Release \ -DONEBIT_HRX=ON -DONEBIT_GPU_PRIVATE="$W/gpu-kernels" \ @@ -273,9 +273,10 @@ stage_test() { run_tests; } # are the engine's own Apache-2.0 source, published in this repository, and without # which a packed install cannot serve a GGUF on the NPU # licensed the closed binaries' LICENSE-BINARY and NOTICE-BINARY are packed beside LICENSE and NOTICE -# npu (with WEEKLY_NPU_KERNELS) every file under npu/kernels but layout.json is sealed, never a -# plaintext ELF; every ELF in the tarball is for the host (x86-64); the packed 1bit has no -# symbol table and its strings name no private source file and no build-machine home +# npu (with WEEKLY_NPU_KERNELS) every set under npu/kernels carries a signed manifest, every +# file it ships is listed in it and matches its sha256, and nothing unlisted rides along; +# every other ELF in the tarball is for the host (x86-64); the packed 1bit has no symbol +# table and its strings name no private source file and no build-machine home # Checks the build's 1bit and every file in the tarball. package_checks() { # package_checks # (no `cmd | grep -q` here: under pipefail grep's early exit fails the pipeline on a match) @@ -327,32 +328,47 @@ package_checks() { # package_checks n=$(find "$t" -type f -printf '%f\n' | LC_ALL=C sort -u | LC_ALL=C comm -12 - <(echo "$names") | tee >(cat >&2) | wc -l) [ "$n" = 0 ] || { echo "FAIL files named like the private sources in the tarball"; bad=1; } [ $bad = 0 ] && echo "ok no private source in $(basename "$1")" - # The NPU kernels, when the package carries them (WEEKLY_NPU_KERNELS): every kernel file is - # sealed (never a plaintext ELF), no ELF in the tarball is for anything but the host (an NPU - # kernel would be), the shipped 1bit has no symbol table, and its strings name no private - # source file and no build-machine home. + # The NPU kernels, when the package carries them (WEEKLY_NPU_KERNELS): each set ships + # layout.json, layer1_*/layer1.elf and a signed manifest; every shipped file is listed in that + # manifest and matches its sha256, so nothing unlisted (a build tree: plaintext MLIR, objects, + # intermediate ELFs) and nothing modified can ride along. The manifest signature itself is + # checked by the engine at load (it holds only the release public key and refuses a file the + # manifest does not cover), so this row verifies the package, not the signature. local nk=0 m privsrc if [ "${WEEKLY_NPU_KERNELS:-1}" = 1 ]; then n=$(find "$t" -path '*/npu/kernels/*/layout.json' -type f | wc -l) [ "$n" -ge 3 ] || { echo "FAIL WEEKLY_NPU_KERNELS=1 but $(basename "$1") carries $n NPU kernel sets (want the 0.6B, 1.7B and 4B sets)"; bad=1; } fi - # The shipped npu tree is an exact allowlist: /layout.json and /layer1_*/layer1.elf. - # Anything else is the kernel method in the clear - layer1.work/ carries plaintext MLIR, object - # files and intermediate ELFs - so a stray file is a leak, not just a wrong shape. while IFS= read -r -d '' f; do rel=${f#"$t"/} - if [ "$(basename "$f")" = layout.json ]; then continue; fi - if [ "$(head -c 8 "$f")" != 1BPKCRYP ]; then - echo "FAIL leaked build artifact in the package (not a sealed kernel): $rel"; nk=$((nk + 1)) - elif [[ ! "$rel" =~ ^[^/]+/npu/kernels/[^/]+/layer1_[^/]+/layer1\.elf$ ]]; then - echo "FAIL unexpected file under npu/kernels (a set ships layout.json + layer1_*/layer1.elf only): $rel"; nk=$((nk + 1)) + setdir=${f%/npu/kernels/*}/npu/kernels/$(echo "${rel#*/npu/kernels/}" | cut -d/ -f1) + case "$(basename "$f")" in + layout.json|manifest|manifest.sig) continue ;; + esac + if [[ "$rel" =~ ^[^/]+/npu/kernels/[^/]+/(layer1_[^/]+/layer1\.elf)$ ]]; then + inner=${BASH_REMATCH[1]} + want=$(awk -v p="$inner" '$1=="file" && $4==p {print $2}' "$t/$setdir/manifest" 2>/dev/null) + if [ -z "$want" ]; then + echo "FAIL $rel is not listed in its set's signed manifest"; nk=$((nk + 1)) + elif [ "$want" != "$(sha256sum "$f" | cut -d' ' -f1)" ]; then + echo "FAIL $rel does not match the sha256 in its set's manifest"; nk=$((nk + 1)) + fi + else + echo "FAIL unexpected file under npu/kernels (a set ships layout.json + layer1_*/layer1.elf + manifest + manifest.sig): $rel"; nk=$((nk + 1)) fi done < <(find "$t" -path '*/npu/kernels/*' -type f -print0) + for lj in $(find "$t" -path '*/npu/kernels/*/layout.json' -type f 2>/dev/null); do + set=$(dirname "$lj") + [ -s "$set/manifest" ] || { echo "FAIL $set has no signed manifest"; nk=$((nk + 1)); } + [ "$(stat -c%s "$set/manifest.sig" 2>/dev/null || echo 0)" = 64 ] || + { echo "FAIL $set/manifest.sig is not a 64-byte Ed25519 signature"; nk=$((nk + 1)); } + done [ "$nk" = 0 ] || bad=1 n=$(find "$t" -path '*/npu/kernels/*' -type f | wc -l) - [ "$n" = 0 ] || [ "$nk" != 0 ] || echo "ok NPU kernels: $n files under npu/kernels, all sealed" + [ "$n" = 0 ] || [ "$nk" != 0 ] || echo "ok NPU kernels: $n files under npu/kernels, every one listed in a signed manifest" nk=0 while IFS= read -r -d '' f; do + case "$f" in */npu/kernels/*) continue ;; esac # AIE kernels, not host programs pack_is_elf "$f" || continue m=$(od -An -t u2 -j 18 -N 2 "$f" | tr -d ' ') [ "$m" = 62 ] || { echo "FAIL $f is an ELF for machine $m, not the host (x86-64)"; nk=$((nk + 1)); } @@ -407,9 +423,9 @@ lemonade_package_checks() { # lemonade_package_checks n=$(find "$t" -path '*/bin/resources/web-app/index.html' -type f | wc -l) if [ "$n" != 0 ]; then echo "ok Lemonade's web app: bin/resources/web-app/index.html and $(find "$t" -path '*/bin/resources/web-app/*' -type f | wc -l) files in $(basename "$1")" else echo "FAIL no Lemonade web app (bin/resources/web-app/index.html) in $(basename "$1")"; rc=1; fi - n=$(find "$t" -maxdepth 2 -name lemond-onebit -type f -exec grep -c '^export ONEBIT_KCRYPT_REQUIRE_SEALED=1$' {} + 2>/dev/null | head -1) - if [ "${n:-0}" != 0 ]; then echo "ok lemond-onebit loads sealed NPU kernels only (ONEBIT_KCRYPT_REQUIRE_SEALED=1)" - else echo "FAIL lemond-onebit does not export ONEBIT_KCRYPT_REQUIRE_SEALED=1"; rc=1; fi + n=$(find "$t" -maxdepth 2 -name lemond-onebit -type f -exec grep -c '^export ONEBIT_KMANIFEST_REQUIRED=1$' {} + 2>/dev/null | head -1) + if [ "${n:-0}" != 0 ]; then echo "ok lemond-onebit loads signed NPU kernel sets only (ONEBIT_KMANIFEST_REQUIRED=1)" + else echo "FAIL lemond-onebit does not export ONEBIT_KMANIFEST_REQUIRED=1"; rc=1; fi rm -rf "$t" return $rc } From 1e727e0eb92251e726a53de100ff8659578a36de Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sat, 24 Oct 2026 04:01:43 -0300 Subject: [PATCH 2/3] npu: a shipped kernel set is manifest + manifest.sig + /layer1.elf Follow-up to the signed-manifest change: the readable layout sidecar no longer ships (the layout recordrides inside the signed manifest) and the kernel directories are opaque ids, so packaging, the image, the release check and the package checks stop looking for layout.json and stop requiring layer1_ directory names. A set that still carries layout.json, or a file the manifest does not list, is a failure. Co-Authored-By: Claude Fable 5.1 --- os/mini/mkimage.sh | 14 +++++++------- os/mini/release-check.sh | 10 +++++----- scripts/package-linux.sh | 17 +++++++++-------- scripts/weekly.sh | 18 +++++++++--------- 4 files changed, 30 insertions(+), 29 deletions(-) diff --git a/os/mini/mkimage.sh b/os/mini/mkimage.sh index b092e496..99564076 100755 --- a/os/mini/mkimage.sh +++ b/os/mini/mkimage.sh @@ -567,8 +567,8 @@ rm -f "$out/data.part"; if [ -d "$build/npu/kernels" ]; then # were making the initramfs 103 MB compressed, which is what grub was failing to load on real # hardware (the machine's own 37 MB initrd loads fine). init.sh links them to the engine's # expected path at boot, so the engine still finds npu/kernels beside itself. - # Only the artifact surface goes on the stick: /layout.json, /layer1_*/layer1.elf, - # manifest and manifest.sig - the allowlist scripts/package-linux.sh packs. A set directory also + # Only the artifact surface goes on the stick: /manifest, /manifest.sig and + # //layer1.elf - the allowlist scripts/package-linux.sh packs. A set directory also # holds layer1.work/ (plaintext # MLIR, object files, intermediate ELFs) because that is where it was built, so copying the tree # would put the kernel method on the stick; the guard that caught that used `find ... | head -3`, @@ -576,10 +576,10 @@ rm -f "$out/data.part"; if [ -d "$build/npu/kernels" ]; then mkdir -p "$out/data.dir/npu/kernels" for set in "$build"/npu/kernels/*/; do setname=$(basename "$set") - [ -f "$set/layout.json" ] || continue + [ -f "$set/manifest" ] || continue mkdir -p "$out/data.dir/npu/kernels/$setname" - cp -a "$set/layout.json" "$set/manifest" "$set/manifest.sig" "$out/data.dir/npu/kernels/$setname/" - for layer in "$set"/layer1_*/; do + cp -a "$set/manifest" "$set/manifest.sig" "$out/data.dir/npu/kernels/$setname/" + for layer in "$set"/*/; do [ -f "$layer/layer1.elf" ] || continue mkdir -p "$out/data.dir/npu/kernels/$setname/$(basename "$layer")" cp -a "$layer/layer1.elf" "$out/data.dir/npu/kernels/$setname/$(basename "$layer")/" @@ -591,10 +591,10 @@ rm -f "$out/data.part"; if [ -d "$build/npu/kernels" ]; then # listed in it with the same sha256: a set that does not verify never reaches a stick. bad=0 for set in "$out/data.dir/npu/kernels"/*/; do - [ -f "$set/layout.json" ] || continue + [ -f "$set/manifest" ] || continue [ -s "$set/manifest" ] && [ "$(stat -c%s "$set/manifest.sig" 2>/dev/null || echo 0)" = 64 ] || { echo "npu/kernels: $(basename "$set") has no signed manifest"; bad=1; continue; } - for elf in "$set"/layer1_*/layer1.elf; do + for elf in "$set"/*/layer1.elf; do [ -f "$elf" ] || continue rel=${elf#"$set"} want=$(awk -v p="$rel" '$1=="file" && $4==p {print $2}' "$set/manifest") diff --git a/os/mini/release-check.sh b/os/mini/release-check.sh index baf3606a..caa17a6e 100755 --- a/os/mini/release-check.sh +++ b/os/mini/release-check.sh @@ -308,17 +308,17 @@ fi # The signature is checked by the engine at load (it holds only the release public key and refuses # a file its set's manifest does not cover), so this row proves the bytes on the stick are the ones # that were signed and that nothing unlisted rides along. -SETS=$(find /data/npu/kernels -name layout.json 2>/dev/null | wc -l) +SETS=$(find /data/npu/kernels -name manifest 2>/dev/null | wc -l) if [ "$SETS" = 0 ]; then - fail npu-kernels "no NPU kernel set on the data partition (find /data/npu/kernels -name layout.json)" + fail npu-kernels "no NPU kernel set on the data partition (find /data/npu/kernels -name manifest)" else BAD=0 for set in $(find /data/npu/kernels -mindepth 1 -maxdepth 1 -type d 2>/dev/null); do - [ -f "$set/layout.json" ] || continue + [ -f "$set/manifest" ] || continue if [ ! -s "$set/manifest" ] || [ "$(stat -c%s "$set/manifest.sig" 2>/dev/null || echo 0)" != 64 ]; then BAD=$((BAD + 1)); say " no signed manifest in $set"; continue fi - for f in "$set"/layout.json "$set"/layer1_*/layer1.elf; do + for f in "$set"/*/layer1.elf; do [ -f "$f" ] || continue rel=${f#"$set"/} want=$(awk -v p="$rel" '$1=="file" && $4==p {print $2}' "$set/manifest" 2>/dev/null) @@ -328,7 +328,7 @@ else done for f in $(find "$set" -type f 2>/dev/null); do rel=${f#"$set"/} - case "$rel" in layout.json|manifest|manifest.sig) continue ;; esac + case "$rel" in manifest|manifest.sig) continue ;; esac [ -n "$(awk -v p="$rel" '$1=="file" && $4==p {print $2}' "$set/manifest" 2>/dev/null)" ] || { BAD=$((BAD + 1)); say " unlisted file: $f"; } done diff --git a/scripts/package-linux.sh b/scripts/package-linux.sh index 91e8aa2a..1a552cc0 100755 --- a/scripts/package-linux.sh +++ b/scripts/package-linux.sh @@ -57,24 +57,25 @@ put ds4/hip ds4-server ds4 ds4-bench # the NPU kernel sets (scripts/weekly.sh, WEEKLY_NPU_KERNELS): packed beside 1bit in npu/kernels, # where it looks for them, each with the signed manifest it is verified against at load. -# Only the artifact surface is shipped: /layout.json, /layer1_*/layer1.elf, manifest and -# manifest.sig. The set directory also holds layer1.work/ (plaintext MLIR, object files, -# intermediate ELFs) because that is where it was built; copying the tree would ship the kernel -# method and feed those static ELFs to patchelf. package_checks asserts this allowlist on the tarball. +# Only the artifact surface is shipped: /manifest, /manifest.sig and //layer1.elf +# (the layout record rides inside the signed manifest; a readable layout.json must not ship). The +# set directory also holds layer1.work/ (plaintext MLIR, object files, intermediate ELFs) because +# that is where it was built; copying the tree would ship the kernel method and feed those static +# ELFs to patchelf. package_checks asserts this allowlist on the tarball. if [ -d "$build/npu/kernels" ]; then mkdir -p "$stage/npu/kernels" for set in "$build"/npu/kernels/*/; do setname=$(basename "$set") # not $name: that is the package name tar uses below - [ -f "$set/layout.json" ] || continue + [ -f "$set/manifest" ] || continue mkdir -p "$stage/npu/kernels/$setname" - cp -a "$set/layout.json" "$set/manifest" "$set/manifest.sig" "$stage/npu/kernels/$setname/" - for layer in "$set"/layer1_*/; do + cp -a "$set/manifest" "$set/manifest.sig" "$stage/npu/kernels/$setname/" + for layer in "$set"/*/; do [ -f "$layer/layer1.elf" ] || continue mkdir -p "$stage/npu/kernels/$setname/$(basename "$layer")" cp -a "$layer/layer1.elf" "$stage/npu/kernels/$setname/$(basename "$layer")/" done done - echo " npu/kernels: $(ls "$stage/npu/kernels" | wc -l) sets, $(find "$stage/npu/kernels" -name layer1.elf | wc -l) kernels, each with its signed manifest" + echo " npu/kernels: $(ls "$stage/npu/kernels" | wc -l) signed sets, $(find "$stage/npu/kernels" -name layer1.elf | wc -l) kernels" fi # The NPU lane's GGUF path runs public helper scripts (the repack, its declared-scale guard and the diff --git a/scripts/weekly.sh b/scripts/weekly.sh index a5266070..0e7049a0 100755 --- a/scripts/weekly.sh +++ b/scripts/weekly.sh @@ -315,7 +315,7 @@ package_checks() { # package_checks [ "$n" = 0 ] || { echo "FAIL source files in the tarball"; bad=1; } # ... and a tarball that carries the NPU kernels must carry all three, or its GGUF-on-NPU path # dies at the first GGUF with "repack script not found" (the bug this list exists to prevent) - if [ -n "$(find "$t" -path '*/npu/kernels/*/layout.json' -type f -print -quit)" ]; then + if [ -n "$(find "$t" -path '*/npu/kernels/*/manifest' -type f -print -quit)" ]; then for h in $helpers; do [ "$(find "$t" -path "*/$h" -type f | wc -l)" = 1 ] || { echo "FAIL the NPU package is missing $h"; bad=1; } done @@ -329,23 +329,23 @@ package_checks() { # package_checks [ "$n" = 0 ] || { echo "FAIL files named like the private sources in the tarball"; bad=1; } [ $bad = 0 ] && echo "ok no private source in $(basename "$1")" # The NPU kernels, when the package carries them (WEEKLY_NPU_KERNELS): each set ships - # layout.json, layer1_*/layer1.elf and a signed manifest; every shipped file is listed in that + # a signed manifest and its layers//layer1.elf files; every shipped file is listed in that # manifest and matches its sha256, so nothing unlisted (a build tree: plaintext MLIR, objects, - # intermediate ELFs) and nothing modified can ride along. The manifest signature itself is + # intermediate ELFs, a readable layout.json) and nothing modified can ride along. The manifest signature itself is # checked by the engine at load (it holds only the release public key and refuses a file the # manifest does not cover), so this row verifies the package, not the signature. local nk=0 m privsrc if [ "${WEEKLY_NPU_KERNELS:-1}" = 1 ]; then - n=$(find "$t" -path '*/npu/kernels/*/layout.json' -type f | wc -l) + n=$(find "$t" -path '*/npu/kernels/*/manifest' -type f | wc -l) [ "$n" -ge 3 ] || { echo "FAIL WEEKLY_NPU_KERNELS=1 but $(basename "$1") carries $n NPU kernel sets (want the 0.6B, 1.7B and 4B sets)"; bad=1; } fi while IFS= read -r -d '' f; do rel=${f#"$t"/} setdir=${f%/npu/kernels/*}/npu/kernels/$(echo "${rel#*/npu/kernels/}" | cut -d/ -f1) case "$(basename "$f")" in - layout.json|manifest|manifest.sig) continue ;; + manifest|manifest.sig) continue ;; esac - if [[ "$rel" =~ ^[^/]+/npu/kernels/[^/]+/(layer1_[^/]+/layer1\.elf)$ ]]; then + if [[ "$rel" =~ ^[^/]+/npu/kernels/[^/]+/([0-9a-f]+/layer1\.elf)$ ]]; then inner=${BASH_REMATCH[1]} want=$(awk -v p="$inner" '$1=="file" && $4==p {print $2}' "$t/$setdir/manifest" 2>/dev/null) if [ -z "$want" ]; then @@ -354,11 +354,11 @@ package_checks() { # package_checks echo "FAIL $rel does not match the sha256 in its set's manifest"; nk=$((nk + 1)) fi else - echo "FAIL unexpected file under npu/kernels (a set ships layout.json + layer1_*/layer1.elf + manifest + manifest.sig): $rel"; nk=$((nk + 1)) + echo "FAIL unexpected file under npu/kernels (a set ships manifest + manifest.sig + /layer1.elf only): $rel"; nk=$((nk + 1)) fi done < <(find "$t" -path '*/npu/kernels/*' -type f -print0) - for lj in $(find "$t" -path '*/npu/kernels/*/layout.json' -type f 2>/dev/null); do - set=$(dirname "$lj") + for mf in $(find "$t" -path '*/npu/kernels/*/manifest' -type f 2>/dev/null); do + set=$(dirname "$mf") [ -s "$set/manifest" ] || { echo "FAIL $set has no signed manifest"; nk=$((nk + 1)); } [ "$(stat -c%s "$set/manifest.sig" 2>/dev/null || echo 0)" = 64 ] || { echo "FAIL $set/manifest.sig is not a 64-byte Ed25519 signature"; nk=$((nk + 1)); } From f77466752b9898e13ffe571f4ec51c19aaa255c8 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 25 Oct 2026 15:07:18 -0300 Subject: [PATCH 3/3] review: keep the scrubbed pin names, derive the manifest path, wrap a comment Rebased onto main (0d0c846) and reviewed against the current packaging, image and release check (#556, #563, #569): - docs/releases.md conflicted on the pins table and the NPU paragraph: the branch put the private repositories' names back into a public file ("gpu-kernels (private, HIP)", "npu-kernels (private, signed)", "npu-runtime (private)"), which main had scrubbed. Resolved keeping main's wording ("private kernel add-on (HIP)", "private NPU kernel set (signed)", "private NPU runtime") and the new shape. - scripts/weekly.sh package_checks: the set's manifest is now derived from `rel` once, instead of rebuilding the path with `${f%/npu/kernels/*}` plus `cut` (which can leave a double slash). Checked that the derived path resolves for /npu/kernels///layer1.elf. - os/mini/init.sh: comment wrapped. Not marked ready: the package stage stages nothing against the pins in this tree (see the PR comment). Co-Authored-By: Claude Fable 5.1 --- os/mini/init.sh | 4 ++-- scripts/weekly.sh | 6 ++++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/os/mini/init.sh b/os/mini/init.sh index 6460d590..4a6eeff6 100644 --- a/os/mini/init.sh +++ b/os/mini/init.sh @@ -94,8 +94,8 @@ else say "[engine] no /data/1bit-llama: '1bit serve' has no HRX llama-server on this boot" fi -# The NPU kernels (each with its signed manifest) live on the stick (they used to be inside the RAM image, which made it too big -# to load): link them to the path the engine looks for, beside itself. +# The NPU kernels (each with its signed manifest) live on the stick - inside the RAM image they made it too +# big to load - and are linked to the path the engine looks for, beside itself. if [ -d /data/npu/kernels ] && [ -d /home ]; then mkdir -p /home/bcloud/wt/npupkg-build/npu 2>/dev/null ln -sfn /data/npu/kernels /home/bcloud/wt/npupkg-build/npu/kernels 2>/dev/null diff --git a/scripts/weekly.sh b/scripts/weekly.sh index 0e7049a0..0d2a69cb 100755 --- a/scripts/weekly.sh +++ b/scripts/weekly.sh @@ -341,13 +341,15 @@ package_checks() { # package_checks fi while IFS= read -r -d '' f; do rel=${f#"$t"/} - setdir=${f%/npu/kernels/*}/npu/kernels/$(echo "${rel#*/npu/kernels/}" | cut -d/ -f1) case "$(basename "$f")" in manifest|manifest.sig) continue ;; esac + # /npu/kernels///layer1.elf -> the set's manifest, without rebuilding the path by hand + manifest=$t/${rel%/npu/kernels/*}/npu/kernels/${rel#*/npu/kernels/} + manifest=${manifest%/*/*}/manifest if [[ "$rel" =~ ^[^/]+/npu/kernels/[^/]+/([0-9a-f]+/layer1\.elf)$ ]]; then inner=${BASH_REMATCH[1]} - want=$(awk -v p="$inner" '$1=="file" && $4==p {print $2}' "$t/$setdir/manifest" 2>/dev/null) + want=$(awk -v p="$inner" '$1=="file" && $4==p {print $2}' "$manifest" 2>/dev/null) if [ -z "$want" ]; then echo "FAIL $rel is not listed in its set's signed manifest"; nk=$((nk + 1)) elif [ "$want" != "$(sha256sum "$f" | cut -d' ' -f1)" ]; then