Skip to content

npu: kernel sets now ship with a signed manifest instead of sealing - #548

Closed
bong-water-water-bong wants to merge 3 commits into
mainfrom
npu/layout-v2
Closed

bong-water-water-bong wants to merge 3 commits into
mainfrom
npu/layout-v2

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Kernel sets now ship with a signed manifest instead of sealing.

Packaging, the image, the release check and the launcher follow the same shape: each set carries its manifest, every file it ships is listed in it and must match its sha256, and anything unlisted under npu/kernels fails the check. Nothing else about the NPU lane changes.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Manifest Verification Logic

The manifest verification logic in mkimage.sh for NPU kernel sets could be improved to avoid potential issues with file paths containing special characters or spaces. Specifically, the use of awk and sha256sum in a loop without proper quoting or escaping may lead to incorrect behavior if filenames contain spaces or other shell metacharacters.

    { echo "npu/kernels: $(basename "$set") has no signed manifest"; bad=1; continue; }
for elf in "$set"/layer1_*/layer1.elf; do
    [ -f "$elf" ] || continue
    rel=${elf#"$set"}
    want=$(awk -v p="$rel" '$1=="file" && $4==p {print $2}' "$set/manifest")
    [ -n "$want" ] && [ "$want" = "$(sha256sum "$elf" | cut -d' ' -f1)" ] ||
        { echo "npu/kernels: $(basename "$set")/$rel does not match its manifest"; bad=1; }
Manifest Verification Logic

Similar to mkimage.sh, the manifest verification logic in release-check.sh could be vulnerable to issues with filenames containing special characters or spaces. The awk and sha256sum commands in the verification loop need to be robust against such cases to ensure correct file validation.

want=$(awk -v p="$rel" '$1=="file" && $4==p {print $2}' "$set/manifest" 2>/dev/null)
if [ -z "$want" ]; then BAD=$((BAD + 1)); say "      not in the manifest: $f"
elif [ "$want" != "$(sha256sum "$f" 2>/dev/null | cut -d' ' -f1)" ]; then
    BAD=$((BAD + 1)); say "      does not match the manifest: $f"; fi
Manifest Verification Logic

The manifest verification logic in package_checks() within scripts/weekly.sh could be susceptible to issues with filenames containing special characters or spaces. The use of awk and sha256sum without proper quoting or escaping may lead to incorrect validation of kernel files.

want=$(awk -v p="$inner" '$1=="file" && $4==p {print $2}' "$t/$setdir/manifest" 2>/dev/null)
if [ -z "$want" ]; then
    echo "FAIL $rel is not listed in its set's signed manifest"; nk=$((nk + 1))
elif [ "$want" != "$(sha256sum "$f" | cut -d' ' -f1)" ]; then
    echo "FAIL $rel does not match the sha256 in its set's manifest"; nk=$((nk + 1))

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Follow-up commit: the readable layout sidecar no longer ships (the layout record rides inside the signed manifest) and kernel directories are opaque ids, so packaging, the image and the checks stop requiring layout.json / layer1_<name> names. A set that still carries layout.json, or any file the manifest does not list, fails the check.

bong-water-water-bong and others added 3 commits October 25, 2026 15:06
The kernel sets carry a signed manifest: every file in a set is listed with its
sha256 and the engine, which holds only the release public key, refuses a file the
manifest does not cover. Packaging, the image, the release check and the launcher
follow the same shape: a set without its manifest, a file that does not match it,
and anything unlisted under npu/kernels is a failure; nothing else about the NPU
lane changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Follow-up to the signed-manifest change: the readable layout sidecar no longer ships (the layout
recordrides inside the signed manifest) and the kernel directories are opaque ids, so packaging, the
image, the release check and the package checks stop looking for layout.json and stop requiring
layer1_<name> directory names. A set that still carries layout.json, or a file the manifest does
not list, is a failure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… comment

Rebased onto main (0d0c846) and reviewed against the current packaging, image and release check
(#556, #563, #569):

- docs/releases.md conflicted on the pins table and the NPU paragraph: the branch put the private
  repositories' names back into a public file ("gpu-kernels (private, HIP)", "npu-kernels (private,
  signed)", "npu-runtime (private)"), which main had scrubbed. Resolved keeping main's wording
  ("private kernel add-on (HIP)", "private NPU kernel set (signed)", "private NPU runtime") and the
  new shape.
- scripts/weekly.sh package_checks: the set's manifest is now derived from `rel` once, instead of
  rebuilding the path with `${f%/npu/kernels/*}` plus `cut` (which can leave a double slash). Checked
  that the derived path resolves for <pkg>/npu/kernels/<set>/<id>/layer1.elf.
- os/mini/init.sh: comment wrapped.

Not marked ready: the package stage stages nothing against the pins in this tree (see the PR comment).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Review: rebased onto main (0d0c846), reviewed against #556/#563/#569 — but NOT ready, and not just for style reasons. Two blockers, one of them mine to fix and one that is not.

1. It stages no kernels with the pins in this tree. config/npu-kernels.config is untouched (17ed5be7a28a), and that pin's release flow produces sealed sets (layout.json + 1BPKCRYP layer1.elf + layer1.work/). The new packer looks for <set>/manifest and continues when it is absent, so the whole NPU lane silently drops out:

$ scripts/package-linux.sh <build> /tmp/pkg548 rc-test
  hrx/llama/bin: 27 files
  npu/kernels: 0 signed sets, 0 kernels

package_checks then fails its "carries 3 NPU kernel sets" row, os/mini/mkimage.sh exits with "carries no layer1.elf", and 1bit.check's new npu-kernels row has nothing to verify. This PR therefore needs a private-side release flow that emits manifest + manifest.sig + plaintext <id>/layer1.elf and its pin bump, in the same merge window.

2. Nothing in the tree implements what the docs now promise. A signed manifest is verified by the loader, and the loader is the private NPU runtime: the public tree has zero references to ONEBIT_KMANIFEST_REQUIRED (or the old ONEBIT_KCRYPT_REQUIRE_SEALED), and the built 1bit contains neither string. So today scripts/lemond-onebit.sh's export ONEBIT_KMANIFEST_REQUIRED=1 is inert, package_checks would bless a package with no enforcement behind it, and config/npu-runtime.config (ce9fa59adf95) is not bumped here either. The private runtime must land with this, or the release ships a launcher flag nothing reads.

And the question that needs the owner, not me: sealing was confidentiality — "a lifted kernel file is ciphertext and does not run". A signature is authenticity/integrity: the same layer1.elf becomes a readable AIE ELF in a public tarball, disassemblable by anyone. This PR deletes the ciphertext sentence from docs/releases.md and docs/npu.md, so it is a deliberate change of the release's IP posture, and it collides with the closed-source rule this lane runs under ("no kernel content in any public file; the packages are the deliverable, not their sources"). If the intent is signing and sealing (sign the manifest, keep the kernels encrypted), that is a different, smaller change and I can take it.

What I changed on the branch (rebased, conflicts resolved, f774667):

  • docs/releases.md: the branch was putting the private repositories' names back into a public file (gpu-kernels (private, HIP), npu-kernels (private, signed), npu-runtime (private)); main had scrubbed them. Resolved to main's naming with the new shape.
  • scripts/weekly.sh package_checks: the set's manifest is derived from rel once instead of rebuilding it with ${f%/npu/kernels/*} + cut (double-slash risk); verified the derived path resolves.
  • os/mini/init.sh: comment wrapped.

Smaller review notes: the manifest line format (awk '$1=="file" && $4==path') is now parsed in three places (package_checks, mkimage.sh, release-check.sh) with no public description of the format — one helper or a documented column layout would be better; dropping the "every ELF is x86-64" rule for */npu/kernels/* is correct (AIE kernels are not host ELFs) and worth keeping; release-check.sh's unlisted-file loop greps the manifest per file, fine at 30 kernels.

I did not run 1bit.check / vmtest: there is no package carrying kernels to check until the private side exists, so those runs would only prove the packer drops the lane — which the output above already shows.

@github-actions

Copy link
Copy Markdown

Preparing review...

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Closing as not planned, on the owner's rule: the NPU kernels stay closed source, and a plaintext layer1.elf in a
public tarball is not acceptable. A signed manifest proves authenticity and integrity but not confidentiality — it would
have shipped the kernel method readable, which is exactly what sealing prevents.

The release format stays sealed sets: <set>/layout.json + <set>/layer1_*/layer1.elf with the 1BPKCRYP magic,
ONEBIT_KCRYPT_REQUIRE_SEALED=1 in scripts/lemond-onebit.sh, and package_checks refusing anything under
npu/kernels that is not a sealed artifact — i.e. main as it stands.

Kept as a possible follow-up (new issue, no deadline): signing the manifest on top of the sealed sets, which adds
authenticity without giving up confidentiality. That is a smaller change than this one and does not touch what ships.

For the record, the review findings on this branch: it staged nothing with the pins in the tree
(npu/kernels: 0 signed sets, 0 kernels, because config/npu-kernels.config still emits sealed sets), and nothing in the
public tree implements the verification it documents — zero references to ONEBIT_KMANIFEST_REQUIRED, and the built
1bit contains neither that string nor the old one, so the private NPU runtime would have had to change in lockstep.
Branch deleted; the rebase/conflict work is not worth keeping for a superseded shape.

@bong-water-water-bong
bong-water-water-bong deleted the npu/layout-v2 branch October 25, 2026 18:32
bong-water-water-bong added a commit that referenced this pull request Oct 27, 2026
)

Sealing is confidentiality only. A set re-sealed under another key, or a
different set that also decrypts, is indistinguishable from ours by the
1BPKCRYP check alone, and #548 was closed as not planned for replacing
sealing instead of adding to it. Authenticity goes on top: every set now
carries manifest.json - each file it ships, with the sha256 of its
ciphertext, plus the set's own name - and manifest.sig, Ed25519 over the
manifest's exact bytes by the release signing key.

  tools/seal_manifest.py    the public half: keygen / write <set> --key /
                            check (digests, no key) / verify (with the
                            release public key) / --self-test. Python 3
                            stdlib only, its own RFC 8032 Ed25519, because
                            os/mini/release-check.sh has to make the same
                            judgement on a busybox stick with no crypto
                            module and no openssl.
  tests/seal_manifest_test.sh + ctest seal_manifest   throwaway key and
                            set in a temp dir: pass on intact, fail on a
                            missing manifest, an unlisted file, a flipped
                            ciphertext byte, and a manifest signed by
                            another key (plus a set moved onto another
                            set's name).
  scripts/package-linux.sh  a sealed set without manifest.json and
                            manifest.sig is not packed at all; the staged
                            copy is checked (verified when ONEBIT_SEAL_PUB
                            names the release public key).
  scripts/weekly.sh         package_checks allows the two new files per set
                            and requires them, checking the tarball's own
                            copy so a manifest that does not list exactly
                            the packed files fails the package.
  os/mini/release-check.sh  the sealed row now also requires every file
                            under /data/npu/kernels to be listed in its
                            set's manifest (seal_manifest_problems, driven
                            by the script's own --self-test), and a new
                            sealed-sig row verifies manifest.sig with the
                            release public key, saying so when the image
                            has no key instead of passing unchecked.
  scripts/lemond-onebit.sh  keeps ONEBIT_KCRYPT_REQUIRE_SEALED=1 and names,
                            without setting, ONEBIT_KCRYPT_REQUIRE_MANIFEST=1:
                            the loader-side switch the private NPU runtime
                            must read, which lands with its pin bump.
  docs/npu.md, docs/releases.md  what a set ships and who checks it.

The loader half, the release key and any real re-seal stay private; no
switch is exported that nothing reads yet (#548).

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant