Skip to content

Security pass over the legacy Kyber code (WP-9, #18) - #42

Merged
Zikithezikit merged 10 commits into
dev-v2from
wp9-legacy-security
Oct 31, 2026
Merged

Zikithezikit merged 10 commits into
dev-v2from
wp9-legacy-security

Conversation

@Zikithezikit

@Zikithezikit Zikithezikit commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

No finding in this pass is High or above, so nothing here needs to hold 2.0.0. Every finding is Low or Info.

This is the single PR for #18 (WP-9). It audits the legacy Kyber code and fixes what the earlier comments on #18 decided to fix, one commit each, and logs everything in SECURITY.md. It does not close #18. Kyber output is byte-identical to v1.0.5: the golden vectors in tests/vectors/legacy-kyber-v1.0.5.json are untouched and pass on every commit.

One thing changes behavior on purpose, as approved: Kyber512/768/1024.from_keys now raises KyberError when the public key is not the copy stored inside the secret key. It is in the CHANGELOG.

Commits, in order

  1. security: put an optimization barrier around the poly_frommsg mask
  2. security: put an optimization barrier in legacy verify and cmov
  3. security: take legacy Kyber randomness from OsRng and fail closed
  4. security: wipe every secret intermediate of legacy decapsulation
  5. fix: make Kyber from_keys reject a public key that is not the one in the secret key
  6. security: wipe the Rust copies of secrets returned to Python
  7. fix: return an error from the unused DummyRng instead of panicking
  8. fix: check buffer and seed lengths in the legacy keypair and encapsulation functions
  9. chore: update rand to 0.8.6 for RUSTSEC-2026-0097
  10. docs: log the legacy Kyber security pass and the CVE review in SECURITY.md

Commits 6 to 9 are the extra checks the task asked for (panics, overflow, input lengths, advisories) that turned up real but small things. The two decided non-fixes, the Compress_11 deviation and the Keccak stack spills, are documented in SECURITY.md and not touched. I agree with both decisions and have no evidence to argue the other way.

Verified (how)

  • fmt, clippy, cargo test, maturin develop and pytest pass on every commit: scripts/check.sh run through git rebase --exec against origin/dev-v2 (result below).
  • Disassembly, rustc 1.95.0, release, one codegen unit, llvm-objdump, x86_64 and aarch64. I built non-inlined probe functions in a scratch crate outside the repo (a path dependency on the worktree) and also disassembled the library's own object. Before and after for the poly_frommsg mask: the original was vectorized with no branch on message bits; after the change the bit and mask go through the stack and the only branches are the bounds check and loop counter. verify and cmov: setg plus vector select before, setne/cset plus vector select after; inside crypto_kem_dec the flag is stored, reloaded and passed to the select with no branch on it.
  • Decaps wipes: the new code zeroes the flag byte and the buffers on both the normal path and the cleanup path; the old code had the normal path only. There is no runtime test of the wipes, because that needs unsafe or an allocator hook.
  • KyberSlash: whole-library release disassembly on x86_64, aarch64, i686 and armv7 has no div instruction and no division helper relocation. A control function with a runtime divisor shows them on all four targets (on armv7 only as a relocation to __aeabi_uidiv, so I scanned relocations too). Covers all three levels and the compress, decompress and tomsg paths, and the ML-KEM code.
  • Randomness: Rust tests with a generator that always fails (all six functions, and the secret key is wiped when the second draw fails), plus end to end with strace -e inject=getrandom:error=EIO: Kyber512/768/1024 keypair and encapsulate all raised RandomnessError, no panic.
  • New implicit-rejection test (tests/legacy_decaps_rejection.rs) derives KDF(z || H(c)) independently and checks it for damaged ciphertexts at all three levels. It also passes on the commit before this series.
  • Panics: about 1000 malformed or wrong-size inputs through the _pykyber functions in a build with overflow checks on; no PanicException, only success or a plain exception.
  • Advisories: OSV queries for the locked runtime crates (see below). cargo audit is not installed here, so this is not its output.

Believed, not verified

  • That the barriers hold on other compilers, flags and CPUs. They are best effort, and SECURITY.md says so.
  • That the multiply-high replacing the constant divisions is constant time on every core.
  • That no adversarial coefficient pattern overflows the NTT arithmetic; the fuzz covered random, all-0xFF and edge values only.
  • That the Python wrapper change to keep the RandomnessError subclass has no caller depending on a bare KyberError class; subclasses still match except KyberError.

Findings

ID: SEC-001. Title: poly_frommsg mask from a message bit. Severity: Low (pattern of CVE-2024-37880, no branch seen with this compiler). Location: src/kyber/poly.rs poly_frommsg. Class: timing. Evidence: not demonstrated; hypothesis for other compilers. Related: CVE-2024-37880. Legacy impact: no. Fix: black_box on bit and mask. Test: golden vectors; the disassembly above.

ID: SEC-002. Title: verify and cmov without a barrier. Severity: Info (no branch seen). Location: src/kyber/verify.rs. Class: timing. Evidence: hypothesis. Related: CVE-2024-37880. Legacy impact: no. Fix: black_box as in mlkem::ct. Test: golden vectors, legacy_decaps_rejection.

ID: SEC-003. Title: thread_rng and panic on RNG failure. Severity: Low. Location: src/lib.rs, kem.rs and indcpa.rs fills. Class: rng. Evidence: demonstrated by reading the code (thread_rng, fill_bytes panics on failure). Related: FIPS 203 section 3.3 for the new code; PLAN-2.0 WP-3. Legacy impact: no (seeded paths and fill order and sizes unchanged). Fix: OsRng, try_fill_bytes, RandomBytesGeneration, RandomnessError, wipe on failure. Test: tests/legacy_rng_failure.rs, python/tests/test_kyber.py TestRandomnessFailure, strace run above.

ID: SEC-004. Title: decaps intermediates, the reject flag and the PRF noise buffers not wiped on every path. Severity: Low. Location: kem.rs crypto_kem_dec*, poly.rs poly_getnoise_*. Class: memory-hygiene. Evidence: demonstrated by code and disassembly (flag byte never zeroed; noise buffers never zeroed). Related: FIPS 203 section 3.3, section 6.3. Legacy impact: no. Fix: Zeroizing wrappers. Test: legacy_decaps_rejection (behavior only); the wipe is checked in the disassembly, not at runtime.

ID: SEC-005. Title: from_keys accepts a public key and secret key from different pairs. Severity: Low (silent misuse, leaks nothing). Location: python/pykyber/_kyber.py from_keys. Class: api-misuse. Evidence: demonstrated in the comment on #18 for 768; the same code is used at 512 and 1024. Related: CVE-2026-115408 (same class, other library). Legacy impact: no change to output; a deliberate change to what from_keys accepts. Fix: compare the key with the copy inside the secret key, offsets from named constants; a wrong-size secret key keeps its size error at use. Test: TestFromKeysBinding, all three levels.

ID: SEC-006. Title: secret keys and shared secrets returned to Python as Vec, freed unwiped. Severity: Low. Location: src/lib.rs. Class: memory-hygiene. Evidence: code reading. Related: FIPS 203 section 3.3. Legacy impact: no. Fix: SecretBytes wrapper that wipes after the bytes object is built. Test: the existing Python suite; the wipe is not tested at runtime.

ID: SEC-007. Title: no length checks in crypto_kem_keypair* and crypto_kem_enc*. Severity: Info (Rust API only; the bindings size every buffer). Location: src/kyber/kem.rs. Class: panic/DoS. Evidence: code reading (slice indexing on caller lengths); I did not run the new tests against the old code. Related: none. Legacy impact: no. Fix: InvalidInput. Test: tests/legacy_length_checks.rs.

ID: SEC-008. Title: DummyRng panics. Severity: Info (never reached from Python). Location: src/kyber/api.rs. Class: panic/DoS. Evidence: code reading. Legacy impact: no. Fix: try_fill_bytes returns an error. Test: derive tests in legacy_length_checks.rs (the stand-in itself is private).

ID: SEC-009. Title: Compress_11 differs from FIPS 203 at x = 1875 and 3125. Severity: Info. Location: polyvec_compress_1024. Class: spec-deviation. Evidence: demonstrated in #28. I wrote the exact fix ((x << 11) + 1664) * 645084 >> 31, which matches exact rounding over all of Z_q, and it moved golden vectors #6 and #7 (Kyber-1024); by estimate (not measured) about 46% of Kyber-1024 ciphertexts would stop matching v1.0.5. Decision: reverted, to keep Kyber-1024 compatible with v1.0.5. Legacy impact of the fix: yes. Not fixed; the pinned test is kept. ML-KEM-1024 uses its own exact routine (poly_compress, d = 11), tested against FIPS 203 at every residue, by ACVP and Wycheproof, and by the differential test against the ml-kem crate, so ML_KEM_1024 has the correct rounding.

ID: SEC-010. Title: Keccak permutation stack spills not wiped. Severity: Low. Location: fips202.rs keccakf1600_statepermute. Class: memory-hygiene. Evidence: re-measured on this branch (rustc 1.95.0 release, x86_64): a 0x78-byte frame and 22 stack stores, not the 0xf8 and about 70 from #28; other targets not looked at. Legacy impact: none expected. Not fixed: only unsafe or a stack scrub whose size depends on the compiler can clear it, the scrub cannot be tested, and other copies of the secrets remain anyway. The reasons are written out in SECURITY.md section 15.

ID: SEC-011. Title: rand 0.8.5, RUSTSEC-2026-0097. Severity: Info (needs the log feature, which is off). Class: supply-chain. Fix: lockfile to 0.8.6.

ID: SEC-012. Title: pyo3 0.27.2, RUSTSEC-2026-0176 and RUSTSEC-2026-0177. Severity: Info (the affected APIs are not used here). Class: supply-chain. Not fixed: the fix is pyo3 0.29.0, a dependency change for the maintainer to decide.

Areas checked with no finding: panics reachable from Python, integer overflow in size handling, length checks before secret-dependent work in decapsulation and the bindings, and the ciphertext compare covering all bytes.

Out of scope, not done

  • ML_KEM_*.from_keys has the same gap as SEC-005 and is not changed.
  • from_keys does not check H(pk) or z in the secret key.
  • api::public(sk) still slices its argument and would panic on a short sk; no Python path calls it.
  • The pyo3 0.29 upgrade (SEC-012).
  • Stack and register copies from moves, and Python bytes copies of secrets, cannot be wiped without unsafe.
  • No dynamic timing test of the legacy code (that is WP-8, not merged).

Sources checked, 2026-10-30

  • https://eprint.iacr.org/2024/1049 (KyberSlash paper): read; two timing bugs from division of secret data, patched in the reference code. Result: not affected, see the disassembly.
  • https://kyberslash.cr.yp.to/: fetch failed with a transport error; the paper above was used as the primary source. Unconfirmed on the project page.
  • NVD CVE-2024-37880: the page did not render; the record was read through the Debian tracker, OSV and INCIBE mirrors (Clang through 18, fixed in pq-crystals/kyber 9b8d306). Result: mitigated.
  • CVE-2026-6330 (wolfSSL, ARM64 NEON compare of half the ciphertext): read through mirrors of the CVE record (MITRE API listing, Debian tracker), not NVD itself. Result: not affected.
  • CVE-2026-115408 (crypto11): read on OpenCVE; NVD status "Received". Result: not affected, same class as SEC-005.
  • https://blog.cr.yp.to/20260704-bugs.html: fetch failed; not read. Unconfirmed.
  • SEALSQ and LightSec 2026: press coverage only, paper not read. Unconfirmed, power and EM attacks are out of scope.
  • AFIT Airtight framework: the scholar.afit.edu page was not found; not run. Not assessed.
  • OSV (RustSec) for pyo3, zeroize, rand, rand_core, getrandom, rand_chacha at the locked versions: see SEC-011 and SEC-012.

Benchmark

Release wheels of the base commit and this branch, python/tests/test_kyber.py -k performance, 100 iterations per op, median of three alternating runs, one noisy machine. Decapsulation is the same within noise (+0.9%, 0.0%, -4.2% at 1024, 512, 768). Key generation and encapsulation are slower by up to about 0.02 ms (Kyber768 keygen 0.051 to 0.074 ms, Kyber1024 keygen 0.087 to 0.113 ms), most likely the getrandom call that replaced thread_rng. Full key exchange moved between -14% and +2%, inside the noise. The first baseline I took was from a debug build and I discarded it.

Checklist

  • Spec sections cited in code for new crypto logic (no new crypto logic; comments cite CVE-2024-37880 and FIPS 203 section 6.3 where relevant)
  • Legacy golden vectors pass unchanged
  • No secret-dependent branches, divisions, or indexing (asm checked: yes, rustc 1.95 release, x86_64 and aarch64, division scan also i686 and armv7; evidence, not proof)
  • Secret intermediates zeroized on all paths (limits above)
  • Inputs validated before secret-dependent work
  • No new runtime deps; no unsafe; no panics on untrusted input
  • No magic numbers: sizes, widths, multipliers and counts are named, documented constants
  • Tests added; fmt/clippy/test/pytest green on every commit (git rebase --exec)
  • Review fixes folded into the commits they fix (no review yet, so none needed); rebased on current dev-v2
  • CHANGELOG updated
  • Sources checked: see the list above

Generated with Claude Code

Zikithezikit and others added 9 commits October 30, 2026 18:39
The mask built from a message bit in the legacy poly_frommsg is the pattern
behind CVE-2024-37880 (Clangover), where a compiler turned it into a branch.
Route the bit and the mask through core::hint::black_box. Output is
unchanged; the golden vectors pass.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Route the accumulated difference, the flag and the select mask through
core::hint::black_box, the same idea as mlkem::ct, so the compiler cannot
prove the mask is 0 or 0xff and turn the select into a branch. The release
disassembly shows setne/cset and a vector select with no branch on the
compare result. Output is unchanged; the golden vectors pass.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Key generation and encapsulation in the Python bindings used thread_rng().
Use rand::rngs::OsRng, call try_fill_bytes in the legacy KEM functions, and
return KyberError::RandomBytesGeneration when the generator fails instead of
panicking. Wipe the half-built secret key when the second draw (z) fails and
hold the partly filled seed in Zeroizing. The Python wrappers keep the
RandomnessError subclass instead of re-wrapping it. Seeded paths and the
golden vectors are unchanged; the random-fill order and sizes are as before.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Hold m', the KDF input and the re-encrypted ciphertext of crypto_kem_dec*
in Zeroizing so they are wiped on every exit, unwinding included, and keep
the implicit-rejection flag in Zeroizing<u8>. The PRF output buffers in
poly_getnoise_* are the secret noise (the key in key generation, r in
encryption) and were never wiped; wipe them too. The release disassembly
shows the wipe of the flag and the buffers on both the normal and the
cleanup path. Add a test for the implicit-rejection output, derived from z
and H(c), for all three levels; it also passes on the commit before this
series. Output is unchanged and the golden vectors pass.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…the secret key

Kyber512/768/1024.from_keys accepted any public key and secret key of the
right sizes, so a pair from two different key pairs built a keypair whose
decapsulate silently returned a different secret than encapsulate had
produced. The round-3 secret key embeds the public key (sk_pke || pk ||
H(pk) || z), so compare the two (both are public) and raise KyberError on a
mismatch. The offset comes from named constants. A wrong-size secret key
keeps its existing size error at use. Approved by the maintainer in the
discussion on issue #18; ML_KEM_*.from_keys is not changed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The legacy bindings returned secret keys and shared secrets as Vec<u8>, which
PyO3 turns into bytes and then frees with the secret still in the allocation.
Hand them over as a SecretBytes wrapper that builds the bytes object and then
wipes the Rust buffer, and wipe the fixed-size copy from the safe API. The
Python object itself cannot be wiped. Output is unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The seeded derive path hands the KEM code a stand-in generator that is never
asked for bytes. Now that the legacy code draws randomness through
try_fill_bytes, make that method return an error, so a future change that
reaches it gets RandomBytesGeneration and not a panic. No path from Python
reaches it today.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ation functions

crypto_kem_keypair* and crypto_kem_enc* indexed caller-supplied pk, sk, ct,
ss and seed slices without checking their lengths, unlike decapsulation, so a
Rust caller with a short buffer panicked. Return InvalidInput instead. The
Python bindings already size every buffer, so no Python path reaches this.
Output is unchanged; the golden vectors pass.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Lockfile-only patch bump of an existing dependency. The advisory needs the
log feature, which pykyber does not enable, so it was not reachable here;
the legacy bindings no longer use thread_rng at all.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@Zikithezikit
Zikithezikit merged commit da445e4 into dev-v2 Oct 31, 2026
46 checks passed
@Zikithezikit
Zikithezikit deleted the wp9-legacy-security branch October 31, 2026 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant