Skip to content

lzma: anchor on the dict low-16-bits, not a fixed props byte (#45) - #57

Merged
nmatt0 merged 1 commit into
masterfrom
fix/lzma-props-anchor
Oct 17, 2026
Merged

nmatt0 merged 1 commit into
masterfrom
fix/lzma-props-anchor

Conversation

@nmatt0

@nmatt0 nmatt0 commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes #45. signatures/lzma.toml anchored on the fixed prefix 5D 00 00 — props byte 0x5D only (lc3/lp0/pb2, the xz --format=lzma default). A legacy LZMA-alone stream built with any other (lc, lp, pb) was never fed to the validator and so was silently missed. The reported case: OpenWrt ath79 firmware (Archer C7 v2) carries its compressed Linux kernel at offset 0x200 with props 0x6D (lc1/lp2/pb2); binwalk detects it, moria did not. For a firmware-triage tool that is a bad failure mode — the highest-value component looks absent rather than undetected.

Fix (the issue's Option B — structural)

The validator was already props-agnostic. Its props <= 224 check is exactly the set of valid (lc<=8, lp<=4, pb<=4) byte values, and it trial-decodes to a clean end-of-stream marker before emitting — so false-positive safety never came from the anchor, only from the decode gate.

The matcher is literal (no wildcard/mask), so Option B is realized by widening the anchor to the format-invariant 00 00: the low 16 bits of the dict_size field, which are always zero because every real encoder uses a dict size that is a multiple of 64 KiB. magic_offset = 1 places the structure start on the props byte that precedes those two zero bytes. This is the same model the existing yaffs2 signature already uses with its common 0xFFFF anchor — a frequent pattern that the validator earns rather than the magic.

Signature plus two code comments only. No validator or decoder logic changed.

Verification

  • OpenWrt 0x6D case now reported verified with correct size/dict.
  • Corpus sweep over a broad firmware slice: 429 real LZMA findings vs 80 with the old anchor (+18 files), 0 crashes, 0 false positives — every new finding independently re-decodes as a valid LZMA-alone stream with a clean end-of-stream marker.
  • Full test suite green; fuzz smoke clean; ASan/UBSan clean on the lzma path, the synthetic fixture, and every file that gained findings.
  • Perf: about +10% (+0.7s) on a 273 MB worst-case image with large low-entropy regions; sub-0.15s delta on typical 8–64 MB firmware — the same tradeoff yaffs2's anchor already makes.
  • New tests/test_lzma.py section covers props 0x6D/0x5A/0x5C/0xB8 identify at an offset plus byte-exact 0x6D extract, using Python's stdlib lzma (no external tool).

signatures/lzma.toml anchored on the fixed prefix 5D 00 00, i.e. props
byte 0x5D only (lc3/lp0/pb2, the xz --format=lzma default). A LZMA-alone
stream built with any other (lc, lp, pb) was invisible: binwalk found the
OpenWrt ath79 kernel at offset 0x200 (props 0x6D = lc1/lp2/pb2) and moria
did not, so a user would conclude the kernel is absent rather than that
detection failed.

The validator is already props-agnostic: its props <= 224 check is exactly
the valid (lc<=8, lp<=4, pb<=4) byte set, and it trial-decodes to a clean
end-of-stream marker, so false-positive safety never came from the anchor.
The matcher is literal (no wildcard), so widen the anchor to the
format-invariant 00 00 (the dict_size low 16 bits, always zero because a
real dict size is a multiple of 64 KiB) at magic_offset = 1, placing the
structure start on the props byte. Same model as the yaffs2 ffff anchor:
a common anchor the validator earns.

Signature plus two comments only; no validator or decoder logic change.

Verification: OpenWrt 0x6D case now reported verified. Corpus sweep over a
375-file slice found 429 real LZMA streams vs 80 with the old anchor (+18
files), 0 crashes, 0 false positives (every new finding re-decodes as valid
LZMA-alone with a clean EOF). Full suite green, fuzz smoke clean, ASan/UBSan
clean on the lzma path, synthetic, and all gained corpus files. New
test_lzma.py section 5 covers props 0x6D/0x5A/0x5C/0xB8 identify + byte-exact
0x6D extract. Also add fuzz_lzx to .gitignore.
@nmatt0
nmatt0 merged commit c184ec5 into master Oct 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LZMA magic anchor only matches props 0x5D — misses OpenWrt kernel at 0x200 (props 0x6D)

1 participant