Repository navigation
lzma: anchor on the dict low-16-bits, not a fixed props byte (#45) - #57
Merged
Merged
Conversation
signatures/lzma.toml anchored on the fixed prefix 5D 00 00, i.e. props byte 0x5D only (lc3/lp0/pb2, the xz --format=lzma default). A LZMA-alone stream built with any other (lc, lp, pb) was invisible: binwalk found the OpenWrt ath79 kernel at offset 0x200 (props 0x6D = lc1/lp2/pb2) and moria did not, so a user would conclude the kernel is absent rather than that detection failed. The validator is already props-agnostic: its props <= 224 check is exactly the valid (lc<=8, lp<=4, pb<=4) byte set, and it trial-decodes to a clean end-of-stream marker, so false-positive safety never came from the anchor. The matcher is literal (no wildcard), so widen the anchor to the format-invariant 00 00 (the dict_size low 16 bits, always zero because a real dict size is a multiple of 64 KiB) at magic_offset = 1, placing the structure start on the props byte. Same model as the yaffs2 ffff anchor: a common anchor the validator earns. Signature plus two comments only; no validator or decoder logic change. Verification: OpenWrt 0x6D case now reported verified. Corpus sweep over a 375-file slice found 429 real LZMA streams vs 80 with the old anchor (+18 files), 0 crashes, 0 false positives (every new finding re-decodes as valid LZMA-alone with a clean EOF). Full suite green, fuzz smoke clean, ASan/UBSan clean on the lzma path, synthetic, and all gained corpus files. New test_lzma.py section 5 covers props 0x6D/0x5A/0x5C/0xB8 identify + byte-exact 0x6D extract. Also add fuzz_lzx to .gitignore.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #45.
signatures/lzma.tomlanchored on the fixed prefix5D 00 00— props byte0x5Donly (lc3/lp0/pb2, thexz --format=lzmadefault). A legacy LZMA-alone stream built with any other(lc, lp, pb)was never fed to the validator and so was silently missed. The reported case: OpenWrt ath79 firmware (Archer C7 v2) carries its compressed Linux kernel at offset0x200with props0x6D(lc1/lp2/pb2); binwalk detects it, moria did not. For a firmware-triage tool that is a bad failure mode — the highest-value component looks absent rather than undetected.Fix (the issue's Option B — structural)
The validator was already props-agnostic. Its
props <= 224check is exactly the set of valid(lc<=8, lp<=4, pb<=4)byte values, and it trial-decodes to a clean end-of-stream marker before emitting — so false-positive safety never came from the anchor, only from the decode gate.The matcher is literal (no wildcard/mask), so Option B is realized by widening the anchor to the format-invariant
00 00: the low 16 bits of thedict_sizefield, which are always zero because every real encoder uses a dict size that is a multiple of 64 KiB.magic_offset = 1places the structure start on the props byte that precedes those two zero bytes. This is the same model the existingyaffs2signature already uses with its common0xFFFFanchor — a frequent pattern that the validator earns rather than the magic.Signature plus two code comments only. No validator or decoder logic changed.
Verification
0x6Dcase now reportedverifiedwith correct size/dict.yaffs2's anchor already makes.tests/test_lzma.pysection covers props0x6D/0x5A/0x5C/0xB8identify at an offset plus byte-exact0x6Dextract, using Python's stdliblzma(no external tool).