fix: detect and deobfuscate <hash>.NN RAR volumes (#87) - #89
Merged
Conversation
Adds a paired reproduction suite for issue #87, where an obfuscated multi-volume RAR set named `<32 hex digits>.NN` is never extracted and the job still reports Completed. Tests are grouped by role: neg_* negative controls — conventional naming works. Green today; they isolate the defect to the filename and act as the regression guard for any fix. pos_* positive reproductions — the obfuscated set. These assert the desired behaviour and therefore FAIL today, so they are #[ignore]d to keep CI green. The fix removes the attributes. guard_* false-positive guards — a naive "any numeric extension is a RAR volume" fix breaks these. They must stay green. obfuscated_rar_volumes.rs covers the detection layer: find_archives returns nothing (Extract reports "No archives found" and skips), has_usable_output classifies raw volumes as payload (which is what lets move_to_history stamp Completed), the volumes are never cleanup candidates, and parse_rar_volume rejects bare numeric extensions. par2_deobfuscation_gate.rs covers the pipeline gate. rename_to_par2_names already recovers original filenames via first-16K MD5 matching, but it sits inside the verify branch, which is skipped when articles_failed==0. The suite is an A/B on identical inputs where only articles_failed differs: the damaged download deobfuscates correctly, the healthy one keeps its <hash>.NN names. Testing that required a parseable PAR2 index, and the repo has no .par2 fixtures while CI has no par2create; rust-par2 exposes no creation API. tests/support/par2_fixture.rs synthesises spec-correct Main and FileDesc packets to fill that gap. Packet MD5s must be exact — the parser silently skips mismatched packets — so the suite self-tests the fixture and verifies its 16K hashes against compute_hash_16k. No production code changes. cargo test -p nzb-postproc # 63 passed, 6 ignored cargo test -p nzb-postproc -- --ignored # 6 failed (the reproductions) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Obfuscated multi-volume RAR sets named `<32 hex digits>.NN` were never
extracted, and the job still reported Completed with raw volumes left in
the output directory. Two independent causes, both fixed here; the tests
added in the previous commit are un-ignored and now pass.
1. PAR2-guided deobfuscation was gated behind damage.
`rename_to_par2_names` recovers original filenames by matching each
file's first-16K MD5 against PAR2 metadata — exactly what an obfuscated
set needs. It was called only from inside the verify branch, and
verification is skipped when articles_failed == 0 because the files are
already CRC-verified. So a *damaged* download self-healed while a clean
one kept its meaningless names. The rename now runs whenever the PAR2
index parses, independent of the verify decision. It is a cheap MD5 pass
over file heads, not a full verification.
This alone fixes the reported case for any post shipping a PAR2 set.
2. Detection matched extensions only.
find_archives, find_cleanup_files and has_usable_output all keyed off
filename patterns, so a bare numeric extension was invisible: no archive
was found (Extract reported "No archives found" and skipped), the
volumes were not cleanup candidates, and has_usable_output — defined as
"anything not recognised as junk is payload" — classified them as usable
output, which let move_to_history stamp Completed.
Detection now reads the RAR4/RAR5 signature. Extension-widening alone
would have been wrong: it misclassifies split 7z volumes and any file
that merely ends in digits, so content is the deciding evidence.
- has_rar_signature(path) new, public
- parse_rar_volume_at(path) new, public; name-based parse plus a
bare-numeric fallback gated on the
signature
- find_rar_files groups obfuscated volumes by stem and
hands unrar the lowest-numbered one
- is_cleanup_candidate_at path-aware cleanup/payload decision
parse_rar_volume keeps its name-only contract and still rejects bare
numerics, so the direct-unpack path in queue_manager is unchanged and
cannot be fed a non-archive. Deliberately not adopted there yet: direct
unpack simply skips obfuscated sets and post-processing handles them.
Not addressed here: has_usable_output remains a denylist ("unrecognised
=> payload"). Inverting it to an allowlist would make future unknown
obfuscation schemes fail loudly rather than silently, but it changes
completion semantics for legitimately extensionless payload and deserves
its own change.
cargo test -p nzb-postproc # 70 passed, 0 ignored
cargo test --workspace # green
cargo fmt --all / cargo clippy --workspace --all-targets # clean
Closes #87
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
thedancingdeveloper
force-pushed
the
test/issue-87-obfuscated-rar-repro
branch
from
August 12, 2026 01:45
4318cab to
8cb5cc9
Compare
thedancingdeveloper
enabled auto-merge (squash)
August 12, 2026 01:45
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Obfuscated multi-volume RAR sets named
<32 hex digits>.NN(e.g.cfd4be79c0fb01d429c52a9f8551ee79.45) were never extracted, and the job stillreported Completed with the raw volumes left in the output directory.
Two commits: the reproduction suite first (every
pos_*test red), then thefix that turns them green. Reviewing them separately shows exactly which
assertion each change buys.
Cause 1 — PAR2-guided deobfuscation was gated behind damage
rename_to_par2_namesrecovers original filenames by matching each file'sfirst-16K MD5 against the PAR2 metadata — exactly what an obfuscated set needs.
It was called only from inside the verify branch, and verification is skipped
when
articles_failed == 0because the files are already CRC-verified. So adamaged download self-healed while a clean one kept its meaningless
names.
The rename now runs whenever the PAR2 index parses, independent of the verify
decision. It is a cheap MD5 pass over file heads, not a full verification. This
alone fixes the reported case for any post shipping a PAR2 set.
Cause 2 — detection matched extensions only
find_archives,find_cleanup_filesandhas_usable_outputall keyed offfilename patterns, so a bare numeric extension was invisible:
Skipped: "No archives found", not a failurehas_usable_output— defined as "anything not recognised as junk ispayload" — classified them as usable, which let the guard at
queue_manager.rs:1977pass andmove_to_historystamp CompletedDetection now reads the RAR4/RAR5 signature. Extension-widening alone would
have been wrong — it misclassifies split 7z volumes and any file that merely
ends in digits — so content is the deciding evidence:
has_rar_signature(path)parse_rar_volume_at(path)find_rar_filesis_cleanup_candidate_atparse_rar_volumekeeps its name-only contract and still rejects barenumerics. That is deliberate: the direct-unpack path in
queue_manager.rs:1485calls it during download, and a name-only parser that accepted
logfile.01would feed non-archives to the unpacker. That path is therefore unchanged.
Tests
70 tests, 0 ignored. Three groups, so the fix is pinned from both sides:
neg_*.rar/.r00,.partNNN.rar,.7z.001, real payload) — guards against the fix breaking ordinary setspos_*guard_*Concert.Recording.1987, a.7z.001split set, alogfile.01. These are why detection sniffs signaturesThe PAR2 gate test is a clean A/B: identical files, identical PAR2, only
articles_faileddiffers. Its negative control passed before the fix too,proving the rename machinery was never the problem — only its placement.
New fixture builder
Testing the gate needs a parseable
.par2, and the repo has no PAR2 fixtureswhile CI has no
par2create;rust-par2exposes no creation API.tests/support/par2_fixture.rssynthesises spec-correct Main andFileDesc packets. Packet MD5s must be exact — the parser silently skips
mismatched packets, so a wrong hash would yield an empty file set rather than a
loud failure. The suite therefore self-tests the fixture and verifies its 16K
hashes against
compute_hash_16k.New dev-deps:
md-5,tokio(macros,rt-multi-thread).Cargo.lockupdated.Verification
Deliberately out of scope
has_usable_outputis still a denylist ("unrecognised ⇒ payload").Inverting it to an allowlist would make future unknown obfuscation schemes
fail loudly rather than silently — a real improvement — but it changes
completion semantics for legitimately extensionless payload and deserves its
own change with its own tests.
post-processing handles them correctly, so this is a missed optimisation, not
a correctness gap. Adopting
parse_rar_volume_atthere is a follow-up.Closes #87