Skip to content

fix: detect and deobfuscate <hash>.NN RAR volumes (#87) - #89

Merged
thedancingdeveloper merged 2 commits into
mainfrom
test/issue-87-obfuscated-rar-repro
Aug 12, 2026
Merged

fix: detect and deobfuscate <hash>.NN RAR volumes (#87)#89
thedancingdeveloper merged 2 commits into
mainfrom
test/issue-87-obfuscated-rar-repro

Conversation

@thedancingdeveloper

@thedancingdeveloper thedancingdeveloper commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Obfuscated multi-volume RAR sets named <32 hex digits>.NN (e.g.
cfd4be79c0fb01d429c52a9f8551ee79.45) were never extracted, and the job still
reported Completed with the raw volumes left in the output directory.

Two commits: the reproduction suite first (every pos_* test red), then the
fix that turns them green. Reviewing them separately shows exactly which
assertion each change buys.

Cause 1 — PAR2-guided deobfuscation was gated behind damage

rename_to_par2_names recovers original filenames by matching each file's
first-16K MD5 against the PAR2 metadata — exactly what an obfuscated set needs.
It was called only from inside the verify branch, and verification is skipped
when articles_failed == 0 because the files are already CRC-verified. So a
damaged download self-healed while a clean one kept its meaningless
names.

The rename now runs whenever the PAR2 index parses, independent of the verify
decision. It is a cheap MD5 pass over file heads, not a full verification. This
alone fixes the reported case for any post shipping a PAR2 set.

Cause 2 — detection matched extensions only

find_archives, find_cleanup_files and has_usable_output all keyed off
filename patterns, so a bare numeric extension was invisible:

  • no archive found → Extract reported Skipped: "No archives found", not a failure
  • volumes were not cleanup candidates → left on disk
  • has_usable_output — defined as "anything not recognised as junk is
    payload"
    — classified them as usable, which let the guard at
    queue_manager.rs:1977 pass and move_to_history stamp Completed

Detection now reads the RAR4/RAR5 signature. Extension-widening alone would
have been wrong
— it misclassifies split 7z volumes and any file that merely
ends in digits — so content is the deciding evidence:

Added
has_rar_signature(path) public
parse_rar_volume_at(path) public; name-based parse plus a bare-numeric fallback gated on the signature
find_rar_files groups obfuscated volumes by stem, hands unrar the lowest-numbered one
is_cleanup_candidate_at path-aware cleanup/payload decision

parse_rar_volume keeps its name-only contract and still rejects bare
numerics. That is deliberate: the direct-unpack path in queue_manager.rs:1485
calls it during download, and a name-only parser that accepted logfile.01
would feed non-archives to the unpacker. That path is therefore unchanged.

Tests

70 tests, 0 ignored. Three groups, so the fix is pinned from both sides:

Prefix Purpose
neg_* Conventional naming (.rar/.r00, .partNNN.rar, .7z.001, real payload) — guards against the fix breaking ordinary sets
pos_* The obfuscated set — each failed before the fix, one per broken layer
guard_* False-positive guards — a text file named Concert.Recording.1987, a .7z.001 split set, a logfile.01. These are why detection sniffs signatures

The PAR2 gate test is a clean A/B: identical files, identical PAR2, only
articles_failed differs.
Its negative control passed before the fix too,
proving the rename machinery was never the problem — only its placement.

New fixture builder

Testing the gate needs a parseable .par2, and the repo has no PAR2 fixtures
while CI has no par2create; rust-par2 exposes no creation API.
tests/support/par2_fixture.rs synthesises spec-correct Main and
FileDesc packets. Packet MD5s must be exact — the parser silently skips
mismatched packets, so a wrong hash would yield an empty file set rather than a
loud failure. The suite therefore self-tests the fixture and verifies its 16K
hashes against compute_hash_16k.

New dev-deps: md-5, tokio (macros, rt-multi-thread). Cargo.lock updated.

Verification

cargo test -p nzb-postproc                              # 70 passed, 0 ignored
cargo test --workspace                                  # green
cargo fmt --all && cargo clippy --workspace --all-targets  # clean

Deliberately out of scope

  • has_usable_output is still a denylist ("unrecognised ⇒ payload").
    Inverting it to an allowlist would make future unknown obfuscation schemes
    fail loudly rather than silently — a real improvement — but it changes
    completion semantics for legitimately extensionless payload and deserves its
    own change with its own tests.
  • Direct unpack doesn't handle obfuscated sets. It skips them and
    post-processing handles them correctly, so this is a missed optimisation, not
    a correctness gap. Adopting parse_rar_volume_at there is a follow-up.

Closes #87

@thedancingdeveloper thedancingdeveloper changed the title test: reproduce obfuscated <hash>.NN RAR extraction failure (#87) fix: detect and deobfuscate <hash>.NN RAR volumes (#87) Aug 11, 2026
thedancingdeveloper and others added 2 commits August 12, 2026 11:45
Adds a paired reproduction suite for issue #87, where an obfuscated
multi-volume RAR set named `<32 hex digits>.NN` is never extracted and
the job still reports Completed.

Tests are grouped by role:

  neg_*   negative controls — conventional naming works. Green today;
          they isolate the defect to the filename and act as the
          regression guard for any fix.
  pos_*   positive reproductions — the obfuscated set. These assert the
          desired behaviour and therefore FAIL today, so they are
          #[ignore]d to keep CI green. The fix removes the attributes.
  guard_* false-positive guards — a naive "any numeric extension is a
          RAR volume" fix breaks these. They must stay green.

obfuscated_rar_volumes.rs covers the detection layer: find_archives
returns nothing (Extract reports "No archives found" and skips),
has_usable_output classifies raw volumes as payload (which is what lets
move_to_history stamp Completed), the volumes are never cleanup
candidates, and parse_rar_volume rejects bare numeric extensions.

par2_deobfuscation_gate.rs covers the pipeline gate. rename_to_par2_names
already recovers original filenames via first-16K MD5 matching, but it
sits inside the verify branch, which is skipped when articles_failed==0.
The suite is an A/B on identical inputs where only articles_failed
differs: the damaged download deobfuscates correctly, the healthy one
keeps its <hash>.NN names.

Testing that required a parseable PAR2 index, and the repo has no .par2
fixtures while CI has no par2create; rust-par2 exposes no creation API.
tests/support/par2_fixture.rs synthesises spec-correct Main and FileDesc
packets to fill that gap. Packet MD5s must be exact — the parser silently
skips mismatched packets — so the suite self-tests the fixture and
verifies its 16K hashes against compute_hash_16k.

No production code changes.

  cargo test -p nzb-postproc              # 63 passed, 6 ignored
  cargo test -p nzb-postproc -- --ignored # 6 failed (the reproductions)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Obfuscated multi-volume RAR sets named `<32 hex digits>.NN` were never
extracted, and the job still reported Completed with raw volumes left in
the output directory. Two independent causes, both fixed here; the tests
added in the previous commit are un-ignored and now pass.

1. PAR2-guided deobfuscation was gated behind damage.

`rename_to_par2_names` recovers original filenames by matching each
file's first-16K MD5 against PAR2 metadata — exactly what an obfuscated
set needs. It was called only from inside the verify branch, and
verification is skipped when articles_failed == 0 because the files are
already CRC-verified. So a *damaged* download self-healed while a clean
one kept its meaningless names. The rename now runs whenever the PAR2
index parses, independent of the verify decision. It is a cheap MD5 pass
over file heads, not a full verification.

This alone fixes the reported case for any post shipping a PAR2 set.

2. Detection matched extensions only.

find_archives, find_cleanup_files and has_usable_output all keyed off
filename patterns, so a bare numeric extension was invisible: no archive
was found (Extract reported "No archives found" and skipped), the
volumes were not cleanup candidates, and has_usable_output — defined as
"anything not recognised as junk is payload" — classified them as usable
output, which let move_to_history stamp Completed.

Detection now reads the RAR4/RAR5 signature. Extension-widening alone
would have been wrong: it misclassifies split 7z volumes and any file
that merely ends in digits, so content is the deciding evidence.

  - has_rar_signature(path)      new, public
  - parse_rar_volume_at(path)    new, public; name-based parse plus a
                                 bare-numeric fallback gated on the
                                 signature
  - find_rar_files               groups obfuscated volumes by stem and
                                 hands unrar the lowest-numbered one
  - is_cleanup_candidate_at      path-aware cleanup/payload decision

parse_rar_volume keeps its name-only contract and still rejects bare
numerics, so the direct-unpack path in queue_manager is unchanged and
cannot be fed a non-archive. Deliberately not adopted there yet: direct
unpack simply skips obfuscated sets and post-processing handles them.

Not addressed here: has_usable_output remains a denylist ("unrecognised
=> payload"). Inverting it to an allowlist would make future unknown
obfuscation schemes fail loudly rather than silently, but it changes
completion semantics for legitimately extensionless payload and deserves
its own change.

  cargo test -p nzb-postproc  # 70 passed, 0 ignored
  cargo test --workspace      # green
  cargo fmt --all / cargo clippy --workspace --all-targets  # clean

Closes #87

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@thedancingdeveloper
thedancingdeveloper force-pushed the test/issue-87-obfuscated-rar-repro branch from 4318cab to 8cb5cc9 Compare August 12, 2026 01:45
@thedancingdeveloper
thedancingdeveloper enabled auto-merge (squash) August 12, 2026 01:45
@thedancingdeveloper
thedancingdeveloper merged commit fd41944 into main Aug 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Obfuscated multi-volume RAR (<hash>.NN) is never extracted and the job still reports Completed

1 participant