Speed up the compressor without changing a single output byte

[AlyxSharkBite]

Speeds up ImplodeEngine under one hard constraint: the output may not change by a single byte. Every optimization was gated on a 589-case corpus of compressed-output SHA-256 hashes (all dictionary sizes × both modes × payloads including block-boundary sizes and an every-length sweep from 1 to 1600) staying identical to the straight port. Decompression is untouched.

Results

Median throughput, 4 MB payloads, binary mode, x86-64 desktop (±3% run-to-run):

Payload	dict	Before	After
Repetitive text	4096	190 MB/s	310 MB/s	+63%
Repetitive text	1024	249 MB/s	441 MB/s	+77%
8-symbol alphabet (worst case)	4096	7.0 MB/s	10.7 MB/s	+53%
8-symbol alphabet	1024	18.1 MB/s	20.4 MB/s	+13%
Mixed runs + noise	4096	105 MB/s	109 MB/s	+4%
Random bytes	4096	46 MB/s	47 MB/s	+2%

What changed

1. Word-at-a-time match measurement. The byte-at-a-time compare loops in FindRepetition and ExtendRepetition are replaced by a helper that XORs two 64-bit loads — zero only when all eight bytes agree — and locates the first mismatch with the trailing-zero count. On x86-64 this JITs to unaligned 64-bit loads plus TZCNT.

This rewrite is only safe because of a property worth writing down: the original counts the second byte of a candidate as matching without comparing it. That is sound, not a bug — within one hash chain the pair hash b0*4 + b1*5 is equal, so equal first bytes force equal second bytes in exact integer arithmetic. The distance bookkeeping (the original computes it from advanced pointers, then decrements once in the max-length case) algebraically collapses to input − matchStart − 1 in every path, which is what the new code stores directly.

Intrinsics that lost: a 32-byte AVX2 compare loop was also measured, per the request — it was slower. Matches are capped at 0x204 bytes and typically far shorter, so the wider vectors cost more in setup than they saved. The 64-bit/TZCNT form won and is what ships.

2. Branchless prechecks (the worst-case win). The candidate scan tested two bytes with && — two hard-to-predict branches on small-alphabet data, where ~half the candidates match on the first byte. Folded into one branchless test, ((a ^ b) | (c ^ d)) == 0, the surviving branch is rarely taken and predicts well: +54% on the degenerate case by itself.

3. Unchecked reads in the chain walks, via Unsafe/MemoryMarshal, with the safety argument documented at each site: every chain is terminated by the entry for the input position itself (so the index cannot leave the table), and every stored offset sits at least MaxRepetitionLength + 4 bytes inside the padded work buffer. No unsafe blocks; AllowUnsafeBlocks stays off.

4. Iterative bit writer. OutputBits recursion unrolled into a loop with the cursor in locals, preserving the original's deliberately-unmasked spill byte (the two-round overlap where the same bits are OR-ed over themselves).

Verification

• 589-hash corpus: identical before and after (re-checked after every individual change)
• dotnet test: 966 passed
• Independent 784-check harness: passed
• dotnet build -warnaserror: clean

README gains a Performance section, and the "no unsafe code" line now states precisely what is and isn't used.