Compute the FairPlay SAP exchange instead of emulating Apple's binary - #1
Compute the FairPlay SAP exchange instead of emulating Apple's binary#1objevovat wants to merge 9 commits into
Conversation
fpsap-helper computed the SAP response by running a 2,711-line ARM64
interpreter over a 165,575-byte snapshot of Apple's signed FairPlay
binary, committed to the repository as a Go byte array. This replaces
both with a native computation of the same function.
The exchange decomposes into three stages, all now implemented directly:
white-box AES over the payload produces a 128-byte GP buffer; nine
MD5-family blocks compress that buffer into a 20-byte digest; the
white-box MD5 network turns the digest into the response.
fpemu's exported API is unchanged, so main.go and the Rust caller in
crates/rotten-crypto need no modification.
Verification, all run before the interpreter was deleted:
- 142/142 archived golden vectors
- 8/8 vectors published by nored/airfry and omarroth/doubletake, which
compute this exchange by emulating Apple's binary and so are
independent of the reverse engineering behind this code
- 227/227 differential comparisons against the interpreter itself,
including 200 random payloads and 20 full 142-byte m2 messages
The helper had no tests; it now has coverage at the exported seam, and
CI builds and tests it, which it did not do before.
Two things this does not claim. The directory gets bigger, not smaller:
roughly 17 MB of generated straight-line code against the ~1.07 MB it
replaces. It is produced by partial evaluation of an execution trace and
is not a compact algorithm -- reducing it to closed form is unfinished
work. And Apple-derived *data* necessarily remains, white-box crypto
being what it is; what is gone is the execution of Apple's instructions.
The replacement is independently reverse-engineered, not derived from
doubletake, and is offered under the Blue Oak Model License 1.0.0. The
directory's GPL-3.0 LICENSE is left untouched -- see PROVENANCE.md for
why that may now be worth revisiting.
Follow-up to the commit that replaced the interpreter. Same outputs, same
API, three changes to how the generated code is produced:
* Apple's dyld-shared-cache addresses are gone -- as folded constants,
as data spilled into scratch memory, and as pointers inside the baked
tables. The earlier PROVENANCE.md claimed 79 of them necessarily
remained "because the hashed buffer really does contain stack memory
holding them". That was wrong: the shared cache is slid on every boot,
so a response depending on an address value would change after a
reboot and the handshake would fail. Confirmed by sliding the whole
cache window for identical output, against a control that inverts
baked bytes and does change it.
* The emitted shapes are re-encoded: byte-at-a-time memory access
becomes word operations, and the 32-bit arithmetic -- which arrived as
nested casts at ~200,000 sites -- is named via a small rt_gen.go.
* Register writes whose results are never read are eliminated, 25% of
statements. The emitter reproduced every intermediate the hardware
computed, including a division by a constant whose quotient is
overwritten on the very next line.
tools/fpsap-helper/internal/ goes from 17 MB to 8.1 MB. It is still large
and still machine-generated straight-line code rather than a compact
algorithm; the trade on offer is unchanged, just cheaper.
Adds internal/layerguard, which pins a digest over all eight generated
functions across 512 payloads to the value recorded before any of this
work, and fails if a shared-cache address ever returns.
Verified: 142/142 golden vectors, 8/8 independent vectors from airfry and
doubletake, the layer digest unchanged, and 287/287 differential
comparisons against the interpreter this replaces -- 250 random payloads
and 30 full m2 messages, driven end to end through both helper binaries.
The research codebase behind this work is private, so the link 404s for anyone reading the PR. Everything needed to verify this directory is in the directory itself.
The bridge was machine-generated straight-line code, produced by partial evaluation of an execution trace. It has since been reduced to a closed form, so the layer packages are gone and internal/fpsapcore computes the same thing in 697 lines of ordinary byte arithmetic. This removes the only real argument against this PR. Earlier revisions grew tools/fpsap-helper from ~1.07 MB to 8.1 MB and asked you to accept that as the price of dropping Apple's snapshot. That trade no longer exists: the directory is now 300 KB, smaller than what it replaces, and the built helper binary drops from 2.96 MB to 2.55 MB. Outputs are unchanged. Verified against the interpreter this replaces on 347 inputs -- 300 random payloads, the edge cases, and 40 full 142-byte m2 messages, driven end to end through both helper binaries -- with zero mismatches, plus 142/142 golden vectors and the 8 independent vectors from airfry and doubletake. PROVENANCE.md is updated: the size section no longer argues a tradeoff that is gone, and the layerguard references are replaced now that the bridge carries no table data at all.
The previous revision said this contribution derives from doubletake "not at all". That was true of the 8.1 MB generated bridge it described, and stopped being true when the closed form replaced it. internal/fpsapcore is taken from omarroth/doubletake at 8ccea5f (LGPL-3.0): fairplay_sap.go and fairplay_md5.go from its internal/airplay package, descriptor.go its descriptor function and constants. Carries NOTICE.md alongside it. This also reverses an argument made here earlier. I suggested that since no doubletake-derived code remained, the GPL-3.0 licence on this directory and the subprocess isolation might be retired. That no longer holds -- keep both. fpbridge and fairplayhash are unaffected and remain independent work under Blue Oak 1.0.0.
fpsapcore already checks its fast descriptor and its collapsed scramble against reference implementations. The ring loop had no equivalent, and it is the part most likely to be got wrong: the 840-step index derivation is irregular for i < 155 purely because i is uint32, so the subtraction wraps through 2^32 before the modulo and (i-155)%210 is (i-109) mod 210 rather than (i+55) mod 210, 2^32 mod 210 being 46. Two optimisations rest on that and neither was covered in a way that would localise a fault -- the first 155 steps read tabulated indices, and the remaining 685 drop the tables for four counters walking contiguous spans. Both are now checked against the naive derivation. Includes the control the repository's own history argues for: assert the wrap genuinely differs from the "obvious" (i+55)%210 over those first 155 steps, so the tables cannot pass by being self-consistently wrong. It differs on all 155.
ae54225 to
6699e71
Compare
Two correctness fixes found after this PR was opened, plus the code sync that carries them. An m2 selects a FairPlay message mode in byte 13, and the mode picks both the CBC IV and the AES round keys for the message body -- so the same 128-byte challenge produces four entirely different responses under modes 0..3. main.go read bytes 14:142 straight out of the m2 and never looked at byte 13, so a mode-0 challenge got a mode-3 answer back: wrong bytes returned confidently, which is worse than an error. Phase 1's tables bake mode 3's key schedule and no parameter could select another, so the fix is to say so. main.go now parses instead of slicing, and any other mode exits 1 naming the mode it got. The m3 framing replayed one captured session. FPSAPExchangeM3's 144-byte prefix is a constant whose body encodes a local SAP captured from a single emulator snapshot; receivers that check the body reject the replay (doubletake#17 is an AppleTV3,2 answering 466 Key Management Error). fpemu.NewFPSAPSession(rand.Reader) generates its own, encrypts it into the m3 body and folds it into the response. Driven with the frozen local SAP it reproduces the captured 164-byte m3 for all 142 golden vectors; given a fresh one it matches doubletake byte for byte across the whole frame. The frozen function stays -- the golden vectors pin it, and crates/rotten-crypto does its own framing. Neither fix changes the 20-byte response for a well-formed mode-3 m2, which is what main.go emits. The 142 golden vectors and the 8 external vectors are unchanged. Also corrected: THIRD_PARTY_NOTICES described doubletake as historical use and the implementation as "of independent provenance". That was true of the ARM64 interpreter's removal but became wrong once the bridge was replaced by doubletake's closed form -- internal/fpsapcore is derived from it. Its licence is LGPL-3.0, not GPL-3.0 as listed. Every file now carries an SPDX header, and internal/fpsapcore/NOTICE.md has the file-by-file breakdown. Verified: gofmt clean, go vet clean, all four packages pass with and without -race, 73/73 files carry an SPDX header.
|
Pushed an update. Two of these are correctness fixes I found after opening this, and one is a licensing correction I got wrong the first time. It answered mode 3 to every m2. An m2 selects a FairPlay message mode in byte 13, and the mode picks both the CBC IV and the AES round keys for the message body — the same 128-byte challenge produces four completely different responses under modes 0–3. The m3 framing replayed one captured session. The 144-byte prefix is a constant whose body encodes a local SAP from a single emulator snapshot. Receivers that check the body reject the replay — omarroth/doubletake#17 is an AppleTV3,2 answering Neither changes the 20-byte response for a well-formed mode-3 m2, which is all A licensing correction. So the licence picture is: Verified: One thing I want to be straight about: none of this has been tested against real hardware. Two independent implementations agreeing on the bytes is the strongest check I can run here, and it is not the same as an Apple TV accepting them. If you have a device to point at it, that is worth more than anything else in this PR. |
The previous commit described internal/fpbridge as Blue Oak 1.0.0 in full.
That was wrong, and wrong in the harmful direction: it understated what is
owed to doubletake.
Four files in that package were written here, but while reading doubletake's
exchangeM3 and validateFPSAPRecord, and it shows in their shape:
fp_sap_session.go the session's structure -- build the record,
encrypt the local SAP into bytes 16..144, fold it
into the descriptor
fp_sap_m3.go field-by-field record validation
mode_identity_test.go four response constants, produced by running
their code
session_xcheck_test.go a local SAP and a 164-byte m3, likewise
All four are now marked LGPL-3.0-or-later in their SPDX headers, and both
THIRD_PARTY_NOTICES.md and internal/fpsapcore/NOTICE.md say so.
Being precise about what is and is not owed, since that is the point of a
notice file. Every constant involved is independently present in data
captured here -- FPLY, the version bytes, the declared length, the mode
byte, the label 8f 1a 9c and the local SAP's 00 01 head are all readable
straight out of m3Prefix, which came from an emulator snapshot rather than
from them. Nothing was copied. What was taken is the reading of the layout:
that byte 12 is the mode, that 13..16 is a label, that a sender randomises
the SAP from byte 2. Knowing where to look is the contribution, and it was
theirs.
No code changes. gofmt clean, go vet clean, all four packages pass with and
without -race, 73/73 files carry an SPDX header.
|
Correction to my previous comment, and it goes against my own interest so I want to be explicit about it. I wrote that Four files in that package were written here, but while reading doubletake's To be precise about what is and isn't owed, since a notice file that just says "some of this is theirs" is not much use: every constant involved is independently present in data captured here. Practically, this changes little for you: No code changed in this push. |
1 participant
What this does
Replaces the ARM64 interpreter in
tools/fpsap-helperwith a native Go implementation of the FairPlay SAP exchange.Deleted:
fpemu/fpexchange_interp.go— 2,711-line ARM64 interpreterfpemu/fpexchange_data.go— 1 MB Go file holding a 165 KB snapshot of Apple's signed FairPlay binaryfpemu's two exported functions keep their signatures and return the same bytes, somain.goandcrates/rotten-cryptoneed no changes.Does it produce the same answers?
I built your current helper and my replacement as two binaries and fed both the same inputs.
4,923 inputs, 0 mismatches. Every single-byte position across the payload, one bit set in each 16-byte block, all-ones with a byte cleared, solid fills, counter ramps, 4,500 random payloads, and 200 full 142-byte m2 messages. The concatenated outputs hash identically.
13/13 error and edge cases behave identically — empty input, odd-length hex, non-hex, off-by-one lengths, trailing newline, uppercase hex, oversized input. Same exit codes, same stderr behaviour, so the subprocess contract
fpsap_helper.rsdepends on is unchanged.I checked the harness can actually fail: perturbing one mask in the bridge makes 52 of 52 sampled inputs disagree.
Other testing
go vetclean on 32- and 64-bitThe helper had no tests before. It now has tests at the exported functions plus a CI job that vets, builds and runs them — CI previously never touched the Go code.
Size
300 KB, replacing ~1.07 MB. The built helper binary shrinks too, 2.96 MB → 2.55 MB.
If you looked at this PR earlier, this is what changed. It used to add 8.1 MB and I argued the trade was "provenance for bulk". That trade is gone. The bridge was machine-generated straight-line code then; it has since been reduced to a closed form, and
internal/fpsapcorecomputes it in 697 lines of ordinary byte arithmetic — ring diffusion, a nonlinear circuit, an MD5-family compression, a fold and a scramble.So it is smaller than what it replaces, and the parts that aren't constant tables can be read.
What still comes from Apple
Data, not code — this is white-box cryptography, so the tables are the cipher. What remains is the white-box AES tables and the Phase-2 network's constants. No Apple binary, no interpreter, no Apple addresses. The bridge carries no table data at all.
Licensing
internal/fpbridge,internal/fairplayhash— my own work, Blue Oak 1.0.0 (permissive)internal/fpsapcore— derived from omarroth/doubletake at8ccea5f, LGPL-3.0, withNOTICE.mdalongside itBoth flow into this directory's existing GPL-3.0 without friction. Keep the
LICENSEand the subprocess isolation as they are.Separate bug you may not know about
M3_PREFIXincrates/rotten-crypto/src/fairplay_setup.rs:45-51is a hardcoded 144-byte constant containing a local SAP captured from one session, so every m3 replays that session. doubletake had the same bug and fixed it ine544a88after issue #17, where an AppleTV3,2 rejected it withRTSP/1.0 466 Key Management Error.This PR doesn't change that. Happy to file it separately.
tools/fpsap-helper/PROVENANCE.mdhas the details.