fix(archive): refuse entries written through an archive's own symlink

[rustytrees]

Fixes the archive half of #789. Split from #790 because the obvious fix here is the wrong one, and the reasoning is worth reviewing on its own.

The bug

isSafeSymlinkTarget allows a target to climb one level above the extraction root. That is deliberate and correct: bottles are built for their installed location and reach sibling formulas by climbing out of the tarball root. The rust bottle is the canonical case, and tests/archive_test.zig already pins it:

rust/1.95.0/lib/rustlib/aarch64-apple-darwin/bin/rust-objcopy
  -> ../../../../../../../opt/llvm/bin/llvm-objcopy

That link dangles at extraction time and resolves once the keg is in place.

The allowance is sound for a leaf link and unsound the moment anything is written through one. The depth arithmetic counts every component of an entry name as a real directory, so once a exists as a symlink the budget no longer describes the filesystem, and entries that each look like "one level up" compose:

a           -> ".."     (accepted, exactly at the budget)
a/b         -> ".."     (accepted; but `a` is already a symlink)
a/b/bin/EVIL            (lands in a sibling of the extraction root)

Against the real bottle layout that reaches <prefix>/bin, which mt shellenv puts on PATH, and extractTarGz returned success. std.tar opens with O_EXCL so nothing is overwritten, but creating a new command name there is enough. A single hop is already an escape on its own — the extraction root's parent is always writable.

Hard links have the same exposure from the other side. follow_symlinks = false only governs the final component; the kernel still resolves every directory on the way there. A target routed through a planted symlink imports an inode the archive never shipped into the keg, where the linker can expose it and a later chmod can widen it.

Why not just forbid escaping targets

That was my first patch, and it fails the compatibility test above — it rejects every bottle that references a sibling formula via opt/, which is most of them. A purely lexical depth budget also can't be made sound: a symlink pointing at a shallower in-tree location makes an entry's real depth lower than its name implies, so a later target that looks like it lands exactly on the root actually lands outside.

What this does instead

The invariant is not "targets may not escape" but "no entry is reached through a symlink this archive created". The pre-scan records each symlink it declares, canonically so ./x and x are one name, and rejects any later entry name — or hard-link target — that traverses one.

That is a property of the archive alone, so it stays in the existing pre-scan pass: still no partial writes, no post-extraction sweep, no new syscalls on the happy path. Leaf links that dangle until install are completely untouched, so the rust shape and every existing isSafeSymlinkTarget test are unchanged.

Residual, deliberately out of scope

A symlink already on disk in a reused destination is not covered, since only the archive's own entries are visible here. The bottle path extracts into a fresh temp dir, so it isn't exposed; a repeat cask or tap install into an existing directory is. Closing that needs per-component O_NOFOLLOW resolution at write time, which is a bigger change than this fix and would sit better as its own piece of work. Called out in the code comment too.

Verification

• zig build test — full suite green, including the existing isSafeSymlinkTarget policy tests, unmodified
• ./scripts/lint-spawn-invariants.sh — clean
• ./scripts/smokes/smoke_security.sh — 8/8

New tests cover: write-through-a-symlink, the chained variant, hard-link-through-a-symlink, the ./-spelled name, plus two compatibility guards — the rust opt/ shape and an ordinary in-tree hard link — that fail if the rule is drawn too tightly.