Skip to content

Add land, the copy-then-verify landing tool (CNA-M07) - #411

Open
mecattaf wants to merge 1 commit into
mainfrom
feat/land-tool
Open

mecattaf wants to merge 1 commit into
mainfrom
feat/land-tool

Conversation

@mecattaf

Copy link
Copy Markdown
Owner

Backlog row CNA-M07: the landing tool land, used by the nightly ~/today sweep and by every one-time landing lane (loose ~/*.md, dated dirs, home canon, Downloads, ~/sept7 prose, root relocations).

Nothing here is wired into a host or a timer. It is a package, an overlay entry and a flake check.

What it does

land copy <src>... --dest <dir> [--exclude GLOB]... [--source-session ID] [--readme-title T]
land verify <packet-dir>
land diff <src> <packet-dir>

copy writes a packet: the copied tree plus preservation-<YYYY-MM-DD>.json and README.md at its root. The manifest is the shape built by hand for the 2026-09-16 orchestration-day packet — a header carrying file count, total bytes and exclusions with reasons, and rows of {source, preserved, bytes, sha256, mtime, source_session}.

Rules the copy obeys, each because a packet is evidence and not a backup:

  • sources are only ever read, never modified, moved or removed;
  • cp -p semantics (mtime and mode preserved), relative structure kept;
  • a regular file over the 95 MiB ceiling is left out, with its size and the reason, and named in the README under "Left out on purpose";
  • a nested .git directory or gitfile is renamed to dot-git in the copy, and the row records renamed_from with the path it had at the source;
  • symlinks are never followed: each becomes a row with kind: symlink and its target, recreated verbatim in the copy;
  • the secret guard runs over the whole plan before the first byte is written, so an abort leaves no half-packet.

verify recomputes sha256 for every row and checks the row count against find -type f minus the packet's two artifacts. diff is the rsync -rn --checksum equivalence check to run before any source is removed.

Secret guard

Names matching *.env, *token*, *secret*, *credential*, id_ed25519*, *.age abort the run unless --allow-secret-names. Text content matching a GitHub token, an OpenAI-style key, an AWS access key id or a PEM private key block aborts with no override. Either way it names the file and never the value.

Backward compatibility

The existing hand-built packet verifies untouched:

$ land verify ~/mecattaf/notes/references/continuity/2026-09-16-orchestration-day
land verify OK  .../2026-09-16-orchestration-day  697 file rows == 697 files  59,791,984 bytes  sha256 all match
$ echo $?
0

The older nyu-2026-09-15 manifest has no mtime and no source_session. Those rows warn, they do not fail — every one of its 209 hashes matches. It still exits non-zero because that manifest genuinely covers 209 of the 261 files in its directory, and the receipt says so in those words rather than blaming sha256.

Tests

checks.land builds its own fixture lane in $TMPDIR — a nested .git, a symlink, a 3-byte file, a glob-excluded file and a sparse file over the ceiling — and drives copy, verify and diff over it. 30 tests. What is pinned: the manifest row count equals find -type f; the dot-git rename and its renamed_from; the symlink row; mode and mtime survive the copy; the sources are byte-identical afterwards; a flipped byte, a stray file and a missing file each fail verify; source drift fails diff; a moved packet still verifies; the guards abort before writing; and a count fault is never reported as a hash fault. rsync is in the check's build inputs so the cross-check runs in CI and not only where the host happens to have it.

Ran 30 tests in 1.9s
OK

Reviewer notes

  • Python 3 standard library only. rsync is a second opinion, never the authority: a packet legitimately differs from its source in two recorded ways (the rename and the exclusions), so diff always hashes and adds the rsync pass only when neither applies.
  • flake.nix shows 55 lines of pre-existing nixfmt drift elsewhere in the file. It was left alone; only the added block is formatted.

🤖 Generated with Claude Code

`land` is the tool the nightly ~/today sweep and every one-time landing lane
(loose ~/*.md, dated dirs, home canon, Downloads, ~/sept7 prose, root
relocations) run to move working files into notes without trusting that the
copy worked.

  land copy <src>... --dest <dir>   copy-then-verify, writes the packet
  land verify <packet-dir>          recompute every hash and count in it
  land diff <src> <packet-dir>      the rsync -rn --checksum equivalence check

The manifest is the shape built by hand for the 2026-09-16 orchestration-day
packet: a header with file count, total bytes and exclusions carrying reasons,
rows of {source, preserved, bytes, sha256, mtime, source_session}. That packet
verifies unchanged (697 rows == 697 files, sha256 all match). Older packets
without mtime or source_session warn rather than fail.

The copy keeps cp -p semantics and relative structure, never follows a symlink
(each becomes a kind: symlink row with its target), renames a nested .git to
dot-git and records renamed_from with the source-side path, and leaves out any
file over the 95 MiB ceiling with its reason in the README. Sources are only
ever read.

The secret guard runs over the whole plan before the first byte is written, so
an abort leaves no half-packet: names matching *.env, *token*, *secret*,
*credential*, id_ed25519* and *.age (overridable with --allow-secret-names),
and text content matching a GitHub token, an OpenAI-style key, an AWS access
key id or a PEM private key block (not overridable). It names the file and
never the value.

checks.land builds its own fixture lane in $TMPDIR — nested .git, symlink,
3-byte file, excluded glob, sparse file over the ceiling — and pins that a
flipped byte or a stray file fails verify, that the sources are untouched
afterwards, and that a count fault is not reported as a hash fault.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mecattaf

Copy link
Copy Markdown
Owner Author

Review note from the first real use (Downloads lanes, 12 packets, 3,222 files, all verify/diff exit 0):

  1. --exclude globs match the packet-relative path, so site/_raw/*.gz silently excludes nothing when the source is beam-dev; only beam-dev/site/_raw/*.gz works. Either match the glob against every suffix of the path (like .gitignore) or make copy fail loudly when an exclude glob matched zero files.
  2. The secret-name guard fired on documentation pages whose names contain token/credentials and on design-token JSON (claude-tokens-*.json); content scans were clean. Consider making the name guard a warning when the content scan is clean, keeping the abort only for content hits and for id_ed25519*/*.age/*.env.
  3. A packet with thousands of excluded files produces a README with one bullet per exclusion (3,131 lines). Summarise by glob with a count and defer to the manifest's excluded array.

None of these blocks the merge; the lanes landed with the correct spellings and an explicit override.

@mecattaf mecattaf added the additive-after-rebase triage bucket 2026-09-20 label Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

additive-after-rebase triage bucket 2026-09-20

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant