Proposal: rewrite checksync in Rust, distributed as a binary-backed npm package

[somewhatabstract]

Summary

Proposal + plan to rewrite checksync from Node/TypeScript to Rust, shipped as
the same checksync npm package so existing users get a seamless update. The
motivation is speed: Node's cold start (~50–150 ms — dominant for the common
pre-commit "few changed files" case) plus single-threaded, per-line stream parsing
of every file (dominant for full-repo CI on a large monorepo) are the two costs a
compiled, parallel implementation removes. This is also a clean moment to clear tech
debt and land several long-requested features.

This starts as a proposal with a cheap go/no-go gate — not a commitment. See
"Validate first" below.

TL;DR

• Worth it? Likely yes, but decide with a ~1-day benchmark (below) before
  committing. The one real risk — silently changing a checksum and churning every
  tag in users' repos — is well-mitigated by the existing __examples__ conformance
  fixtures + published checksum vectors.
• Language: Rust. The core work (parallel, .gitignore-aware walking + globbing)
  maps directly onto ripgrep's ignore + globset crates; serde deletes the
  JSON-schema dependency outright.
• Distribution: the esbuild/biome/swc model — checksync keeps its name and
  ships no binary itself, declaring one optionalDependency per platform
  (@checksync/<os>-<cpu>); bin/checksync.js stays a tiny Node shim that
  spawnSyncs the matching binary. npx / global / pnpm all keep working. Ships
  as a major (11.0.0), identical behavior.
• Programmatic API (checkSync, loadConfigurationFile): preserved via a
  spawn-based JS shim (requires a new checksync --print-config subcommand).

Validate first (Phase −1 — cheap go/no-go, ~1 day, before any rewrite)

The rewrite is a performance bet, and performance is measurable up front without
building the real tool.

1. Decompose the current runtime (hours, read-only): time node -e '' and
   checksync --version for the fixed Node-startup tax; time a small run (a few
   files) vs. a full-repo run to separate startup from walk+parse; optionally
   node --cpu-prof to split fast-glob walk vs. readline parse vs. hash.
2. Walk-hash PoC (~half a day): ~40 lines of Rust (ignore parallel walker +
   read + adler hash over every candidate file) establishes the performance
   floor — it does the expensive ~80% and skips the cheap logic. Compare
   wall-clock vs. checksync on the same real corpus, cold and warm OS page cache.

Decision rule — Go if small-run latency drops from ~150–300 ms to ⪅20 ms
(near-guaranteed by dropping Node) and the full-repo PoC is ⪆3× faster. Stop if
the PoC is within ~1.5× (bottleneck is unavoidable disk I/O, which no language change
fixes) or real invocations are already comfortably sub-100 ms.

The single number that swings the verdict: how many files checksync actually
parses in real usage. A narrow include glob ⇒ the parse win is small and only startup
matters; a whole-tree scan ⇒ parallelism wins big.

Distribution design (binary-backed, seamless update)

Convert to a pnpm workspace: packages/checksync (main) + packages/@checksync/*
(6 platform packages). Six targets, static musl on Linux (no C deps ⇒ one Linux
package per arch covers glibc and musl):

npm package	os / cpu	Rust target
@checksync/linux-x64	linux / x64	x86_64-unknown-linux-musl
@checksync/linux-arm64	linux / arm64	aarch64-unknown-linux-musl
@checksync/darwin-x64	darwin / x64	x86_64-apple-darwin
@checksync/darwin-arm64	darwin / arm64	aarch64-apple-darwin
@checksync/win32-x64	win32 / x64	x86_64-pc-windows-msvc
@checksync/win32-arm64	win32 / arm64	aarch64-pc-windows-msvc

• Each platform package contains only its executable and declares os/cpu; the
  main package lists all six as exact-pinned optionalDependencies and keeps zero
  runtime dependencies of its own.
• bin/checksync.js resolves @checksync/${process.platform}-${process.arch} via
  require.resolve (works across npm hoisting, pnpm's isolated store, and global
  installs), spawnSyncs the binary with stdio:"inherit" and argv.slice(2),
  re-raises signals, and propagates the exit code. No platform match (unsupported
  arch, or --no-optional) ⇒ actionable error, exit 5.
• Rejected alternative: postinstall-download (runs code at install, needs network,
  no lockfile integrity, breaks --ignore-scripts).
• checksync.schema.json stays in the main package for editor $schema
  autocomplete; optionally embed it in the binary behind checksync --print-schema.

Release pipeline: extend the existing Changesets flow — put all 7 packages in a
Changesets fixed group so they share a version; add a build-binaries GitHub
Actions matrix (Linux both arches via cargo-zigbuild static musl; both macOS arches
on macos-14; both Windows arches on windows-latest); on Release-PR merge, build
all binaries, lay each into its platform package, then publish the 6 platform
packages first and the main package last. Fold in npm provenance / trusted
publishing (OIDC) — closes #2340.

The fidelity contract (byte-exact — what must not change)

The whole test strategy pins these. Getting any wrong silently churns users'
checksums or output ordering.

1. Checksum recipe (src/checksum.ts): adler32(("\n" + join(each content line + "\n")).utf8), reinterpreted as i32, decimal-formatted, leading -
   stripped. Golden vectors:
   checksum(["\n","\n","a test\n","more test\n","\n"]) == "1043727889" and
   checksum(["Some super important content!"]) == "1472197848".
2. Two variants per marker (src/parse-file.ts): contentChecksum (content only
   → LOCAL comparisons) and selfChecksum (content + root-relative,
   forward-slash-normalized path appended as a final element with no trailing \n
   → REMOTE/self comparisons + migration writes).
3. Line model = Node readline crlfDelay:Infinity (split on \n, strip
   trailing \r, no final empty line when the file ends in newline, lone \r not a
   separator). Replicate exactly — not str::lines(). Retain each line's original
   terminator for When auto-fixing, added line-ending should match line-ending of file being edited #665.
4. Root detection = nearest ancestor dir (incl. the file's own) containing the
   marker; default marker package.json. Feeds selfChecksum.
5. JS key-ordering (highest-churn trap): targets keyed by line number →
   numeric-ascending → BTreeMap; everything else → insertion order → IndexMap.
6. Regex classes (src/marker-parser.ts): JS \w/\s are ASCII-only; Rust
   regex is Unicode by default → compile with (?-u). Each line is JS
   String.trim()-ed before tag matching.
7. Output: text/verbose logs are byte-exact (color-stripped) — includes
   ". "→newline splitting, fixed label widths, 2-space console.group indent.
   --json is JSON.stringify({version, launchString, files}, null, 4) but
   per-error field order is inconsistent across factories, so JSON is compared
   structurally and Rust picks one canonical order.
8. Exit codes are public: SUCCESS 0, NO_FILES 1, PARSE_ERRORS 2,
   DESYNCHRONIZED_BLOCKS 3, UNKNOWN_ARGS 4, CATASTROPHIC 5, BAD_CONFIG 6, BAD_CACHE 7.
9. Launch string reproduced across the shim boundary (shim passes launcher env so
   the binary renders npx checksync / pnpm checksync correctly).

Phased plan (each phase gates green against the conformance oracle on ubuntu/macOS/windows)

• Phase −1 — Validate the performance thesis (see "Validate first" above).
• Phase 0 — Conformance oracle (no Rust): a committed harness that runs the
  real current CLI as a subprocess over every __examples__ dir × 4 scenarios
  (default+--output-cache; check-only; --json; --update-tags --dry-run;
  migrate_all → --migrate all), plus CLI-surface cases and single-file inputs
  (seed for When running against a file, it should check the files that it links to as well #2320), capturing stdout/stderr/exit/--json/resulting file bytes as
  normalized golden fixtures. Also emit a checksum vector table (ASCII/CRLF/Unicode/
  emoji/empty). Uses real process output, not the Jest StringLogger snapshot.
• Phase 1 — Core: checksum + line model + parser (checksum.ts, line model,
  marker-parser.ts, types.ts). Verify vs golden vectors + parser suite + checksum
  table.
• Phase 2 — Filesystem: walk, glob, ignore, symlinks, root (get-files.ts,
  ignore-*, get-normalized-path-info.ts, path/root utils, ancesdir). Crates:
  ignore, globset, std::fs::canonicalize, dunce.
• Phase 3 — Cache assembly + error generation + output/exit codes:
  get-markers-from-files.ts (two-pass, aliases), generate-errors-for-file.ts,
  errors.ts, determine-migration.ts, output-sink.ts, formatters. Crates:
  owo-colors, serde_json, rayon. Gate: 21 × {check-only, json} green on 3
  OSes. Fold in the GitHub Actions formatter (Output in a format compatible with github annotations so that running in GitHub workflow will annotate code #2096) here.
• Phase 4 — Autofix + migration writes + cache: fix-file.ts,
  output-cache/load-cache, load-migration-config.ts. When auto-fixing, added line-ending should match line-ending of file being edited #665 lands here as a
  deliberate divergence (preserve the file's EOL) behind its own CRLF fixture.
• Phase 5 — CLI surface + config loading: parse-args.ts (yargs→clap),
  option resolution, load-configuration-file.ts (schema→serde), help.ts, cli.ts.
  Add --root (Add a --root argument or similar to explicitly specify the root starting directory rather than using a marker file #2315) and --print-config (for the API shim).
• Phase 6 — Packaging + CI + release: shim, 6 platform packages, cross-compile
  matrix, lockstep versioning, provenance. Verify tarball install + npx checksync +
  the --no-optional fallback on each OS.
• Phase 7 — Cutover: run the Rust binary over a large real corpus + the examples
  and diff vs Node; publish checksync@11.0.0 (MAJOR changeset with migration notes).
  Keep a 10.x maintenance branch.

Open issues folded in

• Free wins: When auto-fixing, added line-ending should match line-ending of file being edited #665 (autofix preserves the file's line ending — the TODO in
  fix-file.ts), Add a --root argument or similar to explicitly specify the root starting directory rather than using a marker file #2315 (--root), Upgrade @hyperjump/json-schema #1261 (obsolete — the ESM JSON-schema dep is
  deleted in favor of serde), Use jest-specific-snapshot or some other mechanism to make integration test snapshots easier to manage #2054 (obsolete — Phase 0 already writes one golden
  file per scenario), Add provenance #2340 (npm provenance / trusted publishing).
• Perf & CI output: Implement parsed file caching to speed things up on larger codebases #2055 (parallel walk/parse addresses most of the "slow"
  motivation with no cache; then a config-agnostic, portable per-file parse artifact
  keyed by content hash that subsumes today's --output-cache/--use-cache), Output in a format compatible with github annotations so that running in GitHub workflow will annotate code #2096
  (GitHub Actions ::error file=…,line=…:: output).
• Design-for-later (shape now, build later): When running against a file, it should check the files that it links to as well #2320 (reverse-direction validation
  of linked files — a targeted extension of the existing two-pass referenced-files
  loader), Support inter-dependency sync tags #887 (cross-repo sync tags — the Implement parsed file caching to speed things up on larger codebases #2055 parse artifact is the interchange
  file), Verification mode or tool that checks the target URLs #2037 (URL verification — kept behind a cli-only network trait so core
  stays pure/offline/deterministic).

Verification

• Golden checksum vectors reproduce exactly, plus a Phase-0 corpus table.
• Differential conformance: Rust binary's stdout/stderr/exit/--json (and autofix
  file bytes) match the Phase-0 golden fixtures for all 21 __examples__ × 4
  scenarios on ubuntu/macOS/windows — byte-exact for text (color-stripped),
  structural for JSON. This is the merge gate for phases 3–5.
• Real-corpus diff before cutover: run both Node and Rust over a large repo and diff
  every reported error and every autofix byte.
• Packaging smoke test: install the packed tarball into a scratch project per OS;
  confirm npx checksync resolves + runs the right binary and the --no-optional
  fallback errors cleanly.

Risks / honest caveats

• The load-bearing risk is checksum/output drift; mitigated by the conformance oracle
  and differential testing, but must be treated as the top priority throughout.
• Non-performance trade-off a benchmark won't capture: you take on a second language
  and a cross-platform binary release pipeline forever, in exchange for erasing
  real tech debt (Upgrade @hyperjump/json-schema #1261, Use jest-specific-snapshot or some other mechanism to make integration test snapshots easier to manage #2054) and unblocking features. Weigh that against the
  measured speedup.
• yargs → clap has quirks to replicate (;/space-split multi-values,
  ignoreFiles=false, tri-state booleans, ? help alias) — needs a small compat
  layer.