Skip to content

nightly-drain: declare the midnight seal and checkout snapshot as a module, and do not attach it to tally-drain.timer #433

Description

@mecattaf

The name collision, first, because it is the trap

KARPATHY-WIKI-MEMORY.md recommends attaching the memory librarian to the existing drain timer. That is the wrong timer.

MEASURED, systemctl --user list-timers 'tally-*' --no-pager and systemctl --user show tally-drain.service -p ExecStart:

tally-drain.timer   next 2026-09-19 23:22:18 CEST, service last started 23:17:18 CEST   (a 5 minute cadence)
ExecStart  .../tally-0.1.0/bin/tally --socket /run/user/1000/tally/tally.sock daemon drain

tally daemon drain drains producer event files into the queue. It fires every five minutes and it has nothing to do with memory. Attaching a nightly session harvest to it would run the harvest 288 times a day under a name that already means something else. The memory drain needs its own unit.

What the module must declare

A new /home/tom/mecattaf/dotfiles/home/nightly-drain.nix, modelled on the existing home/harness-records.nix:

  • a coordinator-gated oneshot user service plus timer, OnCalendar 00:01, Persistent = true, RandomizedDelaySec = 0
  • After = nightly-record.service, Nice = 10, IOSchedulingClass = idle
  • ExecStart = %h/.local/bin/nightly-drain
  • the scripts that graduate into home/dot_local/bin/: the mirror refresh, the index, the day seal, the brief extractor, the checkout snapshot, the receipt writer
  • retention: 14 nightly checkout snapshots plus the first of each month; the sealed days/ tree is never pruned

00:01 is chosen because midnight is contended. MEASURED from the same timer listing: tally-retention.timer and nightly-record at 00:00, the util row at 00:05, update-center-seed at 00:45 and 01:20. 00:01 is free.

The five comments the module must carry, because each one is a rule that was learned the hard way

  1. A root session belongs to day D if its last timestamp, converted from UTC to Europe/Paris, falls on D. The jsonl timestamps end in Z and the file mtimes are local; comparing the raw strings misfiles every session started between 22:00 and midnight.
  2. Quiet detection uses find -mmin, never -newermt '-N minutes', which silently matches nothing on this tree.
  3. The running orchestrator is excluded by session id, not by a heuristic. A row that fails a guard goes to SEAL.json as deferred_live with the guard named, never dropped, and the day index is keyed by session id and append-only so a later reseal inserts it.
  4. The repo drain saves the current checkout, not the repository: an rsync of each repo's git file list with --link-dest against the newest previous snapshot, resolved by listing the snapshot directory and never by date -d yesterday.
  5. A thin git bundle create --all --not --remotes per object store that holds commits on no remote. Bundling reads the object store and writes only the bundle. No repository is modified.

Measured scale, so the retention numbers are not guesses

MEASURED tonight across the 47 checkouts under /home/tom/mecattaf and its siblings: 118,451 files and 4,554,885,335 bytes in the git file list view, against 17.8 GB for the naive directory tree, of which the notes checkout alone is 4.12 GB. 28 of 47 checkouts carry commits on no remote, across 24 object stores, about 19 of which qualify for a bundle. The ten dotfiles worktrees share one object store.

Where it lands

Workbench under /home/tom/today/drain/days/<day>/ and durable under /home/tom/archives/session-archive/, in the exact relative shape E-ARCHIVE-LAYOUT.md gives the NAS tree, so the eventual NAS move is one rsync -a with no renaming. Nothing goes to the notes repository: index rows graduate there later, the bytes never do.

First hand run

Tonight's 00:01 pass is the first run by hand, before any timer exists, per the standing rule. Its receipt will be at /home/tom/archives/session-archive/manifests/nightly/2026-09-19.json. Cite it in the module review rather than re-deriving the numbers.

tier: Opus

The first hand run happened, 00:01 CEST 2026-09-20 (ruling 22)

MEASURED, receipt at /home/tom/archives/session-archive/manifests/nightly/2026-09-19.json: seal ok, 5 root sessions in the day, 1 sealed and harvestable (e9bd0f99, already in the v3 corpus, byte-identical brief), 4 deferred_live, 78 spool rows; 48 checkouts snapshotted (122,640 files, 4,538,676,141 bytes) and 11 thin bundles of local-only commits, all verify_rc 0; handoff file_only. Scripts: /home/tom/today/drain/{index-mirror.py,seal-day.py,drain-extract.py,snapshot-checkouts.sh,write-receipt.py}.

One defect the hand run surfaced, which the module must fix before any timer: guard G5 (no live claude, codex or pi process with cwd inside the session's project dir) is unusable when the project dir is /home/tom itself. On the first pass it matched the Claude chrome-native-host helper (cwd /home/tom/.claude/chrome); on the reseal with exact matching it matched the orchestrating session's own process (cwd /home/tom), so the three closed -home-tom sessions of the day (10f85da0, 3c713c85, f4e20c9a) were deferred while the running session was correctly excluded by identity. The guard needs to exclude the orchestrator's own process tree, or match the session id in the process arguments, not the cwd. Deferred rows are kept in SEAL.json and a later --reseal inserts them, so nothing was lost.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions