Final cleanup and HANDOFF.md rewrite for manuscript handoff by jaredgalloway · Pull Request #320 · matsengrp/multidms

jaredgalloway · 2026-08-18T22:22:59Z

Closes #297

Phase 7 of EPIC #290 — the final cleanup pass plus a rewritten HANDOFF.md.
The ordering is load-bearing: all cleanup (B1–B7) lands before the handoff
is written, because the document describes the paths, configs, and remote
layout that the cleanup changes. Written first, it would ship stale.

What changed

Deleted (B1–B3):

experiments/convergence-lab/ (25 files) and its five citation sites. It was
an evidence record, not a pipeline; the live configs now say what each knob
does rather than citing a deleted README. Also removes
tests/test_convergence_lab_harness.py, whose path-based import harness
was a collection-time ImportError once the lab went — that aborts the
whole pytest run, not one test.
experiments/loss-normalization/ (8 files). Nothing referenced it and it
still called the pre-tier-split single-argument load_config().
The three COMING SOON docs placeholders, plus their toctree entries and the
two absolute-URL intro bullets that would otherwise 404 on the published
site. Content tracked in the stub Write the three missing docs pages (biophysical model, BA.1 fitting example, simulation validation) #318.

Config comment trim (B4): simulation 171 → 63 lines, spike 83 → 61 (65
after the n_processes comment block noted below). Cuts superseded A/B tables,
CORRECTION: self-arguments, and per-issue archaeology. Keeps the
operational safety warnings and the overloaded-maxiter disambiguation. Values
unchanged — proven by comparing the parsed YAML before and after, key by key.

Fit-cache integrity: config.yaml is a fit-tier rule input:, so even a
comment-only edit re-hashes it and marks the ~2h20m spike fit out of date.
Restored with snakemake --touch, then verified: fit_collection.pkl is
byte-identical to the pre-edit baseline in both pipelines
(spike b2b4736073e475a6fd7b1b5260063d6c / 1761133462 B; sim
f602baf4801bb9257a1f22281da99f49 / 649380559 B), both dry-run
Nothing to be done, and a forced dry run against manuscript_figures
re-runs only that rule and the all aggregator — no fit-tier rule appears.

Remote tidy (B5–B6) on ermine: 9 result payloads + 5 loose top-level dirs
moved into archive/archive-2026-08-18/; the clone moved from 217-share-alpha
onto an up-to-date main with one payload per pipeline and live results
symlinks; all 9 remote worktrees removed. Nothing was deleted — only moved.
Both promoted pickles md5-verified against local before any removal.

B7: PR #239 closed unmerged (untouched since 2026-05-07) and the bug
re-raised as #319 — it is still live on main at
model_collection.py:1471. The branch is kept.

`HANDOFF.md`

Rewritten for a reader who knows the science but has never read this codebase.
Seven sections, ordered by what that reader needs first:

Where things stand
Getting the results and looking at them
Issue status
Figure status
Running the pipeline
Building the docs
Science to carry forward

§2 is the load-bearing addition: a five-step checklist taking a first-time
cloner from git clone to viewing figure PDFs and exploring fits in the
dashboard. Fetch and symlink are a single copy-pasteable block ending in
pixi run check-results, because a fetch without the ln -sfn step leaves the
docs build and every .nblink broken with a bare "No such file or directory".
Neither manuscript payload is reachable via pixi run run-pull — it derives
its directory name from the current branch with no override — so the recipe
uses rsync directly.

§5 covers running new experiments. The first draft told the reader the
compute was done and to be suspicious of anything proposing a refit. That is
true of the current manuscript but wrong as guidance: whoever picks this up
may well need a new arm, a different λ grid, or a reviewer's ablation. §5 covers
test-before-prod, the remote launch/poll/fetch cycle, and how to change what
gets fit.

Preserved verbatim: the verified figure manifest, both deliberate filename
traps, the Phase 3 descope, and the notation table. The config tier split and
the --touch guidance moved to experiments/scv2-spike/README.md, where the
rest of that material already lives.

Review pass

A fresh-context agent fact-checked every claim in the document against the
repo and GitHub — 25 issue/PR numbers, every path and pixi task, every
number, the figure manifest against the actual TeX, and the pinned legacy
notebook blob. Five claims were wrong and are fixed in 0dc2c40:

Claim	Truth
S13's filename trap is "documented in the Snakefile's `FIGURE_NAMES` comment"	That comment covers only the Figure 3 trap. S13 appears nowhere outside `HANDOFF.md` — it has no producer yet. Now stated, with a note to carry the warning into whatever producer gets written.
"#240's bug is live", in a section listing open issues	#240 is closed; the bug was never fixed in code. Both facts now stated.
Forced figure-target dry run reported `total: 1`	Re-measured: `total: 2` (`manuscript_figures` + `all`). The load-bearing claim — no fit-tier rule re-runs — is unchanged.
"two commented-out template placeholders"	Four. The 22-figure headline count was already correct.
"13 open issues"	14 today; 13 once this lands and closes #297.

The review also confirmed the figure manifest maps to the right S-numbers by
walking all 17 si.tex labels in order, re-derived both pickle md5s, and
confirmed the A419S fold-changes match the executed notebook digit for digit.

Verification

Check	Result
`pixi run test`	✅ 134 passed
`pixi run lint` / `fmt-check`	✅ pass
`ruff` / `black` on `experiments/` (explicit paths)	✅ pass
`pixi run docs`	✅ build succeeded
Both pickles vs pre-edit baseline	✅ byte-identical
Both pipelines dry-run	✅ `Nothing to be done`
Forced figure-target dry run (both pipelines)	✅ no fit-tier rule; `manuscript_figures` + `all` only (`total: 2`)
CI	✅ 4/4 green

Deviations from spec

results-prod-sim-vpl500-tol1e5 does not exist on the remote. All nine
remote payloads are spike runs, so B6's "sourced from the payloads rescued
in B5a" holds only for spike. The 1.2 GB simulation payload was uploaded
from local instead.
The beta-clip-* payloads were not remote-only — full local copies
existed, so the spec's "losing them loses the run" warning was wrong. They
were still moved first; the risk was simply lower than stated.

Beyond the spec (scope increase): the two stray local
.worktrees/beta-clip-* checkouts were removed, not merely "checked".
beta-clip-50-l2reg-10x had no upstream and carried three commits found
nowhere else, so the branch was pushed to origin first. Removal was gated
on branch-pushed + zero-unpushed + clean-tree + payload-md5-archived.

Beyond the spec — two commits cherry-picked onto this PR: cd5ed49
(117 lines documenting the XLA JIT mapping leak → CLAUDE.md) and f0efaa5
(spike n_processes 6 → 20, one worker per fit). This is a fit-tier
config value change, but n_processes is execution-parallelism only —
model_collection.py:292,389 uses it solely to pick between a serial path
and a Pool size, so it never enters the objective and results are identical
at 6 or 20 workers. It is sequenced after the --touch verification
precisely so the comment-only proof for the other edits stays honest, then
re-touched and re-verified byte-identical. A key-by-key YAML comparison
confirms n_processes is the only value that moved. The third commit,
f4edecd (the beta_clip_range [-50,50] exploratory arm), was deliberately
not picked — it is an experiment, not a fix, and stays on its branch.
294-naive-baseline-arm on the remote was on a detached HEAD, not a
branch. git worktree remove handled it identically.
A stray empty archive/archive- directory (a typo artifact from an
earlier probe) was removed with rmdir, which refuses non-empty dirs.
The plan's Task 1 Step 4 stop condition fired, and was diagnosed as
benign. Both pipelines dry-ran dirty in a fresh worktree before any
edit — because git worktree add stamps every tracked file at checkout
time, which is newer than the rsynced result payloads. Not a stale baseline.
Resolved with a pre-edit --touch, legitimate here because the md5
baseline was captured first and re-verified identical after, which is what
makes the later before/after comparison meaningful rather than confounded
by checkout mtimes.

Open follow-up

The remote --touch (plan Task 9 Step 4) must run post-merge. The trimmed
configs only reach the canonical clone on ermine via git pull after this
lands, so the remote DAG cannot be restored until then. Everything else in the
remote cleanup is complete.

Baseline captured before any edit: spike b2b4736073e475a6fd7b1b5260063d6c / 1761133462 B sim f602baf4801bb9257a1f22281da99f49 / 649380559 B A fresh worktree checkout stamps all tracked files at checkout time, which is newer than the rsynced result payloads, so both pipelines dry-ran dirty before any edit. Resolved with a pre-edit --touch; both pickles verified byte-identical afterward and both pipelines report 'Nothing to be done'.

The lab was an evidence record, not a pipeline. Its measured findings are recoverable from git history; the live configs now state what each knob does rather than citing a deleted README. Also deletes tests/test_convergence_lab_harness.py, whose path-based import of the harness was a collection-time ImportError once the lab went.

Nothing referenced it and it still called the pre-tier-split single-argument load_config(), so it could not run today.

They shipped as empty placeholders on the public site. Removed from both the toctree and the two absolute-URL intro bullets, which would otherwise 404. Content tracked in the stub issue filed alongside this change (#318).

Cuts superseded A/B tables, self-corrections, and per-issue archaeology. Keeps operational safety warnings (n_processes null-is-dangerous) and the overloaded-maxiter disambiguation. Values are unchanged - verified by comparing the parsed YAML before and after. simulation 171 -> 63 lines; spike 83 -> 61 lines.

Both pipelines dry-run fully cached. fit_collection.pkl byte-identical before and after (spike b2b4736073e475a6fd7b1b5260063d6c / 1761133462 B; sim f602baf4801bb9257a1f22281da99f49 / 649380559 B). Forced dry run against manuscript_figures reports figure-tier only.

n_processes must be >= the number of fits. JAX/XLA leaks executable JIT mappings across sequential fits in one process, so a worker handling more than ~5 fits dies with 'Unable to allocate section memory' on a host with 1.4 TB free. Failures land by queue position, not hyperparameter: a serial run of the spike grid passed fits 0-4 and failed 5-19 across both replicates and every fusionreg. Under a pool this masquerades as 'Failed fitting 1 of 20 parameter sets'. Also documents what this run had to rediscover the hard way: the manual launch path when the forwarded 1Password agent blocks remote git, the shared /fh/fast filesystem, cumulative-CPU-time as the only honest liveness probe (%CPU is a lifetime average that hides deadlocks), the Pool deadlock signature, and the 429 / stale-lock / incomplete-file / MaxStartups snags. (cherry picked from commit cd5ed49)

Fits fail by position, not by hyperparameter: a serial run of the 20-set grid passed indices 0-4 and failed 5-19, across both replicates and every fusionreg including 0.0. Each fit JIT-compiles fresh XLA modules whose executable mappings are never released (xla_jit_dylib_12/19/31 counter climbing), so a process dies on its ~6th compile with a 980-byte suballocation failure on a host with 1.4 TB free. 20 workers for 20 fits makes p.map chunksize 1, so each fit gets a fresh process and the leak cannot accumulate. (cherry picked from commit f0efaa5)

n_processes is execution-parallelism only - it selects the Pool size and never enters the objective - so the fit outputs are valid at 6 or 20 workers. Snakemake hashes the config file rather than its semantics, so the DAG went stale on a change that cannot alter results. Verified: fit_collection.pkl byte-identical to the Task 1 baseline, and a key-by-key comparison of the parsed YAML shows n_processes as the only value that moved.

Both checkouts removed after pushing beta-clip-50-l2reg-10x to origin -- it had no upstream and carried three commits found nowhere else, including the XLA JIT mapping leak writeup (cherry-picked onto this branch in the preceding commits). Both branches survive on origin; payloads md5-verified in the ermine archive before removal.

Reorients the document around where the work stands rather than how to install it. Seven sections: status, issues, figures, results location, dashboard, docs, science-to-carry-forward. Describes the post-cleanup repo and remote layout. Preserves verbatim the verified figure manifest, both filename traps, the refuted-S10 analysis, the Phase 3 descope, the notation table, and the config tier split -- with n_processes corrected to the pinned 20.

All nine remote worktrees removed (zero failures) after the md5 gate confirmed both promoted pickles match the local baseline. The shared directory is now multidms/ + archive/ + pixi dotfiles only; the clone is on up-to-date main with one payload per pipeline behind live results symlinks; nine payloads plus the stale April results are preserved under archive/archive-2026-08-18/. No tmux sessions left running. Remaining: the remote 'snakemake --touch' runs post-merge, once the trimmed configs reach the clone via git pull.

The handoff told the reader the compute was done and to "be suspicious of anything that proposes" refitting. That is right about the current manuscript but wrong as guidance: whoever picks this up may well need a new arm, a different lasso grid, or an ablation a reviewer asked for. Adds section 5 covering test-before-prod, the remote launch/poll/fetch cycle, how to change what gets fit, and checking convergence. Reframes section 1 so the caution lands on refitting *by accident* — via the config tier split — rather than on running the pipeline at all. Promotes the XLA JIT leak from a hedged aside in the dashboard section ("if you ever *do* re-run a fit") to a first-class subsection where a runner will find it, since it presents as a bad lasso rung rather than as an infrastructure failure. Renumbers the trailing sections and fixes the cross-references that shifted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Promotes results retrieval to section 2, ahead of issue status, and merges the dashboard section into it, so the document opens with the path a first-time cloner actually needs: environment, remote config, fetch, symlink, figures, dashboard. Covers both pipelines; the old version documented only spike. Three claims in that path were checked against the scripts and were wrong as first drafted: - pixi.lock is gitignored, not tracked, so a fresh clone re-solves. That matters because the jax/jaxopt pins exist to avoid a SIGABRT in jaxopt's proximal solver, and the lock is what would have pinned it. - `run-pull` cannot fetch either manuscript payload: it derives results-<profile>-<branch>, which on main is results-prod-main, and both payloads are named for the landed branches that produced them. It also mkdir -p's the local dir before rsync, so a failed attempt leaves an empty plausibly-named directory behind. - check-results suggests that same broken command when it finds nothing, which is exactly a fresh clone's situation. Also trims material that belongs elsewhere or nowhere: the EPIC phase table, the 40-line S10 erratum forensics (the operative warning stays, the notebook-cell analysis lives in #293), the standalone live-bug section (folded into the issue table with its benign-for-figures rationale), and the vocabulary-retirement section. Six issues the user closed are dropped, #302 is recorded as a duplicate of #192, and the config-variants note moves to the spike README. Adds the legacy analysis notebook at SARS-CoV-2_spike_multidms@6c98b7b to the reference table as the old-vs-new comparison baseline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Removes material that was either fluff or already documented elsewhere: - The section 1 preamble arguing that running the pipeline is allowed. - The run-pull warning. The fetch recipe now just shows the rsync that works, with the ln -sfn link step folded into the same block so fetching and linking are one action rather than two. - The n_processes subsection. The config already pins it to 20, one worker per fit, so the guidance was explaining a setting nobody has to make; the reasoning stays in CLAUDE.md and the config comment. - The convergence-checking step, which is run hygiene, not handoff. - The config tier split, which the spike README documents in more detail. Ports the two things the README lacked -- the --touch guidance and the overloaded-maxiter note -- rather than losing them, and repoints the one reference at the README. - The S10 block in "Science to carry forward". Nothing there was carried into the manuscript; that the linear arm needs re-addressing is already visible in the issue table and the figure manifest, both pointing at #293. Section 7 now holds only the two measured results and the notation table. 557 -> 437 lines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

A fresh-context review fact-checked every claim against the repo and GitHub. Five were wrong: - The S13 filename trap was attributed to the Snakefile's FIGURE_NAMES comment, which documents only the Figure 3 trap. S13 appears nowhere outside this document, because S13 has no producer yet. Say so, and tell whoever writes one to carry the warning across. - #240 was described as live without noting it is closed. The bug is real and unfixed in code; the issue is not open. Both facts now stated. - The forced figure-target dry run reports total: 2 (manuscript_figures plus the all aggregator), not total: 1. Re-measured. The load-bearing claim -- no fit-tier rule re-runs -- is unchanged and now exact. - Four commented-out template placeholders, not two. The 22-figure count was already correct and is unchanged. - 14 open issues today, 13 once this lands and closes #297. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

jaredgalloway and others added 16 commits August 18, 2026 15:56

chore: delete the dead loss-normalization experiment

b2d2478

Nothing referenced it and it still called the pre-tier-split single-argument load_config(), so it could not run today.

docs: retire the three COMING SOON placeholder pages

c1bbd59

They shipped as empty placeholders on the public site. Removed from both the toctree and the two absolute-URL intro bullets, which would otherwise 404. Content tracked in the stub issue filed alongside this change (#318).

jaredgalloway merged commit 038bfa1 into main Aug 19, 2026
4 checks passed

jaredgalloway mentioned this pull request Aug 19, 2026

EPIC: Complete the spike manuscript analyses — convergence, baselines, and figures #290

Closed

Navigation Menu

Uh oh!

Final cleanup and HANDOFF.md rewrite for manuscript handoff - #320

Final cleanup and HANDOFF.md rewrite for manuscript handoff#320
jaredgalloway merged 16 commits into
mainfrom
297-handoff-and-cleanup

jaredgalloway commented Aug 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jaredgalloway commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

HANDOFF.md

Review pass

Verification

Deviations from spec

Open follow-up

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jaredgalloway commented Aug 18, 2026 •

edited

Loading

`HANDOFF.md`