research(055): train-once pilot — invariance prior works, texture pathway missing; DO NOT SCALE YET - #144
Merged
Conversation
…op duplicates - crop_sim.py writes only cropped photo sheets + detail patches + the homography in meta.json; warp_channel() exported as the single warp convention. - dataset.py GlassDelightDataset(crop_view=True) lazily warps photo + all GT channels at load time, in FILE space (before nonlinear decodes) — exact match to the old materialized crops. - verify_lazy_crop_053b.py: equivalence test on the 68 dual-representation pilot samples — 68/68 OK, 986 channel files BIT-EXACT (float EXR maxdiff 0.0), loader wiring maxdiff 0.0. Run before slimming (exit 0 gate). - Retro-slim executed: pilot 12G -> 4.0G (918 redundant files deleted). - 053 report storage claim amended (render-only figure); 053b report supersedes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CTO catch: crop_sim's patches were emitted but dataset.py never referenced them. GlassDelightDataset(patch_prob=0.2) now serves a native-resolution registered detail patch on a fraction of draws (fine texture the 768 work-res destroys); reflect-pad to crop size with valid=0 in the pad (uniform batch shapes, no loss from invented content). Patch inherits its sample's holdout split. Unit-tested: registration bit-exact, loads at shape, split respected both directions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ports a5577a2+36bc550+f6bd8c1 from research/delighting-037 (drift-aware: 8-channel sigma_s AuxHead + phone-ISP loader preserved): latent-space T loss (MSE vs z_T_hat, shadow map adaptive-pooled), need_T opt-in decode, conf gated on decoded steps. test_grad_flow.py ported + extended to 7 heads and wired UNCONDITIONALLY at train_loop top. Acceptance: all heads nonzero grad (T 0.60), mutation-verified (detached z_T_hat fails naming ['T']), 20-step smoke T loss moves 0.1241->0.1003->0.1065 (was bit-frozen under the bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…et draws _cache was unbounded (~10-19GB RAM over a full pilot); now an OrderedDict LRU capped at 64 (~2.2GB). Sheet-path sample_crop now guarantees exactly self.crop via reflect-pad + valid=0 (non-square crop_view grids could crash np.stack). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t device row eval_foundation ran only clean linear renders; now every test photo also runs through each ISP preset (deterministic rng), metrics per-preset with wide_edge (held-out device) broken out, clean row kept as headline for continuity. Verified on the smoke adapter: 172 test samples x 6 conditions. Report updated with the full pre-flight fix section + acceptance evidence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…boards refreshed Disk guard stopped the farm cleanly at 348 samples (68 original + 280 net new, one disk-guard-interrupted partial deleted) -- 74% of the planned 468, per the lead's decision not to resume rendering. 17/17 recipes, 23 HDRIs, shadow 31%/ front-light 66% (near target), storage 60.6MB/sample mean (confirms the 053b lazy-crop fix holds at scale). Per-stage timing reconstructed from the raw [TIMING] log (dump_timings never ran -- process was killed by the guard): gt_render dominates at 68.6% of render time, the cost lever for cloud pricing. Boards refreshed over the full set (overview capped to 40 representative samples for legibility). Backup to gdrive in progress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, checkpoints, resume The 053b two-view diet was only half-wired into train_loop: `patch_prob` defaulted on, but `crop_view` did not, so the "sheet" half of the diet was the FULL uncropped render rather than the simulated user crop the 053 pilot exists to model. Expose both, defaulting crop_view=True, and print the per-root domain mix so a mixed-domain run is self-documenting. Also the durability wiring a 12-24h detached MPS run needs: - --save-every writes adapter + optimizer/scheduler state via a tmp+rename, so a kill mid-write cannot corrupt the checkpoint; - --resume continues from train_state.pt at the saved step (the run survives a session boundary instead of restarting from zero); - the loss curve is appended to train_log.jsonl as it goes — the end-of-run train_log.json alone loses the entire curve on an interrupt, which is exactly how the 040 adapters were lost; - RESUME.md breadcrumb rewritten next to the adapter at every save. eval_foundation gains the matching --no-crop-view switch and defaults to crop_view=True so the held-out set is scored in the same sheet geometry the model trained on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rness --snapshot-every keeps immortal adapter_step<N>.pt alongside the rolling adapter: with a 135-identity pool a long run can overfit, and the rolling checkpoint alone leaves no way to score an earlier point without retraining. real_photo_panels.py runs a trained adapter over the CTO's 15 untouched Pixel captures and lays the model's T / sigma_s / conf and a sigma_s-driven checker relight beside the frozen classical extractor's T / h on the SAME crop. Crop table (per-photo class, mark region and sheet rect) was verified by eye via --crops-only before any model ran. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… plots realpairs_consistency.py scores the reviewer's clause-1 criterion on the report-033 real pair corpus: ORB-register capture B onto A (the frozen 033 registration), then score BOTH the model and the classical extractor with register_pair.registered_t_agreement on the IDENTICAL registration and the IDENTICAL valid mask, so the comparison isolates the T estimator. Pool = 164 registrable cross-capture pairs / 79 products, excluding finished-product photos; the clear-textured-glass subset (42 pairs) is broken out because clause 3 asks specifically whether confidence catches clear-glass failures. Glass class for the classical route is mapped from brand/caution flags (documented heuristic — the catalog carries no class field); the model takes no class input, which is part of what is compared. plot_curves.py renders small multiples, one panel per channel: the channels differ by an order of magnitude and a shared axis would flatten all but one. T and sigma_s are titled as the two regression checks they are (040 severed gradient, 048 upward drift). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Real photos carry no authored gt_T, so the frozen family-2 instrument runs with the CLASSICAL T as reference and the model's T as test: how much of the fine structure the classical route resolves does the model still carry. Read against the same landmarks as the synthetic table (the 8-sigma blur control collapses retained-energy to 0.03), so a model drifting toward zero here is flattening real texture — clause 2 measured in the deployment domain rather than inferred from synthetic alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ix, preflight Banks the sections that do not depend on results: the 16 GB memory finding that forced bs1 (and the measurement showing the loader was never the bottleneck — a concurrent rclone was), the partial second-root restore, the domain mix with the sigma_s caveat, and the per-head gradient preflight table. Results sections marked TBD. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
323 held-out samples x 6 ISP conditions is 1938 forwards; --limit makes a bounded smoke run possible without committing to the full pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ide the worktree The 040 adapters were lost to a worktree-cleanup race. Every save now also lands in a plain local directory, best-effort (a failed mirror must never take down a 14-hour run). This is the PRIMARY hedge: unlike the Drive upload it keeps working when remote auth is down. The 055 run itself was already in flight when this landed, so it is covered by an equivalent external sync daemon; this makes the behaviour intrinsic for future runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…al-photo panels Training finished 17,999 steps in 16.6h. T fell 85% with all 721 logged values distinct (the 040 severed-gradient check passes on real data, not just in preflight); sigma_s read on SUPERVISED draws only fell 74% (the 048 upward drift is resolved) -- the naive average is misleading because the has_sigma_s gate zeroes 37% of bs1 steps. The 'inf band' is exactly one value: B=inf at step 1650, caused by one old-root sample whose gt_B.exr has 26 non-finite pixels and a max finite value of 65248 -- an fp16 EXR write overflowing the half-float ceiling. Harmless because L1-on-log1p backpropagates the SIGN of the difference, so the gradient stayed finite; the final adapter has 0 non-finite values across 1,727,432 params. The real-photo panels are the honest bad news: model T is a smooth desaturated field that loses the relief, the marker scribble and the swirl structure the classical route keeps, and confidence is nearly constant (0.471-0.552 across all 15 captures) including on the clear-glass sheet where T fails hardest. Part of that flattening is structural: since 040, T is supervised as a 64x64 latent and pixel-space T is never in the loss. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…val resolution to 512 The review's '145 screened cross-capture pairs across 64 products' does not reproduce from any predicate over the restored manifest, and should not: it is a historical snapshot of trunk 8c8877a, and the corpus was re-harvested afterwards (aggregate_033.json itself records 213/98 against today's 209/99). The pool used is 164 pairs / 79 products = registrable cross-capture excluding finished-product photos; 42 are clear-textured-glass, the subset clause 3 is about. Every predicate tried is tabulated so the choice is auditable. Pairs are scored at 512 rather than register_pair's 700 default: it is the resolution the model trained at (the UNet sees its 64x64 latent, not an 88x88 one) and it turns a ~7h run into ~1.5h. Both routes are measured at the same size so the comparison stays valid, with an explicit note that the row is not byte-comparable to pre-055 classical numbers taken at 700. Also adds --sample for stratified subsampling that keeps the clear-glass evidence whole. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sed consistency TWO CORRECTIONS. The pair pool was wrong. I read the restored manifest off the data disk rather than the canonical committed manifest_033.json, and applied none of the docs/REAL_PAIRS_DATASET.md 9.3 screens. With the full predicate -- cross_capture, not finished_product, neither image key contaminated, product not non_transmissive_mirror/multi_sheet_listing, and NOT (residual_mad < 15 AND inliers >= 200) -- the committed manifest yields 145 pairs / 64 products exactly. --pool screened is now the default; --pool broad keeps the unscreened 164/79 as a sensitivity row. The same-photo clause is the one that matters: near-duplicate crops posing as cross-capture pairs inflate the consistency of any method, and inflate it most for the smoothest one, so leaving them in biased the comparison toward the model. A plain band-limited MAE would have made the laundering worse, not better: two flat maps have ~0 high-pass residual each, so their difference is ~0 and a flattened route scores 'perfectly consistent'. band_agreement therefore reports the disagreement WITH the band energy and normalises by it, so flattening drives numerator and denominator to zero together and cannot buy a good score. Early reading: classical band energy 14.17 vs model 0.32, and model retained-energy vs classical 0.041 against a frozen flatten control of 0.03. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… control in-table ORB was dropping ~1 pair in 3 at 512 for inliers<12 -- a pure resolution artifact, since a 512-px downsample simply has fewer keypoints, while inference must stay at 512 (the resolution whose 64x64 latent the UNet trained on). Registration and inference are now decoupled: fit H at reg_size=700 where the features are, then map it onto the work grid by conjugation, H_work = S H_reg S^-1 with S = diag(s,s,1). 4/4 registered in smoke where 1 in 3 had failed; pairs that still fail at full res are genuinely unregistrable and are counted in n_skipped_unregistrable rather than silently dropped. Also scores the FROZEN 8-sigma flatten control on the same pairs and the same raw metric. Early numbers make the point better than prose: flatten control 26.91 vs classical 34.06 raw T-agreement -- deliberately destroying texture 'improves' cross-capture consistency by 21%. That is why clause 1 cannot be read alone, and it now sits in the table as evidence rather than as an argument. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ee-clause verdict eval_foundation was SIGKILLed partway through the 323-sample test split: it appended every prediction to a list and reduced at the end, which at 323 samples x 6 ISP conditions x ~12.6 MB of full-res maps is ~24 GB. It now streams grouped by identity -- per-sample metrics reduce to scalars immediately, and the one metric that genuinely needs two maps at once (family-1 cross-lighting invariance, which compares T within a (recipe,seed) group) is computed inside its own group. The statistics are unchanged by construction. Pair results, 140 of 145 screened pairs scored (5 unregistrable even at full res), 64 products, with the broad 161/79 pool reproducing every conclusion: clause 1 PASSES and not by cheating -- model 13.50 vs classical 38.05 raw T-agreement, better on 85% of pairs, while the 8-sigma flatten control only reaches 33.77. Destroying texture buys 11%; the model delivers 65%. The backbone's invariance prior is doing real work, and that is the pilot's positive finding. clause 2 FAILS decisively -- fine retained-energy 0.095 overall and 0.0094 on clear glass against classical 1.27 and a flatten control of 0.03. The model's T carries LESS fine-band energy (0.37) than a deliberate 8-sigma blur (0.66). Normalised, the consistency advantage inverts: band nmae 0.788 vs classical 0.710. clause 3 FAILS -- Spearman(conf,error) -0.069 overall, -0.131 on clear glass; right sign, useless magnitude. Cause is the 4%-of-steps supervision cadence found in the curves. Verdict DO-NOT-SCALE-YET, with the fix list enumerated and not implemented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ET verdict (lead finish) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The external review's prescribed experiment, executed and scored on its three-clause criterion. Clause 1 (registered consistency) PASSES genuinely — model 13.5 vs classical 38.0 MAE on 140 real pairs, far beyond the flatten control's 33.8. Clause 2 (texture) FAILS decisively — 9.5% retained fine energy (0.9% on clear glass), below the blur control. Clause 3 (confidence) FAILS — conf trained on 4% of steps via a logging-cadence coupling. Verdict: DO NOT SCALE YET, with a costed fix list (conf cadence decouple; the latent-T texture ceiling incl. a 056 learned-decoder framing; fp32 GT AOVs) and a concrete re-run gate. Bonus finding: zero degradation on the held-out phone preset — the 053 device-ISP training delivered.
🤖 Generated with Claude Code