From e1fb88830150b0af243a863eb1b3f92913042f10 Mon Sep 17 00:00:00 2001 From: Donglai Wei Date: Wed, 2 Sep 2026 13:09:54 -0400 Subject: [PATCH] j0126 tutorial: fix voxel resolution, add a data-download step, split docs The volume is 9 x 9 x 20 nm (native FFN mip 0), not 10 nm isotropic. Verified: im_align_10nm.zarr/0 is byte-identical to rawdata_realigned mip 0 at the same coordinates on scattered blocks, and its voxel grid [5700, 10913, 10664] matches mip 0 exactly -- a 20 -> 10 nm z-resample would have doubled z. So: * resolution [10,10,10] -> [20,9,9] in both step-1 configs (inert for training, where it only feeds nnU-Net source_spacing, but it is what the configs document about the data); * the zero-shot config no longer instructs a resample that never happened and that the reference runs never used; * params.data.raw_10nm -> raw_em, with the misnomer documented. The data.test `name` stays im_align_10nm: it is baked into the artifact filenames step 2 reads. 2_abiss.yaml's resolution_xyz [10,10,10] is left alone: it is fail-closed checked against already-built precomputed layers, and error correction already uses the correct [20,9,9] (skeletonize.py NATIVE_RESOLUTION_ZYX_NM), so only the neuroglancer display scale is affected. 1_affinity_supervised.yaml now trains on a real 25/8 held-out split (sorted cube names, every 4th from index 2 to val), instead of validating on 3 cubes that were also in train. The released checkpoint and every number in the results table predate this split and are noted as such. The recipe itself is unchanged from base_banis+_zebrafinch_heavy: lr 1e-3, 200k steps, batch 8, patch [48,96,96], aug_em_neuron, from scratch. Docs: resource budget -> RESOURCE.md, staged storage cleanup -> CLEANUP.md, and a new Step 0 that downloads the labelled cubes from HuggingFace and the EM volume with the new scripts/download_precompute.py (source location as argument, mip 0, resumable, shardable). Smoke-tested: a downloaded crop is bit-identical to the local reference zarr. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/download_precompute.py | 158 ++++++++++++ .../neuron_j0126/1_affinity_supervised.yaml | 228 ++++++++++-------- .../neuron_j0126/1_affinity_zeroshot.yaml | 26 +- tutorials/neuron_j0126/CLEANUP.md | 61 +++++ tutorials/neuron_j0126/README.md | 99 ++++---- tutorials/neuron_j0126/RESOURCE.md | 14 ++ tutorials/neuron_j0126/params.yaml | 6 +- 7 files changed, 424 insertions(+), 168 deletions(-) create mode 100644 scripts/download_precompute.py create mode 100644 tutorials/neuron_j0126/CLEANUP.md create mode 100644 tutorials/neuron_j0126/RESOURCE.md diff --git a/scripts/download_precompute.py b/scripts/download_precompute.py new file mode 100644 index 00000000..03bd43b5 --- /dev/null +++ b/scripts/download_precompute.py @@ -0,0 +1,158 @@ +"""Download a Neuroglancer `precomputed` volume into a local zarr (ZYX). + +The source location is the only required argument, so this works for any public +precomputed layer, image or segmentation: + + # whole j0126 EM volume at mip 0 (9 x 9 x 20 nm) -- ~660 GB uint8, shard it + python scripts/download_precompute.py \ + gs://j0126-nature-methods-data/GgwKmcKgrcoNxJccKuGIzRnQqfit9hnfK1ctZzNbnuU/rawdata_realigned \ + --out /path/to/j0126_em.zarr --mip 0 --tile-xy 2048 --slab 64 + + # one 1008^3 test chunk instead of the whole volume (a few minutes, ~1 GB) + python scripts/download_precompute.py --out crop.zarr --mip 0 \ + --bbox 2900 3908 5000 6008 5000 6008 + +Point a config at the array inside the store, e.g. `image: /path/to/crop.zarr/main`. + +Notes carried over from the measured full-volume runs +(`dev/zebrafinch/download_ffn_gcs.py`, which this generalizes): + +* `--parallel` stays 1 by default. CloudVolume's multiprocessing path measured + ~10x SLOWER here, and combined with `--fill-missing` a failed worker returns + SILENT ZEROS. Shard the job instead of raising `--parallel`. +* `--fill-missing` is OFF by default so a gap in the source raises instead of + writing zeros that look like real EM downstream. +* Jobs are (z-slab x XY-tile) blocks. They are disjoint and chunk-aligned, so + several `--shard-id` processes can fill one store concurrently, and each shard + resumes from its own `.progress.` sidecar. +* Run `--init-only` once before launching a sharded array job, so the shards + never race on creating the array. +""" + +import argparse +import time +from pathlib import Path + +import numpy as np +import zarr +from cloudvolume import CloudVolume + + +def parse_args(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("source", help="precomputed location, e.g. gs://bucket/path/layer (precomputed:// optional)") + ap.add_argument("--out", required=True, help="output zarr store path") + ap.add_argument("--dataset", default="main", help="array name inside the store (default: main)") + ap.add_argument("--mip", type=int, default=0, help="source mip level (default: 0, full resolution)") + ap.add_argument( + "--bbox", + type=int, + nargs=6, + metavar=("Z0", "Z1", "Y0", "Y1", "X0", "X1"), + help="ZYX voxel bounds at the chosen mip; default is the whole volume", + ) + ap.add_argument("--slab", type=int, default=64, help="Z voxels per job (default: 64)") + ap.add_argument("--tile-xy", type=int, default=0, help="XY tile per job; 0 = whole XY plane per slab") + ap.add_argument("--parallel", type=int, default=1, help="CloudVolume workers (see module docstring)") + ap.add_argument("--fill-missing", action="store_true", help="return zeros for missing source chunks") + ap.add_argument("--init-only", action="store_true", help="create the zarr array and exit") + ap.add_argument("--shard-id", type=int, default=0) + ap.add_argument("--num-shards", type=int, default=1) + return ap.parse_args() + + +def main(): + args = parse_args() + source = args.source if "://" in args.source else f"precomputed://{args.source}" + + cv = CloudVolume( + source, + mip=args.mip, + parallel=args.parallel, + use_https=True, + progress=False, + fill_missing=args.fill_missing, + ) + size_x, size_y, size_z = (int(v) for v in cv.volume_size) + res_xyz = [int(v) for v in cv.resolution] + res_zyx = [res_xyz[2], res_xyz[1], res_xyz[0]] + + if args.bbox: + z0, z1, y0, y1, x0, x1 = args.bbox + else: + z0, z1, y0, y1, x0, x1 = 0, size_z, 0, size_y, 0, size_x + for lo, hi, limit, axis in ((z0, z1, size_z, "z"), (y0, y1, size_y, "y"), (x0, x1, size_x, "x")): + if not 0 <= lo < hi <= limit: + raise SystemExit(f"--bbox {axis} range [{lo}, {hi}) is outside the mip{args.mip} extent [0, {limit})") + shape = (z1 - z0, y1 - y0, x1 - x0) + + print( + f"{source}\n mip{args.mip}: volume (Z,Y,X)=({size_z},{size_y},{size_x}) " + f"res(zyx)={res_zyx} nm dtype={cv.dtype}\n" + f" writing (Z,Y,X)={shape} from z[{z0}:{z1}] y[{y0}:{y1}] x[{x0}:{x1}] -> {args.out}/{args.dataset}", + flush=True, + ) + + store = zarr.open(args.out, mode="a") + if args.dataset in store: + arr = store[args.dataset] + assert tuple(arr.shape) == shape, (arr.shape, shape) + else: + arr = store.create_array( + args.dataset, + shape=shape, + chunks=(min(args.slab, 64), 256, 256), + dtype=cv.dtype, + ) + arr.attrs["resolution_zyx_nm"] = res_zyx + arr.attrs["source"] = source + arr.attrs["mip"] = args.mip + arr.attrs["bbox_zyx"] = [z0, z1, y0, y1, x0, x1] + + if args.init_only: + print(f"initialized {args.out}/{args.dataset}", flush=True) + return + + tile = args.tile_xy or max(shape[1], shape[2]) + jobs = [ + (zs, ys, xs) + for zs in range(z0, z1, args.slab) + for ys in range(y0, y1, tile) + for xs in range(x0, x1, tile) + ] + mine = jobs[args.shard_id :: args.num_shards] + + suffix = "" if args.num_shards == 1 else f".{args.shard_id}" + prog = Path(f"{args.out}.progress{suffix}") + done = set(prog.read_text().split()) if prog.exists() else set() + print( + f"shard {args.shard_id}/{args.num_shards}: {len(mine)} of {len(jobs)} jobs " + f"(slab {args.slab}, tile {tile}); resuming: {len(done)} already done", + flush=True, + ) + + t0 = time.time() + n = 0 + for zs, ys, xs in mine: + key = f"{zs}_{ys}_{xs}" + if key in done: + continue + ze, ye, xe = min(zs + args.slab, z1), min(ys + tile, y1), min(xs + tile, x1) + block = cv[xs:xe, ys:ye, zs:ze] # (dx, dy, dz, channels) + block = np.asarray(block[..., 0]).transpose(2, 1, 0) # -> (dz, dy, dx) + arr[zs - z0 : ze - z0, ys - y0 : ye - y0, xs - x0 : xe - x0] = block + with open(prog, "a") as f: + f.write(f"{key}\n") + n += 1 + elapsed = time.time() - t0 + left = (len(mine) - len(done) - n) * elapsed / n / 60 + print( + f" z[{zs}:{ze}] y[{ys}:{ye}] x[{xs}:{xe}] done " + f"({n} this run, {elapsed:.0f}s, {elapsed / n:.0f}s/job, ~{left:.0f} min left)", + flush=True, + ) + print(f"DONE -> {args.out}/{args.dataset}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/tutorials/neuron_j0126/1_affinity_supervised.yaml b/tutorials/neuron_j0126/1_affinity_supervised.yaml index f135bbfa..47b944dc 100644 --- a/tutorials/neuron_j0126/1_affinity_supervised.yaml +++ b/tutorials/neuron_j0126/1_affinity_supervised.yaml @@ -1,77 +1,59 @@ -# j0126 step 1, VARIANT: affinity from the SUPERVISED from-scratch model. +# j0126 step 1, VARIANT: the SUPERVISED reference affinity. # -# Train it (200k steps, 4 GPUs, ~batch 8/GPU on the 33 dense GT cubes): -# -# python scripts/main.py --config tutorials/neuron_j0126/1_affinity_supervised.yaml --mode train -# -# then infer with the resulting checkpoint: -# -# python scripts/main.py --config tutorials/neuron_j0126/1_affinity_supervised.yaml --mode test \ -# --checkpoint outputs/1_affinity_supervised//checkpoints/step=00200000.ckpt -# -# The reference run used -# outputs/nisb_base_banis_plus_zebrafinch_heavy/20260726_114349/checkpoints/step=00200000.ckpt +# train: python scripts/main.py --config tutorials/neuron_j0126/1_affinity_supervised.yaml --mode train +# infer: python scripts/main.py --config tutorials/neuron_j0126/1_affinity_supervised.yaml --mode test \ +# --checkpoint /checkpoints/step=00200000.ckpt # # ############################################################################ # THIS VARIANT IS NOT GROUND-TRUTH-FREE. It is not part of the 3-step pipeline. # ############################################################################ -# -# The checkpoint is MedNeXt-L/k3 (62M) trained from scratch for 200k steps ON THE 33 -# ZEBRAFINCH DENSE GT CUBES (`tutorials/neuron_nisb/base_banis+_zebrafinch_heavy.yaml`, -# heavy `aug_em_neuron`). Those cubes are j0126 ground truth, so this model has seen -# labelled j0126 tissue and the README's headline claim -- "no step uses ground truth" -- -# does NOT hold for any run that starts here. Use it as the SUPERVISED UPPER REFERENCE -# that tells you what the GT-free path is giving up, not as a drop-in for `1_affinity_zeroshot.yaml`. -# -# (The 33 cubes come from a different alignment of the volume than the test skeletons, so -# this is not voxel-level test contamination. It is still supervision on the target domain.) -# -# The native-window full-volume affinity/decode is being produced separately. The prior -# whole-volume score 0.444376 came from this checkpoint inferred at [144,144,144]: its -# affinity path has the default suffix and is `arm0_win144`, not this config. Do not attach -# that number or its 0.941244 union-only oracle to the native-window artifact. -# -# Output: the checkpoint-derived test directory, with one float16 CZYX HDF5 per chunk under -# a store whose name ends `_win48x96x96.h5.chunks`. -# -# --------------------------------------------------------------------------- -# WHY THE WINDOW CHANGES, AND WHY IT MUST -# -# This model was trained at patch [48,96,96] (anisotropic -- near-isotropic in nm for -# 9x9x20 voxels), NOT at the 128^3 the pretrain model used. MedNeXt normalizes with -# `nn.GroupNorm(num_groups=C, num_channels=C)`, i.e. per-sample per-channel InstanceNorm -# with NO running statistics -# (lib/MedNeXt/nnunet_mednext/network_architecture/mednextv1/blocks.py:42). Normalization -# statistics are therefore computed over the *window's* spatial extent at every block, so -# THE FORWARD PASS IS WINDOW-SIZE DEPENDENT. Running this checkpoint at the inherited -# [144,144,144] would be a 3x Z / 1.5x XY extent mismatch and would invert the trained -# Z-thin anisotropy. Native-window affinities ranked better on the fixed local chunks; a -# whole-volume funlib/mt50 value must wait for the native store and decode to finish. -# -# Divisibility, which the [48,96,96] choice has to satisfy on two counts: -# * MedNeXt-L is 4-stage / stride-16: 48/16 = 3 and 96/16 = 6. -# * At the inherited `overlap: 0.5` the strides are (24,48,48); 1008 % 24 == 0 and -# 1008 % 48 == 0, so every chunk's window lattice has the same phase and chunk seams -# stay reproducible. (This is the invariant the "1008 = 14 * 72" comment in -# base_banis+_zebrafinch.yaml protects for the 144 window.) -# Total voxel-forwards per chunk land within 10% of the [144,144,144] run (24,863 small -# windows vs 3,375 large), so wall-clock is comparable despite the 7.4x window count. -# -# sw_batch_size 32: the pretrain model measures ~3.46 GB per batch unit at 144^3 (+0.77 GB -# fixed). A [48,96,96] window is 6.75x fewer voxels => ~0.51 GB/unit => 32 units ~= 17 GB, -# which fits both the 40 GB A100 (g003-g006) and the 49 GB L40S (g010-g018). Drop to 16 if -# it OOMs. -# -# BOTH window keys MUST live in YAML, never on the CLI: `inference.window` is copied into -# the runtime-owned `inference.sliding_window` alias (which the lazy/zarr path actually -# reads) by `sync_inference_runtime_aliases`, and the `default.inference` stage merge can -# clobber a pre-resolution CLI override. Original diagnosis in -# base_banis+_zebrafinch_zarr_b4.yaml; the same class of bug silently reverted an ROI -# override via runtime/sharding.py:57. -# -# Everything else -- input zarr, 1008^3 grid, halo 72, reflect padding, roi -# [5700,10913,10664], select_channel [0,1,2], scale_sigmoid, float16 -- is inherited -# unchanged, so these chunks are directly comparable in layout with `1_affinity_zeroshot.yaml`'s. +# MedNeXt-L/k3 (62M) trained from scratch for 200k steps on the j0126 dense GT +# cubes. It has seen labelled j0126 tissue, so the README's headline claim -- +# "no step uses ground truth" -- does NOT hold for any run that starts here. +# Use it as the SUPERVISED UPPER REFERENCE that says what the GT-free path is +# giving up. (The cubes come from a different alignment of the volume than the +# test skeletons, so this is target-domain supervision, not voxel-level test +# contamination.) +# +# DATA -- the PADDED cubes (`im_raw_4-32-32` / `seg_gt_4-32-32`), not the raw +# ones. The [4,32,32]-per-side pad is real EM context on the image side and -1 +# on the label side, so the loss skips the border and no mask volume is needed. +# Paths are listed explicitly rather than globbed: a bare `gt_*.h5` also matches +# the `gt_*_skeleton.h5` siblings and would pair 66 labels against 33 images. +# +# SPLIT -- 25 train / 8 val, disjoint. Rule: sort the 33 cube names, take every +# 4th starting at index 2 for val. That holds out 8 cubes including 2 of the 4 +# large 128x256x256 ones, so val is not systematically the small cubes. +# The released checkpoint (`pytc/j0126 affinity_scratch_48x96x96.ckpt`) and +# every number in the README results table PREDATE this split: that run +# trained on all 33 cubes and validated on a 3-cube subset that was also in +# train, so its val curve was not a generalization estimate. Retraining with +# this config gives an honest held-out curve and a slightly different model; +# it does not reproduce that checkpoint bit for bit. +# +# RESOLUTION -- 20 x 9 x 9 nm in ZYX (9 x 9 x 20 nm in XYZ), the native FFN +# mip-0 grid. The cubes, the inference volume and the evaluation skeletons all +# live on it. Nothing here is 10 nm isotropic despite the `im_align_10nm.zarr` +# name: that store is byte-for-byte equal to mip 0 on the same coordinates. +# Patch [48,96,96] is therefore 960 x 864 x 864 nm -- near-isotropic in physical +# space, which is the point of the anisotropic shape. MedNeXt-L is 4-stage / +# stride-16, and 48/16 = 3, 96/16 = 6, both integer. +# +# WINDOW -- infer at the trained [48,96,96]. MedNeXt normalizes with per-sample +# `GroupNorm(num_groups=C, num_channels=C)` and NO running statistics, so the +# forward pass depends on the window's spatial extent; the zero-shot +# [144,144,144] would invert the trained Z-thin anisotropy. Both window keys +# must live in YAML, never on the CLI: `inference.window` is copied into the +# runtime-owned `inference.sliding_window` alias, and the `default.inference` +# stage merge can clobber a pre-resolution CLI override. At the inherited +# `overlap: 0.5` the strides are (24,48,48) and 1008 % 24 == 1008 % 48 == 0, so +# every chunk's window lattice keeps the same phase and the seams stay +# reproducible. `sw_batch_size: 32` is ~17 GB; drop to 16 if it OOMs. +# +# Everything else -- input volume, 1008^3 grid, halo 72, reflect padding, ROI +# [5700,10913,10664], select_channel [0,1,2], scale_sigmoid, float16 -- is +# inherited from `1_affinity_zeroshot.yaml` unchanged, so these chunks are +# directly comparable in layout with the zero-shot ones. _base_: - 1_affinity_zeroshot.yaml @@ -79,9 +61,9 @@ _base_: experiment_name: neuron_j0126_affinity_arm0_96 description: >- Supervised reference affinity for j0126: from-scratch MedNeXt-L/k3 trained 200k steps on - the 33 zebrafinch dense GT cubes, inferred at its native [48,96,96] window. NOT - ground-truth-free -- provided as the upper reference against the zero-shot - 1_affinity_zeroshot.yaml, not as part of the GT-free pipeline. + the j0126 dense GT cubes (25 train / 8 held-out val), inferred at its native [48,96,96] + window. NOT ground-truth-free -- the upper reference against 1_affinity_zeroshot.yaml, + not a drop-in for it. save_path: ${params.paths.output_root}/affinity_arm0_96 @@ -89,10 +71,6 @@ default: system: num_gpus: 4 model: - # Anisotropic patch matching physical isotropy for 9x9x20 nm voxels: - # [48,96,96] = ~864 x 864 x 960 nm, versus the inherited 128^3 = ~1152 x 1152 - # x 2560 nm (Z-heavy, 5x more voxels). MedNeXt-L is 4-stage / stride-16: - # 48/16 = 3 and 96/16 = 6, both integer. input_size: [48, 96, 96] # From scratch. This is the explicit ablation against the init-from-NISB # finetune line, so do NOT set an external_weights_path here. @@ -109,14 +87,14 @@ default: use_preloaded_cache_val: true # 5x fewer voxels per patch than 128^3, so batch 8 fits where 2 was the # ceiling -- more random crops per step, which regularizes a 62M-param - # model on 33 cubes. + # model on 25 cubes. batch_size: 8 # Preflight requires model.input_size == data.dataloader.patch_size. patch_size: [48, 96, 96] augmentation: # DeepEM-matched heavy augmentation: defect_mutex + elastic + ±50% contrast - # + slice shift/drop + lost_section + motion blur + missing parts. With only - # 33 cubes this carries the regularization instead of shrinking the model. + # + slice shift/drop + lost_section + motion blur + missing parts. With so + # few cubes this carries the regularization instead of shrinking the model. profile: aug_em_neuron inference: window: @@ -125,7 +103,7 @@ default: decoding: # Chunked raw prediction naming still uses the decode suffix even though # decoding is disabled in step 1. Without this override the native-window - # run targets the complete 144-window store and silently skip-on-exists. + # run targets the complete 144-window store and silently skips on exists. save_suffix: zebrafinch_chunk_raw_grid1008_halo72_win48x96x96 train: @@ -139,28 +117,82 @@ train: t_max: 200000 data: train: - # All 33 dense GT cubes, as [4,32,32]-per-side padded images. The padding - # carries a -1 ignore border, so the loss skips it and no separate mask - # volume is needed. - # - # The `*[0-9]` glob is load-bearing: a bare `gt_*.h5` also matches the - # `gt_*_skeleton.h5` caches, which would pair 66 labels against 33 images. - path: "" - resolution: [10, 10, 10] - image: ${params.data.dense_images}/im_*[0-9].h5 - label: ${params.data.dense_labels}/gt_*[0-9].h5 - val: - # A 3-cube subset that is also in train. It drives EMA validation and - # checkpoint selection only -- no cube is held out, and it is NOT a - # generalization estimate. Real evaluation is step 3 against the 50 test - # skeletons. + # The 25 training cubes: sorted cube names minus every 4th from index 2. path: "" - resolution: [10, 10, 10] + resolution: [20, 9, 9] image: + - ${params.data.dense_images}/im_z255-383_y1407-1663_x1535-1791.h5 + - ${params.data.dense_images}/im_z2559-2687_y4991-5247_x4863-5119.h5 + - ${params.data.dense_images}/im_z2834-2984_y5311-5461_x5077-5227.h5 + - ${params.data.dense_images}/im_z2868-3018_y5744-5894_x5157-5307.h5 + - ${params.data.dense_images}/im_z2874-3024_y5707-5857_x5304-5454.h5 + - ${params.data.dense_images}/im_z3096-3246_y5954-6104_x5813-5963.h5 + - ${params.data.dense_images}/im_z3118-3268_y6538-6688_x6100-6250.h5 + - ${params.data.dense_images}/im_z3126-3276_y6857-7007_x5694-5844.h5 + - ${params.data.dense_images}/im_z3438-3588_y2775-2925_x3476-3626.h5 - ${params.data.dense_images}/im_z3456-3606_y3188-3338_x4043-4193.h5 - - ${params.data.dense_images}/im_z3914-4064_y9035-9185_x2573-2723.h5 + - ${params.data.dense_images}/im_z3492-3642_y7888-8038_x8374-8524.h5 + - ${params.data.dense_images}/im_z3596-3746_y3888-4038_x3661-3811.h5 + - ${params.data.dense_images}/im_z3604-3754_y4101-4251_x3493-3643.h5 + - ${params.data.dense_images}/im_z3608-3758_y3829-3979_x3423-3573.h5 + - ${params.data.dense_images}/im_z3710-3860_y8691-8841_x2889-3039.h5 + - ${params.data.dense_images}/im_z3722-3872_y4548-4698_x2879-3029.h5 + - ${params.data.dense_images}/im_z3734-3884_y4315-4465_x2209-2359.h5 + - ${params.data.dense_images}/im_z4102-4252_y6330-6480_x1899-2049.h5 + - ${params.data.dense_images}/im_z4312-4462_y9341-9491_x2419-2569.h5 + - ${params.data.dense_images}/im_z4440-4590_y7294-7444_x2350-2500.h5 + - ${params.data.dense_images}/im_z4905-5055_y928-1078_x1729-1879.h5 + - ${params.data.dense_images}/im_z4951-5101_y9415-9565_x2272-2422.h5 + - ${params.data.dense_images}/im_z5001-5151_y9426-9576_x2197-2347.h5 + - ${params.data.dense_images}/im_z5405-5555_y10490-10640_x3406-3556.h5 - ${params.data.dense_images}/im_z734-884_y9561-9711_x563-713.h5 label: + - ${params.data.dense_labels}/gt_z255-383_y1407-1663_x1535-1791.h5 + - ${params.data.dense_labels}/gt_z2559-2687_y4991-5247_x4863-5119.h5 + - ${params.data.dense_labels}/gt_z2834-2984_y5311-5461_x5077-5227.h5 + - ${params.data.dense_labels}/gt_z2868-3018_y5744-5894_x5157-5307.h5 + - ${params.data.dense_labels}/gt_z2874-3024_y5707-5857_x5304-5454.h5 + - ${params.data.dense_labels}/gt_z3096-3246_y5954-6104_x5813-5963.h5 + - ${params.data.dense_labels}/gt_z3118-3268_y6538-6688_x6100-6250.h5 + - ${params.data.dense_labels}/gt_z3126-3276_y6857-7007_x5694-5844.h5 + - ${params.data.dense_labels}/gt_z3438-3588_y2775-2925_x3476-3626.h5 - ${params.data.dense_labels}/gt_z3456-3606_y3188-3338_x4043-4193.h5 - - ${params.data.dense_labels}/gt_z3914-4064_y9035-9185_x2573-2723.h5 + - ${params.data.dense_labels}/gt_z3492-3642_y7888-8038_x8374-8524.h5 + - ${params.data.dense_labels}/gt_z3596-3746_y3888-4038_x3661-3811.h5 + - ${params.data.dense_labels}/gt_z3604-3754_y4101-4251_x3493-3643.h5 + - ${params.data.dense_labels}/gt_z3608-3758_y3829-3979_x3423-3573.h5 + - ${params.data.dense_labels}/gt_z3710-3860_y8691-8841_x2889-3039.h5 + - ${params.data.dense_labels}/gt_z3722-3872_y4548-4698_x2879-3029.h5 + - ${params.data.dense_labels}/gt_z3734-3884_y4315-4465_x2209-2359.h5 + - ${params.data.dense_labels}/gt_z4102-4252_y6330-6480_x1899-2049.h5 + - ${params.data.dense_labels}/gt_z4312-4462_y9341-9491_x2419-2569.h5 + - ${params.data.dense_labels}/gt_z4440-4590_y7294-7444_x2350-2500.h5 + - ${params.data.dense_labels}/gt_z4905-5055_y928-1078_x1729-1879.h5 + - ${params.data.dense_labels}/gt_z4951-5101_y9415-9565_x2272-2422.h5 + - ${params.data.dense_labels}/gt_z5001-5151_y9426-9576_x2197-2347.h5 + - ${params.data.dense_labels}/gt_z5405-5555_y10490-10640_x3406-3556.h5 - ${params.data.dense_labels}/gt_z734-884_y9561-9711_x563-713.h5 + val: + # The 8 held-out cubes. Disjoint from train, so val/loss is a real + # generalization signal for checkpoint selection. End-to-end evaluation is + # still step 3 against the 50 test skeletons. + path: "" + resolution: [20, 9, 9] + image: + - ${params.data.dense_images}/im_z2815-2943_y5631-5887_x4607-4863.h5 + - ${params.data.dense_images}/im_z2934-3084_y5115-5265_x5140-5290.h5 + - ${params.data.dense_images}/im_z3436-3586_y599-749_x2779-2929.h5 + - ${params.data.dense_images}/im_z3492-3642_y841-991_x381-531.h5 + - ${params.data.dense_images}/im_z3702-3852_y9605-9755_x2244-2394.h5 + - ${params.data.dense_images}/im_z3914-4064_y9035-9185_x2573-2723.h5 + - ${params.data.dense_images}/im_z4801-4951_y10154-10304_x1972-2122.h5 + - ${params.data.dense_images}/im_z5119-5247_y1023-1279_x1663-1919.h5 + label: + - ${params.data.dense_labels}/gt_z2815-2943_y5631-5887_x4607-4863.h5 + - ${params.data.dense_labels}/gt_z2934-3084_y5115-5265_x5140-5290.h5 + - ${params.data.dense_labels}/gt_z3436-3586_y599-749_x2779-2929.h5 + - ${params.data.dense_labels}/gt_z3492-3642_y841-991_x381-531.h5 + - ${params.data.dense_labels}/gt_z3702-3852_y9605-9755_x2244-2394.h5 + - ${params.data.dense_labels}/gt_z3914-4064_y9035-9185_x2573-2723.h5 + - ${params.data.dense_labels}/gt_z4801-4951_y10154-10304_x1972-2122.h5 + - ${params.data.dense_labels}/gt_z5119-5247_y1023-1279_x1663-1919.h5 diff --git a/tutorials/neuron_j0126/1_affinity_zeroshot.yaml b/tutorials/neuron_j0126/1_affinity_zeroshot.yaml index 2740bd00..4f20a015 100644 --- a/tutorials/neuron_j0126/1_affinity_zeroshot.yaml +++ b/tutorials/neuron_j0126/1_affinity_zeroshot.yaml @@ -23,7 +23,7 @@ _base_: experiment_name: neuron_j0126_affinity description: >- Zero-shot affinity prediction on j0126 with the NISB-trained banis+ MedNeXt-L/k3 - model. Chunked sliding-window inference over the 10 nm isotropic volume. + model. Chunked sliding-window inference over the native 9 x 9 x 20 nm volume. save_path: ${params.paths.output_root}/affinity @@ -65,7 +65,7 @@ default: # Each chunk reads an overlapping halo and writes only its core. halo: [72, 72, 72] axes: all - # True EM geometry (ZYX voxels at 10 nm). The im_align volume is rounded out + # True EM geometry (ZYX voxels at 20 x 9 x 9 nm). The im_align volume is rounded out # to 5700 x 12288 x 12288, so the last ~2 chunks in Y/X are pure zero padding. # Clipping the grid to this ROI skips them: 1014 -> 726 chunks. roi: [5700, 10913, 10664] @@ -100,25 +100,29 @@ test: affinity_mask_path: "" data: test: - # 10 nm isotropic resample of the public j0126 volume. To rebuild it from the - # public mirror (Januszewski et al. 2018, Nat Methods): + # The public j0126 EM volume at NATIVE mip 0. Mirror of Januszewski et al. + # 2018, Nat Methods: # # gs://j0126-nature-methods-data/GgwKmcKgrcoNxJccKuGIzRnQqfit9hnfK1ctZzNbnuU/ # rawdata_realigned uint8, 9 x 9 x 20 nm (x,y,z) <- what FFN segmented # ffn_segmentation uint64, same grid <- the 0.5390 reference # - # cloud-volume returns XYZC; transpose to ZYX and resample to 10 nm isotropic - # (scale 2 in Z, 1.8 in Y/X) before pointing `image` at the result. The model was - # trained at 9 nm isotropic, so the resolution match matters more than the exact - # grid; do not feed native 9x9x20 without resampling. + # Fetch it with `scripts/download_precompute.py --mip 0 --out .zarr`, + # which returns ZYX (see Step 0 of the README). Do NOT resample: mip 0 is the grid + # FFN published and the grid the evaluation skeletons index, and it is what the + # reference runs actually consumed. `im_align_10nm.zarr` is a MISNOMER kept for + # artifact-path compatibility -- it is byte-for-byte equal to mip 0 on the same + # coordinates (checked on scattered blocks), XY-padded out to 12288, and its voxels + # are 9 x 9 x 20 nm, not 10 nm isotropic. # - # Dense multiscale OME-Zarr, level 0 = full 10 nm resolution. `dataset_type: null` + # Dense multiscale OME-Zarr, level 0 = full resolution. `dataset_type: null` # selects the generic volume backend and the .zarr suffix picks the zarr reader. Do # NOT point this at the tiled-PNG pyramid: the lazy window re-decodes 4096x4096 # tiles per window and runs ~19 h/chunk with the GPU idle, versus ~30 min/chunk here. dataset_type: null - image: ${params.data.raw_10nm} + image: ${params.data.raw_em} label: null skeleton: null + # Legacy label: it is baked into the artifact filenames step 2 reads. name: im_align_10nm - resolution: [10, 10, 10] + resolution: [20, 9, 9] diff --git a/tutorials/neuron_j0126/CLEANUP.md b/tutorials/neuron_j0126/CLEANUP.md new file mode 100644 index 00000000..d6135fc9 --- /dev/null +++ b/tutorials/neuron_j0126/CLEANUP.md @@ -0,0 +1,61 @@ +# j0126 staged storage cleanup + +How to reclaim space while the whole-volume run is in flight. See +[RESOURCE.md](RESOURCE.md) for the totals these steps bring down, and +[README.md](README.md) for the workflow. + +Cleanup trades storage for restartability. Run each block only after the named downstream artifact exists. Replace the first path with the resolved `params.paths.output_root`; the guards refuse an empty path, `/`, or a run without the expected output markers. + +After EC `sizes` has written its aggregated inventory and ABISS has completed, remove the ABISS affinity copy, watershed, chunk map, resume scratch, and run workspace. This normally recovers several TiB while preserving the final ABISS segmentation and its parameter file: + +```bash +J0126_OUTPUT_ROOT="/absolute/path/to/outputs/neuron_j0126" +ABISS_ROOT="$J0126_OUTPUT_ROOT/abiss" +EC_ROOT="$J0126_OUTPUT_ROOT/error_correction_v7" + +case "$J0126_OUTPUT_ROOT" in ""|"/") exit 2;; esac +test -f "$ABISS_ROOT/precomputed/seg/info" || exit 2 +test -s "$EC_ROOT/segment_sizes.data" || exit 2 + +rm -r -- \ + "$ABISS_ROOT/precomputed/affinity" \ + "$ABISS_ROOT/precomputed/ws" \ + "$ABISS_ROOT/chunkmap" \ + "$ABISS_ROOT/scratch" \ + "$ABISS_ROOT/run" +``` + +After `skeletons`, `contact_graph`, and `junction_features` complete, their per-chunk caches are redundant. Removing them preserves the aggregated morphology, contact graph, and junction features used by the resolver: + +```bash +J0126_OUTPUT_ROOT="/absolute/path/to/outputs/neuron_j0126" +EC_ROOT="$J0126_OUTPUT_ROOT/error_correction_v7" + +case "$J0126_OUTPUT_ROOT" in ""|"/") exit 2;; esac +test -s "$EC_ROOT/segment_skeleton_graph.h5" || exit 2 +test -s "$EC_ROOT/contact_graph.npz" || exit 2 +test -s "$EC_ROOT/junction_features_raw.npz" || exit 2 + +rm -r -- \ + "$EC_ROOT/skeleton_chunks" \ + "$EC_ROOT/contact_chunks" \ + "$EC_ROOT/skeleton_cache" +``` + +The original 2.5–3 TiB affinity store is needed through the EC `contacts` stage, but not after the final output verifies. At that point it can be archived or deleted; set the exact store explicitly rather than deleting the whole affinity output directory: + +```bash +J0126_OUTPUT_ROOT="/absolute/path/to/outputs/neuron_j0126" +EC_ROOT="$J0126_OUTPUT_ROOT/error_correction_v7" +AFFINITY_STORE="/absolute/path/to/the/affinity.h5.chunks" + +case "$J0126_OUTPUT_ROOT" in ""|"/") exit 2;; esac +case "$AFFINITY_STORE" in ""|"/") exit 2;; esac +test -f "$EC_ROOT/error_correction_manifest.json" || exit 2 +find "$AFFINITY_STORE" -maxdepth 1 -name 'chunk_z*_y*_x*.h5' -print -quit \ + | grep -q . || exit 2 + +rm -r -- "$AFFINITY_STORE" +``` + +Keep `precomputed/seg`, `error_correction_manifest.json`, `segment_sizes.data`, the resolver reports, and `resolver/v7/frozen_junction_merges.{npz,json}`. Those are the compact result and provenance record. Also keep the source affinity if exact contact regeneration matters more than storage. diff --git a/tutorials/neuron_j0126/README.md b/tutorials/neuron_j0126/README.md index ae97c1b2..e8040334 100644 --- a/tutorials/neuron_j0126/README.md +++ b/tutorials/neuron_j0126/README.md @@ -1,6 +1,6 @@ # j0126: conservative segmentation, morphology-based reconnecting -This tutorial turns a 10 nm j0126 EM volume into a neuron segmentation in three steps: +This tutorial turns the j0126 EM volume (9 x 9 x 20 nm) into a neuron segmentation in three steps: 1. Predict voxel affinities. 2. Decode them conservatively with ABISS, preferring splits to false merges. @@ -29,75 +29,54 @@ The table separates the affinity source, conservative decoder, and optional corr On the synthetic affinity the nucleus certificate is **inert**: its scan finds 0 multi-nucleus watershed objects, so it publishes zero repairs and that row is also the exclusion-mask baseline. Whether the certificate has anything to correct is a property of the watershed, not of the nucleus mask — the scratch affinity fuses 8 soma pairs at the watershed stage and this one fuses none. An independent replay of the same affinity scores 0.383 at the same tolerance, so the no-op is confirmed rather than assumed. The synthetic row trails scratch on NERL and VOI split; it is a zero-shot transfer result, not a tuned one. -## Resource budget +## Step 0 — get the data -These are planning figures for the complete 5,700×10,913×10,664 voxel volume, not guarantees. They assume the configured 726 affinity chunks, a 48 GB-class GPU, a parallel filesystem, and no queue time. With the staged cleanup below, expect a **6–7 TiB peak** and reserve **8 TiB**. Keeping every intermediate or a second affinity arm can require 10 TiB. After final verification, retaining only the result and audit artifacts should take well below 1 TiB; retaining the source affinity raises that to roughly 3–4 TiB. +Two inputs, and the first one is only needed if you retrain the supervised +reference affinity in step 1. -| Step | Compute specification | Estimated wall time | Storage while running | -|---|---|---:|---:| -| 1. Affinity | 1 GPU with at least 48 GB per shard; 726 independent shards | ~30 min/shard; ~6 h at 64 GPUs, ~5 h at 80 GPUs | 2.5–3 TiB for the float16 chunk store | -| 2. ABISS | Layer-aware CPU fleet; 16 CPU / 130 GB nodes for atomic chunks, then 1–8 CPU workers with 40–100 GB per composite chunk | 3.75 h measured on 40×16 CPU workers; ~1.9 h projected with the multi-node layout below | input affinity plus ~0.7 TiB resumable scratch, a precomputed affinity layer, and ~45 GB final segmentation; budget 4 TiB additional | -| 3. Error correction | 80 array tasks, 8 CPU workers/task and 64 GB/task; reductions run serially | 6–12 h estimate; this has not yet been benchmarked end-to-end | reuse steps 1–2 inputs; reserve 0.5–1 TiB for skeleton, contact, and output artifacts | - -The step-1 timing is measured from the chunked Zarr input path. On a 40 GB GPU, lower `sw_batch_size` from 12 before running; that increases the per-chunk time. Reading tiled PNGs instead can take roughly 19 h **per chunk** and is not a usable production path. ABISS scratch and `work/` artifacts are only needed for resume; retain the final precomputed segmentation, parameter file, and manifests after recording the result, then reclaim scratch space. - -### Staged storage cleanup - -Cleanup trades storage for restartability. Run each block only after the named downstream artifact exists. Replace the first path with the resolved `params.paths.output_root`; the guards refuse an empty path, `/`, or a run without the expected output markers. - -After EC `sizes` has written its aggregated inventory and ABISS has completed, remove the ABISS affinity copy, watershed, chunk map, resume scratch, and run workspace. This normally recovers several TiB while preserving the final ABISS segmentation and its parameter file: +**Labelled cubes** (395 MB) — 33 densely labelled subvolumes: ```bash -J0126_OUTPUT_ROOT="/absolute/path/to/outputs/neuron_j0126" -ABISS_ROOT="$J0126_OUTPUT_ROOT/abiss" -EC_ROOT="$J0126_OUTPUT_ROOT/error_correction_v7" - -case "$J0126_OUTPUT_ROOT" in ""|"/") exit 2;; esac -test -f "$ABISS_ROOT/precomputed/seg/info" || exit 2 -test -s "$EC_ROOT/segment_sizes.data" || exit 2 - -rm -r -- \ - "$ABISS_ROOT/precomputed/affinity" \ - "$ABISS_ROOT/precomputed/ws" \ - "$ABISS_ROOT/chunkmap" \ - "$ABISS_ROOT/scratch" \ - "$ABISS_ROOT/run" +wget https://huggingface.co/datasets/pytc/zebrafinch-j0126/resolve/main/j0126-train-33vol.zip +unzip j0126-train-33vol.zip -d /train/ ``` -After `skeletons`, `contact_graph`, and `junction_features` complete, their per-chunk caches are redundant. Removing them preserves the aggregated morphology, contact graph, and junction features used by the resolver: +That gives `im_raw/` + `seg_gt/` and the padded pair `im_raw_4-32-32/` + +`seg_gt_4-32-32/`, which is what [1_affinity_supervised.yaml](1_affinity_supervised.yaml) +reads: the padding is real EM context on the image side and `-1` on the label +side, so the loss ignores the border and no mask volume is needed. + +**EM volume** — the public FFN mirror (Januszewski et al. 2018), uint8 at +9 × 9 × 20 nm (x, y, z), i.e. `[20, 9, 9]` in the ZYX order the configs use: -```bash -J0126_OUTPUT_ROOT="/absolute/path/to/outputs/neuron_j0126" -EC_ROOT="$J0126_OUTPUT_ROOT/error_correction_v7" - -case "$J0126_OUTPUT_ROOT" in ""|"/") exit 2;; esac -test -s "$EC_ROOT/segment_skeleton_graph.h5" || exit 2 -test -s "$EC_ROOT/contact_graph.npz" || exit 2 -test -s "$EC_ROOT/junction_features_raw.npz" || exit 2 - -rm -r -- \ - "$EC_ROOT/skeleton_chunks" \ - "$EC_ROOT/contact_chunks" \ - "$EC_ROOT/skeleton_cache" +``` +gs://j0126-nature-methods-data/GgwKmcKgrcoNxJccKuGIzRnQqfit9hnfK1ctZzNbnuU/rawdata_realigned ``` -The original 2.5–3 TiB affinity store is needed through the EC `contacts` stage, but not after the final output verifies. At that point it can be archived or deleted; set the exact store explicitly rather than deleting the whole affinity output directory: +Running one or two chunks? Point the config straight at the bucket and skip the +copy. For a whole-volume run, or anything you will read more than a few times, +download mip 0 once into a local zarr: ```bash -J0126_OUTPUT_ROOT="/absolute/path/to/outputs/neuron_j0126" -EC_ROOT="$J0126_OUTPUT_ROOT/error_correction_v7" -AFFINITY_STORE="/absolute/path/to/the/affinity.h5.chunks" - -case "$J0126_OUTPUT_ROOT" in ""|"/") exit 2;; esac -case "$AFFINITY_STORE" in ""|"/") exit 2;; esac -test -f "$EC_ROOT/error_correction_manifest.json" || exit 2 -find "$AFFINITY_STORE" -maxdepth 1 -name 'chunk_z*_y*_x*.h5' -print -quit \ - | grep -q . || exit 2 - -rm -r -- "$AFFINITY_STORE" +# whole volume: 5700 x 10913 x 10664 uint8 = ~660 GB; shard it across an array job +# (the 5.3 TB uint64 FFN segmentation layer took ~2 h at 8 shards, so this is less) +python scripts/download_precompute.py \ + gs://j0126-nature-methods-data/GgwKmcKgrcoNxJccKuGIzRnQqfit9hnfK1ctZzNbnuU/rawdata_realigned \ + --out /j0126_em.zarr --mip 0 --tile-xy 2048 --slab 64 \ + --shard-id "$SLURM_ARRAY_TASK_ID" --num-shards 8 + +# or one 1008^3 crop for a smoke test (~1 GB) +python scripts/download_precompute.py \ + --out /tmp/j0126_crop.zarr --mip 0 --bbox 2016 3024 5040 6048 5040 6048 ``` -Keep `precomputed/seg`, `error_correction_manifest.json`, `segment_sizes.data`, the resolver reports, and `resolver/v7/frozen_junction_merges.{npz,json}`. Those are the compact result and provenance record. Also keep the source affinity if exact contact regeneration matters more than storage. +Then point `params.data.raw_em` at the array, e.g. `/j0126_em.zarr/main`. + +Take mip 0 and do not resample. It is the grid FFN published, the grid the +evaluation skeletons index, and the grid the reference affinity run actually +used — the local `im_align_10nm.zarr` is byte-for-byte equal to mip 0 on the +same coordinates despite its name, so 9 × 9 × 20 nm anisotropic voxels are what +every number in the results table was produced on. ## Before running @@ -105,6 +84,8 @@ Edit **only** [params.yaml](params.yaml). It contains the repository checkout, d The zero-shot affinity path needs a NISB-trained checkpoint. The supervised affinity YAML is included as a target-domain reference only: it uses j0126 dense labels, so it is not part of the zero-shot pipeline; its trained checkpoint can be downloaded instead of retrained (see [Step 1](#supervised-reference-affinity)). +Whole-volume planning figures live in [RESOURCE.md](RESOURCE.md), and the staged storage cleanup that keeps the peak down is in [CLEANUP.md](CLEANUP.md). + ## Step 1 — affinity prediction Run zero-shot inference with an NISB checkpoint: @@ -126,7 +107,7 @@ The output is chunked float16, three-channel affinity under `output_root/affinit ### Supervised reference affinity -`1_affinity_supervised.yaml` is the target-domain reference: MedNeXt-L/k3 trained from scratch for 200k steps on the 33 j0126 dense-GT cubes. It has seen labelled j0126 tissue, so any run that starts here is not ground-truth-free. +`1_affinity_supervised.yaml` is the target-domain reference: MedNeXt-L/k3 trained from scratch for 200k steps on the j0126 dense-GT cubes, 25 for training and 8 held out for validation. It has seen labelled j0126 tissue, so any run that starts here is not ground-truth-free. Training it costs roughly four GPU-days. Download the reference checkpoint instead: @@ -137,6 +118,8 @@ python scripts/main.py --config tutorials/neuron_j0126/1_affinity_supervised.yam --mode test --checkpoint ckpt/affinity_scratch_48x96x96.ckpt ``` +The released checkpoint predates the held-out split: it trained on all 33 cubes and validated on a 3-cube subset that was also in training, so its validation curve was not a generalization estimate. Every number in the results table comes from that checkpoint. Retraining with the current YAML gives an honest held-out curve and a slightly different model — it does not reproduce the released weights bit for bit. + It must be inferred at its native `[48, 96, 96]` window, which the YAML already sets: MedNeXt normalizes without running statistics, so the forward pass depends on the window extent, and the zero-shot config's `[144, 144, 144]` would invert the trained Z-thin anisotropy. Its affinity lands under `output_root/affinity_arm0_96`, so step 2's `source_affinity_h5` has to be repointed there. ## Step 2 — conservative ABISS decode diff --git a/tutorials/neuron_j0126/RESOURCE.md b/tutorials/neuron_j0126/RESOURCE.md new file mode 100644 index 00000000..3461ea4f --- /dev/null +++ b/tutorials/neuron_j0126/RESOURCE.md @@ -0,0 +1,14 @@ +# j0126 resource budget + +Planning figures for the whole-volume run. See [README.md](README.md) for the +workflow itself and [CLEANUP.md](CLEANUP.md) for reclaiming space as it runs. + +These are planning figures for the complete 5,700×10,913×10,664 voxel volume, not guarantees. They assume the configured 726 affinity chunks, a 48 GB-class GPU, a parallel filesystem, and no queue time. With the [staged cleanup](CLEANUP.md), expect a **6–7 TiB peak** and reserve **8 TiB**. Keeping every intermediate or a second affinity arm can require 10 TiB. After final verification, retaining only the result and audit artifacts should take well below 1 TiB; retaining the source affinity raises that to roughly 3–4 TiB. + +| Step | Compute specification | Estimated wall time | Storage while running | +|---|---|---:|---:| +| 1. Affinity | 1 GPU with at least 48 GB per shard; 726 independent shards | ~30 min/shard; ~6 h at 64 GPUs, ~5 h at 80 GPUs | 2.5–3 TiB for the float16 chunk store | +| 2. ABISS | Layer-aware CPU fleet; 16 CPU / 130 GB nodes for atomic chunks, then 1–8 CPU workers with 40–100 GB per composite chunk | 3.75 h measured on 40×16 CPU workers; ~1.9 h projected with the per-layer multi-node layout described in [Step 2](README.md#step-2--conservative-abiss-decode) | input affinity plus ~0.7 TiB resumable scratch, a precomputed affinity layer, and ~45 GB final segmentation; budget 4 TiB additional | +| 3. Error correction | 80 array tasks, 8 CPU workers/task and 64 GB/task; reductions run serially | 6–12 h estimate; this has not yet been benchmarked end-to-end | reuse steps 1–2 inputs; reserve 0.5–1 TiB for skeleton, contact, and output artifacts | + +The step-1 timing is measured from the chunked Zarr input path. On a 40 GB GPU, lower `sw_batch_size` from 12 before running; that increases the per-chunk time. Reading tiled PNGs instead can take roughly 19 h **per chunk** and is not a usable production path. ABISS scratch and `work/` artifacts are only needed for resume; retain the final precomputed segmentation, parameter file, and manifests after recording the result, then reclaim scratch space. diff --git a/tutorials/neuron_j0126/params.yaml b/tutorials/neuron_j0126/params.yaml index afc1cc9f..fe1bb825 100644 --- a/tutorials/neuron_j0126/params.yaml +++ b/tutorials/neuron_j0126/params.yaml @@ -16,6 +16,10 @@ params: dataset_root: /projects/weilab/dataset/zebrafinch data: - raw_10nm: ${params.paths.dataset_root}/im_align_10nm.zarr/0 + # Native FFN mip 0: 9 x 9 x 20 nm (x,y,z) = [20, 9, 9] in ZYX. The + # `im_align_10nm.zarr` name is historical -- the store is byte-for-byte + # equal to mip 0 on the same coordinates, not a 10 nm isotropic resample. + # Any ZYX zarr from `scripts/download_precompute.py --mip 0` works here. + raw_em: ${params.paths.dataset_root}/im_align_10nm.zarr/0 dense_images: ${params.paths.dataset_root}/train/im_raw_4-32-32 dense_labels: ${params.paths.dataset_root}/train/seg_gt_4-32-32