Make v2 preprocessing scripts location-independent and add a runbook

[Choumingzhao]

Both preprocessing shell scripts required cd tessera_preprocessing before
they would find their sibling processors/binaries, and their output
subdirectories were hardcoded. Resolve SCRIPT_DIR so each can be invoked from
anywhere, expose the output subdirectories as env vars so the layout lines up
across steps, and add a runnable end-to-end runbook to the README.

• s1_s2_downloader.sh: resolve SCRIPT_DIR and invoke s1/s2_fast_processor.py
  (plus the existence checks) via "$SCRIPT_DIR/...". Expose S1_RAW_SUBDIR /
  S2_RAW_SUBDIR (defaults data_sar_raw / data_raw) so the raw-output layout is
  configurable and matches what the stacker reads.
• s1_s2_stacker.sh: resolve SCRIPT_DIR and invoke "$SCRIPT_DIR/s1_stack" /
  "$SCRIPT_DIR/s2_stack". Make BASE_DIR, S1_RAW_SUBDIR, S2_RAW_SUBDIR,
  PROCESSED_SUBDIR and DOWNSAMPLE_RATE env-var overridable, so the stacker
  picks up the downloader's output layout automatically and produces a
  step-numbered data_processed dir.
• stitch_tiled_representation.py: add --out_name (default
  stitched_representation) so the final .npy/.tif can be named after the
  region/year; convert_npy2tiff.py already reuses the npy basename for the .tif.
• README: add an "End-to-end runbook" section — one-time setup (venv, base deps
  with a fiona / gdal-config install note, PyTorch, v2 deps + Medium weights),
  a shared variable block, step-numbered commands (Step 0-5 → 0.roi /
  1.data_* / 2.data_processed / 3.retiled_d_pixel / 4.embeddings_v2 /
  5.result), and two one-click copy-paste blocks.

Co-Authored-By: GLM-5 noreply@z.ai

[Choumingzhao]

Summary

Running the v2 embedding pipeline end-to-end required a lot of manual
fiddling — cd tessera_preprocessing, editing hardcoded paths/values inside
several scripts, and re-deriving the directory layout for each step. This PR
reduces all of that to a few top-level variables (DATA_DIR, PYTHON_ENV,
BASENAME, YEAR, ROI_TIFF) so each step can be run from the repo root by
copy-paste, and surfaces parameters that were previously buried in main() as
proper CLI flags. It is split into two focused commits.

Commit 1 — CLI flags for the convert helpers

convert_npy2tiff.py and convert_shp_to_tiff.py previously required editing
hardcoded paths/values in main() before running. They now use argparse with
the same --snake_case convention as stitch_tiled_representation.py and
dpixel_retiler.py.

• convert_npy2tiff.py: --npy_path / --ref_tiff_path / --out_dir
  (required), --downsample_rate (default 1).
• convert_shp_to_tiff.py: --shp_path (required), --tiff_path (defaults to
  the shapefile's name/dir), --pixel_size (default 10), --force_crs
  (default: auto UTM via CRS.from_user_input).

Commit 2 — Location-independent scripts, configurable layout, and a runbook

Preprocessing scripts run from anywhere (no cd). Both
s1_s2_downloader.sh and s1_s2_stacker.sh now resolve SCRIPT_DIR and invoke
their sibling processors/binaries via "$SCRIPT_DIR/...", so they can be
launched from the repo root.

Output layout is now configurable and consistent across steps. The raw
output subdirs (data_sar_raw / data_raw), the processed subdir
(data_processed), BASE_DIR, and DOWNSAMPLE_RATE are exposed as
env-var-overridable defaults, so the stacker picks up the downloader's output
layout automatically.

stitch_tiled_representation.py gains --out_name so the final
.npy / .tif can be named after the region/year (<BASENAME>); the existing
convert_npy2tiff.py already reuses the npy basename for the .tif.

New "End-to-end runbook" in the README walks through the whole pipeline as
step-numbered commands with parameter explanations:

Step	Output dir
0 — Shapefile → ROI GeoTIFF	0.roi/
1 — Download S1/S2	1.data_sar_raw/, 1.data_raw/
2 — Stack	2.data_processed/
3 — Retile d_pixel	3.retiled_d_pixel/
4 — Inference (v2)	4.embeddings_v2/
5 — Stitch & convert	5.result/<BASENAME>.npy, .tif

It includes a one-time Setup block (venv → base deps → PyTorch → v2 deps +
Medium weights), a shared variable block, and two one-click copy-paste blocks
(Step 0+1, then Step 2→5). Step 1 (download) is kept isolated since it is
failure-prone and benefits from retries.

Known install gotcha

While following the tutorial, installing the base requirements.txt failed
while building fiona with a gdal-config: not found error. The fix is to
install the GDAL system packages and re-run pip:

sudo apt install gdal-bin libgdal-dev

All defaults in the shell scripts are generic placeholders (/absolute/path/to/your/...); no machine-specific paths are committed.

🤖 Co-authored with GLM-5.

[Choumingzhao]

fb8c898 — two runbook refinements**
1. Downloads default to overwrite=false (resume-friendly).
The downloader commands (Step 1 + Block A) now pass
S1_OVERWRITE=false / S2_OVERWRITE=false.
2. Non-download steps use the activated python directly.
Steps 0/2/3/4/5 and the one-click Block B now call python <script> instead of
"${PYTHON_ENV}" <script>. PYTHON_ENV is now confined to the downloader: it
prefixes commands with $PYTHON_ENV and has an -x existence check, so it
still needs an absolute interpreter path, whereas every other step runs after
the venv is activated in the Setup block, so python is enough. (PYTHON_ENV
stays in the variable block — annotated — plus the two downloader invocations.)

[FrankFeng-23]

thanks for the PR! the v2 codebase is still in its very early stage. do you manage to get it running?

[Choumingzhao]

thanks for the PR! the v2 codebase is still in its very early stage. do you manage to get it running?

yeah, following the README and got one year v2 embedding of my ROI. Haven't test multi-year yet.

This PR is intend to make it easier for people to run their ROI quickly. I'm testing multi-year now. Will let you know when finished.

[Choumingzhao]

Re-write runbook and make it easier to run v2 with only 5 configs.

DATA_DIR=/absolute/path/to/your/data_dir              # all outputs are written under here
ROI_SHP=/absolute/path/to/your/roi.shp
PYTHON_ENV=/absolute/path/to/python_env/bin/python    # run `which python` when you have activated venv
BASENAME=myregion                                     # final result file name (.npy and .tif)
YEAR=2025                                             # data year, range [2017-2025]

Test it and it works well.

[FrankFeng-23]

thank you! will review it later!

[Choumingzhao]

Commit: 542d314

1. Add a standalone runbook script, we can get the embeddings with MINIMAL configs now (tested multiple times and worked D); We can simply re-run the runbook if something wrong
2. Add a RESOLUTION config option in runbook. We can download with a lower resolution, which saves download(?) ¹ and process time (tested with resolution 40m multiple times and worked)
3. Fix a hard-coding resultion print statement (10m) in tessera_infer/convert_npy2tiff.py. Now resolution are retrieved from reference raster.

Following screenshots are 10m and 40m embeddings from same area.

Footnotes

1. it seems that actual downloaded files are still of original resolution (10m for S1 and main spectral band in S2). ↩