Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions .claude/PHYSICS-CORRECTIONS.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,17 @@
# `ddr_match` — the two physics paths

`params.ddr_match: bool` (default **true**) selects which forward chain runs.
`true` reproduces DDR bit-for-bit and keeps `compare_ddr_sandbox` an ABSOLUTE
MATCH (invariant 1). `false` enables the corrected physics.
**DEPRECATED (2026-08-19): default flipped to `false`.** DDR ported these
corrections in DeepGroundwater/ddr#192 as a clean break (no flag), so the
corrected physics is now what BOTH implementations run by default, the sandbox
fixture is regenerated from post-#192 DDR, and `compare_ddr_sandbox` holds its
ABSOLUTE MATCH on the corrected path (verified 2026-08-19: max abs 1.5e-5 m³/s,
f32 floor). Setting `ddr_match: true` emits a deprecation WARN; the legacy path
remains only to reproduce pre-#192 results and for `use_cuda_graphs: true`
(whose captured kernel hardcodes the legacy 5/3 celerity), and will be removed.

`params.ddr_match: bool` selects which forward chain runs.
`true` reproduces PRE-#192 DDR bit-for-bit. `false` (default) enables the
corrected physics.

```
forward_chain_inner (src/routing/mmc_op.rs)
Expand Down
29 changes: 29 additions & 0 deletions .claude/skills/ddrs-dev/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ Full commands and rationale in `references/testing.md`.
| `src/nn/`, `Cargo.toml` rskan tag | **B** | 4-test KAN fixture sweep, then Tier A |
| `src/config.rs`, `src/training/`, other `src/` | **C** | `cargo test --lib`, `cargo test`, `compare_ddr_sandbox` |
| `config/**/*.yaml` only | **D** | `ddrs plan --config … --workspace …` exits 0, no drift |
| `examples/juniata/**` (bundle, config, README) | — | `cargo test --test juniata_bundle` (never skips — bundle is committed), and `ddrs --config examples/juniata/ddrs.yaml plan` exits 0 from the repo root |
| Plotting / analysis scripts only | — | no gate |
| `epochs`, `learning_rate`, `batch_size`, loss weights within documented ranges | — | no gate |

Expand Down Expand Up @@ -151,6 +152,34 @@ warm start: no <path>.mpk — Adam starts cold

There is no `"precip loading"` string — two retired skills told you to grep for it.

## Juniata single-catchment sample (`examples/juniata/`)

The fastest full end-to-end exercise of the CLI, and the mirror of DDR's
`examples/juniata` (DeepGroundwater/ddr PR #193): one gauge (USGS 01567000,
8,657 km², 213 reaches), 8.9 MB committed bundle, no external stores or CUDA.
Run **from the repo root** (data paths in the config are repo-root-relative);
the workspace intentionally lands beside the config at `examples/juniata/.ddrs/`
(gitignored) — the one sanctioned exception to fact 2 above.

```bash
target/release/ddrs --config examples/juniata/ddrs.yaml plan
target/release/ddrs --config examples/juniata/ddrs.yaml run --workflow train-and-test --backend cpu
```

Verified 2026-08-19: `plan` baseline NSE 0.695 / KGE 0.819 (matches DDR's
Python readers to rounding — a live cross-implementation check); 30-epoch CPU
train-and-test finishes in ~21 s at routed NSE 0.790 / KGE 0.881 vs DDR-Python's
0.784 / 0.877 (residual = window-sampling RNG streams only; exact match is
impossible by construction; ddrs 4-seed spread NSE 0.790–0.800). Both examples
run the corrected physics — since 2026-08-19 `ddr_match` is DEPRECATED and
defaults to `false`, matching DDR post-#192. (Historical: with legacy physics
this gauge scored 0.840 / 0.913 — *better* here, but not the same model.)
Two deviations from DDR's bundle: `data/statistics/*.json` is **committed**
(ddrs never recomputes statistics), and `.gitignore` carries
`!examples/juniata/ddrs.yaml` so the example config survives the global
`ddrs.yaml` ignore. Regenerate the bundle in the ddr repo
(`extract_bundle.py`), then re-copy `data/` plus the generated statistics JSON.

## Maintenance

This skill and `ddrs-eval-plots` are the only two skills in this repo. When a run,
Expand Down
10 changes: 6 additions & 4 deletions .claude/skills/ddrs-dev/references/build-and-env.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,12 @@ Regenerate with:
```bash
cd ~/projects/ddr && uv run python ~/projects/ddrs/scripts/export_ddr_sandbox.py
```
> **Only the desktop's `~/projects/ddr` working tree is a valid reference.** It
> carries unpushed `geometry/trapezoidal.py` work. A fixture regenerated from a
> clean DDR clone diverges ~1% (max abs ≈ 0.55 m³/s) at *every* ddrs commit —
> that is a wrong reference, not a port bug. Details:
> **Any DDR checkout at or past DeepGroundwater/ddr#192 is a valid reference**
> (2026-08-19: fixture regenerated from post-#192 master; comparison matches at
> the f32 floor on the corrected physics, which is now ddrs's default —
> `ddr_match` is deprecated). The old rule — only the desktop's working tree
> with unpushed `geometry/trapezoidal.py` work — applied before #192 landed;
> a PRE-#192 clean clone still diverges ~1% (max abs ≈ 0.55 m³/s). Details:
> `docs/reference/ddr-comparison.md` §Regenerating fixtures.

**KAN parity** — `tests/fixtures/` (tracked, unlike `/fixtures/`), loaded behind
Expand Down
4 changes: 3 additions & 1 deletion .claude/skills/ddrs-dev/references/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,8 @@ Ten production `input_var_names`: `SoilGrids1km_clay`, `aridity`, `meanelevation
| Key | Default | Notes |
|---|---|---|
| `sparse_solver` | `cpu` | On a non-CUDA backend, `cuda` **silently WARN-falls-back** to cpu. An unrecognized value **panics** |
| `use_cuda_graphs` | **false** | `config/merit_training.yaml` *sets* true — that is a YAML value, not the code default |
| `use_cuda_graphs` | false | Requires the DEPRECATED `ddr_match: true` (the captured kernel hardcodes the legacy 5/3 celerity); rejected alongside the corrected-physics default. `config/merit_training.yaml` set it true until 2026-08-19 |
| `ddr_match` | **false** (since 2026-08-19) | DEPRECATED. `true` = legacy pre-#192 DDR physics (5/3 celerity, X ≡ 0.3, upstream-cols readout) — parses with a WARN, kept only for pre-#192 reproduction and CUDA graphs. DDR itself runs the corrected physics since DeepGroundwater/ddr#192. See `.claude/PHYSICS-CORRECTIONS.md` |
| `use_leakance` | false | |
| `leakance_losing_only` | **true** | Clamps `head = max(0, depth − d_gw)`, so gaining reaches produce `zeta ≡ 0` |
| `leakance_impervious_threshold` | 0.7 | Masks reaches whose `corridor_impervious` is **`>`** this value (not `≥`) |
Expand Down Expand Up @@ -177,6 +178,7 @@ Four validators run at `Config::from_yaml_file`, plus one at dataset open.
| | neither adjacency nor fabric | `"adjacency sources are missing"` |
| | `geospatial_fabric_layer` on a non-gpkg | `"geospatial_fabric_layer"` + `".gpkg"` |
| `validate_leakance` | `use_leakance` + `use_cuda_graphs` | both key names |
| `validate_ddr_match` | `use_cuda_graphs: true` without the deprecated `ddr_match: true` | `"use_cuda_graphs: true` requires the DEPRECATED `ddr_match: true"` |
| `validate_disagg_pretrained` | `freeze: true` without `pretrained_checkpoint` | `"freeze: true requires pretrained_checkpoint"` |
| `validate_grad_accum` | `grad_accum_steps: 0` | `"grad_accum_steps: 0"` |
| | `use_grad_accum: true` with steps < 2 | `"requires grad_accum_steps: N with N >= 2"` |
Expand Down
8 changes: 5 additions & 3 deletions .claude/skills/ddrs-dev/references/traps.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,11 @@ worktree, silently replacing the worktree build.
4. `cargo test --test sparse_gradcheck`. **Also failing ⇒ the algorithm changed.
V1-only ⇒ kernel ordering or arithmetic fusion.**

**Wrong-reference failure mode:** a fixture regenerated from a clean DDR clone
diverges ~1% (max abs ≈ 0.55 m³/s) at *every* ddrs commit. Only the desktop's
`~/projects/ddr` working tree is valid — see `build-and-env.md`.
**Wrong-reference failure mode:** a fixture regenerated from a PRE-#192 DDR
clone diverges ~1% (max abs ≈ 0.55 m³/s) at *every* ddrs commit. Since
2026-08-19 any DDR checkout at or past DeepGroundwater/ddr#192 is a valid
reference (fixture + comparison both on corrected physics) — see
`build-and-env.md`.

## T3 — CUDA graphs mask NaN

Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,6 @@ __pycache__/
.ddrs/
.ddrs-synthetic-n-*/
ddrs.yaml
# …but the Juniata sample's config is a committed example, not a personal
# workspace config (its .ddrs/ workspace is still ignored by the rule above).
!examples/juniata/ddrs.yaml
11 changes: 7 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,13 @@ overview is in `~/projects/ddr/CLAUDE.md`.
fixture is generated by `scripts/export_ddr_sandbox.py` run under DDR's
`uv` venv. Re-run this benchmark after every change to `src/routing/`,
`src/geometry.rs`, or `src/sparse/`.
**Caveat (2026-06-06):** the reference DDR state lives only in the
desktop's `~/projects/ddr` working tree (unpushed `geometry/trapezoidal.py`
work). A fixture regenerated from a clean DDR clone diverges ~1% at every
ddrs commit — wrong reference, not a port bug. See
**Update (2026-08-19):** the fixture is regenerated from DDR master
post-#192 (DDR ported ddrs's corrected physics as a clean break), and the
comparison now runs — and matches at the f32 floor — on the corrected
physics, which is ddrs's default since `ddr_match` was deprecated (default
flipped to `false`; see `.claude/PHYSICS-CORRECTIONS.md`). The old
2026-06-06 "desktop-only reference" caveat is obsolete: any DDR checkout at
or past #192 is a valid reference. See
`docs/reference/ddr-comparison.md` §Regenerating fixtures.
2. **f32 throughout the routing core.** No mixed precision. The DDR comparison
sits at the f32 precision floor (~1e-7 rel diff per reach); any cast to
Expand Down
6 changes: 4 additions & 2 deletions config/merit_training.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,10 @@ params:
# backend; remove or set to cpu to fall back to the CPU solver).
sparse_solver: cuda
# Capture the routing forward pass as a CUDA graph and replay it each
# timestep — faster, CUDA backend only.
use_cuda_graphs: true
# timestep — faster, CUDA backend only. OFF because the captured kernel
# hardcodes the legacy (deprecated `ddr_match: true`) celerity; it cannot
# be used with the corrected physics that is now the default.
use_cuda_graphs: false

# Testing-mode overlay: when mode: testing (workflow: eval), these keys
# REPLACE the matching experiment: keys; absent keys inherit.
Expand Down
11 changes: 10 additions & 1 deletion docs/reference/ddr-comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,16 @@ If you regenerate fixtures, V1 must still pass. If it suddenly fails
after regeneration, DDR's solver moved — investigate the DDR diff before
touching ddrs.

### The desktop-only DDR reference (2026-06-06)
### The desktop-only DDR reference (2026-06-06) — OBSOLETE since 2026-08-19

> **Update (2026-08-19):** DDR merged the corrected physics
> (DeepGroundwater/ddr#192, porting ddrs's `ddr_match: false` path as a clean
> break), so the caveat below no longer binds. The V1 fixture is regenerated
> from post-#192 DDR master and `compare_ddr_sandbox` — now running the
> corrected physics, since `ddr_match` is deprecated with default `false` —
> reports ABSOLUTE MATCH at the f32 floor (max abs 1.5e-5 m³/s, max rel
> ~2e-7). Any DDR checkout at or past #192 is a valid reference. The section
> below is kept for interpreting PRE-#192 fixtures.

> **The reference DDR state is NOT a pushed commit (as of 2026-06-06).**
> The port mirrors the desktop checkout of `~/projects/ddr`, which
Expand Down
81 changes: 81 additions & 0 deletions examples/juniata/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Juniata single-catchment sample

ddrs on one basin: the Juniata River at Newport, PA (USGS 01567000,
8,657 km², 213 MERIT reaches). Everything needed is in `data/` — no HPC,
S3, CUDA, or external stores. This is the ddrs mirror of DDR's
`examples/juniata` (DeepGroundwater/ddr PR #193); the two run on the
byte-identical bundle with the same hyperparameters, so results are
directly comparable across implementations.

## Quickstart

From the **repo root** (the config's data paths are repo-root-relative):

cargo build --release --bin ddrs
target/release/ddrs --config examples/juniata/ddrs.yaml plan
target/release/ddrs --config examples/juniata/ddrs.yaml run --workflow train-and-test --backend cpu

`plan` validates the bundle and prints the summed-Q' baseline (no learned
parameters — the bar routing has to beat). `run` trains the KAN head for
30 optimizer steps (one random 90-day window per epoch on a single gauge),
then evaluates over water years 1996–2010. The workspace lands beside the
config at `examples/juniata/.ddrs/` (gitignored); inspect a finished run
with:

target/release/ddrs --config examples/juniata/ddrs.yaml show <run-id>

30 steps demonstrates learning and physically plausible parameters, not a
converged CONUS-grade model. The full train-and-test workflow takes well
under a minute on CPU.

Reference results (CPU, 30 epochs, test 1995-10-01 – 2010-09-30):

| | NSE | KGE |
|---|---|---|
| ddrs routed | 0.790 | 0.881 |
| DDR (Python) routed | 0.784 | 0.877 |
| summed-Q' baseline | 0.695 | 0.819 |

The ddrs baseline reproduces DDR's (0.695 / 0.820) to rounding — the two
implementations read the same bundle through independent readers. The
routed numbers agree to well within sampling noise: the residual comes
only from the 30 random training windows being drawn from different RNG
streams (torch's global RNG vs ChaCha12), so exact-match is impossible by
construction. With 30 noisy single-gauge steps, expect seed-to-seed
spread in the routed metrics: ddrs seeds {42, 7, 123, 2026} scored NSE
0.790–0.800 / KGE 0.881–0.886.

Both examples run the corrected physics (trapezoid-exact celerity β,
Cunge-matched X, own-reach gauge readout): DDR removed its legacy path in
PR #192, and ddrs deprecated `params.ddr_match` on 2026-08-19 — the
default is now `false` (corrected), so this config needs no flag at all.

## What's in the bundle

| File | Contents |
|---|---|
| `juniata_qprime.ic` | icechunk, `Qr(divide_id, time)` daily m³/s, 213 divides, 1980–2010 (dHBV2 UH retrospective) |
| `juniata_obs.ic` | icechunk, `streamflow(gage_id, time)` daily m³/s, USGS 01567000, 1980–2010 |
| `juniata_attributes.nc` | 10 KAN input attributes per COMID |
| `juniata_conus_adjacency.zarr` | binsparse COO subgraph + `length_m`, `slope`, `order` (compact 0..212 indexing, topologically ordered, lower-triangular) |
| `juniata_gages_adjacency.zarr` | single-gage COO group, same schema as the CONUS store |
| `juniata_gage.csv` | one-row gage metadata (gages_3000 schema) |
| `statistics/…json` | attribute normalization statistics over the 213 catchments |

One deviation from DDR's bundle: `statistics/` is **committed** here.
DDR computes it on first run (`set_statistics`); ddrs never recomputes
statistics (`src/data/statistics.rs`), so the JSON ships with the bundle.

Both icechunk stores start 1980-01-01 (required by the readers'
positional time origin). The bundle itself is regenerated from the CONUS
stores by DDR's `examples/juniata/extract_bundle.py` (maintainer-run);
this copy is taken verbatim from the ddr repo — regenerate there, then
re-copy `data/` and re-copy the generated statistics JSON.

## Gate

`tests/juniata_bundle.rs` asserts the bundle contract through the real
ddrs readers (213 reaches / 212 edges, lower-triangular adjacency,
1980-01-01 time origins, all 10 KAN inputs present). It runs in the
default `cargo test` sweep; if you touch the bundle or `src/data/`, make
sure it still passes.
Binary file added examples/juniata/data/juniata_attributes.nc
Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"shape": [
212
],
"data_type": "int32",
"chunk_grid": {
"name": "regular",
"configuration": {
"chunk_shape": [
212
]
}
},
"chunk_key_encoding": {
"name": "default",
"configuration": {
"separator": "/"
}
},
"fill_value": 0,
"codecs": [
{
"name": "bytes",
"configuration": {
"endian": "little"
}
},
{
"name": "zstd",
"configuration": {
"level": 0,
"checksum": false
}
}
],
"attributes": {},
"zarr_format": 3,
"node_type": "array",
"storage_transformers": []
}
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"shape": [
212
],
"data_type": "int32",
"chunk_grid": {
"name": "regular",
"configuration": {
"chunk_shape": [
212
]
}
},
"chunk_key_encoding": {
"name": "default",
"configuration": {
"separator": "/"
}
},
"fill_value": 0,
"codecs": [
{
"name": "bytes",
"configuration": {
"endian": "little"
}
},
{
"name": "zstd",
"configuration": {
"level": 0,
"checksum": false
}
}
],
"attributes": {},
"zarr_format": 3,
"node_type": "array",
"storage_transformers": []
}
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"shape": [
213
],
"data_type": "float32",
"chunk_grid": {
"name": "regular",
"configuration": {
"chunk_shape": [
213
]
}
},
"chunk_key_encoding": {
"name": "default",
"configuration": {
"separator": "/"
}
},
"fill_value": 0.0,
"codecs": [
{
"name": "bytes",
"configuration": {
"endian": "little"
}
},
{
"name": "zstd",
"configuration": {
"level": 0,
"checksum": false
}
}
],
"attributes": {},
"zarr_format": 3,
"node_type": "array",
"storage_transformers": []
}
Binary file not shown.
Loading
Loading