From 1e54aa340a7ff7862bbb228d15219cfefc33b10b Mon Sep 17 00:00:00 2001 From: Mehul Date: Wed, 14 Oct 2026 00:21:39 +0530 Subject: [PATCH 1/7] Add phase 2 plan: the model --- .../2026-10-14-stratavision-phase-2-model.md | 301 ++++++++++++++++++ 1 file changed, 301 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-14-stratavision-phase-2-model.md diff --git a/docs/superpowers/plans/2026-10-14-stratavision-phase-2-model.md b/docs/superpowers/plans/2026-10-14-stratavision-phase-2-model.md new file mode 100644 index 0000000..9139b59 --- /dev/null +++ b/docs/superpowers/plans/2026-10-14-stratavision-phase-2-model.md @@ -0,0 +1,301 @@ +# StrataVision Phase 2 (Model) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** The calculation that says, for any point and depth in the city, what the ground probably is and how sure that is, plus the check that measures how often it is right. No screens. + +**Architecture:** Pure TypeScript in `app/src/model/`, tested with Vitest. Borewells are prepared once per load (each layer turned into a height range and a drilling group); `predictColumn` then answers for one point by a weighted vote of nearby prepared borewells, and `checkModel` hides each borewell in turn and scores the prediction against it. One small Rust change sends the soil-type list with the records, because hand-added soil types are only known to the database. + +**Tech Stack:** TypeScript ~7.0.2, Vitest ^5.0.2 (the version StrataField's `core` uses), `@strata/core` for types, ground heights and distances. Rust only in Task 1. + +**Spec:** `docs/superpowers/specs/2026-10-13-stratavision-design.md`, section "The model" (drilling groups, question answered, how the vote works, accuracy check, speed) and the model lines of "Testing". + +## Global Constraints + +- Branch `phase-2-model`, cut from `phase-1-skeleton` if pull request #1 is not merged yet, otherwise from `main`. One pull request; the owner merges. No AI attribution lines. +- StrataField (`strata/`) is not changed. +- Depths and heights inside the model are in **feet**. Height above sea level of a depth = ground height in metres × `METRES_TO_FEET` − depth in feet. A borewell whose `depthUnit` is `'m'` has its layer depths multiplied by `METRES_TO_FEET` first. +- Ground height of a borewell: its own `groundElevationM` when not null, otherwise `heightAt(grid, latitude, longitude)`. Ground height of an asked point: `heightAt` only. +- Every number below is a named export of `app/src/model/constants.ts`, with exactly these values: `STEP_FT = 5`, `RADIUS_KM = 5`, `MAX_VOTERS = 12`, `MIN_DISTANCE_KM = 0.025`, `SAME_SITE_KM = 0.1`, `QUALITY_WEIGHT = { good: 1, fair: 0.7, unknown: 0.7, poor: 0.4 }`, `ADDRESS_WEIGHT = 0.5`, `LIKELY = { nearestKm: 1, sites: 3, share: 0.6 }`, `ROUGH = { nearestKm: 3, sites: 2 }`, `MIN_BOREWELLS_TO_JUDGE = 10`, `ODD = { fraction: 0.5, minSteps: 10 }`. +- The model never invents data: no voter means no answer; a "Not recorded" layer, a gap, or a layer with no soil type linked never votes. +- Not in this phase: screens, running the check off the main thread, the real elevation file (tests build small grids by hand), tuning the constants against real records (phase 3). + +## Review Focus + +1. Many borewells at one spot (the real records have plenty): they must count as one site for confidence and share one vote, and must not crowd every other site out of an answer. (Task 3; the 12-voter limit is the known weak point and has its own test.) +2. A borewell much shallower than its neighbours: below its bottom it must stop voting, not vote its last layer downwards. (Task 3) +3. A point outside the ground-height grid, or a borewell there with no recorded ground height: no answer and no crash. (Task 3) +4. A borewell logged in metres among borewells logged in feet. (Task 3) +5. The check on very few borewells, or on borewells that are all far apart, must say it cannot judge instead of reporting 0% or dividing by zero. (Task 4) + +--- + +### Task 1: Send the soil-type list with the records + +**Files:** +- Modify: `app/src-tauri/src/data.rs`, `app/src-tauri/src/lib.rs`, `app/src/lib/api.ts`, `app/src/App.tsx` + +**Interfaces:** +- Produces (Rust): `pub struct Records { pub sites: Vec, pub materials: Vec }` (camelCase), `pub fn load(data_dir: &Path) -> Result`; command `load_data` returns it. +- Produces (TypeScript): `export interface Records { sites: SiteRecord[]; materials: Material[] }`, `loadData(): Promise`. + +- [ ] **Step 1: Change the tests first.** In `data.rs` every `load(...).unwrap()` that reads sites now reads `.sites`; add to `loads_after_a_clean_close`: + +```rust +let records = load(tmp.path()).unwrap(); +assert!(records.materials.iter().any(|m| m.id == "clay" && m.lithology_class.as_deref() == Some("CLAY"))); +``` + +- [ ] **Step 2: Run** `cargo test -p stratavision`. Expected: does not compile (`Records` not defined, no field `sites`). +- [ ] **Step 3: Implement.** `load` reads `strata_db::repo::materials::list(&tx)` inside the same transaction as the borewells and returns `Records`. Update `lib.rs`, `api.ts` and `App.tsx` (state holds `Records`; the screen behaves as before). +- [ ] **Step 4: Run** `cargo test -p stratavision`, `cargo clippy --workspace --all-targets -- -D warnings`, `npm run typecheck`, `npm run lint`, then the debug build and `npm run smoke`. Expected: 8 tests pass, the rest clean, smoke OK. +- [ ] **Step 5: Commit** — `Send the soil-type list with the records` + +--- + +### Task 2: Drilling groups, and a test runner + +**Files:** +- Create: `app/src/model/constants.ts`, `app/src/model/groups.ts`, `app/src/model/groups.test.ts` +- Modify: `app/package.json` (dev dependency `vitest ^5.0.2`, script `"test": "vitest run"`), root `package.json` (script `"test": "npm test -w app"`), `.github/workflows/ci.yml` (a `Test` step running `npm test` in the `check` job, after Lint) + +**Interfaces:** +- Produces: + +```ts +export type Group = "clay" | "fineSand" | "waterBearing" | "hard"; +export const GROUPS: Group[]; // in that order; ties in a vote go to the earlier one +/** The drilling group of a layer, or null if it must not vote. */ +export function groupOf(layer: Pick, materials: Map): Group | null; +export const materialsById: (materials: Material[]) => Map; +``` + +- [ ] **Step 1: Write `groups.test.ts`.** `materials` is `materialsById(DEFAULT_MATERIALS)` from `@strata/core` plus one hand-added soil type `{ id: "x-grit", lithologyClass: null, lithologyFamily: "SAND" }` and one `{ id: "x-fill", lithologyClass: null, lithologyFamily: "OTHER" }`. + +```ts +it.each([ + ["clay", "clay"], ["silty_clay", "clay"], ["sandy_clay", "clay"], ["silt", "clay"], + ["fine_sand", "fineSand"], + ["medium_sand", "waterBearing"], ["coarse_sand", "waterBearing"], ["yellow_sand", "waterBearing"], + ["gravel", "waterBearing"], ["sandy_gravel", "waterBearing"], + ["kankar", "hard"], ["clay_kankar", "hard"], ["sandy_kankar", "hard"], ["rock", "hard"], ["boulder", "hard"], +])("%s is %s", (id, group) => expect(groupOf({ materialId: id, waterBearing: false }, materials)).toBe(group)); + +it("a hand-added soil type falls back on its family", …"x-grit" → "waterBearing"); +it("never votes: not recorded, other, unlinked, unknown id", …"not_recorded", "x-fill", null, "no-such-id" → null); +it("the driller's water-bearing tick upgrades fine sand", …{ materialId: "fine_sand", waterBearing: true } → "waterBearing"); +it("the tick does not turn clay into water-bearing", …{ materialId: "clay", waterBearing: true } → "clay"); +it("the tick does not turn kankar into water-bearing", …{ materialId: "sandy_kankar", waterBearing: true } → "hard"); +``` + +- [ ] **Step 2: Run** `npm test`. Expected: fails, `groupOf` not found. +- [ ] **Step 3: Implement `constants.ts` and `groups.ts`.** Class table as in the spec. Family fallback when the class is null or not in the table: CLAY → clay, SAND → waterBearing, ROCK → hard, anything else → null. The tick upgrades only when the soil type's family is SAND. +- [ ] **Step 4: Run** `npm test`, `npm run typecheck`, `npm run lint`. Expected: all pass. +- [ ] **Step 5: Commit** — `Add drilling groups` + +--- + +### Task 3: The vote + +**Files:** +- Create: `app/src/model/model.ts`, `app/src/model/model.test.ts`, `app/src/model/testGround.ts` (test helpers only) + +**Interfaces:** +- Consumes: `Group`, `GROUPS`, `groupOf`, `materialsById`, the constants; from `@strata/core`: `Borewell`, `StrataLayer`, `Material`, `ElevationGrid`, `heightAt`, `kmBetween`, `METRES_TO_FEET`. +- Produces: + +```ts +export interface ModelSite { borewell: Borewell; strata: StrataLayer[] } + +/** One borewell made ready to vote. */ +export interface Prepared { + id: string; borewellId: string; + latitude: number; longitude: number; + groundFt: number; // height above sea level + weight: number; // record quality × location source + runs: { topFt: number; bottomFt: number; group: Group }[]; // heights above sea level, top first +} +/** Leaves out borewells with no location, no ground height, or no layer that can vote. */ +export function prepare(sites: ModelSite[], materials: Material[], grid: ElevationGrid): Prepared[]; + +export type Confidence = "likely" | "rough" | "none"; +export interface Source { id: string; borewellId: string; km: number; weight: number } +export interface Step { + top: number; bottom: number; // feet below ground at the asked point + shares: Record | null; // adds up to 1; null when nobody voted + likely: Group | null; // null when confidence is "none" + confidence: Confidence; + sources: Source[]; // nearest first +} +/** null when the ground height at the point is unknown. `exclude` and `groundM` are for the accuracy check. */ +export function predictColumn(latitude: number, longitude: number, prepared: Prepared[], grid: ElevationGrid, + options?: { exclude?: string; groundM?: number }): Step[] | null; +``` + +- Produces (test helpers, `testGround.ts`): `flatGrid(metres: number): ElevationGrid` covering latitude 26.6 to 27.1, longitude 80.7 to 81.2; `slopeGrid(westMetres, eastMetres)`; `site(id, latitude, longitude, layers: [top, bottom, materialId][], overrides?: Partial): ModelSite`; `east(km)` and `north(km)` giving a longitude or latitude that far from `[26.85, 80.95]`. + +**The rules** (`predictColumn`), in order: + +1. Ground at the point: `options.groundM`, else `heightAt`; null → return null. +2. Candidates: prepared borewells other than `exclude`, within `RADIUS_KM`, sorted nearest first. None → return `[]`. +3. The column runs from 0 down to the deepest candidate's bottom, in the point's own depths, rounded up to a whole step. +4. For each step, the voters are the first `MAX_VOTERS` candidates that have a run covering the height of the step's middle. +5. Each voter's weight is `prepared.weight / max(km, MIN_DISTANCE_KM)² / crowd`, where `crowd` is the number of that step's voters within `SAME_SITE_KM` of it, itself included. +6. `shares` are the weights by group divided by their total. +7. Sites: walk the voters nearest first; a voter starts a new site unless it is within `SAME_SITE_KM` of the first voter of an existing site. +8. Confidence: `likely` if nearest voter ≤ `LIKELY.nearestKm` and sites ≥ `LIKELY.sites` and largest share ≥ `LIKELY.share`; else `rough` if nearest ≤ `ROUGH.nearestKm` and sites ≥ `ROUGH.sites`; else `none`. + +- [ ] **Step 1: Write `model.test.ts`.** All on `flatGrid(120)` unless stated; "the point" is `[26.85, 80.95]`. `stepAt(column, depthFt)` returns the step containing that depth. + +```ts +it("recovers a flat layer between borewells", … + // three borewells 0.5 km away to the north, east and south-west, each clay 0–50, sand 50–100 + // stepAt(col, 20).likely === "clay"; stepAt(col, 70).likely === "waterBearing"; both "likely"; col.length === 20 +); +it("names the borewells behind a step, nearest first", …sources ids in distance order, km within 0.01); +it("stops a shallow borewell voting below its bottom", … + // two deep clay borewells 0.5 km away (0–200), one borewell 0.1 km away sand 0–50 only + // stepAt(col, 20).likely === "waterBearing"; stepAt(col, 120).shares.clay === 1; its sources do not include the shallow one +); +it("a crowded site shares one vote", … + // ten clay borewells within 30 m of each other 1 km east; one sand borewell 1 km west + // stepAt(col, 20).shares.clay is within 0.05 of 0.5 +); +it("a crowded site is one site for confidence", … + // the same ten alone: confidence "none" (one site), likely null, shares.clay === 1 +); +it("ponytail limit: more than 12 borewells at the nearest spot leave room for nobody else", … + // thirteen clay borewells within 30 m, 0.2 km away; one sand borewell 0.5 km away + // documents today's behaviour: sources.length === 12, shares.waterBearing === 0, confidence "none" +); +it("matches layers by height, not depth, on sloping ground", … + // slopeGrid(110, 130). Sand lies between 250 and 300 ft above sea level in borewells west and east of the point. + // At the point the sand step is at the depth that height corresponds to there, not at either borewell's depth. +); +it("uses a borewell's own recorded ground height before the grid", …groundElevationM: 130 on flatGrid(120) shifts its layers by 10 m); +it("confidence falls with distance", …three agreeing borewells at 0.5 km → "likely"; at 2 km → "rough"; at 4 km → "none"); +it("confidence falls with disagreement", …three sites at 0.5 km, shares 0.5/0.5 → "rough"); +it("counts poor records and address-located borewells less", … + // good clay at 1 km vs poor sand at 1 km → shares.clay ≈ 1/1.4; same with locationSource "address" → ≈ 1/1.5 +); +it("converts borewells logged in metres", …depthUnit "m", clay 0–10 m → clay down to 30 ft, nothing at 40 ft); +it("not recorded and gaps never vote", …a "not_recorded" layer 0–50 and a gap 50–60: steps there have shares null when it is the only borewell); +it("gives no answer outside the ground-height grid", …predictColumn(28, 80.95, …) === null); +it("leaves out borewells it cannot place", …prepare drops: no latitude; outside the grid with no groundElevationM; only not-recorded layers); +it("gives an empty column with no borewell within 5 km", …[]); +it("exclude hides a borewell from its own prediction", …); +``` + +- [ ] **Step 2: Run** `npm test`. Expected: fails, `predictColumn` not found. +- [ ] **Step 3: Implement `testGround.ts`, then `prepare` and `predictColumn` in `model.ts`** following the rules above. Leave this comment on the voter limit: `// ponytail: the 12 nearest borewells, not sites; a spot with 12 or more logs crowds out everyone else. Limit by site if real records show it.` +- [ ] **Step 4: Run** `npm test`, `npm run typecheck`, `npm run lint`. Expected: all pass. +- [ ] **Step 5: Commit** — `Add the weighted vote` + +--- + +### Task 4: The accuracy check and odd ones out + +**Files:** +- Create: `app/src/model/check.ts`, `app/src/model/check.test.ts` + +**Interfaces:** +- Consumes: `Prepared`, `predictColumn`, `Step` from Task 3; `median` from `@strata/core`. +- Produces: + +```ts +export interface Tally { steps: number; right: number } +export interface OddOne { id: string; borewellId: string; steps: number; right: number; typical: number } +export interface CheckResult { + canJudge: boolean; // at least MIN_BOREWELLS_TO_JUDGE borewells were compared + compared: number; // borewells with at least one comparable step + overall: Tally; likely: Tally; rough: Tally; + byDistance: { under1: Tally; from1to3: Tally; over3: Tally }; // by distance to the nearest other borewell + oddOnesOut: OddOne[]; // worst first +} +export function checkModel(prepared: Prepared[], grid: ElevationGrid): CheckResult; +export const share = (t: Tally) => (t.steps === 0 ? null : t.right / t.steps); +``` + +**The rules:** + +1. For each prepared borewell, `predictColumn` at its own place with `exclude` its id and `groundM` its own ground height. +2. A step is comparable when the prediction's `likely` is not null and the hidden borewell has a run covering the step's middle. It is right when the two groups are the same. +3. Every comparable step is added to `overall`, to `likely` or `rough` by its confidence, and to the band of its borewell. +4. Odd ones out: among borewells with at least `ODD.minSteps` comparable steps, `typical` is the median of their own right-fractions within their band; a borewell is odd when its right-fraction is below `ODD.fraction × typical`. Sorted by right-fraction, lowest first. + +- [ ] **Step 1: Write `check.test.ts`** + +```ts +it("scores a uniform city as always right", … + // 16 borewells on a 4 × 4 grid 0.5 km apart, all clay 0–50, sand 50–100 + // canJudge true; share(overall) === 1; oddOnesOut empty; byDistance.under1.steps === overall.steps +); +it("finds a borewell that does not fit its surroundings", … + // the same city with one borewell logged upside down (sand 0–50, clay 50–100) + // oddOnesOut[0].borewellId is that one; oddOnesOut.length === 1; share(overall) < 1 +); +it("cannot judge on fewer than ten borewells", …9 borewells → canJudge false, oddOnesOut empty); +it("does not divide by zero when nothing is comparable", … + // 12 borewells each more than 5 km from every other: compared === 0, canJudge false, share(overall) === null +); +it("puts borewells in the band of their nearest neighbour", …pairs 0.5 km, 2 km and 4 km apart land in under1, from1to3, over3); +``` + +- [ ] **Step 2: Run** `npm test`. Expected: fails, `checkModel` not found. +- [ ] **Step 3: Implement `check.ts`** following the rules. +- [ ] **Step 4: Run** `npm test`, `npm run typecheck`, `npm run lint`. Expected: all pass. +- [ ] **Step 5: Commit** — `Add the accuracy check and odd ones out` + +--- + +### Task 5: A made-up city of 500 borewells + +Proves the model on ground whose true answer is known, at the real size and with the real overlap, inside time limits. + +**Files:** +- Create: `app/src/model/madeUpCity.ts` (test helper), `app/src/model/city.test.ts` + +**Interfaces:** +- Produces (test helper): + +```ts +/** What the made-up ground really is at a place and a height above sea level, in feet. */ +export function truth(latitude: number, longitude: number, heightFt: number): Group; +/** `count` borewells logged from `truth`, always the same for the same seed. */ +export function madeUpCity(count: number, seed?: number): { sites: ModelSite[]; grid: ElevationGrid }; +``` + +**The made-up ground:** `slopeGrid(110, 125)`. Going down by height above sea level: clay, then water-bearing sand, then fine sand, then hard, then water-bearing sand to the bottom. The four boundaries start at 330, 280, 250 and 220 ft above sea level and each moves up and down by 15 ft across the city as a smooth wave about 8 km long, each with a different direction. Borewells: a small seeded random-number function (no dependency); 70% spread evenly over latitude 26.75 to 26.95, longitude 80.85 to 81.05; 30% placed within 50 m of an earlier one; depth between 150 and 400 ft; layers read from `truth` every 5 ft and merged; record quality `good`. + +- [ ] **Step 1: Write `city.test.ts`** + +```ts +const { sites, grid } = madeUpCity(500); +const prepared = prepare(sites, DEFAULT_MATERIALS, grid); + +it("is right at least 7 times in 10 where it says likely", … + const r = checkModel(prepared, grid); + // r.canJudge; share(r.likely) >= 0.7; share(r.likely) > share(r.rough); r.likely.steps > r.overall.steps * 0.5 +); +it("flags few honest borewells as odd", …r.oddOnesOut.length <= 10); +it("gets better with more borewells", …share(checkModel of madeUpCity(100)).overall) < share(checkModel of madeUpCity(500)).overall)); +it("agrees with the truth at a place nobody drilled", … + // predictColumn at [26.87, 80.97]: of its "likely" steps, at least 70% equal truth at that step's middle +); +it("is fast enough", … + // one column < 20 ms; 200 columns along a line < 1 s; 100 × 100 points at one depth (whole columns) < 5 s; + // checkModel(prepared, grid) < 5 s. Each measured with performance.now(); the test's own timeout is 30 s. +); +``` + +- [ ] **Step 2: Run** `npm test`. Expected: fails, `madeUpCity` not found. +- [ ] **Step 3: Implement `madeUpCity.ts`.** +- [ ] **Step 4: Run** `npm test`. Expected: all pass. If an accuracy or time assertion fails, the model is what needs to change, using superpowers:systematic-debugging; the thresholds come from the spec and stay. +- [ ] **Step 5: Run** `npm run typecheck`, `npm run lint`. **Commit** — `Prove the model on a made-up city of 500 borewells` + +--- + +### Task 6: Pull request + +- [ ] **Step 1: Run everything:** `npm test`, `npm run typecheck`, `npm run lint`, `cargo fmt --all --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace`, the debug build, `npm run smoke`. Expected: all pass. +- [ ] **Step 2: Push `phase-2-model` and open the pull request** into `main` (or into `phase-1-skeleton` if #1 is still open, saying so). Report the made-up city's numbers in the description: share right for likely and rough steps, by distance band, and the four timings. Turn Auto-fix on. The owner merges. From 44696a390ac51d804c2241e3b731fa9c5647269e Mon Sep 17 00:00:00 2001 From: Mehul Date: Wed, 14 Oct 2026 00:23:21 +0530 Subject: [PATCH 2/7] Send the soil-type list with the records --- app/src-tauri/src/data.rs | 36 ++++++++++++++++++++++++++++-------- app/src-tauri/src/lib.rs | 4 ++-- app/src/App.tsx | 8 ++++---- app/src/lib/api.ts | 13 ++++++++++--- 4 files changed, 44 insertions(+), 17 deletions(-) diff --git a/app/src-tauri/src/data.rs b/app/src-tauri/src/data.rs index d20f55a..7a03cbd 100644 --- a/app/src-tauri/src/data.rs +++ b/app/src-tauri/src/data.rs @@ -6,8 +6,10 @@ use serde::Serialize; use std::path::{Path, PathBuf}; use std::time::Duration; use strata_db::db::DB_FILE_NAME; -use strata_db::models::{Borewell, PipeSegment, SearchFilters, StrataLayer, WaterReading}; -use strata_db::repo::{borewells, layers}; +use strata_db::models::{ + Borewell, Material, PipeSegment, SearchFilters, StrataLayer, WaterReading, +}; +use strata_db::repo::{borewells, layers, materials}; use strata_db::schema; /// Name of the folder, inside the roaming app-data folder, that every Strata app shares. @@ -23,6 +25,15 @@ pub struct SiteRecord { pub water_readings: Vec, } +/// Everything read in one go, all as it was at the same moment. +#[derive(Debug, Clone, Serialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct Records { + pub sites: Vec, + /// Every soil type, built-in and added by hand; layers refer to them by id. + pub materials: Vec, +} + /// Why the records could not be read. The screens choose the wording from `kind`. #[derive(Debug, Clone, Serialize, PartialEq)] #[serde(tag = "kind", rename_all = "camelCase")] @@ -55,11 +66,11 @@ pub fn data_dir(roaming_app_data: &Path) -> PathBuf { } } -/// Every borewell that is not in the Recycle bin, read from `/strata.db` as it is at +/// Every borewell that is not in the Recycle bin, and the soil types, read from `/strata.db` as it is at /// this moment. The file is opened read-only: the records are never upgraded or changed, and the /// folder and the file are never created. SQLite may add its two bookkeeping files (`-wal` and /// `-shm`) beside an existing database, exactly as StrataField itself does. -pub fn load(data_dir: &Path) -> Result, LoadError> { +pub fn load(data_dir: &Path) -> Result { let path = data_dir.join(DB_FILE_NAME); if !path.exists() { return Err(LoadError::NoDatabase); @@ -87,7 +98,10 @@ pub fn load(data_dir: &Path) -> Result, LoadError> { strata: item.strata, }); } - Ok(sites) + Ok(Records { + sites, + materials: materials::list(&tx)?, + }) } #[cfg(test)] @@ -202,7 +216,13 @@ mod tests { assert!(!tmp.path().join("strata.db-shm").exists()); let before = std::fs::read(&db_file).unwrap(); - let got = load(tmp.path()).unwrap(); + let records = load(tmp.path()).unwrap(); + // The soil types come with the records: hand-added ones are only known to the database. + assert!(records + .materials + .iter() + .any(|m| m.id == "clay" && m.lithology_class.as_deref() == Some("CLAY"))); + let got = records.sites; assert_eq!(got.len(), 2); let full = site(&got, "BW-1"); assert_eq!(full.strata.len(), 1); @@ -226,7 +246,7 @@ mod tests { fn loads_while_another_app_has_it_open() { let tmp = tempfile::tempdir().unwrap(); let _open = seeded(tmp.path()); - assert_eq!(load(tmp.path()).unwrap().len(), 2); + assert_eq!(load(tmp.path()).unwrap().sites.len(), 2); } #[test] @@ -243,7 +263,7 @@ mod tests { }) .unwrap(); drop(db); - let got = load(tmp.path()).unwrap(); + let got = load(tmp.path()).unwrap().sites; assert_eq!(got.len(), 1); assert_eq!(got[0].borewell.borewell_id, "BW-2"); } diff --git a/app/src-tauri/src/lib.rs b/app/src-tauri/src/lib.rs index 1355388..5f2fa7e 100644 --- a/app/src-tauri/src/lib.rs +++ b/app/src-tauri/src/lib.rs @@ -2,9 +2,9 @@ mod data; use tauri::Manager; -/// Every borewell in StrataField's records, read afresh each time the screens ask. +/// StrataField's records, read afresh each time the screens ask. #[tauri::command] -fn load_data(app: tauri::AppHandle) -> Result, data::LoadError> { +fn load_data(app: tauri::AppHandle) -> Result { let roaming = app.path().data_dir()?; data::load(&data::data_dir(&roaming)) } diff --git a/app/src/App.tsx b/app/src/App.tsx index 62f8a1f..d9342ff 100644 --- a/app/src/App.tsx +++ b/app/src/App.tsx @@ -2,10 +2,10 @@ import { useCallback, useEffect, useState } from "react"; import { RefreshCw } from "lucide-react"; import { Button } from "@/components/ui/button"; import { SiteMap, located } from "@/components/SiteMap"; -import { loadData, type LoadError, type SiteRecord } from "@/lib/api"; +import { loadData, type LoadError, type Records } from "@/lib/api"; import { text } from "@/text"; -type State = { sites: SiteRecord[] } | { problem: LoadError } | null; +type State = { records: Records } | { problem: LoadError } | null; export default function App() { const [state, setState] = useState(null); @@ -14,7 +14,7 @@ export default function App() { // Reading again keeps what is on screen until the new records arrive, so the map does not blink. const load = useCallback(() => { loadData().then( - (sites) => setState({ sites }), + (records) => setState({ records }), (e) => setState({ problem: isLoadError(e) ? e : { kind: "other", message: String(e) } }), ); }, []); @@ -43,7 +43,7 @@ function Body({ state, selectedId, onSelect }: { state: State; selectedId: strin const p = state.problem; return {text.problems[p.kind]}; } - const { sites } = state; + const { sites } = state.records; if (sites.length === 0) return {text.empty}; const shown = located(sites).length; diff --git a/app/src/lib/api.ts b/app/src/lib/api.ts index a63ee06..a1c3380 100644 --- a/app/src/lib/api.ts +++ b/app/src/lib/api.ts @@ -1,5 +1,5 @@ import { invoke } from "@tauri-apps/api/core"; -import type { Borewell, PipeSegment, StrataLayer, WaterReading } from "@strata/core"; +import type { Borewell, Material, PipeSegment, StrataLayer, WaterReading } from "@strata/core"; /** One borewell with everything the analysis uses, as read from StrataField's records. */ export interface SiteRecord { @@ -9,8 +9,15 @@ export interface SiteRecord { waterReadings: WaterReading[]; } +/** Everything read in one go, all as it was at the same moment. */ +export interface Records { + sites: SiteRecord[]; + /** Every soil type, built-in and added by hand; layers refer to them by id. */ + materials: Material[]; +} + /** Why the records could not be read (see `LoadError` in src-tauri/src/data.rs). */ export type LoadError = { kind: "noDatabase" | "newer" | "older" } | { kind: "other"; message: string }; -/** Every borewell that is not in the Recycle bin. Rejects with a `LoadError`. */ -export const loadData = () => invoke("load_data"); +/** Every borewell that is not in the Recycle bin, and the soil types. Rejects with a `LoadError`. */ +export const loadData = () => invoke("load_data"); From 57a13343a2717311f40f0d058ac1a7e005a750b7 Mon Sep 17 00:00:00 2001 From: Mehul Date: Wed, 14 Oct 2026 00:25:28 +0530 Subject: [PATCH 3/7] Add drilling groups --- .github/workflows/ci.yml | 5 ++++- app/package.json | 6 ++++-- app/src/model/constants.ts | 42 ++++++++++++++++++++++++++++++++++++ app/src/model/groups.test.ts | 35 ++++++++++++++++++++++++++++++ app/src/model/groups.ts | 36 +++++++++++++++++++++++++++++++ package-lock.json | 4 ++-- package.json | 3 ++- 7 files changed, 125 insertions(+), 6 deletions(-) create mode 100644 app/src/model/constants.ts create mode 100644 app/src/model/groups.test.ts create mode 100644 app/src/model/groups.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dc477c1..f06c299 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -17,7 +17,7 @@ concurrency: jobs: check: - name: Typecheck, lint, frontend build + name: Typecheck, lint, tests, frontend build runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 @@ -37,6 +37,9 @@ jobs: - name: Lint run: npm run lint + - name: Test + run: npm test + - name: Build frontend run: npm run build -w app diff --git a/app/package.json b/app/package.json index beb7062..82d22c0 100644 --- a/app/package.json +++ b/app/package.json @@ -8,7 +8,8 @@ "build": "tsc && vite build", "typecheck": "tsc --noEmit", "preview": "vite preview", - "tauri": "tauri" + "tauri": "tauri", + "test": "vitest run" }, "dependencies": { "@base-ui/react": "^1.8.0", @@ -35,6 +36,7 @@ "shadcn": "^4.21.1", "tailwindcss": "^4.3.3", "typescript": "~7.0.2", - "vite": "^8.0.16" + "vite": "^8.0.16", + "vitest": "^5.0.4" } } diff --git a/app/src/model/constants.ts b/app/src/model/constants.ts new file mode 100644 index 0000000..13c043c --- /dev/null +++ b/app/src/model/constants.ts @@ -0,0 +1,42 @@ +/** + * Every number the model uses, in one place. These are starting values: they are adjusted once + * against the real records using the accuracy check (check.ts), with the aim that a step the model + * calls "likely" is right at least 7 times in 10. + */ +import type { RecordQuality } from "@strata/core"; + +/** The column is answered in steps this thick, in feet. */ +export const STEP_FT = 5; + +/** Borewells further than this from a point say nothing about it. */ +export const RADIUS_KM = 5; + +/** At most this many borewells vote on a step: the nearest ones that reach it. */ +export const MAX_VOTERS = 12; + +/** A borewell closer than this counts as this far away, so one at the very spot is not infinitely loud. */ +export const MIN_DISTANCE_KM = 0.025; + +/** Borewells this close together are one site: they share a vote and count once for confidence. */ +export const SAME_SITE_KM = 0.1; + +/** How much a borewell's vote counts, by how good its record is. */ +export const QUALITY_WEIGHT: Record = { good: 1, fair: 0.7, unknown: 0.7, poor: 0.4 }; + +/** A borewell placed by looking up its address is only roughly where the map says. */ +export const ADDRESS_WEIGHT = 0.5; + +/** "Likely": a borewell this near, at least this many separate sites, and this much agreement. */ +export const LIKELY = { nearestKm: 1, sites: 3, share: 0.6 }; + +/** "Rough guess": a borewell this near and at least this many separate sites. */ +export const ROUGH = { nearestKm: 3, sites: 2 }; + +/** With fewer borewells than this to compare, the accuracy check says it cannot judge. */ +export const MIN_BOREWELLS_TO_JUDGE = 10; + +/** + * A borewell is an odd one out when it matches its surroundings less than this fraction as often as + * is typical for borewells with neighbours as near, judged on at least this many steps. + */ +export const ODD = { fraction: 0.5, minSteps: 10 }; diff --git a/app/src/model/groups.test.ts b/app/src/model/groups.test.ts new file mode 100644 index 0000000..6157979 --- /dev/null +++ b/app/src/model/groups.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from "vitest"; +import { DEFAULT_MATERIALS, type Material } from "@strata/core"; +import { groupOf, materialsById } from "./groups"; + +const handAdded = (id: string, lithologyFamily: Material["lithologyFamily"]): Material => + ({ id, name: id, color: "#000000", pattern: "dots", isCustom: true, lithologyClass: null, lithologyFamily }); +const materials = materialsById([...DEFAULT_MATERIALS, handAdded("x-grit", "SAND"), handAdded("x-fill", "OTHER")]); +const group = (materialId: string | null, waterBearing = false) => groupOf({ materialId, waterBearing }, materials); + +describe("groupOf", () => { + it.each([ + ["clay", "clay"], ["silty_clay", "clay"], ["sandy_clay", "clay"], ["silt", "clay"], + ["fine_sand", "fineSand"], + ["medium_sand", "waterBearing"], ["coarse_sand", "waterBearing"], ["yellow_sand", "waterBearing"], + ["gravel", "waterBearing"], ["sandy_gravel", "waterBearing"], + ["kankar", "hard"], ["clay_kankar", "hard"], ["sandy_kankar", "hard"], ["rock", "hard"], ["boulder", "hard"], + ])("%s is %s", (id, expected) => expect(group(id)).toBe(expected)); + + it("covers every built-in soil type except Not recorded", () => { + for (const m of DEFAULT_MATERIALS) expect(group(m.id) === null).toBe(m.id === "not_recorded"); + }); + + it("a hand-added soil type falls back on its family", () => expect(group("x-grit")).toBe("waterBearing")); + + it("never votes: not recorded, other, unlinked, unknown id", () => { + expect(group("not_recorded")).toBeNull(); + expect(group("x-fill")).toBeNull(); + expect(group(null)).toBeNull(); + expect(group("no-such-id")).toBeNull(); + }); + + it("the driller's water-bearing tick upgrades fine sand", () => expect(group("fine_sand", true)).toBe("waterBearing")); + it("the tick does not turn clay into water-bearing", () => expect(group("clay", true)).toBe("clay")); + it("the tick does not turn kankar into water-bearing", () => expect(group("sandy_kankar", true)).toBe("hard")); +}); diff --git a/app/src/model/groups.ts b/app/src/model/groups.ts new file mode 100644 index 0000000..58fcbec --- /dev/null +++ b/app/src/model/groups.ts @@ -0,0 +1,36 @@ +/** + * Drilling groups: the few kinds of ground that matter when drilling for water. Coarse enough for + * logs written by different drillers to agree, fine enough to say where the water-bearing sand is. + */ +import type { LithologyClass, LithologyFamily, Material, StrataLayer } from "@strata/core"; + +export type Group = "clay" | "fineSand" | "waterBearing" | "hard"; + +/** In this order everywhere; a tied vote goes to the earlier one. */ +export const GROUPS: Group[] = ["clay", "fineSand", "waterBearing", "hard"]; + +const BY_CLASS: Partial> = { + CLAY: "clay", SILTY_CLAY: "clay", SANDY_CLAY: "clay", SILT: "clay", + FINE_SAND: "fineSand", + // Yellow Sand as water-bearing is an assumption to confirm with DWO. + MEDIUM_SAND: "waterBearing", COARSE_SAND: "waterBearing", YELLOW_SAND: "waterBearing", GRAVEL: "waterBearing", SANDY_GRAVEL: "waterBearing", + KANKAR: "hard", CLAY_KANKAR: "hard", SANDY_KANKAR: "hard", ROCK: "hard", BOULDER: "hard", +}; + +/** For soil types added by hand, which have a family but no class in the table above. */ +const BY_FAMILY: Partial> = { CLAY: "clay", SAND: "waterBearing", ROCK: "hard" }; + +export const materialsById = (materials: Material[]) => new Map(materials.map((m) => [m.id, m])); + +/** + * The drilling group of a layer, or null if it must not vote: "Not recorded", a soil type that is + * neither clay, sand nor rock, or a layer whose soil name was never linked to a soil type. + */ +export function groupOf(layer: Pick, materials: Map): Group | null { + const m = layer.materialId ? materials.get(layer.materialId) : undefined; + if (!m) return null; + // The driller's own tick. It only ever upgrades sand: it is not filled in on every record, so an + // unticked layer keeps its class, and a tick on clay or kankar is taken as a slip. + if (layer.waterBearing && m.lithologyFamily === "SAND") return "waterBearing"; + return (m.lithologyClass && BY_CLASS[m.lithologyClass]) ?? (m.lithologyFamily && BY_FAMILY[m.lithologyFamily]) ?? null; +} diff --git a/package-lock.json b/package-lock.json index 6004be1..60ed01b 100644 --- a/package-lock.json +++ b/package-lock.json @@ -46,7 +46,8 @@ "shadcn": "^4.21.1", "tailwindcss": "^4.3.3", "typescript": "~7.0.2", - "vite": "^8.0.16" + "vite": "^8.0.16", + "vitest": "^5.0.4" } }, "app/node_modules/typescript": { @@ -2202,7 +2203,6 @@ "integrity": "sha512-uMzsv9stDdofIZBx4UGh4Z9ZgXnGC9seRDscilBcTHAeOFk67MZg/k9bWVfO+bk4qZmG+l52hRnLwUiH2LnJTg==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "undici-types": "~8.9.0" } diff --git a/package.json b/package.json index b399e7f..7851775 100644 --- a/package.json +++ b/package.json @@ -17,7 +17,8 @@ "build": "npm run tauri -w app -- build", "typecheck": "npm run typecheck -w app", "lint": "eslint app e2e", - "smoke": "node e2e/smoke.mjs" + "smoke": "node e2e/smoke.mjs", + "test": "npm test -w app" }, "devDependencies": { "@eslint/js": "^10.0.0", From 6ba366eebb11fa70a9fe85c1932e66ef79b85815 Mon Sep 17 00:00:00 2001 From: Mehul Date: Wed, 14 Oct 2026 00:26:21 +0530 Subject: [PATCH 4/7] Add the weighted vote --- app/src/model/model.test.ts | 193 ++++++++++++++++++++++++++++++++++++ app/src/model/model.ts | 159 +++++++++++++++++++++++++++++ app/src/model/testGround.ts | 34 +++++++ 3 files changed, 386 insertions(+) create mode 100644 app/src/model/model.test.ts create mode 100644 app/src/model/model.ts create mode 100644 app/src/model/testGround.ts diff --git a/app/src/model/model.test.ts b/app/src/model/model.test.ts new file mode 100644 index 0000000..c2030e7 --- /dev/null +++ b/app/src/model/model.test.ts @@ -0,0 +1,193 @@ +import { describe, expect, it } from "vitest"; +import { DEFAULT_MATERIALS, type ElevationGrid } from "@strata/core"; +import { predictColumn, prepare, type ModelSite, type Step } from "./model"; +import { CENTRE, east, flatGrid, north, site, slopeGrid } from "./testGround"; + +const flat = flatGrid(120); +const CLAY_OVER_SAND: [number, number, string][] = [[0, 50, "clay"], [50, 100, "medium_sand"]]; +const CLAY: [number, number, string][] = [[0, 200, "clay"]]; +const SAND: [number, number, string][] = [[0, 200, "medium_sand"]]; + +/** The column at the centre, from these borewells. */ +const column = (sites: ModelSite[], grid: ElevationGrid = flat) => { + const col = predictColumn(CENTRE[0], CENTRE[1], prepare(sites, DEFAULT_MATERIALS, grid), grid); + if (!col) throw new Error("no column"); + return col; +}; +const stepAt = (col: Step[], depthFt: number) => { + const step = col.find((s) => s.top <= depthFt && depthFt < s.bottom); + if (!step) throw new Error(`no step at ${depthFt} ft`); + return step; +}; +/** Three borewells with the same layers, `km` from the centre to the north, east and south-west. */ +const around = (km: number, layers: [number, number, string][], name = "a") => [ + site(`${name}1`, north(km), east(0), layers), + site(`${name}2`, north(0), east(km), layers), + site(`${name}3`, north(-km * 0.7), east(-km * 0.7), layers), +]; +/** `count` borewells within 30 m of each other, `km` east of the centre. */ +const crowd = (count: number, km: number, layers: [number, number, string][]) => + Array.from({ length: count }, (_, i) => site(`c${i}`, north(i * 0.002), east(km), layers)); + +describe("predictColumn", () => { + it("recovers a flat layer between borewells", () => { + const col = column(around(0.5, CLAY_OVER_SAND)); + expect(col).toHaveLength(20); + expect(stepAt(col, 20)).toMatchObject({ top: 20, bottom: 25, likely: "clay", confidence: "likely" }); + expect(stepAt(col, 70)).toMatchObject({ likely: "waterBearing", confidence: "likely" }); + expect(stepAt(col, 20).shares).toEqual({ clay: 1, fineSand: 0, waterBearing: 0, hard: 0 }); + }); + + it("names the borewells behind a step, nearest first", () => { + const col = column([site("far", north(-0.8), east(0), CLAY), site("near", north(0.3), east(0), CLAY), site("mid", north(0), east(0.5), CLAY)]); + const { sources } = stepAt(col, 20); + expect(sources.map((s) => s.id)).toEqual(["near", "mid", "far"]); + expect(sources[0].borewellId).toBe("NEAR"); + expect(sources[0].km).toBeCloseTo(0.3, 2); + expect(sources[0].weight).toBeGreaterThan(sources[1].weight); + }); + + it("stops a shallow borewell voting below its bottom", () => { + const col = column([site("deep1", north(0.5), east(0), CLAY), site("deep2", north(0), east(0.5), CLAY), site("shallow", north(-0.1), east(0), [[0, 50, "medium_sand"]])]); + expect(stepAt(col, 20).likely).toBe("waterBearing"); + expect(stepAt(col, 120).shares?.clay).toBe(1); + expect(stepAt(col, 120).sources.map((s) => s.id).sort()).toEqual(["deep1", "deep2"]); // equally far, either order + }); + + it("a crowded site shares one vote", () => { + const col = column([...crowd(10, 1, CLAY), site("alone", north(0), east(-1), SAND)]); + expect(stepAt(col, 20).shares?.clay).toBeCloseTo(0.5, 1); + }); + + it("a crowded site is one site for confidence", () => { + const step = stepAt(column(crowd(10, 0.5, CLAY)), 20); + expect(step.shares?.clay).toBe(1); + expect(step).toMatchObject({ confidence: "none", likely: null }); + }); + + it("ponytail limit: more than 12 borewells at the nearest spot leave room for nobody else", () => { + // Today's behaviour, written down so that changing it is a decision: the voters are the 12 + // nearest borewells, not sites. + const step = stepAt(column([...crowd(13, 0.2, CLAY), site("other", north(0), east(-0.5), SAND)]), 20); + expect(step.sources).toHaveLength(12); + expect(step.shares?.waterBearing).toBe(0); + expect(step.confidence).toBe("none"); + }); + + it("matches layers by height, not depth, on sloping ground", () => { + // The ground climbs about 8 ft per km eastwards. Sand lies between 250 and 300 ft above sea + // level everywhere, so it is deeper below ground in the east than in the west. + const slope = slopeGrid(60, 180); + const groundFt = (lon: number) => (60 + ((lon - 80.7) / 0.5) * 120) * 3.28084; + const logged = (id: string, lon: number) => { + const top = groundFt(lon) - 300, bottom = groundFt(lon) - 250; + return site(id, north(0), lon, [[0, top, "clay"], [top, bottom, "medium_sand"], [bottom, 200, "clay"]]); + }; + const col = column([logged("west", east(-2)), logged("east", east(2))], slope); + // At the centre the sand is 93.7 to 143.7 ft down. The west borewell has sand at 87 ft and the + // east one has clay at 142 ft: matching by depth would mix them. + expect(stepAt(col, 87).shares?.clay).toBeCloseTo(1, 6); + expect(stepAt(col, 97).shares?.waterBearing).toBeCloseTo(1, 6); + expect(stepAt(col, 142).shares?.waterBearing).toBeCloseTo(1, 6); + expect(stepAt(col, 147).shares?.clay).toBeCloseTo(1, 6); + expect(stepAt(col, 97).confidence).toBe("rough"); + }); + + it("uses a borewell's own recorded ground height before the grid", () => { + // Surveyed 10 m above what the grid says, so its sand starts 16.9 ft down at the centre, not 50. + const col = column([site("surveyed", north(0), east(0.5), CLAY_OVER_SAND, { groundElevationM: 130 })]); + expect(stepAt(col, 12).shares?.clay).toBe(1); + expect(stepAt(col, 22).shares?.waterBearing).toBe(1); + expect(col).toHaveLength(14); + }); + + it("confidence falls with distance", () => { + expect(stepAt(column(around(0.5, CLAY)), 20).confidence).toBe("likely"); + expect(stepAt(column(around(2, CLAY)), 20).confidence).toBe("rough"); + expect(stepAt(column(around(4, CLAY)), 20)).toMatchObject({ confidence: "none", likely: null }); + }); + + it("confidence falls with disagreement", () => { + const step = stepAt(column([site("a", north(0.5), east(0), CLAY), site("b", north(0), east(0.5), SAND), site("c", north(-3), east(0), CLAY)]), 20); + expect(step).toMatchObject({ confidence: "rough", likely: "clay" }); + expect(step.shares!.clay).toBeLessThan(0.6); + }); + + it("gives a tie to the earlier group", () => { + const step = stepAt(column([site("a", north(0.5), east(0), SAND), site("b", north(-0.5), east(0), CLAY)]), 20); + expect(step.likely).toBe("clay"); + }); + + it("counts poor records and address-located borewells less", () => { + const against = (overrides: Parameters[4]) => + stepAt(column([site("good", north(0), east(1), CLAY), site("weak", north(0), east(-1), SAND, overrides)]), 20).shares!.clay; + expect(against({ recordQuality: "poor" })).toBeCloseTo(1 / 1.4, 3); + expect(against({ recordQuality: "fair" })).toBeCloseTo(1 / 1.7, 3); + expect(against({ locationSource: "address" })).toBeCloseTo(1 / 1.5, 3); + }); + + it("converts borewells logged in metres", () => { + const col = column([site("metric", north(0), east(0.5), [[0, 10, "clay"]], { depthUnit: "m" })]); + expect(col).toHaveLength(7); // 10 m is 32.8 ft + expect(stepAt(col, 27).shares?.clay).toBe(1); + }); + + it("not recorded and gaps never vote", () => { + const col = column([site("patchy", north(0), east(0.5), [[0, 50, "not_recorded"], [60, 100, "clay"]])]); + expect(col).toHaveLength(20); + for (const depth of [20, 52]) expect(stepAt(col, depth)).toMatchObject({ shares: null, likely: null, confidence: "none", sources: [] }); + expect(stepAt(col, 70).shares?.clay).toBe(1); + }); + + it("the driller's tick makes fine sand water-bearing", () => { + const col = column([site("ticked", north(0), east(0.5), [[0, 50, "fine_sand", true], [50, 100, "fine_sand"]])]); + expect(stepAt(col, 20).shares?.waterBearing).toBe(1); + expect(stepAt(col, 70).shares?.fineSand).toBe(1); + }); + + it("gives no answer outside the ground-height grid", () => { + expect(predictColumn(28, 80.95, prepare(around(0.5, CLAY), DEFAULT_MATERIALS, flat), flat)).toBeNull(); + }); + + it("gives an empty column with no borewell within 5 km", () => { + expect(column([site("far", north(6), east(0), CLAY)])).toEqual([]); + }); + + it("exclude hides a borewell from its own prediction", () => { + const prepared = prepare([site("here", CENTRE[0], CENTRE[1], CLAY)], DEFAULT_MATERIALS, flat); + expect(predictColumn(CENTRE[0], CENTRE[1], prepared, flat)).toHaveLength(40); + expect(predictColumn(CENTRE[0], CENTRE[1], prepared, flat, { exclude: "here" })).toEqual([]); + }); + + it("groundM stands in for the grid at the asked point", () => { + const prepared = prepare([site("a", north(0), east(0.5), CLAY_OVER_SAND)], DEFAULT_MATERIALS, flat); + // Standing 10 m higher than the grid says, the sand is 32.8 ft further down. + const col = predictColumn(CENTRE[0], CENTRE[1], prepared, flat, { groundM: 130 })!; + expect(stepAt(col, 77).shares?.clay).toBe(1); + expect(stepAt(col, 87).shares?.waterBearing).toBe(1); + }); +}); + +describe("prepare", () => { + it("leaves out borewells it cannot place", () => { + const prepared = prepare([ + site("ok", north(0), east(0), CLAY), + site("surveyed-outside", 28, 80.95, CLAY, { groundElevationM: 125 }), + site("no-location", null, null, CLAY), + site("outside-grid", 28, 80.95, CLAY), + site("nothing-recorded", north(1), east(0), [[0, 100, "not_recorded"]]), + site("unlinked", north(2), east(0), [[0, 100, null]]), + ], DEFAULT_MATERIALS, flat); + expect(prepared.map((p) => p.id)).toEqual(["ok", "surveyed-outside"]); + }); + + it("turns depths into heights above sea level, in feet", () => { + const [p] = prepare([site("a", north(0), east(0), CLAY_OVER_SAND)], DEFAULT_MATERIALS, flat); + expect(p.groundFt).toBeCloseTo(393.7, 1); + expect(p.runs).toHaveLength(2); + expect(p.runs[1]).toMatchObject({ group: "waterBearing" }); + expect(p.runs[1].topFt).toBeCloseTo(343.7, 1); + expect(p.runs[1].bottomFt).toBeCloseTo(293.7, 1); + expect(p.weight).toBe(1); + }); +}); diff --git a/app/src/model/model.ts b/app/src/model/model.ts new file mode 100644 index 0000000..d2f0735 --- /dev/null +++ b/app/src/model/model.ts @@ -0,0 +1,159 @@ +/** + * The model: at this point, at this depth, what is the ground probably made of, and how sure are we? + * + * A weighted vote of nearby borewells, depth by depth. Layers are compared by height above sea + * level, not depth below ground: the layers were laid flat, the ground above them was not. Nothing + * is stored; the answer is worked out from the records each time, so a new borewell simply joins + * the vote. Every answer can name the borewells behind it. + */ +import { METRES_TO_FEET, heightAt, type Borewell, type ElevationGrid, type Material, type StrataLayer } from "@strata/core"; +import { ADDRESS_WEIGHT, LIKELY, MAX_VOTERS, MIN_DISTANCE_KM, QUALITY_WEIGHT, RADIUS_KM, ROUGH, SAME_SITE_KM, STEP_FT } from "./constants"; +import { GROUPS, groupOf, materialsById, type Group } from "./groups"; + +export interface ModelSite { + borewell: Borewell; + strata: StrataLayer[]; +} + +/** One borewell made ready to vote. Heights are feet above sea level. */ +export interface Prepared { + id: string; + borewellId: string; + latitude: number; + longitude: number; + groundFt: number; + /** The lowest height any of its layers reaches. */ + bottomFt: number; + /** How much its vote counts before distance: record quality × how its location was found. */ + weight: number; + /** Its layers that can vote, top first. */ + runs: { topFt: number; bottomFt: number; group: Group }[]; +} + +export type Confidence = "likely" | "rough" | "none"; + +export interface Source { + id: string; + borewellId: string; + km: number; + weight: number; +} + +export interface Step { + /** Feet below ground at the asked point. */ + top: number; + bottom: number; + /** Each group's share of the vote, adding up to 1; null when no borewell reaches this step. */ + shares: Record | null; + /** The group with the largest share; null when confidence is "none". */ + likely: Group | null; + confidence: Confidence; + /** The borewells that voted, nearest first. */ + sources: Source[]; +} + +/** + * Turns borewells into voters, once per load. Left out: borewells with no location, with no known + * ground height (none recorded and outside the grid), or with no layer that can vote. + */ +export function prepare(sites: ModelSite[], materials: Material[], grid: ElevationGrid): Prepared[] { + const byId = materialsById(materials); + const prepared: Prepared[] = []; + for (const { borewell: b, strata } of sites) { + if (b.latitude == null || b.longitude == null) continue; + // A borewell's own recorded ground height is always preferred to the grid. + const groundM = b.groundElevationM ?? heightAt(grid, b.latitude, b.longitude); + if (groundM == null) continue; + const groundFt = groundM * METRES_TO_FEET; + const toFeet = b.depthUnit === "m" ? METRES_TO_FEET : 1; + const runs: Prepared["runs"] = []; + for (const layer of [...strata].sort((x, y) => x.startDepth - y.startDepth)) { + const group = groupOf(layer, byId); + if (group && layer.endDepth > layer.startDepth) { + runs.push({ topFt: groundFt - layer.startDepth * toFeet, bottomFt: groundFt - layer.endDepth * toFeet, group }); + } + } + if (runs.length === 0) continue; + prepared.push({ + id: b.id, borewellId: b.borewellId, latitude: b.latitude, longitude: b.longitude, groundFt, runs, + bottomFt: Math.min(...runs.map((r) => r.bottomFt)), + weight: (QUALITY_WEIGHT[b.recordQuality] ?? QUALITY_WEIGHT.unknown) * (b.locationSource === "address" ? ADDRESS_WEIGHT : 1), + }); + } + return prepared; +} + +/** What this borewell recorded at a height, or null in a gap, above its top or below its bottom. */ +export function groupAt(p: Prepared, heightFt: number): Group | null { + for (const run of p.runs) if (run.topFt >= heightFt && heightFt > run.bottomFt) return run.group; + return null; +} + +/** + * The ground at a point, in steps from the surface down to the deepest that any borewell within + * reach goes. Returns null when the ground height at the point is unknown, and an empty column when + * no borewell is within reach. `exclude` (a borewell's id) and `groundM` are for the accuracy check. + */ +export function predictColumn( + latitude: number, longitude: number, prepared: Prepared[], grid: ElevationGrid, + options: { exclude?: string; groundM?: number } = {}, +): Step[] | null { + const groundM = options.groundM ?? heightAt(grid, latitude, longitude); + if (groundM == null) return null; + const groundFt = groundM * METRES_TO_FEET; + + // Positions in km east and north of the point: flat-earth, accurate to well under 1% across a city. + const kx = 111.32 * Math.cos((latitude * Math.PI) / 180); + const near: { p: Prepared; x: number; y: number; km: number }[] = []; + for (const p of prepared) { + if (p.id === options.exclude) continue; + const x = (p.longitude - longitude) * kx, y = (p.latitude - latitude) * 110.57, km = Math.hypot(x, y); + if (km <= RADIUS_KM) near.push({ p, x, y, km }); + } + near.sort((a, b) => a.km - b.km); + + const deepestFt = Math.max(0, ...near.map((c) => groundFt - c.p.bottomFt)); + const column: Step[] = []; + for (let top = 0; top < deepestFt; top += STEP_FT) { + const middle = groundFt - (top + STEP_FT / 2); + // ponytail: the 12 nearest borewells, not sites; a spot with 12 or more logs crowds out everyone else. Limit by site if real records show it. + const voters: { c: (typeof near)[number]; group: Group }[] = []; + for (const c of near) { + const group = groupAt(c.p, middle); + if (group && voters.push({ c, group }) === MAX_VOTERS) break; + } + column.push({ top, bottom: top + STEP_FT, ...vote(voters) }); + } + return column; +} + +const apart = (a: { x: number; y: number }, b: { x: number; y: number }) => Math.hypot(a.x - b.x, a.y - b.y); + +function vote(voters: { c: { p: Prepared; x: number; y: number; km: number }; group: Group }[]): Pick { + if (voters.length === 0) return { shares: null, likely: null, confidence: "none", sources: [] }; + + const shares: Record = { clay: 0, fineSand: 0, waterBearing: 0, hard: 0 }; + const sources: Source[] = []; + let total = 0; + for (const { c, group } of voters) { + // Crowded sites don't shout: borewells close together share their vote. + const crowd = voters.filter((other) => apart(other.c, c) <= SAME_SITE_KM).length; + const weight = c.p.weight / Math.max(c.km, MIN_DISTANCE_KM) ** 2 / crowd; + shares[group] += weight; + total += weight; + sources.push({ id: c.p.id, borewellId: c.p.borewellId, km: c.km, weight }); + } + for (const g of GROUPS) shares[g] /= total; + + // Separate sites, nearest first: a voter starts a new one unless it is beside an earlier site. + const sites: (typeof voters)[number]["c"][] = []; + for (const { c } of voters) if (!sites.some((s) => apart(s, c) <= SAME_SITE_KM)) sites.push(c); + + const best = GROUPS.reduce((a, b) => (shares[b] > shares[a] ? b : a)); + const nearestKm = voters[0].c.km; + const confidence: Confidence = + nearestKm <= LIKELY.nearestKm && sites.length >= LIKELY.sites && shares[best] >= LIKELY.share ? "likely" + : nearestKm <= ROUGH.nearestKm && sites.length >= ROUGH.sites ? "rough" + : "none"; + return { shares, likely: confidence === "none" ? null : best, confidence, sources }; +} diff --git a/app/src/model/testGround.ts b/app/src/model/testGround.ts new file mode 100644 index 0000000..d4d9f18 --- /dev/null +++ b/app/src/model/testGround.ts @@ -0,0 +1,34 @@ +/** Made-up ground and borewells for the model's tests. Not used by the app. */ +import type { Borewell, ElevationGrid, StrataLayer } from "@strata/core"; +import type { ModelSite } from "./model"; + +/** The middle of the test area; tests ask about this point unless they say otherwise. */ +export const CENTRE: [number, number] = [26.85, 80.95]; + +/** A latitude `km` north of the centre (negative for south). */ +export const north = (km: number) => CENTRE[0] + km / 110.57; +/** A longitude `km` east of the centre (negative for west). */ +export const east = (km: number) => CENTRE[1] + km / (111.32 * Math.cos((CENTRE[0] * Math.PI) / 180)); + +const AREA = { west: 80.7, east: 81.2, south: 26.6, north: 27.1 }; + +/** Ground that rises evenly from `westMetres` at the west edge to `eastMetres` at the east edge. */ +export const slopeGrid = (westMetres: number, eastMetres: number): ElevationGrid => + ({ cols: 2, rows: 2, ...AREA, data: Int16Array.from([westMetres, eastMetres, westMetres, eastMetres], (m) => m * 10) }); + +export const flatGrid = (metres: number) => slopeGrid(metres, metres); + +/** A borewell with the given layers, each `[top, bottom, soil type id]` or with a fourth `true` for the water-bearing tick. */ +export function site( + id: string, latitude: number | null, longitude: number | null, + layers: [number, number, string | null, boolean?][], overrides: Partial = {}, +): ModelSite { + const borewell = { + id, borewellId: id.toUpperCase(), latitude, longitude, + locationSource: "gps", groundElevationM: null, depthUnit: "ft", recordQuality: "good", + ...overrides, + } as Borewell; + const strata = layers.map(([startDepth, endDepth, materialId, waterBearing = false], i) => + ({ id: `${id}-${i}`, borewellId: id, startDepth, endDepth, material: materialId ?? "", materialId, color: "#000000", pattern: "dots", remarks: "", waterBearing }) as StrataLayer); + return { borewell, strata }; +} From 6dc9412e7bc6c48ad9387ef3d5e8d6344182cb9d Mon Sep 17 00:00:00 2001 From: Mehul Date: Wed, 14 Oct 2026 00:27:49 +0530 Subject: [PATCH 5/7] Add the accuracy check and odd ones out --- app/src/model/check.test.ts | 82 +++++++++++++++++++++++++++++ app/src/model/check.ts | 101 ++++++++++++++++++++++++++++++++++++ app/src/model/model.ts | 5 +- 3 files changed, 187 insertions(+), 1 deletion(-) create mode 100644 app/src/model/check.test.ts create mode 100644 app/src/model/check.ts diff --git a/app/src/model/check.test.ts b/app/src/model/check.test.ts new file mode 100644 index 0000000..a5f328a --- /dev/null +++ b/app/src/model/check.test.ts @@ -0,0 +1,82 @@ +import { describe, expect, it } from "vitest"; +import { DEFAULT_MATERIALS } from "@strata/core"; +import { checkModel, share } from "./check"; +import { prepare, type ModelSite } from "./model"; +import { east, flatGrid, north, site } from "./testGround"; + +const flat = flatGrid(120); +const CLAY_OVER_SAND: [number, number, string][] = [[0, 50, "clay"], [50, 100, "medium_sand"]]; +const SAND_OVER_CLAY: [number, number, string][] = [[0, 50, "medium_sand"], [50, 100, "clay"]]; + +/** `side` × `side` borewells `km` apart, all with the same layers. */ +const town = (side: number, km: number, layers = CLAY_OVER_SAND): ModelSite[] => + Array.from({ length: side * side }, (_, i) => site(`t${i}`, north(Math.floor(i / side) * km), east((i % side) * km), layers)); +const check = (sites: ModelSite[]) => checkModel(prepare(sites, DEFAULT_MATERIALS, flat), flat); + +describe("checkModel", () => { + it("scores a uniform town as always right", () => { + const r = check(town(4, 0.5)); + expect(r.canJudge).toBe(true); + expect(r.compared).toBe(16); + expect(r.overall).toEqual({ steps: 320, right: 320 }); + expect(r.likely).toEqual(r.overall); + expect(r.byDistance.under1).toEqual(r.overall); + expect(r.oddOnesOut).toEqual([]); + }); + + it("finds a borewell that does not fit its surroundings", () => { + const sites = town(4, 0.5); + sites[5] = site("upside-down", sites[5].borewell.latitude, sites[5].borewell.longitude, SAND_OVER_CLAY); + const r = check(sites); + expect(r.oddOnesOut).toHaveLength(1); + expect(r.oddOnesOut[0]).toMatchObject({ id: "upside-down", borewellId: "UPSIDE-DOWN", steps: 20, right: 0, typical: 1 }); + expect(share(r.overall)).toBe(300 / 320); + }); + + it("lists the worst first", () => { + const sites = town(5, 0.5); + sites[6] = site("all-wrong", sites[6].borewell.latitude, sites[6].borewell.longitude, SAND_OVER_CLAY); + // Right for the top 30 ft only: 6 steps of 20. + sites[18] = site("mostly-wrong", sites[18].borewell.latitude, sites[18].borewell.longitude, [[0, 30, "clay"], [30, 50, "medium_sand"], [50, 100, "clay"]]); + expect(check(sites).oddOnesOut.map((o) => o.id)).toEqual(["all-wrong", "mostly-wrong"]); + }); + + it("cannot judge on fewer than ten borewells", () => { + const sites = town(3, 0.5); + sites[4] = site("upside-down", sites[4].borewell.latitude, sites[4].borewell.longitude, SAND_OVER_CLAY); + const r = check(sites); + expect(r.compared).toBe(9); + expect(r.canJudge).toBe(false); + expect(r.oddOnesOut).toEqual([]); + }); + + it("does not divide by zero when nothing is comparable", () => { + // Twelve borewells, each more than 5 km from every other. + const r = check(Array.from({ length: 12 }, (_, i) => site(`far${i}`, north((Math.floor(i / 4) - 1) * 6), east(((i % 4) - 1.5) * 6), CLAY_OVER_SAND))); + expect(r.compared).toBe(0); + expect(r.canJudge).toBe(false); + expect(r.overall).toEqual({ steps: 0, right: 0 }); + expect(share(r.overall)).toBeNull(); + expect(r.oddOnesOut).toEqual([]); + }); + + it("puts borewells in the band of their nearest neighbour, and scores guesses the model would not show", () => { + // Three pairs, far from each other; the borewells of a pair are 0.5, 2 and 4 km apart. + const pair = (name: string, y: number, gap: number) => [site(`${name}a`, north(y), east(-10), CLAY_OVER_SAND), site(`${name}b`, north(y), east(-10 + gap), CLAY_OVER_SAND)]; + const r = check([...pair("near", -12, 0.5), ...pair("mid", 0, 2), ...pair("far", 12, 4)]); + const both = { steps: 40, right: 40 }; + expect(r.byDistance).toEqual({ under1: both, from1to3: both, over3: both }); + // One borewell is one site, which the model never shows as an estimate; the check still scores it. + expect(r.none).toEqual({ steps: 120, right: 120 }); + expect(r.likely.steps + r.rough.steps).toBe(0); + }); + + it("compares by height, using the hidden borewell's own ground", () => { + // Surveyed 10 m above its neighbours, and logged accordingly: the same layers by height. + const sites = town(4, 0.5); + const b = sites[5].borewell; + sites[5] = site("on-a-mound", b.latitude, b.longitude, [[0, 82.8, "clay"], [82.8, 132.8, "medium_sand"]], { groundElevationM: 130 }); + const r = check(sites); + expect(r.oddOnesOut).toEqual([]); + }); +}); diff --git a/app/src/model/check.ts b/app/src/model/check.ts new file mode 100644 index 0000000..60f7107 --- /dev/null +++ b/app/src/model/check.ts @@ -0,0 +1,101 @@ +/** + * The honesty check: hide each borewell in turn, predict it from the others, and count how often + * the model's best guess matched what was really logged. This is the number that should go up as + * borewells are added, and it shows whether "likely" really means likely on the real records. + * + * It also finds the odd ones out: borewells that disagree with their surroundings far more than is + * usual, which are most often a slip in typing or a pin in the wrong place. + */ +import { METRES_TO_FEET, kmBetween, type ElevationGrid } from "@strata/core"; +import { MIN_BOREWELLS_TO_JUDGE, ODD } from "./constants"; +import { groupAt, largest, predictColumn, type Prepared } from "./model"; + +/** How many steps were compared, and in how many the best guess was right. */ +export interface Tally { + steps: number; + right: number; +} + +export interface OddOne { + id: string; + borewellId: string; + steps: number; + right: number; + /** The usual share right for borewells with neighbours as near as this one's. */ + typical: number; +} + +type Band = "under1" | "from1to3" | "over3"; + +export interface CheckResult { + /** False with too few borewells to compare; the numbers are then not worth showing. */ + canJudge: boolean; + /** Borewells for which at least one step could be compared. */ + compared: number; + overall: Tally; + /** By how sure the model said it was. "none" steps are guesses it would not show as estimates. */ + likely: Tally; + rough: Tally; + none: Tally; + /** By how far each borewell is from its nearest neighbour, in km. */ + byDistance: Record; + /** Worst first. */ + oddOnesOut: OddOne[]; +} + +/** The share right, or null when nothing was compared. */ +export const share = (t: Tally) => (t.steps === 0 ? null : t.right / t.steps); + +const tally = (): Tally => ({ steps: 0, right: 0 }); + +/** The middle value. (core's `median` rounds to one decimal, which is too coarse for shares.) */ +function middle(values: number[]): number | null { + if (values.length === 0) return null; + const s = [...values].sort((a, b) => a - b), m = s.length >> 1; + return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2; +} + +export function checkModel(prepared: Prepared[], grid: ElevationGrid): CheckResult { + const result: CheckResult = { + canJudge: false, compared: 0, overall: tally(), likely: tally(), rough: tally(), none: tally(), + byDistance: { under1: tally(), from1to3: tally(), over3: tally() }, oddOnesOut: [], + }; + const each: { p: Prepared; band: Band; own: Tally }[] = []; + + for (const p of prepared) { + // Predicted standing where it stands: same place, same ground height, itself left out. + const column = predictColumn(p.latitude, p.longitude, prepared, grid, { exclude: p.id, groundM: p.groundFt / METRES_TO_FEET }); + if (!column) continue; + let nearestKm = Infinity; + for (const q of prepared) if (q !== p) nearestKm = Math.min(nearestKm, kmBetween(p.latitude, p.longitude, q.latitude, q.longitude)); + const band: Band = nearestKm < 1 ? "under1" : nearestKm <= 3 ? "from1to3" : "over3"; + + const own = tally(); + for (const step of column) { + const logged = groupAt(p, p.groundFt - (step.top + step.bottom) / 2); + if (!step.shares || !logged) continue; // nobody else reaches this step, or it logged nothing here + const right = largest(step.shares) === logged ? 1 : 0; + for (const t of [own, result.overall, result[step.confidence], result.byDistance[band]]) { + t.steps++; + t.right += right; + } + } + if (own.steps > 0) each.push({ p, band, own }); + } + + result.compared = each.length; + result.canJudge = each.length >= MIN_BOREWELLS_TO_JUDGE; + if (!result.canJudge) return result; + + const judged = each.filter((e) => e.own.steps >= ODD.minSteps); + for (const band of ["under1", "from1to3", "over3"] as const) { + const inBand = judged.filter((e) => e.band === band); + const typical = middle(inBand.map((e) => e.own.right / e.own.steps)); + if (typical == null) continue; + for (const { p, own } of inBand) { + if (own.right / own.steps < ODD.fraction * typical) result.oddOnesOut.push({ id: p.id, borewellId: p.borewellId, ...own, typical }); + } + } + result.oddOnesOut.sort((a, b) => a.right / a.steps - b.right / b.steps); + return result; +} diff --git a/app/src/model/model.ts b/app/src/model/model.ts index d2f0735..4d976f9 100644 --- a/app/src/model/model.ts +++ b/app/src/model/model.ts @@ -127,6 +127,9 @@ export function predictColumn( return column; } +/** The group with the largest share; a tie goes to the earlier group. */ +export const largest = (shares: Record) => GROUPS.reduce((a, b) => (shares[b] > shares[a] ? b : a)); + const apart = (a: { x: number; y: number }, b: { x: number; y: number }) => Math.hypot(a.x - b.x, a.y - b.y); function vote(voters: { c: { p: Prepared; x: number; y: number; km: number }; group: Group }[]): Pick { @@ -149,7 +152,7 @@ function vote(voters: { c: { p: Prepared; x: number; y: number; km: number }; gr const sites: (typeof voters)[number]["c"][] = []; for (const { c } of voters) if (!sites.some((s) => apart(s, c) <= SAME_SITE_KM)) sites.push(c); - const best = GROUPS.reduce((a, b) => (shares[b] > shares[a] ? b : a)); + const best = largest(shares); const nearestKm = voters[0].c.km; const confidence: Confidence = nearestKm <= LIKELY.nearestKm && sites.length >= LIKELY.sites && shares[best] >= LIKELY.share ? "likely" From 18d11c7ea46a3e1f939fa0dd91486dc3db25e2e0 Mon Sep 17 00:00:00 2001 From: Mehul Date: Wed, 14 Oct 2026 00:33:17 +0530 Subject: [PATCH 6/7] Prove the model on a made-up city of 500 borewells --- app/src/model/city.test.ts | 77 ++++++++++++++++++++++++++++++++++++ app/src/model/madeUpCity.ts | 78 +++++++++++++++++++++++++++++++++++++ app/src/model/model.ts | 72 +++++++++++++++++++++++----------- 3 files changed, 205 insertions(+), 22 deletions(-) create mode 100644 app/src/model/city.test.ts create mode 100644 app/src/model/madeUpCity.ts diff --git a/app/src/model/city.test.ts b/app/src/model/city.test.ts new file mode 100644 index 0000000..a678ae4 --- /dev/null +++ b/app/src/model/city.test.ts @@ -0,0 +1,77 @@ +/** + * The model on a made-up city of 500 borewells whose true ground is known: is it right where it + * says it is sure, does it improve with more borewells, and is it fast enough for the screens? + */ +import { describe, expect, it } from "vitest"; +import { DEFAULT_MATERIALS, METRES_TO_FEET, heightAt } from "@strata/core"; +import { checkModel, share } from "./check"; +import { madeUpCity, truth } from "./madeUpCity"; +import { predictColumn, prepare } from "./model"; + +const city = (count: number) => { + const { sites, grid } = madeUpCity(count); + return { grid, prepared: prepare(sites, DEFAULT_MATERIALS, grid) }; +}; +const { grid, prepared } = city(500); +const result = checkModel(prepared, grid); + +/** Milliseconds taken by `work`. */ +const timed = (work: () => void) => { + const start = performance.now(); + work(); + return performance.now() - start; +}; + +describe("a made-up city of 500 borewells", () => { + it("has many borewells at the same spots, like the real records", () => { + expect(prepared).toHaveLength(500); + expect(result.compared).toBe(500); + expect(result.canJudge).toBe(true); + }); + + it("is right at least 7 times in 10 where it says likely", () => { + console.log("made-up city, share right:", { + overall: share(result.overall), likely: share(result.likely), rough: share(result.rough), none: share(result.none), + under1: share(result.byDistance.under1), from1to3: share(result.byDistance.from1to3), over3: share(result.byDistance.over3), + likelySteps: result.likely.steps, allSteps: result.overall.steps, odd: result.oddOnesOut.length, + }); + expect(share(result.likely)!).toBeGreaterThanOrEqual(0.7); + expect(share(result.likely)!).toBeGreaterThan(share(result.rough)!); + // "Likely" must not be bought by hardly ever saying it. + expect(result.likely.steps).toBeGreaterThan(result.overall.steps * 0.5); + }); + + it("flags few honest borewells as odd", () => { + expect(result.oddOnesOut.length).toBeLessThanOrEqual(10); + }); + + it("gets better with more borewells", () => { + const small = city(100); + expect(share(checkModel(small.prepared, small.grid).overall)!).toBeLessThan(share(result.overall)!); + }); + + it("agrees with the truth at a place nobody drilled", () => { + const [latitude, longitude] = [26.87, 80.97]; + const groundFt = heightAt(grid, latitude, longitude)! * METRES_TO_FEET; + const sure = predictColumn(latitude, longitude, prepared, grid)!.filter((s) => s.confidence === "likely"); + expect(sure.length).toBeGreaterThan(20); + const right = sure.filter((s) => s.likely === truth(latitude, longitude, groundFt - (s.top + s.bottom) / 2)); + expect(right.length / sure.length).toBeGreaterThanOrEqual(0.7); + }); + + it("is fast enough", { timeout: 30_000 }, () => { + const times = { + column: timed(() => predictColumn(26.87, 80.97, prepared, grid)), + // A section: 200 columns along a line across the city. + section: timed(() => { for (let i = 0; i < 200; i++) predictColumn(26.76 + i * 0.0009, 80.86 + i * 0.0009, prepared, grid); }), + // A city map: 100 × 100 points. + map: timed(() => { for (let i = 0; i < 100; i++) for (let j = 0; j < 100; j++) predictColumn(26.75 + i * 0.002, 80.85 + j * 0.002, prepared, grid); }), + check: timed(() => checkModel(prepared, grid)), + }; + console.log("made-up city, milliseconds:", times); + expect(times.column).toBeLessThan(20); + expect(times.section).toBeLessThan(1000); + expect(times.map).toBeLessThan(5000); + expect(times.check).toBeLessThan(5000); + }); +}); diff --git a/app/src/model/madeUpCity.ts b/app/src/model/madeUpCity.ts new file mode 100644 index 0000000..5c86200 --- /dev/null +++ b/app/src/model/madeUpCity.ts @@ -0,0 +1,78 @@ +/** + * A made-up city whose true ground is known everywhere, for testing the model at the real size + * (about 500 borewells, many of them at the same spots). Not used by the app. + */ +import { METRES_TO_FEET, heightAt, type ElevationGrid } from "@strata/core"; +import type { Group } from "./groups"; +import type { ModelSite } from "./model"; +import { CENTRE, site, slopeGrid } from "./testGround"; + +const SOIL: Record = { clay: "clay", waterBearing: "medium_sand", fineSand: "fine_sand", hard: "kankar" }; + +/** + * Going down: clay, water-bearing sand, fine sand, hard ground, then water-bearing sand again. + * Each boundary sits at `base` feet above sea level and rises and falls by 15 ft across the city + * as a smooth wave 8 km long, each running in its own direction. + */ +const BOUNDARIES = [ + { base: 330, direction: 0.3, shift: 0 }, + { base: 280, direction: 1.4, shift: 2 }, + { base: 250, direction: 2.5, shift: 4 }, + { base: 220, direction: 4.0, shift: 1 }, +]; +const BELOW: Group[] = ["clay", "waterBearing", "fineSand", "hard", "waterBearing"]; + +/** What the made-up ground really is at a place and a height above sea level, in feet. */ +export function truth(latitude: number, longitude: number, heightFt: number): Group { + const x = (longitude - CENTRE[1]) * 111.32 * Math.cos((CENTRE[0] * Math.PI) / 180), y = (latitude - CENTRE[0]) * 110.57; + let ceiling = Infinity; + for (let i = 0; i < BOUNDARIES.length; i++) { + const { base, direction, shift } = BOUNDARIES[i]; + const wave = Math.sin((2 * Math.PI * (x * Math.cos(direction) + y * Math.sin(direction))) / 8 + shift); + ceiling = Math.min(ceiling, base + 15 * wave); // a boundary never rises above the one over it + if (heightFt > ceiling) return BELOW[i]; + } + return BELOW[BOUNDARIES.length]; +} + +/** A small repeatable random-number function (mulberry32), so the city is the same on every run. */ +function random(seed: number) { + return () => { + seed = (seed + 0x6d2b79f5) | 0; + let t = Math.imul(seed ^ (seed >>> 15), 1 | seed); + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +/** + * `count` borewells logged from `truth`: 70% spread over the city, 30% within 50 m of an earlier + * one, each 150 to 400 ft deep. Always the same for the same seed. + */ +export function madeUpCity(count: number, seed = 1): { sites: ModelSite[]; grid: ElevationGrid } { + const grid = slopeGrid(110, 125); + const next = random(seed); + const sites: ModelSite[] = []; + for (let i = 0; i < count; i++) { + let latitude: number, longitude: number; + if (i > 0 && next() < 0.3) { + const beside = sites[Math.floor(next() * sites.length)].borewell; + latitude = beside.latitude! + ((next() - 0.5) * 0.07) / 110.57; + longitude = beside.longitude! + ((next() - 0.5) * 0.07) / 99.3; + } else { + latitude = 26.75 + next() * 0.2; + longitude = 80.85 + next() * 0.2; + } + const groundFt = heightAt(grid, latitude, longitude)! * METRES_TO_FEET; + const depth = 150 + Math.floor(next() * 51) * 5; + const layers: [number, number, string][] = []; + for (let top = 0; top < depth; top += 5) { + const soil = SOIL[truth(latitude, longitude, groundFt - (top + 2.5))]; + const last = layers[layers.length - 1]; + if (last && last[2] === soil) last[1] = top + 5; + else layers.push([top, top + 5, soil]); + } + sites.push(site(`m${i}`, latitude, longitude, layers)); + } + return { sites, grid }; +} diff --git a/app/src/model/model.ts b/app/src/model/model.ts index 4d976f9..3bca4e5 100644 --- a/app/src/model/model.ts +++ b/app/src/model/model.ts @@ -89,6 +89,9 @@ export function groupAt(p: Prepared, heightFt: number): Group | null { return null; } +/** The group with the largest share; a tie goes to the earlier group. */ +export const largest = (shares: Record) => GROUPS.reduce((a, b) => (shares[b] > shares[a] ? b : a)); + /** * The ground at a point, in steps from the surface down to the deepest that any borewell within * reach goes. Returns null when the ground height at the point is unknown, and an empty column when @@ -104,59 +107,84 @@ export function predictColumn( // Positions in km east and north of the point: flat-earth, accurate to well under 1% across a city. const kx = 111.32 * Math.cos((latitude * Math.PI) / 180); - const near: { p: Prepared; x: number; y: number; km: number }[] = []; + const near: Near[] = []; for (const p of prepared) { if (p.id === options.exclude) continue; const x = (p.longitude - longitude) * kx, y = (p.latitude - latitude) * 110.57, km = Math.hypot(x, y); - if (km <= RADIUS_KM) near.push({ p, x, y, km }); + if (km <= RADIUS_KM) near.push({ p, x, y, km, run: 0 }); } near.sort((a, b) => a.km - b.km); const deepestFt = Math.max(0, ...near.map((c) => groundFt - c.p.bottomFt)); const column: Step[] = []; + let who: Near[] = [], said: Group[] = [], seats: Seats | null = null, counted: Counted | null = null; for (let top = 0; top < deepestFt; top += STEP_FT) { const middle = groundFt - (top + STEP_FT / 2); // ponytail: the 12 nearest borewells, not sites; a spot with 12 or more logs crowds out everyone else. Limit by site if real records show it. - const voters: { c: (typeof near)[number]; group: Group }[] = []; + const nowWho: Near[] = [], nowSaid: Group[] = []; + // Going down, each borewell is followed through its own layers (`run`), and dropped for good + // once the step is below its bottom. This is `groupAt` without starting from the top each time. + let kept = 0; for (const c of near) { - const group = groupAt(c.p, middle); - if (group && voters.push({ c, group }) === MAX_VOTERS) break; + const runs = c.p.runs; + while (c.run < runs.length && runs[c.run].bottomFt >= middle) c.run++; + if (c.run === runs.length) continue; + near[kept++] = c; + if (nowWho.length < MAX_VOTERS && runs[c.run].topFt >= middle) { + nowWho.push(c); + nowSaid.push(runs[c.run].group); + } } - column.push({ top, bottom: top + STEP_FT, ...vote(voters) }); + near.length = kept; + // From one step down to the next the same borewells usually vote, and often the same way, so + // the work is shared: steps of one column may hold the same `shares` and `sources`. + const sameWho = seats !== null && nowWho.length === who.length && nowWho.every((c, i) => c === who[i]); + if (!seats || !sameWho) seats = seat(nowWho); + if (!counted || !sameWho || nowSaid.some((g, i) => g !== said[i])) counted = count(seats, nowSaid); + who = nowWho; + said = nowSaid; + column.push({ top, bottom: top + STEP_FT, ...counted }); } return column; } -/** The group with the largest share; a tie goes to the earlier group. */ -export const largest = (shares: Record) => GROUPS.reduce((a, b) => (shares[b] > shares[a] ? b : a)); +/** A borewell within reach of the asked point: `x`, `y` and `km` are from that point; `run` is the layer it has got down to. */ +type Near = { p: Prepared; x: number; y: number; km: number; run: number }; -const apart = (a: { x: number; y: number }, b: { x: number; y: number }) => Math.hypot(a.x - b.x, a.y - b.y); +/** Who votes on a step and how much each counts, before anyone says what they logged. */ +type Seats = { sources: Source[]; total: number; sites: number; nearestKm: number }; +type Counted = Pick; -function vote(voters: { c: { p: Prepared; x: number; y: number; km: number }; group: Group }[]): Pick { - if (voters.length === 0) return { shares: null, likely: null, confidence: "none", sources: [] }; +const beside = (a: Near, b: Near) => (a.x - b.x) ** 2 + (a.y - b.y) ** 2 <= SAME_SITE_KM ** 2; - const shares: Record = { clay: 0, fineSand: 0, waterBearing: 0, hard: 0 }; +function seat(voters: Near[]): Seats { const sources: Source[] = []; let total = 0; - for (const { c, group } of voters) { + for (const c of voters) { // Crowded sites don't shout: borewells close together share their vote. - const crowd = voters.filter((other) => apart(other.c, c) <= SAME_SITE_KM).length; + let crowd = 0; + for (const other of voters) if (beside(other, c)) crowd++; const weight = c.p.weight / Math.max(c.km, MIN_DISTANCE_KM) ** 2 / crowd; - shares[group] += weight; total += weight; sources.push({ id: c.p.id, borewellId: c.p.borewellId, km: c.km, weight }); } - for (const g of GROUPS) shares[g] /= total; - // Separate sites, nearest first: a voter starts a new one unless it is beside an earlier site. - const sites: (typeof voters)[number]["c"][] = []; - for (const { c } of voters) if (!sites.some((s) => apart(s, c) <= SAME_SITE_KM)) sites.push(c); + const sites: Near[] = []; + for (const c of voters) if (!sites.some((s) => beside(s, c))) sites.push(c); + return { sources, total, sites: sites.length, nearestKm: voters[0]?.km ?? Infinity }; +} + +function count({ sources, total, sites, nearestKm }: Seats, said: Group[]): Counted { + if (sources.length === 0) return { shares: null, likely: null, confidence: "none", sources }; + + const shares: Record = { clay: 0, fineSand: 0, waterBearing: 0, hard: 0 }; + for (let i = 0; i < said.length; i++) shares[said[i]] += sources[i].weight; + for (const g of GROUPS) shares[g] /= total; // divided once at the end, so a unanimous vote is exactly 1 const best = largest(shares); - const nearestKm = voters[0].c.km; const confidence: Confidence = - nearestKm <= LIKELY.nearestKm && sites.length >= LIKELY.sites && shares[best] >= LIKELY.share ? "likely" - : nearestKm <= ROUGH.nearestKm && sites.length >= ROUGH.sites ? "rough" + nearestKm <= LIKELY.nearestKm && sites >= LIKELY.sites && shares[best] >= LIKELY.share ? "likely" + : nearestKm <= ROUGH.nearestKm && sites >= ROUGH.sites ? "rough" : "none"; return { shares, likely: confidence === "none" ? null : best, confidence, sources }; } From 9dcf39ff4a022b6952d0551a4683026aedbb203a Mon Sep 17 00:00:00 2001 From: Mehul Date: Wed, 14 Oct 2026 00:45:13 +0530 Subject: [PATCH 7/7] Fixes from review: feet only, a seat limit per spot, a stricter accuracy check, a harder made-up city - Layer depths are always feet in StrataField; the metres conversion was wrong and is gone. - No spot gets more than 3 of the 12 votes, so a place with many logs cannot crowd out the rest. - The accuracy check hides everything at a borewell's spot, takes the distance step by step, keeps guesses it would not show out of the headline, and needs 5 borewells to call one odd. - The made-up ground now has layers that thin out and vanish; the model must beat guessing by height alone, and must find most logs planted upside down. - The map timing limit is looser on CI runners. --- app/src/model/check.test.ts | 26 ++++++- app/src/model/check.ts | 70 ++++++++++++------- app/src/model/city.test.ts | 51 ++++++++++++-- app/src/model/constants.ts | 8 ++- app/src/model/madeUpCity.ts | 25 +++++-- app/src/model/model.test.ts | 28 +++++--- app/src/model/model.ts | 32 ++++++--- .../specs/2026-10-13-stratavision-design.md | 21 +++--- 8 files changed, 190 insertions(+), 71 deletions(-) diff --git a/app/src/model/check.test.ts b/app/src/model/check.test.ts index a5f328a..313f8c0 100644 --- a/app/src/model/check.test.ts +++ b/app/src/model/check.test.ts @@ -41,6 +41,24 @@ describe("checkModel", () => { expect(check(sites).oddOnesOut.map((o) => o.id)).toEqual(["all-wrong", "mostly-wrong"]); }); + it("hides everything at the spot, so borewells side by side cannot vouch for each other", () => { + const sites = town(4, 0.5); + const b = sites[5].borewell; + sites[5] = site("twin-a", b.latitude, b.longitude, SAND_OVER_CLAY); + sites.push(site("twin-b", b.latitude! + 0.03 / 110.57, b.longitude, SAND_OVER_CLAY)); + expect(check(sites).oddOnesOut.map((o) => o.id).sort()).toEqual(["twin-a", "twin-b"]); + }); + + it("does not blame a borewell for what lies far below its neighbour", () => { + // An honest borewell 4 km out of town in different ground, with a shallow one beside it. + const far = east(1.5 + 4); + const r = check([...town(4, 0.5), site("edge-deep", north(0), far, [[0, 30, "clay"], [30, 100, "kankar"]]), site("edge-shallow", north(0.03), far, [[0, 30, "clay"]])]); + expect(r.oddOnesOut).toEqual([]); + expect(r.byDistance.under1).toEqual({ steps: 320, right: 320 }); + expect(r.byDistance.over3.steps).toBeGreaterThan(0); + expect(r.byDistance.over3.right).toBeLessThan(r.byDistance.over3.steps); + }); + it("cannot judge on fewer than ten borewells", () => { const sites = town(3, 0.5); sites[4] = site("upside-down", sites[4].borewell.latitude, sites[4].borewell.longitude, SAND_OVER_CLAY); @@ -66,9 +84,10 @@ describe("checkModel", () => { const r = check([...pair("near", -12, 0.5), ...pair("mid", 0, 2), ...pair("far", 12, 4)]); const both = { steps: 40, right: 40 }; expect(r.byDistance).toEqual({ under1: both, from1to3: both, over3: both }); - // One borewell is one site, which the model never shows as an estimate; the check still scores it. + // One borewell is one site, which the model never shows as an estimate; the check still scores it, + // but keeps it out of the headline. expect(r.none).toEqual({ steps: 120, right: 120 }); - expect(r.likely.steps + r.rough.steps).toBe(0); + expect(r.overall).toEqual({ steps: 0, right: 0 }); }); it("compares by height, using the hidden borewell's own ground", () => { @@ -77,6 +96,7 @@ describe("checkModel", () => { const b = sites[5].borewell; sites[5] = site("on-a-mound", b.latitude, b.longitude, [[0, 82.8, "clay"], [82.8, 132.8, "medium_sand"]], { groundElevationM: 130 }); const r = check(sites); - expect(r.oddOnesOut).toEqual([]); + expect(r.overall.steps).toBe(320); + expect(r.overall.right).toBe(320); }); }); diff --git a/app/src/model/check.ts b/app/src/model/check.ts index 60f7107..d56882f 100644 --- a/app/src/model/check.ts +++ b/app/src/model/check.ts @@ -3,11 +3,14 @@ * the model's best guess matched what was really logged. This is the number that should go up as * borewells are added, and it shows whether "likely" really means likely on the real records. * + * Everything at the hidden borewell's spot is hidden with it. The records have many borewells side + * by side, and two logs of the same spot agreeing says nothing about a place nobody drilled. + * * It also finds the odd ones out: borewells that disagree with their surroundings far more than is * usual, which are most often a slip in typing or a pin in the wrong place. */ import { METRES_TO_FEET, kmBetween, type ElevationGrid } from "@strata/core"; -import { MIN_BOREWELLS_TO_JUDGE, ODD } from "./constants"; +import { MIN_BOREWELLS_TO_JUDGE, ODD, SAME_SITE_KM } from "./constants"; import { groupAt, largest, predictColumn, type Prepared } from "./model"; /** How many steps were compared, and in how many the best guess was right. */ @@ -19,25 +22,28 @@ export interface Tally { export interface OddOne { id: string; borewellId: string; + /** Its own score where it fits worst: steps compared, and how many matched. */ steps: number; right: number; - /** The usual share right for borewells with neighbours as near as this one's. */ + /** The usual share right for borewells judged from as far away. */ typical: number; } type Band = "under1" | "from1to3" | "over3"; +const BANDS: Band[] = ["under1", "from1to3", "over3"]; export interface CheckResult { /** False with too few borewells to compare; the numbers are then not worth showing. */ canJudge: boolean; /** Borewells for which at least one step could be compared. */ compared: number; + /** Steps the model would show as an estimate: likely and rough together. */ overall: Tally; - /** By how sure the model said it was. "none" steps are guesses it would not show as estimates. */ likely: Tally; rough: Tally; + /** Best guesses the model would not show ("not enough data"), scored all the same. */ none: Tally; - /** By how far each borewell is from its nearest neighbour, in km. */ + /** Every scored step, by how far away the nearest borewell behind that step was, in km. */ byDistance: Record; /** Worst first. */ oddOnesOut: OddOne[]; @@ -47,6 +53,10 @@ export interface CheckResult { export const share = (t: Tally) => (t.steps === 0 ? null : t.right / t.steps); const tally = (): Tally => ({ steps: 0, right: 0 }); +const add = (t: Tally, right: number) => { + t.steps++; + t.right += right; +}; /** The middle value. (core's `median` rounds to one decimal, which is too coarse for shares.) */ function middle(values: number[]): number | null { @@ -60,42 +70,54 @@ export function checkModel(prepared: Prepared[], grid: ElevationGrid): CheckResu canJudge: false, compared: 0, overall: tally(), likely: tally(), rough: tally(), none: tally(), byDistance: { under1: tally(), from1to3: tally(), over3: tally() }, oddOnesOut: [], }; - const each: { p: Prepared; band: Band; own: Tally }[] = []; + /** Each compared borewell's own score, by how far away it was judged from. */ + const each: { p: Prepared; own: Record }[] = []; for (const p of prepared) { - // Predicted standing where it stands: same place, same ground height, itself left out. - const column = predictColumn(p.latitude, p.longitude, prepared, grid, { exclude: p.id, groundM: p.groundFt / METRES_TO_FEET }); + const spot = new Set(); + for (const q of prepared) if (kmBetween(p.latitude, p.longitude, q.latitude, q.longitude) <= SAME_SITE_KM) spot.add(q.id); + // Predicted standing where it stands: same place, same ground height, its own spot left out. + const column = predictColumn(p.latitude, p.longitude, prepared, grid, { exclude: spot, groundM: p.groundFt / METRES_TO_FEET }); if (!column) continue; - let nearestKm = Infinity; - for (const q of prepared) if (q !== p) nearestKm = Math.min(nearestKm, kmBetween(p.latitude, p.longitude, q.latitude, q.longitude)); - const band: Band = nearestKm < 1 ? "under1" : nearestKm <= 3 ? "from1to3" : "over3"; - const own = tally(); + const own: Record = { under1: tally(), from1to3: tally(), over3: tally() }; + let any = false; for (const step of column) { const logged = groupAt(p, p.groundFt - (step.top + step.bottom) / 2); if (!step.shares || !logged) continue; // nobody else reaches this step, or it logged nothing here const right = largest(step.shares) === logged ? 1 : 0; - for (const t of [own, result.overall, result[step.confidence], result.byDistance[band]]) { - t.steps++; - t.right += right; - } + // A deep borewell beside a shallow one is judged from far away below the shallow one's bottom, + // so the distance is taken step by step, not once for the borewell. + const km = step.sources[0].km; + const band: Band = km < 1 ? "under1" : km <= 3 ? "from1to3" : "over3"; + add(result[step.confidence], right); + if (step.confidence !== "none") add(result.overall, right); + add(result.byDistance[band], right); + add(own[band], right); + any = true; } - if (own.steps > 0) each.push({ p, band, own }); + if (any) each.push({ p, own }); } result.compared = each.length; result.canJudge = each.length >= MIN_BOREWELLS_TO_JUDGE; if (!result.canJudge) return result; - const judged = each.filter((e) => e.own.steps >= ODD.minSteps); - for (const band of ["under1", "from1to3", "over3"] as const) { - const inBand = judged.filter((e) => e.band === band); - const typical = middle(inBand.map((e) => e.own.right / e.own.steps)); - if (typical == null) continue; - for (const { p, own } of inBand) { - if (own.right / own.steps < ODD.fraction * typical) result.oddOnesOut.push({ id: p.id, borewellId: p.borewellId, ...own, typical }); + // A borewell is odd where it matches less than half as often as is usual at that distance. It is + // listed once, with its score where it fits worst. + const worst = new Map(); + for (const band of BANDS) { + const judged = each.filter((e) => e.own[band].steps >= ODD.minSteps); + if (judged.length < ODD.minBorewells) continue; + const typical = middle(judged.map((e) => share(e.own[band])!))!; + for (const { p, own } of judged) { + const mine = share(own[band])!; + const before = worst.get(p.id); + if (mine < ODD.fraction * typical && (!before || mine < before.right / before.steps)) { + worst.set(p.id, { id: p.id, borewellId: p.borewellId, ...own[band], typical }); + } } } - result.oddOnesOut.sort((a, b) => a.right / a.steps - b.right / b.steps); + result.oddOnesOut = [...worst.values()].sort((a, b) => a.right / a.steps - b.right / b.steps); return result; } diff --git a/app/src/model/city.test.ts b/app/src/model/city.test.ts index a678ae4..a10a446 100644 --- a/app/src/model/city.test.ts +++ b/app/src/model/city.test.ts @@ -6,7 +6,8 @@ import { describe, expect, it } from "vitest"; import { DEFAULT_MATERIALS, METRES_TO_FEET, heightAt } from "@strata/core"; import { checkModel, share } from "./check"; import { madeUpCity, truth } from "./madeUpCity"; -import { predictColumn, prepare } from "./model"; +import { GROUPS, type Group } from "./groups"; +import { predictColumn, prepare, type Prepared } from "./model"; const city = (count: number) => { const { sites, grid } = madeUpCity(count); @@ -15,6 +16,26 @@ const city = (count: number) => { const { grid, prepared } = city(500); const result = checkModel(prepared, grid); +/** + * How often you would be right knowing only the height above sea level and nothing about the + * place: at each height, guess whatever is commonest there across the whole city. + */ +function guessingByHeightAlone(all: Prepared[]) { + const steps: { band: number; group: Group }[] = []; + for (const p of all) for (const run of p.runs) for (let h = run.topFt - 2.5; h > run.bottomFt; h -= 5) steps.push({ band: Math.floor(h / 5), group: run.group }); + const seen = new Map>(); + for (const { band, group } of steps) { + const counts = seen.get(band) ?? { clay: 0, fineSand: 0, waterBearing: 0, hard: 0 }; + counts[group]++; + seen.set(band, counts); + } + const commonest = (band: number) => GROUPS.reduce((a, b) => (seen.get(band)![b] > seen.get(band)![a] ? b : a)); + return steps.filter((s) => commonest(s.band) === s.group).length / steps.length; +} + +/** GitHub's shared runners can be a few times slower than a desk computer. */ +const SLOW = (globalThis as { process?: { env: Record } }).process?.env.CI ? 3 : 1; + /** Milliseconds taken by `work`. */ const timed = (work: () => void) => { const start = performance.now(); @@ -25,10 +46,28 @@ const timed = (work: () => void) => { describe("a made-up city of 500 borewells", () => { it("has many borewells at the same spots, like the real records", () => { expect(prepared).toHaveLength(500); - expect(result.compared).toBe(500); + const sharing = prepared.filter((p) => prepared.some((q) => q !== p && Math.hypot((p.latitude - q.latitude) * 110.57, (p.longitude - q.longitude) * 99.3) <= 0.1)); + expect(sharing.length).toBeGreaterThan(150); expect(result.canJudge).toBe(true); }); + it("is ground where the place matters, and the model uses the place", () => { + const blind = guessingByHeightAlone(prepared); + console.log("made-up city, right by height alone:", blind); + expect(blind).toBeLessThan(0.75); + expect(share(result.overall)!).toBeGreaterThan(blind + 0.15); + }); + + it("finds logs typed in upside down, and few honest ones", () => { + const { sites, grid: g, planted } = madeUpCity(500, 1, 10); + const odd = checkModel(prepare(sites, DEFAULT_MATERIALS, g), g).oddOnesOut.map((o) => o.id); + const found = planted.filter((id) => odd.includes(id)); + console.log("made-up city, upside-down logs found:", found.length, "of", planted.length, "honest flagged:", odd.length - found.length); + expect(planted).toHaveLength(50); + expect(found.length).toBeGreaterThanOrEqual(35); // most of them + expect(odd.length - found.length).toBeLessThanOrEqual(10); + }); + it("is right at least 7 times in 10 where it says likely", () => { console.log("made-up city, share right:", { overall: share(result.overall), likely: share(result.likely), rough: share(result.rough), none: share(result.none), @@ -69,9 +108,9 @@ describe("a made-up city of 500 borewells", () => { check: timed(() => checkModel(prepared, grid)), }; console.log("made-up city, milliseconds:", times); - expect(times.column).toBeLessThan(20); - expect(times.section).toBeLessThan(1000); - expect(times.map).toBeLessThan(5000); - expect(times.check).toBeLessThan(5000); + expect(times.column).toBeLessThan(20 * SLOW); + expect(times.section).toBeLessThan(1000 * SLOW); + expect(times.map).toBeLessThan(5000 * SLOW); + expect(times.check).toBeLessThan(5000 * SLOW); }); }); diff --git a/app/src/model/constants.ts b/app/src/model/constants.ts index 13c043c..86ccdbb 100644 --- a/app/src/model/constants.ts +++ b/app/src/model/constants.ts @@ -14,6 +14,9 @@ export const RADIUS_KM = 5; /** At most this many borewells vote on a step: the nearest ones that reach it. */ export const MAX_VOTERS = 12; +/** At most this many of those come from one spot (borewells within `SAME_SITE_KM` of each other). */ +export const MAX_PER_SITE = 3; + /** A borewell closer than this counts as this far away, so one at the very spot is not infinitely loud. */ export const MIN_DISTANCE_KM = 0.025; @@ -37,6 +40,7 @@ export const MIN_BOREWELLS_TO_JUDGE = 10; /** * A borewell is an odd one out when it matches its surroundings less than this fraction as often as - * is typical for borewells with neighbours as near, judged on at least this many steps. + * is typical at that distance from the nearest other site, judged on at least this many steps and + * against at least this many borewells. */ -export const ODD = { fraction: 0.5, minSteps: 10 }; +export const ODD = { fraction: 0.5, minSteps: 10, minBorewells: 5 }; diff --git a/app/src/model/madeUpCity.ts b/app/src/model/madeUpCity.ts index 5c86200..7a5ebd9 100644 --- a/app/src/model/madeUpCity.ts +++ b/app/src/model/madeUpCity.ts @@ -11,8 +11,10 @@ const SOIL: Record = { clay: "clay", waterBearing: "medium_sand", /** * Going down: clay, water-bearing sand, fine sand, hard ground, then water-bearing sand again. - * Each boundary sits at `base` feet above sea level and rises and falls by 15 ft across the city - * as a smooth wave 8 km long, each running in its own direction. + * Each boundary sits at `base` feet above sea level and rises and falls by 40 ft across the city + * as a smooth wave 8 km long, each running in its own direction. The waves are taller than the + * layers are thick, so layers thin out and vanish in places, as sand lenses do: knowing the height + * alone is not enough to guess the ground, the place matters. */ const BOUNDARIES = [ { base: 330, direction: 0.3, shift: 0 }, @@ -29,7 +31,7 @@ export function truth(latitude: number, longitude: number, heightFt: number): Gr for (let i = 0; i < BOUNDARIES.length; i++) { const { base, direction, shift } = BOUNDARIES[i]; const wave = Math.sin((2 * Math.PI * (x * Math.cos(direction) + y * Math.sin(direction))) / 8 + shift); - ceiling = Math.min(ceiling, base + 15 * wave); // a boundary never rises above the one over it + ceiling = Math.min(ceiling, base + 40 * wave); // a boundary never rises above the one over it if (heightFt > ceiling) return BELOW[i]; } return BELOW[BOUNDARIES.length]; @@ -48,11 +50,15 @@ function random(seed: number) { /** * `count` borewells logged from `truth`: 70% spread over the city, 30% within 50 m of an earlier * one, each 150 to 400 ft deep. Always the same for the same seed. + * + * With `wrongEvery`, every so-many-th borewell is logged upside down (as if the log had been typed + * in from the bottom); their ids are returned as `planted`. */ -export function madeUpCity(count: number, seed = 1): { sites: ModelSite[]; grid: ElevationGrid } { +export function madeUpCity(count: number, seed = 1, wrongEvery = 0): { sites: ModelSite[]; grid: ElevationGrid; planted: string[] } { const grid = slopeGrid(110, 125); const next = random(seed); const sites: ModelSite[] = []; + const planted: string[] = []; for (let i = 0; i < count; i++) { let latitude: number, longitude: number; if (i > 0 && next() < 0.3) { @@ -72,7 +78,14 @@ export function madeUpCity(count: number, seed = 1): { sites: ModelSite[]; grid: if (last && last[2] === soil) last[1] = top + 5; else layers.push([top, top + 5, soil]); } - sites.push(site(`m${i}`, latitude, longitude, layers)); + if (wrongEvery && i % wrongEvery === wrongEvery - 1) { + planted.push(`m${i}`); + let top = 0; + const upsideDown = layers.reverse().map(([from, to, soil]): [number, number, string] => [top, (top += to - from), soil]); + sites.push(site(`m${i}`, latitude, longitude, upsideDown)); + } else { + sites.push(site(`m${i}`, latitude, longitude, layers)); + } } - return { sites, grid }; + return { sites, grid, planted }; } diff --git a/app/src/model/model.test.ts b/app/src/model/model.test.ts index c2030e7..447ad92 100644 --- a/app/src/model/model.test.ts +++ b/app/src/model/model.test.ts @@ -65,13 +65,19 @@ describe("predictColumn", () => { expect(step).toMatchObject({ confidence: "none", likely: null }); }); - it("ponytail limit: more than 12 borewells at the nearest spot leave room for nobody else", () => { - // Today's behaviour, written down so that changing it is a decision: the voters are the 12 - // nearest borewells, not sites. + it("a busy spot takes at most three of the twelve seats", () => { const step = stepAt(column([...crowd(13, 0.2, CLAY), site("other", north(0), east(-0.5), SAND)]), 20); - expect(step.sources).toHaveLength(12); - expect(step.shares?.waterBearing).toBe(0); - expect(step.confidence).toBe("none"); + expect(step.sources.map((s) => s.id)).toEqual(["c0", "c1", "c2", "other"]); + expect(step.shares!.waterBearing).toBeGreaterThan(0.1); + expect(step.confidence).toBe("rough"); + }); + + it("a busy spot does not hide the sites around it", () => { + // Twelve clay logs at one spot, nearer than six separate sites that all found sand. + const ring = Array.from({ length: 6 }, (_, i) => site(`r${i}`, north(0.4 * Math.cos(i)), east(0.4 * Math.sin(i)), SAND)); + const step = stepAt(column([...crowd(12, 0.3, CLAY), ...ring]), 20); + expect(step).toMatchObject({ likely: "waterBearing", confidence: "likely" }); + expect(step.sources).toHaveLength(9); }); it("matches layers by height, not depth, on sloping ground", () => { @@ -126,10 +132,10 @@ describe("predictColumn", () => { expect(against({ locationSource: "address" })).toBeCloseTo(1 / 1.5, 3); }); - it("converts borewells logged in metres", () => { - const col = column([site("metric", north(0), east(0.5), [[0, 10, "clay"]], { depthUnit: "m" })]); - expect(col).toHaveLength(7); // 10 m is 32.8 ft - expect(stepAt(col, 27).shares?.clay).toBe(1); + it("reads layer depths as feet whatever the borewell's unit says", () => { + // StrataField keeps every layer in feet; the unit on an old record only labels its total depth. + const col = column([site("old", north(0), east(0.5), [[0, 10, "clay"]], { depthUnit: "m" })]); + expect(col).toHaveLength(2); }); it("not recorded and gaps never vote", () => { @@ -156,7 +162,7 @@ describe("predictColumn", () => { it("exclude hides a borewell from its own prediction", () => { const prepared = prepare([site("here", CENTRE[0], CENTRE[1], CLAY)], DEFAULT_MATERIALS, flat); expect(predictColumn(CENTRE[0], CENTRE[1], prepared, flat)).toHaveLength(40); - expect(predictColumn(CENTRE[0], CENTRE[1], prepared, flat, { exclude: "here" })).toEqual([]); + expect(predictColumn(CENTRE[0], CENTRE[1], prepared, flat, { exclude: new Set(["here"]) })).toEqual([]); }); it("groundM stands in for the grid at the asked point", () => { diff --git a/app/src/model/model.ts b/app/src/model/model.ts index 3bca4e5..0647d45 100644 --- a/app/src/model/model.ts +++ b/app/src/model/model.ts @@ -7,7 +7,7 @@ * the vote. Every answer can name the borewells behind it. */ import { METRES_TO_FEET, heightAt, type Borewell, type ElevationGrid, type Material, type StrataLayer } from "@strata/core"; -import { ADDRESS_WEIGHT, LIKELY, MAX_VOTERS, MIN_DISTANCE_KM, QUALITY_WEIGHT, RADIUS_KM, ROUGH, SAME_SITE_KM, STEP_FT } from "./constants"; +import { ADDRESS_WEIGHT, LIKELY, MAX_PER_SITE, MAX_VOTERS, MIN_DISTANCE_KM, QUALITY_WEIGHT, RADIUS_KM, ROUGH, SAME_SITE_KM, STEP_FT } from "./constants"; import { GROUPS, groupOf, materialsById, type Group } from "./groups"; export interface ModelSite { @@ -44,12 +44,12 @@ export interface Step { top: number; bottom: number; /** Each group's share of the vote, adding up to 1; null when no borewell reaches this step. */ - shares: Record | null; + shares: Readonly> | null; /** The group with the largest share; null when confidence is "none". */ likely: Group | null; confidence: Confidence; /** The borewells that voted, nearest first. */ - sources: Source[]; + sources: readonly Source[]; } /** @@ -65,12 +65,13 @@ export function prepare(sites: ModelSite[], materials: Material[], grid: Elevati const groundM = b.groundElevationM ?? heightAt(grid, b.latitude, b.longitude); if (groundM == null) continue; const groundFt = groundM * METRES_TO_FEET; - const toFeet = b.depthUnit === "m" ? METRES_TO_FEET : 1; + // Layer depths are always feet: StrataField converts logs written in metres when it reads them. + // (`depthUnit` on an old record only labels its total depth.) const runs: Prepared["runs"] = []; for (const layer of [...strata].sort((x, y) => x.startDepth - y.startDepth)) { const group = groupOf(layer, byId); if (group && layer.endDepth > layer.startDepth) { - runs.push({ topFt: groundFt - layer.startDepth * toFeet, bottomFt: groundFt - layer.endDepth * toFeet, group }); + runs.push({ topFt: groundFt - layer.startDepth, bottomFt: groundFt - layer.endDepth, group }); } } if (runs.length === 0) continue; @@ -90,16 +91,16 @@ export function groupAt(p: Prepared, heightFt: number): Group | null { } /** The group with the largest share; a tie goes to the earlier group. */ -export const largest = (shares: Record) => GROUPS.reduce((a, b) => (shares[b] > shares[a] ? b : a)); +export const largest = (shares: Readonly>) => GROUPS.reduce((a, b) => (shares[b] > shares[a] ? b : a)); /** * The ground at a point, in steps from the surface down to the deepest that any borewell within * reach goes. Returns null when the ground height at the point is unknown, and an empty column when - * no borewell is within reach. `exclude` (a borewell's id) and `groundM` are for the accuracy check. + * no borewell is within reach. `exclude` (borewell ids) and `groundM` are for the accuracy check. */ export function predictColumn( latitude: number, longitude: number, prepared: Prepared[], grid: ElevationGrid, - options: { exclude?: string; groundM?: number } = {}, + options: { exclude?: ReadonlySet; groundM?: number } = {}, ): Step[] | null { const groundM = options.groundM ?? heightAt(grid, latitude, longitude); if (groundM == null) return null; @@ -109,7 +110,7 @@ export function predictColumn( const kx = 111.32 * Math.cos((latitude * Math.PI) / 180); const near: Near[] = []; for (const p of prepared) { - if (p.id === options.exclude) continue; + if (options.exclude?.has(p.id)) continue; const x = (p.longitude - longitude) * kx, y = (p.latitude - latitude) * 110.57, km = Math.hypot(x, y); if (km <= RADIUS_KM) near.push({ p, x, y, km, run: 0 }); } @@ -120,7 +121,8 @@ export function predictColumn( let who: Near[] = [], said: Group[] = [], seats: Seats | null = null, counted: Counted | null = null; for (let top = 0; top < deepestFt; top += STEP_FT) { const middle = groundFt - (top + STEP_FT / 2); - // ponytail: the 12 nearest borewells, not sites; a spot with 12 or more logs crowds out everyone else. Limit by site if real records show it. + // The nearest borewells that reach this step vote, but no spot gets more than a few seats: + // the records have many logs at the same places, and one busy colony must not fill the room. const nowWho: Near[] = [], nowSaid: Group[] = []; // Going down, each borewell is followed through its own layers (`run`), and dropped for good // once the step is below its bottom. This is `groupAt` without starting from the top each time. @@ -130,7 +132,7 @@ export function predictColumn( while (c.run < runs.length && runs[c.run].bottomFt >= middle) c.run++; if (c.run === runs.length) continue; near[kept++] = c; - if (nowWho.length < MAX_VOTERS && runs[c.run].topFt >= middle) { + if (nowWho.length < MAX_VOTERS && runs[c.run].topFt >= middle && seatsBeside(nowWho, c) < MAX_PER_SITE) { nowWho.push(c); nowSaid.push(runs[c.run].group); } @@ -157,6 +159,12 @@ type Counted = Pick; const beside = (a: Near, b: Near) => (a.x - b.x) ** 2 + (a.y - b.y) ** 2 <= SAME_SITE_KM ** 2; +function seatsBeside(voters: Near[], c: Near) { + let n = 0; + for (const v of voters) if (beside(v, c)) n++; + return n; +} + function seat(voters: Near[]): Seats { const sources: Source[] = []; let total = 0; @@ -169,6 +177,8 @@ function seat(voters: Near[]): Seats { sources.push({ id: c.p.id, borewellId: c.p.borewellId, km: c.km, weight }); } // Separate sites, nearest first: a voter starts a new one unless it is beside an earlier site. + // "Beside" is measured pair by pair, so a string of borewells each within reach of the next can + // count as two sites while sharing votes as one long one. A starting rule, not a survey. const sites: Near[] = []; for (const c of voters) if (!sites.some((s) => beside(s, c))) sites.push(c); return { sources, total, sites: sites.length, nearestKm: voters[0]?.km ?? Infinity }; diff --git a/docs/superpowers/specs/2026-10-13-stratavision-design.md b/docs/superpowers/specs/2026-10-13-stratavision-design.md index 538a312..e70e2ad 100644 --- a/docs/superpowers/specs/2026-10-13-stratavision-design.md +++ b/docs/superpowers/specs/2026-10-13-stratavision-design.md @@ -100,10 +100,12 @@ surface down to the deepest depth any voter reaches. Each step has: 1. **Common height.** Each borewell's layers are placed by height above sea level: ground height minus depth. Ground height is the borewell's own `groundElevationM` when recorded, otherwise the - elevation grid. Depths in metres are converted to feet. The asked point's ground height comes from - the grid; results are shown as depth below ground at that point. + elevation grid. Layer depths are always feet (StrataField converts logs written in metres when it + reads them). The asked point's ground height comes from the grid; results are shown as depth below + ground at that point. 2. **Who votes.** For a step, a borewell votes if it is within 5 km, has a location, and has a - recorded layer covering the middle of that step's height. The 12 nearest such borewells are used. + recorded layer covering the middle of that step's height. The 12 nearest such borewells are used, + with at most 3 from any one spot, so a place with many logs cannot fill the room. 3. **Weight.** `1 / distance²` (distance floored at 25 m), multiplied by: - record quality: good 1.0, fair or unknown 0.7, poor 0.4; - location: 0.5 if the location came from an address lookup, otherwise 1.0; @@ -120,14 +122,17 @@ least 7 times in 10. ### Accuracy check -`checkModel(borewells, grid)` hides each borewell in turn, predicts its column from the rest, and -compares step by step against what was recorded. It returns: +`checkModel(borewells, grid)` hides each borewell in turn, together with every other borewell at the +same spot (two logs of one spot agreeing says nothing about a place nobody drilled), predicts its +column from the rest, and compares step by step against what was recorded. It returns: - the share of steps where `likely` matched the recorded group, overall and separately for `likely` and `rough` steps; -- the same split by distance to the nearest other borewell: under 1 km, 1 to 3 km, over 3 km; -- **odd ones out**: borewells whose own match is far below the others near them (under half the - typical match for their distance band, with at least 10 comparable steps), worst first. +- the best guess scored for every step, split by how far away the nearest borewell behind that step + was: under 1 km, 1 to 3 km, over 3 km; +- **odd ones out**: borewells whose own match is far below what is usual at that distance (under + half the typical match in a distance band, on at least 10 comparable steps and against at least 5 + borewells), worst first. With fewer than 10 usable borewells it reports that it cannot judge yet.